Tactile perception in combination with vision for successful robotic housekeeping. Part 1

In recent years, tactile perception and the fusion of tactile and visual senses (called tactile-visual fusion) have become innovative approaches in this direction. The introduction of tactile perception in robotics signifies a radical shift in the capabilities of machines, providing them with the ability to perceive the environment through touch, much like humans.

Traditional robots have largely relied on visual perception, rarely taking into account the tactile sensations necessary for interaction in complex and dynamic environments.

Tactile sensors combine various tactile characteristics, such as pressure, temperature, texture, and material properties, endowing robots with rich sensory organs and allowing them to interact more skillfully with the environment.

In manufacturing, robots equipped with tactile sensors will perform fine assembly tasks with higher precision and adaptability.

In healthcare, tactile feedback potentially helps surgeons during minimally invasive operations, increasing the accuracy of manipulating surgical instruments.

Machine learning algorithms play a key role in facilitating the integration of these diverse sensory data within the framework of tactile-visual fusion. For example, deep learning architectures such as convolutional neural networks (CNN) and recurrent neural networks (RNN) demonstrate exceptional capabilities in capturing complex correlations between multimodal perception data and objects to be recognized.

However, there are still many factors that affect the ability of robots to perform complex tasks in real-world scenarios, and, consequently, they are difficult to replace human labor. First, there is the problem of creating robots with sensory capabilities similar to humans. Take, for example, pouring hot water from a cup - in this process, the robot requires visual perception to locate the cup, and it needs tactile information such as pressure, temperature, and slip to perform the pouring action. Just like humans, for effective interaction of robots with the environment, multimodal sensory capabilities such as vision, pressure, temperature, thermal characteristics, textures, and slip are necessary. In addition, fast and sensitive perception capabilities are required. For example, the lack of high ability for fast and sensitive slip detection can cause the robot to drop an object when trying to grasp it, leading to losses or endangering people.

We offer a robotic flexible tactile sensor based on thermosensation, allowing simultaneous multimodal perception of pressure upon contact, temperature, thermal conductivity, texture, and slip (Fig.1). The proposed tactile-visual fusion strategy is used for grasping various objects, where vision helps determine the object's position and pose, and slip-based tactile feedback control ensures dexterous robotic grasping. It is also worth noting that the robot is capable of stably grasping a cup with minimal force using tactile perception to ensure that the cup does not get crushed or slip when water is poured into it.

We apply the proposed tactile-visual fusion method to the task of cleaning a desk, where the robot automatically performs a series of actions such as locating the object, stable holding of the object, and precise object recognition to ensure sorting of multiple items and cleaning on the desk.

This confirms the feasibility and superiority of the proposed multimodal tactile sensor, as well as the tactile-visual fusion architecture, demonstrating the promising potential of intelligent robots for household chores.

The robot has multimodal perception capabilities, including vision and touch (perception of pressure upon contact, temperature, thermal conductivity, and texture of objects, as well as slippage). Thanks to these perception capabilities and the proposed tactile-visual fusion strategy, the robot can perform a number of tasks, such as object localization, stable grasping, object recognition, and sorting.

Principles of operation of the multimodal flexible tactile sensor

The sensor consists of an upper sensitive layer, a lower sensitive layer, a PDMS layer, and a porous material in the middle (Fig. 2a). The upper and lower sensitive layers have the same sensitive structure, which includes two concentric platinum (Pt) thermistors deposited on a flexible polyimide substrate. The inner thermistor has low resistance (~ 50 Ohms), which acts as a thermoelement (referred to as a hot film). The outer thermistor has higher resistance (~ 500 Ohms, referred to as a cold film) and works as a local temperature sensor. The radii of the hot film and cold film are 0.95 mm and 3.2 mm, respectively (Fig. 2a). The perception principle is based on conductive heat transfer. The hot film on the sensitive layer serves as both a heater and its own temperature sensor, which is sensitive to the thermal conductivity of the environment.

a The structure of the tactile sensor consists of the upper sensitive layer, the lower sensitive layer, PDMS, and a porous material in the middle.b The principle of operation of the lower sensitive layer.c The response to pressure U_p of the lower sensitive layer during loading and unloading. The inset shows the lower detection limit.d The principle of operation of the upper sensitive layer. Heat transfer of the upper sensitive layer in the modes of no contact, contact, and sliding.e The output signals of the tactile sensor respond to the temperature of the environment and the object, respectively.f The response of the upper sensitive layer when touching materials with different thermal conductivities.g The response of the upper sensor layer during contact and sliding on PA and PEEK, respectively.h Sliding and texture are determined by the macro- and microelements of the signals of the upper sensitive layer. Macro- and microelements are extracted from the signals of the upper sensor signal by filtering during sliding. The error bars shown in c,e and frepresent the standard deviations of five repeated measurements of one tactile sensor.

In the lower sensitive layer, contact pressure causes elastic deformation of the middle porous material, changing the thermal conductivity of the porous material due to its piezothermal conversion (Fig. 2b). Consequently, the lower sensitive layer responds to contact pressure (the output signal is denoted as U_p). Fig. 2c shows the sensor's response curve during loading and unloading under pressure. The sensor has a wide measurement range of 20 N and a low detection limit of 0.01 N, as well as low hysteresis of 2.4%.

As for the upper sensitive layer, when the sensor comes into contact with an object, the thermal conductivity of the object affects the heat transfer of the hot film and thus can be detected. In particular, the hot film in the upper sensitive layer is electrically heated using a constant temperature difference (CTD) scheme, providing a higher temperature than the environment and creating a thermal field in the contact object. When slipping, the hot film moves to a cooler area of the object it is in contact with, resulting in a change in heat transfer that is immediately detected by the hot film. Essentially, slipping is determined by the change in heat transfer of the hot film. After slipping, the heat transfer returns to the state before slipping, as the sensor remains in contact with the heated area. Thus, the slipping of the object is detected by the output signal of the upper sensor layer (designated as U_i) (Fig. 2d). In addition, as mentioned earlier, cold film thermistors in the lower and upper sensitive layers perceive the ambient temperature and the temperature of the object, respectively, as shown in Fig. 2e. Using the CTD scheme, the tactile sensor is not subject to fluctuations in ambient temperature. Fig. 2f shows the change in response of the upper sensitive layer when touching objects with different thermal conductivities. As the thermal conductivity of the object increases, the sensor output signal also increases. Fig. 2g shows the dynamic response of the sensor when transitioning from non-contact mode to contact, slipping, and back to contact with objects. Since the thermal conductivity of polyamide (PA) is higher, it can be seen that the response in contact and slipping is higher than that of polyetheretherketone (PEEK) material. It is worth noting that changes in contact pressure do not affect slip detection. More importantly, since the sensor is sensitive to the surface relief of the object, the response during slipping is filtered to extract its macro-characteristics (U_macro) and micro-characteristics (U_micro) (Fig. 2h). These functions can be used to simultaneously determine the state of slipping and the texture of the object's surface. In addition, the tactile sensor is tested in reciprocating contact disconnection cycles up to 1000 times, indicating good stability and reliability of the sensor.

Detection of Slippage and Texture Recognition

Macrofunctions in the response signals of the upper sensitive layer can be used to characterize the state of slippage of an object on the sensor. We select various materials (bakelite, PE, PA, PEEK, and PPS) for testing and record sensor responses at sliding speeds of 0 (no slippage), 5, 10, 15, 20, 30, and 50 mm/s respectively (Fig. 3a). The results show that as the thermal conductivity of the material increases and the sliding speed accelerates, the corresponding macro-characteristics (U_macro) increase. When the thermal conductivity of the object is known from the previous state of stable contact, U_macro can be used to determine the sliding speed of the object when grasped by a robot. In addition, we use PPS material to determine the minimum detection threshold and response time for slippage detection. As shown in Fig. 3b, at a small sliding speed of 0.05 mm/s, the sensor still demonstrates a clear response. Furthermore, after the occurrence of slippage, the sensor response exceeds the noise level in the absence of slippage in just 4 ms (shown in Fig. 3c). Taken together, these results indicate that the proposed tactile sensor can be used for ultra-sensitive and ultra-fast detection of slippage across various materials.

a Macro-characteristics for various materials with different sliding speeds. b The tactile sensor has a low slip detection limit - 0.05 mm/s. c The tactile sensor has a fast response time - 4 ms for slip detection. d Micro-functions at different sliding speeds. e FFT analysis of microelements at different sliding speeds. f FFT analysis of microelements at different sliding speeds. g Tactile reactions and microphotographs of four types of fabrics, including polyester spandex, elastic polyester knit fabric, nylon, and artificial silk with encrypted code (F1–F4). h Tactile response of the F1 and F3 key combination. z Confusion matrix for fabric recognition.

Microparticles in the response signals of the upper sensory layer can be used to determine the morphology of the micro-surface of objects, facilitating texture recognition. Figure 3d shows the sensor microelement signals received when the PA material slides over the surface of the object at a normal pressure of 2N at different speeds (0.5, 1, 3, and 5 mm/s). As the sliding speed increases, since the surface relief of the object remains unchanged, the maximum number of signals over the same period of time also increases. Analyzing these signals using Fast Fourier Transform (FFT) in the frequency domain gives the spectrum shown in Figure 3e. It can be seen from the graph that the frequencies corresponding to the spectral peaks also increase with increasing sliding speed. At a sliding speed of 1 mm/s, the corresponding PA frequency is 4.03 Hz, so the lattice period on the PA surface is approximately 248 μm. Visualization of the Short-Time Fourier Transform (STFT) using microelements provides time-dependent frequency information, displaying the magnitude of the STFT intensity over time (Figure 3f). As the sliding speed increases, the frequency distribution gradually shifts to the high-frequency region. Texture determination is also performed for other various materials.

Thus, the thermal conductivity, sliding speed, and surface texture of the object can be independently determined using the upper signal of the hot film sensor (U_i) from contact to sliding process. When the sensor comes into contact with the material, the response of the hot film in the upper sensitive layer changes significantly, and the stabilized voltage (U_i) can be used to determine the thermal conductivity of the material.

When slippage occurs during the robotic gripping process, the sliding speed can be determined based on the macro-characteristics (U_macro) extracted from the upper perception signal of the hot film, in combination with the predetermined thermal conductivity.

Additionally, by performing FFT for microelements (U_micro) extracted from the upper signal of the hot film sensor, the lattice period of the material surface, reflecting the texture, can be determined.

The use of tactile sensors to measure the thermal properties of the material and determine the texture also allows for accurate differentiation of various fabrics. We select ten different fabrics (polyester spandex, polyester knit stretch fabric, nylon, encrypted artificial silk, cotton canvas, denim, fleece, wool-polyester fabric, cardboard and linen fabric, as well as lycra to obtain tactile sensor responses (Fig. 3g) when sliding over the surface of these fabrics. In addition, the tactile response of the fabric combination is also shown in Fig. 3h, indicating that the sensor consistently contacts two fabrics during the sliding process, and the corresponding macroscopic and microscopic characteristics of the sensor signal can be used to identify fabrics in sequence. Subsequently, we extract macro and micro elements from the tactile sensor data to perform fabric recognition. Using this approach, we achieve accurate recognition of these 10 fabrics with an overall accuracy of 94.3%.