Blog

Nov 05, 2024

Magnetostrictive bi-perceptive flexible sensor for tracking bend and position of human and robot hand | Scientific Reports

Scientific Reports volume 14, Article number: 20781 (2024) Cite this article 527 Accesses 1 Citations Metrics details The sensor that simultaneously perceives bending strain and magnetic field has the

Scientific Reports volume 14, Article number: 20781 (2024) Cite this article

527 Accesses

1 Citations

Metrics details

The sensor that simultaneously perceives bending strain and magnetic field has the potential to detect the finger bending state and hand position of the human and robot. Based on unique magneto-mechanical coupling effect of magnetostrictive materials, the proposed a bi-perceptive flexible sensor, consisting of the Co–Fe film and magnetic sensing plane coils, can realize dual information perception of strain/magnetic field through the change of magnetization state. The sensor structure and interface circuit of the sensing system are designed to provide high sensitivity and fast response, based on the input–output characteristics of the simulation model. An asynchronous multi-task deep learning method is proposed, which takes the output of the position task as the partial input of the bending state task to analyze the output information of the sensor quickly and accurately. The sensing system, integrating with the proposed model, can better predict the bending state and approach distance of human or robot hand.

The dexterity and perceptual capabilities of the human hand allow it to perform complex and delicate tasks, and similar manipulators continue to expand their range of applications through technological innovation1,2,3. The tracking of finger bending and hand position enables motion capture and gesture interaction of the human hand, as well as precise perception and control of the manipulator. Typically, vision, tactile, and other sensing technologies are utilized to track the movements of human and robot hand4,5,6. The utilization of cameras, laser radar, and other equipment facilitates the capture of images depicting the hand, with visual algorithms to achieve motion recognition7. However, the quality of tracking is susceptible to significant degradation owing to external environmental factors such as fluctuations in ambient light, the presence of objects with similar colors or shapes within the tracking area, and occlusions obstructing the line of sight8. Another approach is using tactile9 or strain10,11 sensors to monitor strain information such as stretching, compression, or bending generated by joint movements to achieve motion state perception of both the human and robot hand12,13. Yet, the lack of spatial position information significantly affects precise hand movements and interactions, making it difficult to realize virtual interactions and robotic grasping tasks14.

The dual-modal flexible sensor perceives stress15,16 and magnetic fields17,18, allowing for tactile and touchless interactivity. Currently, the sensitive components of these sensors are composed of polymer that is dispersed with magnetic particles19. When external stress or magnetic field is applied to this type of sensor, magnetic particles will move and lead to the changes of magnetoresistance20, resistance21, capacitance22, friction electric23, displacement reaction voltage24, thus achieving normal stress and proximity sensing. Nevertheless, these sensor can’t be used to detect bend and position of human and robot hand, because magnetic particles aren’t perceive to bending strain and out-of-plane magnetic fields25.

Magnetostrictive materials, due to their unique magneto-mechanical coupling effect, can realize dual information perception through the change of magnetization state under the strain/magnetic field26. The integration of magnetostrictive alloy and microelectromechanical systems technology (MEMS) realized flexible and lightweight design of sensors. Suwa presented that the pressure sensor of 0.5 mm in width and 2 μm in thickness was composed of two magnetostrictive films and a planar coils layer sandwiched between them27. Kwon prepared magnetostrictive Co–Fe and Ni–Fe/Co–Fe films with spin valve structure to characterize bending stress by changes in magnetic anisotropy28. The magnetometer combined Fe58.1Co24.9B16Si1 films of 10 µm thickness with the graphics circuit by epoxy glue29, which exhibited a sensitivity of 1985 V/T.

Although the single mode magnetostrictive sensors for magnetic field or strain sensing have been well developed, they are limited to detecting specific stress or magnetic fields sources and thus unable to satisfy the growing demand for multi-information acquisition30. The Co–Fe film, as a magnetostrictive material, can be utilized in bi-perceptive sensor due to its large magnetostrictive coefficient and excellent soft magnetic properties under low fields31,32. Co–Fe films can be prepared rapidly, simply, and inexpensively through electrodeposition33,34. These films are highly flexible and lightweight35,36, allowing them to fit the flexibility and dexterity requirements of limb surfaces, without impeding joint movement. Coil sensing is used to detect the magnetization in magnetostrictive films37,38,39, which can reflect strain and magnetic field signals for the development of sensors. The interface circuit and multi-task algorithm of the sensor unit are designed, and the sensor system can detect and analyse the hand motion information. A new multi-task learning framework40,41,42 is proposed to simultaneously learn and predict the bending state and object position tasks using the correlation of movements during grasping.

Magnetostrictive Co–Fe alloys have both excellent soft magnetic and magnetoelastic coupling properties, and can be prepared as thin film by electrodeposition methods, which are excellent sensitive materials for flexible strain-magnetic field sensing applications. Based on the magnetostriction and magnetization effects, the magnetic domain distribution of Co–Fe film is affected by the force/magnetic field, and thus dual-information perception is achieved through the fast response of its own magnetized state to the applied strain-magnetic field stimulus. In order to illustrate the response of magnetostrictive film to strain and magnetic field, we use COMSOL Multiphysics to investigate the multi-physics field coupling process of magnetostrictive film under bending strain and magnetic field. A simulation model is used to analyze the magnetization distribution of Co–Fe film under external stimulation. By visualizing the magnetization distribution and changes, a sensing method can be designed to capture the signals of changes in the magnetization state.

Figure 1a,b show the magnetization direction and intensity distribution along the z-axis within the Co–Fe films under strain and magnetic field conditions. By observing the magnetization direction (indicated by red arrows), magnetization is mainly distributed in out-of-plane and in-plane directions under strain and magnetic field, respectively. Considering that planar magnetic sensing is usually measured in the out-of-plane direction, the magnetization strength along the z-axis is preferentially evaluated. When applying strain load or permanent magnet along the normal direction at the film center, changes in z-axis magnetization state exhibit symmetry along the y axis direction. Furthermore, the magnitude of z-axis magnetization under strain and magnetic field influence is similar, which helps to detect and compare the changes of bending strain and magnetic field simultaneously. This symmetry ensures that the sensing signal of one physical quantity is not overshadowed by the other, thereby facilitating accurate and independent detection of both strain and magnetic field variations. Considering the symmetrical distribution of the sensor along the y-axis, the z-axis magnetization at the representative major axis centerline of the film is selected to observe its amplitude changes under different curvatures and different distances from the permanent magnet, as shown in Fig. 1c,d. Across curvature radius ranging from − 10 and − 60 to + 60 and + 10 mm, as well as distances from the permanent magnet spanning from − 45 and − 115 to + 115 and + 45 mm, changes in magnetization state are predominantly concentrated at the center.

Magnetization distribution in magnetostrictive Co–Fe film under bending strain and magnetic field. Magnetization distribution of the Co–Fe film under bending strain (a) and permanent magnet (b), where the red arrow indicates the direction of the magnetization and the color map indicates the magnitude of the magnetization along the z axis. (c) Magnitude of magnetization Mz along the major axis centerline under ρ of − 10 to − 60 mm and + 60 to + 10 mm. (d) Magnitude of magnetization Mz along the major axis centerline under distance from permanent magnet range from − 45 and − 115 to + 115 and + 45 mm.

Under external dynamic magnetic field excitation, the magnetic induction strength of magnetostrictive films is used to reflect the behavior of material magnetization43. Therefore, a pick-up coil for detecting the magnetic induction intensity and an exciting coil for providing an external dynamic magnetic field are constructed for sensing the magnetization behavior under bending strain and magnetic field. And based on the analysis of the magnetization distribution, the coil is placed along the center of the main axis of the film, which can optimally collect the magnetization changes in the vertical direction. Through the construction of a three-dimensional hierarchical structural model as depicted in Fig. 2a, the sensor’s composition is delineated into distinct layers: the Co–Fe electrodeposition layer (illustrated in gray film), the insulating PI substrate (depicted in orange film), the pick-up coil (embodied by the yellow planar coil), and the exciting coil (represented by the red planar coil), both wound in congruent directions. To evaluate the response characteristics, the sensor is subjected to bending strain and magnetic field stimulation by applying the strain load along the short axis of the sensor, and by adjustments to the distance between the permanent magnet (depicted as the blue cylinder) and the sensor.

Simulation model and result. (a) Simulation model construction. (b) Cloud map of magnetic flux density distribution on Co–Fe film. (c) Distribution of magnetic flux density on the x-axis of Co–Fe film. (d) Distribution of magnetic flux density on the y-axis of Co–Fe film. (e) The pick-up coil output induced voltage under the curvature radius ρ ranging − 10 to – 60 mm and + 60 to + 10 mm. (f) The pick-up coil output induced voltage under the distance from the magnet ranging − 45 mm to − 115 mm and + 115 mm to + 45 mm.

The excitation current applied to the exciting coil, results in the distribution of magnetic flux density along the normal direction (Z-axis) within the film plane, as illustrated in Fig. 2b. A notable disparity in magnetic field strength between the interior and exterior of the coil is apparent, with a relatively uniform distribution of field strength in the central region of the film. Figure 2c,d offer insights into the flux density distribution along the major axis (x-axis) and minor axis (y-axis) of the Co–Fe film respectively, as the excitation current is varied from 100 to 150 mA. Notably, the distribution of field strength within the Co–Fe film exhibits significant amplitudes in proximity to the coil position. To achieve a more uniform magnetic field distribution, an excitation current of 100 mA is deemed optimal. Under this condition, the average magnetic flux density within the coil can reach 50 mT, resulting in a relatively uniform magnetic field distribution with the average value of 36.3 mT and the standard deviation of 1.1 mT across the film.

With the excitation current set at 100 mA and a frequency of 10 kHz, the signal waveform of the output voltage is recorded while curvature radius ranging from − 10 and − 60 to + 60 and + 10 mm, as depicted in Fig. 2e. Similarly, when the distance from the magnet varies between − 45 to − 115 and + 115 to + 45 mm, the signal waveform of the output voltage is captured, as shown in Fig. 2f. As the curvature radius and the magnet position transition from negative to positive, the amplitude of the output voltage signal increases monotonically. The simulation demonstrates the feasibility of the planar coil sensing structure for strain/magnetic field sensing, which is crucial for understanding and optimizing the performance of the sensor.

Based on the material's sensitivity to strain and magnetic field, the structure of the bi-perceptive flexible sensor is designed by analyzing the electromagnetic response of film under bending and magnet. Figure 3a illustrates the basic configuration of the new sensor, which consists of three-layer hierarchical structure comprising two-layer flexible circuit and one-layer electrodeposited magnetostrictive alloy film. There are two coils, including exciting coil and pick-up coil, in the two-layer flexible circuit, which can detect the changes in magnetization state of the magnetostrictive alloy film under external stress or magnetic field. The turns ratio of exciting coil to pick-up coil is 1:1, and number of windings is both 15. The routing width and clearance of circuit are both 0.07 mm. The length, width and thickness of the sensor are 35 mm, 9 mm and 40 µm, respectively.

Fabrication and output characteristics of sensor system. (a) Schematic illustration of the bi-perceptive sensor unit: and preparation process. (b) Schematic of analog circuit. (c) Output waveforms of analog circuit. (d) Sensor output voltage under ρ of 10–65 mm. (e) Sensor output voltage under H of 1–11 kA/m. (f) Response time and recovery time of the sensor at ρ of 10mm and H of 11 kA/m. (g) Output voltage of sensor under dynamic force and magnetic field at 1–4 Hz.

Based on the input–output characteristics of the sensor in the simulation model, appropriate interface circuits need to be designed for the input and output signals of the exciting and pick-up coil. Given the disparity between the output signal of the sensor and the input voltage requirements of the single-chip microcomputer ADC, the design of a signal processing circuit becomes imperative to continuously extract the signal amplitude. Based on the analysis of the induced voltage signal in the simulation, the alternating signal induced by the coil is extracted through differential amplification, half-wave shaping and peak detection, and finally converted into digital signal to characterize the external strain-magnetic stimulation. The output voltage generated by the sensor pick-up coil first gets through differential amplifier (output is Uamp1), then is transmitted to the 1.25 V-bias half-wave amplifier (Uamp2) and finally converted into the peak detection voltage amplifier (UPKD), as shown in Fig. 3b. Output waveforms of signal processing circuit, including Uamp1, Uamp2, and UPKD, are shown in Fig. 3c. The excitation circuit is designed to provide excitation current for the sensing unit. The dual-output 3 A MOSFET driver provides the ± 10 V square wave as the excitation current. The excitation signal is controlled by the MCU, and the digital signals collected from several sensors are sent to the host computer for display and analysis.

The electrodeposited magnetostrictive layer of the sensor unit is facing upwards and the coil layer is facing downwards. The symbols of strain and magnetic field applied to the front of the sensor are defined as positive. Figure 3d depicts the output signal of the sensor under ρ of 10–65 mm along two opposite directions by loading different deflections to sensor. Figure 3e illustrates the output signal of the sensor under H of 1–11 kA/m along two opposite directions by providing different magnetic field. The maximal sensitivities of the sensor are 22 mV/mm within the curvature radius of 10–65 mm and 1.78 mV/(kA/m) within the magnetic field of 1–11 kA/m, respectively. It’s important to note that the direction of the applied strain and magnetic field can significantly affect the sensor’s output. Experimental results indicate that the sensor exhibits good static characteristics, and its output shows consistent behavior in both directions, enabling flexible design of application scenarios. Figure 3f characterizes the load and unload output amplitude under ρ of 65 mm and H of 11 kA/m for testing hysteresis of the sensor. The maximum response and recovery time of the sensor are 17 ms and 39 ms, respectively. Considering the load and unload long-stroke, the output hysteresis is acceptable for detecting movement with a general frequency not exceeding 4 Hz. Figure 3g shows the dynamic output characteristics of the sensor under curvature and magnetic field with frequencies ranging from 1 to 4 Hz. The changes of sensitivity are less than 1.6% at 1–4 Hz dynamic bending strains or magnetic field, which indicates good dynamic stability.

The curvature and magnetic field bi-perceptive ability of the sensor is applied to the interaction between the manipulator and the target object, which can be used to detect the position of the grasped object and the bending attitude information of the manipulator. Figure 4a shows that the schematic diagram of the three-finger manipulator and grasped object. The magnetic beacons and the sensor with control circuit are installed on the grasped object and three-finger manipulator respectively for comprehensive sense and fusion of finger attitude and object position. The flexible sensor is closely attached to the inside of the finger joint of the manipulator through adhesive tape, which does not prevent the manipulator from doing any action and ensures the flexibility of the manipulator. The sensor outputs positive signal for characteristic of curvature when subjected to bending. The magnet is fixed on the surface of the object and point at sensor that outputs negative signal by adjusting the field of the magnetic beacons in polarity and strength. The three-finger manipulator with sensor grasps the object with magnetic beacons, which displacement and grasping speed of the manipulator were set to 50 mm/s and 30 mm/s respectively. The manipulator motor encoder data and sensor output voltage are collected. Considering that the signal acquisition amplification process will produce random high-frequency noise, the amplitude limiting jitter filtering method is used to process the data collected by the sensor system, which has a good filtering effect for slowly changing measured parameters.

Robotic grasping experiments and results. (a) Assembly drawing of sensor and magnet on the robotic hand and object. (b) Change of the sensor output where the finger bearing the sensor approaches, grasps, loosens and retreats from the magnet-decorated object.

For data analysis and processing, the sensor system was calibrated using the maximum and minimum output signal values of the sensor units during the establishment of the dataset. The maximum bend of the manipulator serves to calibrate the maximum output values of the sensor unit. The value is minimum, while the magnetic beacons are placed closest to the sensor. After calibration, the sensor system can be used to monitor manipulator motion data and send it to the host computer built based on MATLAB GUI. Then, the signal is mapped to the range of 0 to 1. Figure 4b shows the variation of the sensor output voltage during the approaching, grasping, loosening and retracting progress. The decrease of output signal shows distance reduction from the object at the stage i, when manipulator approaches the object and fingers unbend. In the stage ii, the object enters the grasp range of the manipulator and finger joint is driven to grasp the object, meanwhile the signal is less than 0.5 and further decreases. The minimum signal is acquired when the finger joint touches the object surface. During the grasping progress at stage iii, the signal rapidly increases and becomes positive values due to curvature of joints. As soon as the manipulator graspes object stably, output maintains constant with maximum curvature of joints at stage vi. The output signal is opposite to the approaching and grasping process when manipulator loosens and retreats the object at stage v, vi, vii. This qualitative signal change reflects the object proximity and finger movement. The sensor is able to collect curvature-magnetic information simultaneously, and further analyses the dual information of object positioning and finger movement by processing the output signal of the sensor.

In this manipulator-object interaction experiment, the magnets fixed on the surface of the object maintain fixed positions relative to each other. Three magnets serve as a unified object entity, enabling the reconstruction of the distance between the manipulator and the object by analyzing magnetic information in the data of three sensor. The actual distance between manipulator and object measured by cable-pull encoder serves to reflect the object position. Simultaneously, the encoder signal from the motor on the manipulator's finger enables real-time understanding of the motor angular position to map the finger bending state. These encoder outputs, ranging between 0 and 255, correlate with the minimum and maximum values representing measured distance or curvature. Subsequently, by comparing the distance information based on magnetic perception with the sensor data reflecting curvature-magnetic perception, the bending information of the three fingers can be determined respectively.

Based on the output signals from the sensor, a multi-task learning network is constructed to predict the manipulator motion state, as shown in Fig. 5a. In the shared layer, the model processes the input data through the shared layer, which enables the model to capture common patterns and representations. Then, the resulting shared features as inputs to the specific subtask layer. And in the task layer, each subtask is processed synchronously to generate outputs for predicting bending states (finger A, B, and C) and object positions. The traditional multi-task learning network relies entirely on the neural network to find correlations between parameters and process each subtask independently. The traditional multi-task learning network lacks explicitly incorporating any a priori knowledge about the tasks or the relationships between them, which leads to sub-optimal learning, especially in complex scenarios where the relationships between tasks is crucial.

Comparison of multi-task learning methods. (a) Traditional multi-task learning. (b) Proposed asynchronous multi-task learning structure. (c) The comparison real value and predicted value by outputting traditional multi-task learning. (d) The comparison real value and predicted value by outputting proposed asynchronous multi-task learning structure. (e) Normalized RMSE trends for prediction with further training epochs. (f) The predicted results comparison of the different objects using RMSE, MAE, and R2 score.

Therefore, we propose an asynchronous multi-task deep learning method to predict object location and finger bending state in two steps, as shown in Fig. 5b. The object position is first predicted based on the features extracted from the shared layer. Then the output of the object position task is used as part of the input for the finger bending state task to infer the finger bending state in combination with the associated bending features of shared layer. Based on the correlation between the bending strain and magnetic field information, the position information that is universally relevant to the subtasks is extracted for establishing the relationship between the subtasks. The introduction of constraint terms based on prior knowledge guides the learning process and encourages the network to prioritize positional relationships between parameters. The proposed method has better overall performance compared to traditional methods that predict all subtasks synchronously.

Considering the temporal correlations inherent in manipulator movements, we leverage Long Short-Term Memory (LSTM) networks to better capture long-term dependencies in the time series data. LSTM networks achieve this by introducing gate mechanisms to model the relationships between time steps. Thereby the model’s performance on sequential data tasks is enhanced. In the shared layers, we construct a regression model comprising two LSTM layers, each containing 10 nodes, and introduce nonlinearity using ReLU activation function. To mitigate overfitting, we incorporate a Maxpooling layer after each LSTM layer. Then the output of shared layer goes through task layer consists of fully connected layer and ReLU activation function. By incorporating task-specific fully connected layers, the model exhibits flexibility in adapting to the requirements of different tasks. Additionally, the ReLU activation function enhances the model’s nonlinear representation capability, thereby contributing to improving performance and generalization. The object position task layer consists of fully connected layers, the rectified linear unit (ReLU) activation functions and Savitzky-Golay (SG) filtering. The SG filtering reduces noise interference in time series data and improve the prediction accuracy of subsequent tasks. The SG filter uses the method of linear least squares to fit consecutive subsets of adjacent data points to low-order polynomials. The filtered object position output, along with the shared representations extracted by the shared layers, infers the bending state of the three fingers. Finally, we optimize the network parameters using the Adam optimizer algorithm to minimize the mean squared error loss function. As the primary error metric quantifying the deviation between predicted values and ground truth, we construct the loss function using the Root Mean Square Error (RMSE).

Where \({\widehat{y}}_{i}\) denotes the prediction of the model ensemble and \({y}_{i}\) is the ground value at sample \(i\).

For completeness and ease of comparison between tasks, we also report the Mean Absolute Error (MAE) and the coefficient of determination (R2 score).

The traditional and proposed multi-task models were trained with the collected data to predict the object position and the finger bending state of each finger. Figure 5c,d illustrate the comparison of prediction results between the two models. It’s obvious that the feature information obtained from the asynchronous multi-task learning model corresponds better to changes in the object approach distance and finger bending actions. The deviation between model predictions and measured ground truth values is less than 7.25%. From the above results, it is evident that adjusting the loss function and filter paraments of position prediction task significantly improves prediction performance. Furthermore, the strong correlation between variables and the object position output fed into other subtasks lead to a noticeable improvement in finger bending state prediction. Figure 5e illustrates that compared to traditional multitask learning networks, the proposed asynchronous learning model achieves better predictions of both object position and finger bending state. The traditional multitask learning model requires over 900 training epochs to achieve a normalized RMSE of 0.065. In contrast, utilizing our developed asynchronous multitask learning model, a normalized RMSE of less than 0.04 is demonstrated within 100 transfer training epochs.

In grasping experiments involving objects of different shapes and materials, we collect data from the sensing system to predict the object positions and finger bending states. We compared the prediction results of various objects using RMSE, MAE, and R2 score, as shown in Fig. 5f. The results indicate that the standardized RMSE ranges from 0.02 to 0.04, MAE ranges from 2.5 to 4, and R2 values range from 0.85 to 0.95 for different shapes and materials of the objects. This suggests that the model exhibits consistently good predictive performance on different objects. However, silicone cubes and rubber spheres exhibit higher RMSE and MAE, along with lower R2 score, compared to other objects. This discrepancy may be attributed to the relatively soft texture of these objects, causing changes in the relative positions of magnetic beacons during grasping. It results in inaccurate estimation of the object position.

The lightweight and curved bi-perceptive sensor can comfortably accommodate human hands or a glove surface for tracking hand movements. Figure 6a shows a motion tracking platform composed of a magnet and a data glove based on bi-perceptive sensor. To ensure the independence of signals between each unit, sensors are installed exclusively at the metacarpophalangeal joints. The sensor is fixed at user’s knuckles in the sewing interlayer of the glove. The sensor and the circuit components are integrated in the glove to construct a data glove and upload the collected data.

Motion tracking platform based on bi-perceptive flexible sensor. (a) Illustration of the motion tracking platform. (b) Interactive operations at different distances with the magnet and the corresponding sensor signals. (c) The distance from magnet and finger bending angle: the predicted value output by the model and the real value obtained by the measurement are compared.

In this study, the sensor at the five joints recorded the movements of the human hand, and a magnetic beacon (NdFeB N35 Φ20 × 5 mm) was placed beneath the palm to locate the position and simulate virtual objects. It allows the operator who simulates real scenes to realize motion tracking, such as space location and gesture control. The operator, wearing the data glove, extends their hand toward the magnet and engages in gestural interactions at varying distances from the magnet. Concurrently, the output of the sensor varies with both distance and gesture, as depicted in Fig. 6b. The strain-magnetic field signal from a single sensor cannot be directly used to characterize the distance from the magnet due to the influence of gestures and finger movements.

Based on the asynchronous multi-task deep learning method, models can be quickly deployed to analyze the time series data of the 5 sensor units in the data glove, predict the distance between the hand and the magnet, and finger bending angle. Figure 6c shows the comparison between the actual value and the predicted value. The actual values are the distance from magnet and finger bending angle measured by the pull rope displacement sensor and flex sensor. The predicted values are the outputs of the asynchronous multi-tasking model based on the acquired data from the sensors. Some deviations occur at the moment of gesture motion, but the distance and bending angle changes can be clearly distinguished in the predicted values.

The developed asynchronous multi-task learning model can be used to jointly detect object position and each finger bending state. Moreover, this architecture is generic in other subtask-related regression prediction applications. Building asynchronous models using prior knowledge contained in multiple related tasks to guide learning information improves prediction performance by comparison with traditional multitasking models. The approach ensures that each task benefits from an understanding of the relevant tasks, which improves the speed, accuracy and reliability of predictions.

In this paper, we propose a bi-perceptive flexible sensor that can be integrated into a manipulator for detecting manipulator bending state and the distance between the manipulator and the object. Regarding the convenience, function and cost-effective fabrication, our methodology exhibits the competitive advantages compared with many commercial systems and reported literatures3,19,44 in the field of the motion tracking and object perception (Table 1). Compared with reported commercial and literature techniques, the system has the advantages of fewer sensors, less data processing, and concise deep learning model, which reduce the cost and power consumption of the whole system. This makes the system more competitive and promises a wide range of robotic applications.

Based on the high sensitivity of magnetized state of magnetostrictive alloy to stress/magnetic field, the Co–Fe film by electrodeposition is proposed to develop strain-magnetic bi-perceptive flexible sensor. Through COMSOL simulation, the coupling relationship between the magnetization intensity of Co–Fe film and the external stress/magnetic field is studied. Combined with the principle of vector magnetic field sensing, a planar coil flexible sensing model is constructed, and the output characteristics of the sensing unit under strain/magnetic field are analyzed. Based on the simulation results, the sensing unit and its interface circuit were designed and fabricated for the strain-magnetic bi-perceptive sensing system. The experiment system of sensor was established and the experiment data proved the veracity of the model and performances of the sensor. The maximal sensitivity of the sensor is 22 mV/mm in the bending of 10–65 mm. The maximal sensitivity of the sensor is 1.78 mV/(kA/m) in the magnetic field of 1–11 kA/m. The changes of sensitivity are less than 1.6% under different ρ and H at 1–4 Hz which indicates good dynamic output stability. Based on the dual perceptual characteristics of sensor data, we propose an asynchronous multi-task deep learning method, which takes the output of the object position task as partial the input of the bending state task to analyze the output information of the sensor quickly and accurately. The normalized RMSE is less than 0.04 within 100 transfer training epochs, showing better prediction of object approach distance and finger curvature state compared to traditional methods. The sensing system, integrate with the proposed model, shows consistent good prediction performance for tracking bend and position of human and robot hand.

The polyimide copper clad laminate, as cathodes and electrodeposited substrate, was cleaned with acetone and rinsed with deionized water. The anode of the platinum strip (30 mm × 30 mm × 0.1 mm, 99.9 wt%) was fixed parallel and center-aligning to the cathodes at a distance of 3 cm. The composition of the baths includes 0.3m/L H3BO3 (buffer), 0.3 m/L NaCl (improve conductivity), 0.4 m/L ascorbic acid (antioxidant), 0.4m/L Glycine (complexing agent), and 0.3 m/L Sodium citrate (complexing agent). The concentration of the main salt of the solution is 0.08 mol/L FeSO4 and 0.3 mol/L CoSO4. The electrolyte pH was controlled at 2.5 with 10% H2SO4. Mild temperature (50 ℃) and stirring conditions were applied to enhance the mass transport of metal from the bulk electrolyte towards the electrode surface. The electric deposition of constant current is achieved using a digital DC source (Agilent 66312A).

The preparation process of sensor involves the flexible circuits and electrodeposition process. Initially, a flexible substrate, made of polyimide (PI), is cleaned thoroughly to remove any contaminants. The substrate is then coated with a thin layer of adhesive, followed by a copper conductive layer, through a lamination process. The thickness of substrate and copper layer is 12.5 µm and 18 µm respectively. Once the copper layer is in place, photolithography is used to define the circuit pattern. A photoresist material is applied to the copper surface, exposed to UV light through a patterned mask, and developed to reveal the desired circuit design. The exposed copper is then etched away, leaving behind the circuit pattern. After etching, the remaining photoresist is stripped off, and the circuit undergoes a cleaning process.

The top layer of the prepared flexible circuit is electrochemically deposited with a thickness of 20 µm Co–Fe film for magnetostrictive applications. The flexible circuit is taped to isolate the non-deposited conductive regions and immersed in a solution containing the desired metal ions. During electrodeposition, electrodes are placed in the plating bath and the copper layer of the flexible circuit is connected to the cathode. A controlled current is passed through the solution to deposit the metal ions onto the conductive areas of the flexible circuit. The thickness and quality of the electrodeposited layer can be controlled by adjusting the current density, bath composition, temperature and plating time. After the desired thickness is achieved, the flexible circuit is removed from the plating bath, rinsed and dried.

A dynamic strain/magnetic testing platform has been established to evaluate the output characteristics and dynamic performance of the sensor system. The AFG1022 signal generator is utilized to produce sinusoidal or square wave signals across a range of frequencies, which are then amplified by the MB500VI power amplifier. The power amplifier is connected to a shaker, responsible for driving the moving coil within it. As the moving coil experiences periodically changing electromagnetic forces, it propels the push rod to reciprocate.

To assess the output characteristics of the sensing system under bending strain and magnetic field, the sensing unit is affixed onto a standard three-point bending fixture. The test is conducted by actuating the push rod to apply external stimulation, while the curvature radius and magnetic field strength are calibrated based on the displacement of the push rod. The contact point on the push rod directly interfaces with the sensor, enabling adjustment of the deflection to load bending strain onto the sensor by modulating the displacement exerted by the push rod. Additionally, the push rod is utilized to drive the permanent magnet (Sm2Co17 28H) in close to the sensor, thereby applying a magnetic field from above. Following the pre-calibrated data, the push rod is driven to load the corresponding curvature radius and magnetic field onto the sensing unit. The acquisition card is employed to record and calibrate the output position of the push rod, and subsequently, the data is uploaded to a computer for further analysis.

The dataset consists of the output signals from the sensors that track finger movements and the corresponding motion state labels (bending and position). The input data and its labeled values are pre-processed (normalization and outlier processing) and are aligned as time series data with the number of data channels 3 and length 80. The total sample size of the sensing data with labels is 120, of which 80% of the data is used to train the model and 20% of the data is used to test the predictive effectiveness of the proposed model.

Multitask learning networks are constructed and trained based on PyTorch library to regress and predict robotic hand movements. According to the proposed network architecture, common features are processed using a shared base network and then task-specific branches are created for each output. Each branch can consist of dense layers with ReLU activation, with the final layer corresponding to each task with the appropriate activation. The global loss function for all subtasks is the sum of the individual loss functions for each task. During the training process, input data from each batch of 20 samples is fed into the network and forward passes compute predictions for all tasks. By comparing the predicted and actual values, the combined loss is calculated and the network weights are adjusted by back-propagation. Our model is trained for 1500 epochs, and the learning rate is fixed at 10-4 for optimal performance. After training, the model is evaluated on a test set for accuracy and generalization ability.

All data needed to evaluate the conclusions in the paper are present in the paper.

Yang, C., Jiang, Y., Li, Z., He, W. & Su, C.-Y. Neural control of bimanual robots with guaranteed global stability and motion precision. IEEE Trans. Ind. Inform. 13, 1162–1171. https://doi.org/10.1109/tii.2016.2612646 (2017).

Article Google Scholar

Cheng, H., Wang, Y. & Meng, M. Q. H. A vision-based robot grasping system. IEEE Sens. J. 22, 9610–9620. https://doi.org/10.1109/jsen.2022.3163730 (2022).

Article ADS Google Scholar

Shi, Q., Sun, Z., Le, X., Xie, J. & Lee, C. Soft robotic perception system with ultrasonic auto-positioning and multimodal sensory intelligence. ACS Nano 17, 4985–4998. https://doi.org/10.1021/acsnano.2c12592 (2023).

Article CAS PubMed Google Scholar

Borras, J., Alenya, G. & Torras, C. A grasping-centered analysis for cloth manipulation. IEEE Trans. Robot. 36, 924–936. https://doi.org/10.1109/tro.2020.2986921 (2020).

Article Google Scholar

Xu, T. et al. High-sensitivity flexible tri-axial capacitive tactile sensor for object grab sensing. Measurement 202, 876. https://doi.org/10.1016/j.measurement.2022.111876 (2022).

Article Google Scholar

Xiao, Y., Jiang, S., Zhao, X., Jiang, H. & Zhang, W. Crack-enhanced mechanosensitivity of cost-effective piezoresistive flexible strain sensors suitable for motion detection. Smart Mater. Struct. 27, 89. https://doi.org/10.1088/1361-665X/aadc89 (2018).

Article Google Scholar

Li, Y., Liu, Y., Ma, Z. & Huang, P. A novel generative convolutional neural network for robot grasp detection on Gaussian guidance. IEEE Trans. Instrum. Meas. 71, 1–10. https://doi.org/10.1109/tim.2022.3203118 (2022).

Article Google Scholar

Ponraj, G. & Ren, H. Sensor fusion of leap motion controller and flex sensors using kalman filter for human finger tracking. IEEE Sens. J. 18, 2042–2049. https://doi.org/10.1109/jsen.2018.2790801 (2018).

Article ADS Google Scholar

Gao, S., Weng, L., Deng, Z., Wang, B. & Huang, W. Biomimetic tactile sensor array based on magnetostrictive materials. IEEE Sens. J. 21, 13116. https://doi.org/10.1109/jsen.2021.3068160 (2021).

Article ADS CAS Google Scholar

Li, Y. et al. Design and output characteristics of magnetostrictive tactile sensor for detecting force and stiffness of manipulated objects. IEEE Trans. Ind. Inform. 15, 1219–1225. https://doi.org/10.1109/tii.2018.2862912 (2019).

Article Google Scholar

Yang, H., Weng, L., Wang, B. & Huang, W. Design and characterization of high-sensitivity magnetostrictive tactile sensor array. IEEE Sens. J. 22, 4004–4013. https://doi.org/10.1109/jsen.2022.3145822 (2022).

Article ADS Google Scholar

Shaw-Cortez, W., Oetomo, D., Manzie, C. & Choong, P. Robust object manipulation for tactile-based blind grasping. Control Eng. Pract. 92, 136. https://doi.org/10.1016/j.conengprac.2019.104136 (2019).

Article Google Scholar

Wu, Y.-H. & Santello, M. Distinct sensorimotor mechanisms underlie the control of grasp and manipulation forces for dexterous manipulation. Sci. Rep. 13, 8. https://doi.org/10.1038/s41598-023-38870-8 (2023).

Article CAS Google Scholar

She, Y. et al. Cable manipulation with a tactile-reactive gripper. Int. J. Robot. Res. 40, 1385–1401. https://doi.org/10.1177/02783649211027233 (2021).

Article ADS Google Scholar

Iqra, M., Anwar, F., Jan, R. & Mohammad, M. A. A flexible piezoresistive strain sensor based on laser scribed graphene oxide on polydimethylsiloxane. Sci. Rep. 12, 1. https://doi.org/10.1038/s41598-022-08801-0 (2022).

Article CAS Google Scholar

Zhang, X. et al. Magnetic flexible tactile sensor via direct ink writing. Sens. Actuators A Phys. 327, 753. https://doi.org/10.1016/j.sna.2021.112753 (2021).

Article CAS Google Scholar

Kwon, J.-H., Kwak, W.-Y. & Cho, B. K. Magnetization manipulation of a flexible magnetic sensor by controlled stress application. Sci. Rep. 8, 1. https://doi.org/10.1038/s41598-018-34036-z (2018).

Article CAS Google Scholar

Makushko, P. et al. Flexible magnetoreceptor with tunable intrinsic logic for on-skin touchless human-machine interfaces. Adv. Funct. Mater. 31, 1089. https://doi.org/10.1002/adfm.202101089 (2021).

Article CAS Google Scholar

Zhou, Y. et al. A multimodal magnetoelastic artificial skin for underwater haptic sensing. Sci. Adv. 10, 8567. https://doi.org/10.1126/sciadv.adj8567 (2024).

Article Google Scholar

Ge, J. et al. A bimodal soft electronic skin for tactile and touchless interaction in real time. Nat. Commun. 10, 4405. https://doi.org/10.1038/s41467-019-12303-5 (2019).

Article ADS CAS PubMed PubMed Central Google Scholar

Huang, P. et al. Bioinspired flexible and highly responsive dual-mode strain/magnetism composite sensor. ACS Appl. Mater. Interfaces 10, 11197–11203. https://doi.org/10.1021/acsami.8b00250 (2018).

Article CAS PubMed Google Scholar

Zhou, Q. et al. Tilted magnetic micropillars enabled dual-mode sensor for tactile/touchless perceptions. Nano Energy 78, 382. https://doi.org/10.1016/j.nanoen.2020.105382 (2020).

Article CAS Google Scholar

Zhao, J. et al. Flexible organic tribotronic transistor for pressure and magnetic sensing. ACS Nano 11, 11566–11573. https://doi.org/10.1021/acsnano.7b06480 (2017).

Article CAS PubMed Google Scholar

Xu, J. et al. Flexible, self-powered, magnetism/pressure dual-mode sensor based on magnetorheological plastomer. Compos. Sci. Technol. 183, 820. https://doi.org/10.1016/j.compscitech.2019.107820 (2019).

Article CAS Google Scholar

Shu, Q. et al. Magnetic flexible sensor with tension and bending discriminating detection. Chem. Eng. J. 433, 424. https://doi.org/10.1016/j.cej.2021.134424 (2022).

Article CAS Google Scholar

Reid, A. H. et al. Beyond a phenomenological description of magnetostriction. Nat. Commun. 9, 388. https://doi.org/10.1038/s41467-017-02730-7 (2018).

Article ADS CAS PubMed PubMed Central Google Scholar

Suwa, Y., Agatsuma, S., Hashi, S. & Ishiyama, K. Study of strain sensor using FeSiB magnetostrictive thin film. IEEE Trans. Magn. 46, 666–669. https://doi.org/10.1109/tmag.2009.2033553 (2010).

Article ADS CAS Google Scholar

Kwon, J. H., Kwak, W. Y., Choi, H. Y., Kim, G. H. & Cho, B. K. Effects of repetitive bending on the magnetoresistance of a flexible spin-valve. J. Appl. Phys. 117, 4533. https://doi.org/10.1063/1.4914533 (2015).

Article CAS Google Scholar

Guo, L. et al. Wide linearity range and highly sensitive MEMS-based micro-fluxgate sensor with double-layer magnetic core made of Fe–Co–B amorphous alloy. Micromachines 8, 352. https://doi.org/10.3390/mi8120352 (2017).

Article PubMed PubMed Central Google Scholar

Cañón Bermúdez, G. S. & Makarov, D. Magnetosensitive E-skins for interactive devices. Adv. Funct. Mater. 31, 7788. https://doi.org/10.1002/adfm.202007788 (2021).

Article CAS Google Scholar

Hunter, D. et al. Giant magnetostriction in annealed Co1−xFex thin-films. Nat. Commun. 2, 1529. https://doi.org/10.1038/ncomms1529 (2011).

Article CAS Google Scholar

Zhao, Y. et al. Magnetostriction and structure characteristics of Co70Fe30 alloy prepared by directional solidification. J. Magn. Magn. Mater. 451, 587–593. https://doi.org/10.1016/j.jmmm.2017.11.046 (2018).

Article ADS CAS Google Scholar

del Carmen Aguirre, M. & Urreta, S. E. Effect of an external magnetic field orthogonal to the electrode surface on the electrocrystallization mechanism of Co–Fe films under pulsed applied potential. J. Alloys Compds. 878, 160347. https://doi.org/10.1016/j.jallcom.2021.160347 (2021).

Article CAS Google Scholar

Nicolenco, A. et al. Mechanical, magnetic and magnetostrictive properties of porous Fe–Ga films prepared by electrodeposition. Mater. Des. 208, 109915. https://doi.org/10.1016/j.matdes.2021.109915 (2021).

Article CAS Google Scholar

Cao, D. et al. Investigation on the structure and dynamic magnetic properties of FeCo films with different thicknesses by vector network analyzer and electron spin resonance spectroscopy. J. Alloys Compds. 688, 917–922. https://doi.org/10.1016/j.jallcom.2016.07.110 (2016).

Article CAS Google Scholar

Alper, M., Kockar, H., Sahin, T. & Karaagac, O. Properties of Co–Fe films: Dependence of cathode potentials. IEEE Trans. Magn. 46, 390–392. https://doi.org/10.1109/tmag.2009.2033711 (2010).

Article ADS CAS Google Scholar

Setiadi, R. N. & Schilling, M. Sideband sensitivity of fluxgate sensors theory and experiment. Sens. Actuators A Phys. 285, 573–580. https://doi.org/10.1016/j.sna.2018.11.049 (2019).

Article ADS CAS Google Scholar

Hristoforou, E., Ktena, A. & Gong, S. Magnetic sensors: Taxonomy, applications, and new trends. IEEE Trans. Magn. 55, 1–14. https://doi.org/10.1109/tmag.2018.2888642 (2019).

Article CAS Google Scholar

Yan, B., Zhu, W., Zhuang, X., Lu, Z. & Fang, G. Coil optimization in a fluxgate magnetometer with Co68.2Fe4.3Si12.5B15 amorphous wire cores for geomagnetic station observation. IEEE Trans. Instrum. Meas. 70, 1–7. https://doi.org/10.1109/tim.2021.3086904 (2021).

Article CAS Google Scholar

Sarkar, P. & Etemad, A. Self-supervised ECG representation learning for emotion recognition. IEEE Trans. Affect. Comput. 13, 1541–1554. https://doi.org/10.1109/taffc.2020.3014842 (2022).

Article Google Scholar

Ma, K. et al. End-to-end blind image quality assessment using deep neural networks. IEEE Trans. Image Process. 27, 1202–1213. https://doi.org/10.1109/tip.2017.2774045 (2018).

Article ADS MathSciNet PubMed Google Scholar

Amyar, A., Modzelewski, R., Li, H. & Ruan, S. Multi-task deep learning based CT imaging analysis for COVID-19 pneumonia: Classification and segmentation. Comput. Biol. Med. 126, 104037. https://doi.org/10.1016/j.compbiomed.2020.104037 (2020).

Article CAS PubMed PubMed Central Google Scholar

Deng, Z. & Dapino, M. J. Review of magnetostrictive materials for structural vibration control. Smart Mater. Struct. 27, 5. https://doi.org/10.1088/1361-665X/aadff5 (2018).

Article Google Scholar

Funabashi, S. et al. Multi-fingered in-hand manipulation with various object properties using graph convolutional networks and distributed tactile sensors. IEEE Robot. Autom. Lett. 7, 2102–2109. https://doi.org/10.1109/lra.2022.3142417 (2022).

Article Google Scholar

Download references

This work was supported by the National Natural Science Foundation of China (No. 51801053, 52077052, 52377007), the Natural Science Foundation of Hebei Province (E2022202067) and Central Guidance on Local Science and Technology Development Fund (No.226Z1704G).

State Key Laboratory of Reliability and Intelligence of Electrical Equipment, Tianjin, 300130, China

Qian Wang, Mingming Li, Pingping Guo, Liang Gao, Ling Weng & Wenmei Huang

The Key Laboratory of Electromagnetic Field and Electrical Apparatus Reliability of Hebei Province, School of Electrical Engineering, Hebei University of Technology, Tianjin, 300130, China

Qian Wang, Mingming Li, Pingping Guo, Liang Gao, Ling Weng & Wenmei Huang

You can also search for this author in PubMed Google Scholar

You can also search for this author in PubMed Google Scholar

You can also search for this author in PubMed Google Scholar

You can also search for this author in PubMed Google Scholar

You can also search for this author in PubMed Google Scholar

You can also search for this author in PubMed Google Scholar

Q.W. conducted the fabrication and design of flexible sensors and led the drafting of the article. Q.W. and L.G. designed with the experiments and interpretations. MML led the research work, designed the experiments. P.P.G., L.W. and W.M.H. reviewed the article draft.

Correspondence to Mingming Li.

The authors declare no competing interests.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.

Reprints and permissions

Wang, Q., Li, M., Guo, P. et al. Magnetostrictive bi-perceptive flexible sensor for tracking bend and position of human and robot hand. Sci Rep 14, 20781 (2024). https://doi.org/10.1038/s41598-024-70661-7

Download citation

Received: 15 April 2024

Accepted: 20 August 2024

Published: 06 September 2024

DOI: https://doi.org/10.1038/s41598-024-70661-7

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative