Using Miniaturized Strain Sensors to Provide a Sense of Touch in a Humanoid Robotic Arm

Introduction

In recent years, autonomous robots have begun performing various complex tasks previously undertaken only by humans. Although robots powered by artificial intelligence have become more advanced, in dynamic environments with unpredictable events such as interactions with moving obstacles and variable-shaped objects, human input is often necessary to provide extra information for the robots. As noted by Billard and Kragic, there is still considerable advancement in dexterous manipulation required to achieve useful capability in robotic handling techniques (Billard and Kragic, 2019). Therefore, master-slave robotic systems still have an important role to enhance the capabilities of both humans and robots in performing tasks in unknown and dangerous environments (Hashtrudi-Zaad and Salcudean, 2001; Aliaga et al., 2004; Tadakuma et al., 2005). Lee et al. note the challenges associated with artificial intelligence (AI) techniques for control of robotic equipment, which adds weight to utility of the approach of a simple human-assisted robotic system (Lee et al., 2019).

Master-slave manipulation has been extensively utilized in minimally invasive surgeries (MIS) (Li et al., 2012; Hwang et al., 2019), rehabilitation (Li et al., 2010), hazardous environments (Lee et al., 2012) including space (Karam et al., 2018; Liu et al., 2019), and agriculture. Various studies have detailed the design of master-slave robotic systems where surgeons' hand motions are mimicked to facilitate complex surgery with minimal invasive action (Wang et al., 2010). In such master-slave robotic systems, the surgeon controls a console with high precision and reliability (Sang et al., 2011). In particular, Da Vinci, by Intuitive Surgical (Guthart and Salisbury, 2000) is a surgical master-slave robotic system capable of highly dexterous movement, designed to improve the clinical outcomes of patients by using precise robotic control to minimize the trauma from surgical procedures. Recently, various types of surface-mountable, miniaturized and low-cost strain (Kottapalli et al., 2012; Xu et al., 2013; Sengupta et al., 2017; Khan et al., 2018) and pressure (Asadnia et al., 2015; Vujaklija and Farina, 2018) sensors have been utilized, allowing accurate sensing and control capability for robotic systems. The systems primarily rely on position control, or force or torque feedback from actuator systems. Such systems are in high demand for sensitive processes such as MIS, where freedom of operation of the surgical instruments is limited (Hwang et al., 2019). Using accurate control systems such as those mentioned above, or a robotic proxy for the human arm, would allow performing delicate tasks ranging from surgery, bomb defusing, or holding fragile objects.

Three-dimensional (3D) printing technology has initiated a revolution in the manufacturing industry because of the high level of customization not possible previously (Hagihghi et al., 2020; Moshizi et al., 2020). 3D printing has been commonly used to manufacture customized products including exoskeletons for rehabilitation purposes (Cabibihan et al., 2008; Syed et al., 2017), customized robots (Kim et al., 2005; Chang et al., 2009; Gao et al., 2016) development of soft actuators and sensors (Lee et al., 2008), biomechanics (Wang et al., 2014; Mukhopadhyay, 2015) and bionic devices (Asadnia et al., 2015, 2016; Nag et al., 2018b). Mick et al. built a 3D printed robotic arm to test human-robot control strategies, usable for teleoperation (Mick et al., 2019). Manufacturing of robotic systems has been significantly advanced through the use of 3D printing technologies. This has meant that the challenges to the development of new generations of robotic systems are now in the domain of advanced sensing and control systems.

Master-slave robotic systems based on tactile sensors have frequently been used for indirect manipulation of objects (Hwang et al., 2019). In addition to the speed and position information required for a master-slave system, force and vibration have also been used to improve the performance and realism of the system (Konyo, 2016). When grasping, touching, and gripping, human hands rely on tactile feedback from nerve endings. In order to mimic the human sense of touch, various force-sensing methods have been developed to control grasping force, which is essential for robots in physical contact with humans (Cabibihan et al., 2008) or the environment. Flexible sensors based on polymer materials have been developed for use on various curved surfaces (Kim et al., 2005; Chang et al., 2009; Khan et al., 2018). In polymer-based sensors, resistive and capacitive sensing mechanisms are frequently used due to their characteristics of good sensitivity, high response time and low power consumption (Gao et al., 2016). Capacitive sensing is known to be very sensitive and is widely used in applications where high-resolution is required (Lee et al., 2008; Wang et al., 2014). Resistive sensing elements can be realized as sensor arrays with good sensitivity and a reliable response (Mukhopadhyay, 2015). More recently, 3D printing and laser-cutting technologies have been used to develop flexible strain sensors for robotic hand development (Nag et al., 2018b). Our group has developed various types of microelectromechanical systems (MEMS) strain (Khan et al., 2018), pressure (Asadnia et al., 2015), and flow (Kottapalli et al., 2015; Asadnia et al., 2016) sensors promising for use in robotic applications.

Force sensing resistors (FSRs) play a crucial role in human-robot interactions (Argall and Billard, 2010). An FSR consists of two polymer membranes made of a flexible substrate with a printed semi-conductor, paired with a flexible substrate with printed interdigitating electrodes. When a force is applied to the active surface, the device experiences a change in the resistance that approximately follows an inverse power-law characteristic. When the active surface is pressed down, the material deforms and pushes against the bottom conductive film, allowing a small electrical current (varying with the applied pressure) to flow. This characteristic suggests the FSR as a promising candidate to mimic the sense of touch of human skin.

E-skin has in recent years progressed significantly in the emulation of characteristics of human skin, which allows for placement of sensors such as the BioTac and force sensors using embedded microfluidic channels on a robotic hand (Vogt et al., 2013). These sensors can measure the effort being applied to pick up objects and judge the required amount of pressure or force. There have also been advances in developing an e-skin that uses a range of pressure and temperature sensors in an integrated device. Dahiya and Ravinder emphasize the importance of touch for sensing multiple parameters, critical for applications as varied as social robotics, throught to Factory 4.0 (Dahiya, 2019). In addition to sensing pressure and temperature, such devices can discriminate between different types of materials, and also perceive air or water movement over the sensor by means of convective heat transfer (Zhao and Zhu, 2017).

The primacy of touch sensing was shown by Taunyazov et al., in their experiments using multimodal sensing techniques to discriminate between various surface types (Taunyazov et al., 2019).

In this paper, we develop a master-slave robotic hand system with a good sense of its environment, able to pick up delicate objects. We aim to shed light on the usage of force resistive sensors in prosthetics, MIS processes, and other fields where robotic hands could be useful. We implement a simple, low cost open loop system capable of picking up arbitrary objects and generating data for basic robots operating with a repetitive task. Using off-the-shelf, components and available technologies, we implement a novel implementation of a robotic hand. The result is a well-developed bespoke control system to suit the equipment available and the application.

The section Fabrication of the Slave Robotic Hand System of this paper describes the technology and processes involved in developing the robotic hand system, The section Discussion and Results provides a details of the results achieved in the testing of the hand's performance when handling objects, and the Conclusion summarizes our work and suggests promising avenues for further study.

Fabrication of the Slave Robotic Hand System

In past years, there have been significant advancements in the area of flexible sensors and electronic skin (e-skin). Control systems such as the wearable device designed and discussed in Dhillon and Horch (2005) are able to measure the positions of each joint in the human arm accurately, but due to the size and structure of the analog design are only effective to the wrist. For accurate measurement of the joint positions of the hand, another control input method such as optical or inertial systems (compared in van den Noort et al., 2016), or the data glove with force sensing capability developed in Fahn and Sun (2005), are required. Direct control by the nervous system of both force and position without the usual feedback is achievable in the case of amputation patients using neural control sensors mounted to peripheral nerve stumps as discussed in Dhillon and Horch (2005), and is shown to increase the successful integration of artificial limbs.

Figure 1 illustrates the arrangement of the master-slave robotic hand system proposed in this study. Flexible tactile sensors on the master hand provide force information that the robotic hand will attempt to match with the force measured on the slave hand sensors as it grasps an object. Force sensors utilizing the Piezoresistive effect represent an advance in methods available to convert mechanical force into an electrical signal for use in systems such as robotics (Nag et al., 2018a). Here we discuss different types of flexible sensors that could be used in conjunction with a robotic hand to measure and identify the forces required for picking up and holding objects.

FIGURE 1

Figure 1. A schematic of the proposed master-slave robot. Tactile sensors are attached to both glove and bionic hand. The controller sends the force command to the bionic hand (slave) based on the signal from the glove-mounted tactile sensors.

Force Sensing Resistors (FSR)

FSR sensors have been developed in various forms, as shown in Figure 2. These sensors are suitable for detecting forces of differing magnitudes but are not on their own suitable for precision measurements. The sensors can be supplied with active surfaces of different sizes and shapes. The FSR sensors have several advantages over comparable sensing techniques, such as small size, low cost, and excellent shock resistance (Flórez and Velásquez, 2010).

FIGURE 2

Figure 2. Different forms of Force Sensing Resistors (A) circle-shaped FSR, (B) square-shaped FSR, and (C) strip-shaped FSR.

A voltage divider can be used to convert the resistance change in the FSR to a change in voltage. The decrease in resistance across the FSR results in a change of voltage at the junction of the voltage divider, which can then be measured with conventional analog instrumentation. An appropriate selection of static resistor in the voltage divider is essential to ensure proper resolution and sensing range (Asadnia et al., 2013). For most applications, the static resistor will be toward the center of sensing range of FSR. Interlink FSR sensors have been utilized for a range of products due to their robustness and ability to detect widely varying forces at high frequencies (Saadeh and Trabia, 2012).

Design of Master-Slave Robotic Hand

In this study, a robotic hand was 3D printed and then equipped with Robotis Dynamixel AX-12A servos, in order to gain a qualitative understanding of how the sensors can be used when gripping an object and to suggest improvements to the system. Preliminary tests were performed to develop a better understanding of the sensors and their response to different forces. This was performed by connecting each sensor to a simple voltage divider, and applying a force to each sensor in turn, then recording the resistance and estimated applied force.

A single FSR sensor was used with the AX-12A Gripper that set the gripper at a specific force as calculated from the force vs. resistance graph for the FSR sensors. A proportional controller was used to make the action as smooth as possible and tests were carried out using varying gripping forces to determine its capabilities.

Initially, the tests were conducted on each of the hand's fingers to find the limits of their movements, to ensure that we were operating the servos within safe limits. Following this, the sensors were attached to the fingers and experiments were conducted various gripping objects. Similar tests were performed with a Dynamixel gripper by adjusting gain and delay values to improve the control of the hand. Objects with different weights and shapes such as a plastic cup and a screwdriver were tested with the robotic hand to evaluate its performance when gripping the various items.

Our primary work was on the interface between the robotic hand and the objects being picked up, so the robustness, responsiveness, and durability of the FSR sensors were important factors. In order to estimate the required amount of feedback, different sized sensors were tested to find the required active area on the slave robotic hand. Flexible epoxy was used to mount the FSR sensors on a smart glove, then a human finger inserted into the glove. As the finger was flexed, we measured the bend as detected by the sensor and applied it as an input to a controller which moved a robotic finger correspondingly. The smaller FSR sensors used had sensitivity in the range of up to 10N whereas the larger sensors can handle forces up to 100N.

After testing a range of sensors, it was found that the 0.5” FSR (SF-SEN-09375) sensors were the most suitable for our application, as their size allows them to be fitted to the tip of the finger on the glove or on the end of a robotic finger, and they provide sufficient contact area to sense an object, while not restricting movement.

Three different Flex sensors and two different sized static resistors with the resistance of 3.3 and 47 kΩ were tested, in order to characterize the sensory system. Table 1 indicates the experimental results of testing different types of FSRs by varying the amounts of force. This table shows that for the 0.5” and small Interlink FSR sensors, static resistors of ~60 and 70 kΩ are needed, while a lower value resistor is required for the Square FSR.

TABLE 1

Table 1. Resistance response from rudimentary pressure application.

A second test was developed using an FSR sensor, a flex sensor, and a small servo. The flex sensor was connected via a voltage divider, and a small servo was used to simulate the robotic finger movement between 0 and 180 degrees. The FSR sensor was used to simulate the robotic finger making contact with an object. When the FSR sensor was pressed, the servomotors move in small movements to simulate gripping the object with varying amounts of force. The electrical schematic for this system used in these experiments is shown in Figure 3.

FIGURE 3

Figure 3. Electrical schematic of the circuit used in the experiments.

To eliminate erratic movement, a modified proportional controller was used such that the controller would take the error difference between the two corresponding sensors on the robotic hand and the smart glove, and when the output of the smart glove sensor was greater than the robotic hand sensor's output, it closed the gripper until it met an object and the signal on the robotic hand sensor increased to match the value on the glove sensor.

The circuit used for the experiments with the control loop can be characterized by the basic equation for a proportional loop control:

$\begin{array}{l}G=\frac{F_a}{F_{sp}}=\frac{1}{1+k_p.k_s}\end{array}$

Where Fa is the force applied by the slave, F_sp is force setpoint from the user glove, k_p is adjustable controller gain constant, and k_s the system gain.

As can be seen by inspection, the accuracy of force matching is determined by the system gain. Maximizing this system gain results in maximum development of the system response, and hence the minimum difference between the force sensed from the glove, and the force applied to the slave hand object. Due to non-linearities and delays in the system, the useful k_p values are limited by the onset of hunting and instability.

We expect that the system timing and accuracy could be improved through the use of a more complex control algorithm, which could select between different control loop parameters, depending on whether the force sensors were in contact with an object or not. A proportional-integral controller may improve accuracy, but likely at the expense of response time and the simplicity of our current system.

This can be seen from the Supplementary Video 1. To find the appropriate pressure values for the proportional controller, experiments were conducted with the AX-12A servo while the sensors were mounted inside the gripper. The test sequence waited for 2 s, then simulated a user force of 2 Newtons (N). The gripper would then close on the object and try to match the 2 N by adjusting the force exerted on the object. After a further 8 s the gripper would release. Different pressure values ranging from 0.02 to 0.08 kPa were tried for different simulated forces of 0.5, 2.0, and 3.0 N.

The initial code available for the Dynamixel Smart Shield controlling the AX-12 Servos had a limited position resolution of 1° rotation of the servo. It was found that this was inadequate for the fine movement needed to adjust the varying amount of force applied to the FSR sensors. We modified the control hardware to directly write to the servos from the Arduino microcontroller in order to bypass the above limitation. This required a minor modification to the communications wiring to adapt the Arduino's standard RS-232 communications to half-duplex configuration. This modification allowed for a quicker response time of the servo as well as a finer position resolution corresponding to better than 0.3° rotation.

3D Printing the Robotic Hand

3D printing technology has been widely employed in developing bionic robotic hands (Vujaklija and Farina, 2018). Two well-known 3D printed robotic hands are the InMoov Robotic hand by Gael Langevin (2014) and the Ada Robotic hand by Open Bionics (Introducing the Hero Arm). The InMoov robotic hand was initially designed as a way to visualize hand gestures and subsequently grew into a fully functioning robotic hand. It features individually movable fingers that operate via tendons connected to five servos located in the forearm of the model. The model has extra hinge points on each of the little finger and ring finger to better resemble a human hand's grip. The configuration of the servo system with two tendons for each finger means one tendon is tightened to close the finger, and the other is tightened to open it. This has the advantage that the tendons are only functional in tension, allowing the 3D printing process to be relatively simple, as it only needs one type of Acrylonitrile butadiene styrene (ABS) filament and does not need to use a flexible filament which could be difficult to print. The Ada Robotic Hand is a fully 3D printable robotic hand developed by Open Bionics (Bionics), derived from the Dextrus robotic hand developed by the Open Hand project (Gibbard, 2013) and now continued by Open Bionics. The Ada Hand is operated by five linear actuators and is printed using standard ABS filament for the main components and Ninjaflex filament for the hinges. It operates from a custom PCB board based on the AVR ATMega 2560 microcontroller by Microchip Technology. This robotic hand uses linear actuators to move the fingers rather than rotational servos. The advantage of this robotic hand is that there is no slack in any tendons; therefore, the actuators can be located in the palm of the hand rather than the forearm. An example of a master-slave robotic hand based on position sensors can be seen in Karam et al. (2018).

For this study, the 3D printed InMoov Open Source prosthetic hand was chosen. The design allowed us flexibility in the choice of operating mechanism, with the servos located in the arm. Light nylon fishing line was used to form the tendons, as they will always be in tension. This saves space and allows a simple way for the force to be transmitted through bends in the structure. The hand was printed by an Omni3D Factory 3D printer using standard 1.75 mm ABS filament with 20% infill, three shells, and no support. The finger components were glued together and each of the holes for the hinges on each of the fingers was drilled out to 3 mm for the outside and 3.2 mm for the inside. Three millimeter filament was then used to make the hinges.

Braided fishing line was used for the tendons as it is low stretch and high strength. The line was threaded through the corresponding holes for each finger. The fishing line was then tied off at the tip of each finger before the final fingertip cover was glued on. Four twisted pair cable pairs were used to connect each of the sensors which were placed the hands' fingertips. The twisted-pair cables were also threaded through their respective holes and exited at the base of the fingers. The wires were not threaded through the actual fingers as they would have impaired the bending of fingers and interfered with the fishing line tendons.

Due to space restrictions, we could not mount the AX-12A servos within the space available is the cavity of the InMoov arm. So five TowerPro MG946R servos were mounted on a servo bed in the forearm of the hand and the five servo pulleys attached to the servos with screws. The fishing line was then threaded through the servo pulleys and tied off taut, while the servos were in the neutral position and each of the fingers was in an open and flat orientation. The caps for each of fingers were then hot glued onto each of the corresponding fingers, the FSR sensors soldered onto the ends of the cabling at the ends of the fingers, and then attached to the tips of the fingers. The Arduino code was modified for control of the TowerPro servos, as these are controlled using Pulse width modulation (PWM), rather than the serial communications used by the Dynamixel devices. This used the standard Arduino Servo library. The extent of movement for each of the fingers was then established, so that appropriate range of motion for the servo could be set.

Five FSR sensors were placed on the fingertips of a leather glove, and those sensors and the sensors on the slave hand were connected to voltage divider circuits with 3.3 k Ω resistors, which were then in turn connected to analog input pins on the Arduino. The final attachment for the sensors to the slave hand fingertips was made using hook and loop material (Velcro) attached to the tip and back of the sensors. This allowed for interchanging of sensors and to trial different locations on the fingertip to achieve maximum contact with the objects being picked up. The sensors were then connected to an identical voltage divider as that for the user's glove with the FSR sensor. Figure 4 shows the different joints in a biological finger and the developed InMoov robotic hand with different finger movements.

FIGURE 4

Figure 4. (a) Diagram showing joints of the fingers (Reprinted with permission from Wheatland et al., 2015), (b) 3D printed InMoov robotic hand showing varying finger movements, and the FSR sensors mounted on the fingers.

The 3D printed hand was assembled with the MG946R servos acting as actuators and braided fishing line as tendons. The original design had the thumb closing to the palm instead of to the fingers when gripping an object. This was later corrected by fabricating and fitting an adjustable thumb whose angle of movement could be changed to allow different objects to be picked up. The position of the force sensors on the fingertips was critical to correct operation, as it needs to directly meet the surface of the object being picked up.

Another problem with the use of the proportional controller with the MG946R servos was that when the proportional gain (kp) value was excessive, as mentioned above the controller would overcompensate the position command sent to the servo, which resulted in instability of the finger position. This meant that the fingers would not settle on the object and would oscillate back and forth instead of gripping it effectively. On the other hand, when the kp value was lowered to below 0.01, the fingers would not start moving until there was a relatively large difference (typically 0.5 N) between the master and slave sensors.

A further significant problem was the correct positioning of sensors on the fingertips, and how the whole finger articulated. With the exception of the thumb, there were three hinges on each of the fingers, and one servo to control each finger, so they each folded inward differently depending on the stiffness of each of the hinges, as shown in Figure 4b. For example, if the metacarpophalangeal (MCP) joint as shown in Figure 4a was tighter than the proximal interphalangeal (PIP) joint and distal interphalangeal (DIP) joint, then a command to close the finger results in a first movement of curling the finger (red line in Figure 4b), whereas if the PIP and DIP joints were stiffer than the MCP joint, the finger would bend from its base first (blue line in Figure 4b. This presented a problem, as different actions were required for holding different shaped objects. The “initially curl” behavior was a problem when gripping a larger round object such as a cup or bottle, because the sensor would not hit the surface of the object orthogonally and the finger would push the object away. The “initially-bend” behavior was much more suited to picking up most objects as the sensor would hit the surface, and valid reading of the force being applied could then be obtained.

The control code was modified to suit the MG946R servos, as they had different characteristics and control methods from that of the AX-12A devices used for the robotic gripper. The Arduino standard servo library has a resolution of only 1° of rotation, which was too coarse for the positioning accuracy required for our experiment. To improve the resolution, we tested the Arduino “writeMicroseconds” function to determine if a better resolution was possible for the servos' movement. WriteMicroseconds is a command that specifies a PWM “on-time” cycle in microseconds to the servo (This would normally be in the range of 1,000 for fully counterclockwise and 2,000 for fully clockwise, with 1,500 as the neutral position). This allows a resolution of 0.18 ° in the command signal. While servos typically have a deadband of 5 μs, for the MG946R this is 1 μs, allowing the system to correctly respond to the more accurate command signal. The code was modified to test whether this could improve the ability of the robotic fingers to reach the required force without instability. The control algorithm shown in Figure 5 processes the input signals to control the output to the slave hand in a number of steps. The input force signals from both the master and the slave fingers are read, scaled and linearized to provide the system with a good representation of the actual force on the sensors. Then the output to the slave servo is adjusted so as to more closely match that of the master. If there is no force on either, the finger will open. Finally the loop will wait for a delay time to provide the required sample period for the system, then repeat the process.

FIGURE 5

Figure 5. Control loop algorithm.

Discussion and Results

Two different objects were initially picked up using the master-slave robotic hand, as shown in Figure 6. The first object was very lightweight (9 g) plastic cup (Figure 6a). For the initial test, instead of measuring the force from the master hands' glove, we applied a steady force setpoint of 0.5 N for the slave hand to match. Following this, a target setpoint obtained from force sensors on the master hand was used to dictate the force used to pick up a plastic cup. The second object tested was a heavier item (screwdriver with a weight of 320 g) (Figure 6b) with a large handle. The testing protocol was the same as for the cup, i.e., a fixed 0.5 N setpoint for a set time, and a test with force derived from the sensors on the master glove. The master-slave robot showed outstanding performance in handling heavy and delicate objects by relying on the force sensors. Details have been further discussed in below sections.

FIGURE 6

Figure 6. Grip test (a) with the robotic hand gripping the plastic cup (9 g weight), and (b) with the robotic hand gripping the screwdriver (weight of 320 g).

Grip Testing 1—Cup

The first experiment simulated a steady force of 0.5 N on the thumb and index finger, to which the robotic hand would then respond, gripping a plastic cup. The actions were trialed with different proportional gain values to test which value resulted in better grip and response. The user's force for the thumb and the index finger were both set to 0.5 N. As illustrated in Figure 7, kp values higher 0.03 made the system unstable with the index finger and thumb not being able to settle at a suitable force while trying to grip a cup at 0.5N.

FIGURE 7

Figure 7. Slave hand position and gripping force on a plastic cup, with a simulated master force of 10 N, (A) applied by the index finger, using system proportional gain of kp = 0.015, (B) applied by the index finger, with gain of kp = 0.03, (C) applied by thumb, with system proportional gain of kp = 0.015, and (D) applied by thumb, with system proportional gain of kp = 0.03.

Various timing modifications were also trialed to improve the system's stability. The response of the control loop was slowed by increasing the sample period as a qualitative test to determine if slowing the system down improved its performance. These results are shown in Figure 7.

From these results, it can be seen how with the set force of 0.5 N for a duration of 8 s, and with kp set to 0.015, the index finger and thumb both respond to the force sensors and grip the cup successfully. Increasing the gain to 0.03 resulted in instability (Figures 7C,D. After testing a range of kp values, we found it possible to maintain a force within 0.1 N of the force signal setpoint. In order to deal with the lag in movement of the robot fingers in response to positioning commands, we then introduced a delay of 60 ms on top of the execution time of the control program of 40 ms. This increased the effective sample time of the system to 100 ms, allowing the signal from each sensor to stabilize. Next, the robotic hand was used as part of a master-slave system in order to test the system's performance in gripping a plastic cup. The user's glove gripped the plastic cup between the index finger and thumb, while the robotic hand was positioned near an identical cup. As shown in Figures 7A,B, the force on the robotic hand closely followed the force signal from the user's glove. The robotic hand has a maximum force of circa 1.5 N (corresponding to the maximum force output of the servo). In some instances, the servo reached the extent of its allowed movement, and in others, we could not achieve the required force due to the fishing line and 3D printed structure stretching and/or deforming under load. The graphs show that when gripping at different forces (as would be common when holding an item like a cup), the robotic hand was able to follow the force required with a reasonable response. The procedure was repeated with the robotic hand around the cup while the user's glove was used to generate the force target for the robotic hand. The user picked up a plastic cup with the index finger and thumb, and the robotic hand did the same with an identical cup, and the results were recorded.

Grip Testing 2—Screwdriver

The next experiment was to pick up a heavier item such as a screwdriver (weight 320 g). The screwdriver needs to be gripped by all of the fingers, with all of the force sensors contacting the surface of the screwdriver. Initially, the hand was tested with a set force of 0.5 N with different values of kp, as shown in Figure 8. These graphs show that the servos took some time to stabilize but gripped the screwdriver well. The glove was then used to pick a similar object, measuring the required force, while the screwdriver was placed in the robotic hand's grip. The control loop was also tested with different kp values. A modified proportional controller was developed in the code, comparing the two analog signals coming from each of the sensors, with the aim of reducing the speed of the gripper as the readings from the two sensors converged. The difference between the two analog signals was taken as the error, then multiplied by an arbitrary kp value. The error value was used to adjust the servo position command in the corresponding direction. Limits were used to keep the gripper inside the bounds of its range. The tests were done at set force commands of 0.5, 2.0, and 3.0 N, using a range of kp values. The results were promising with the apparatus have a settling time of 0.5 s in most cases and a steady-state error of <0.25 N in most cases. The results are shown in Figure 8. The experiment was successful as the robotic hand could hold the screwdriver firmly without dropping it. This again demonstrated that the system can apply a force closely tracking the force measured by the user's glove.

FIGURE 8

Figure 8. Gripping screwdriver with force setpoint derived from user's glove. (A) Index finger, (B) middle finger, (C) ring finger, (D) little finger, and (E) thumb.

The thumb was also found to have an important role in how well the hand held objects. Originally the thumb movement was to fold to directly into the palm and not toward the fingers, resulting in a lack of ability to grip round objects effectively, as the sensor on the thumb did not press against the object being held. When the thumb's action was altered so that it closed toward the other fingers in a more natural manner, the sensor would touch the surface of the object and allow reasonable control of the force exerted by the thumb. These results indicate the potential for this approach to be extended to other systems requiring fine control of force, in fields as diverse as surgery, bomb disposal, and others.

Conclusion

In this paper, we developed a master-slave robotic hand based on force-sensing resistive sensors. A 3D model of a bionic hand was designed, and 3D printed to function as a slave robotic hand. The hand was fitted with servo actuators (serving as an analog of human muscles) connected to nylon fishing line (serving as the tendons) to move the fingers. A specially fitted out glove with flexible sensors attached was used as a master hand, which measured force signals for the slave to follow. The motion of the slave robotic hand was controlled by the master hand using the signal from the force sensors attached to both master and slave hands. With careful adjustment of the mechanical structure and the dynamics of the control system, the robotic hand was able to grasp objects of different sizes and shapes. Using two types of actuator, we achieved steady control exerting a fixed force on a selection of objects. The system proved quite sensitive to the geometry of the hand structure and to the tuning of the control loops. As the system has a relatively low data rate requirement between the master and the slave hand, it shows good potential for eventual use in remote master-slave applications as diverse as remote surgery, or explosives disposal.

Data Availability Statement

All datasets generated for this study are included in the article/Supplementary Material.

Author Contributions

AB has extensive practical experience with control and automation, his work was associated with experiments and interpretation of the results and made a major contribution to the manuscript. SM has experience with biomimetic sensors and sensing, he helped with the measurement and sensing aspects of the project and contributed to the manuscript. AR has extensive experience with materials and coatings, his contribution provided important input into the sensing and grip aspects of the experiments. MA is greatly experienced in biomimetic sensing, particularly relating to piezo-resistive devices, he was responsible for originating and supervising the project. All authors contributed to the article and approved the submitted version.

Funding

This study was supported by the Australian Research Council Centre Discovery Early Career Researcher Award (DECRA) DE180100688.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fmech.2020.550328/full#supplementary-material

References

Aliaga, I., Rubio, A., and Sanchez, E. (2004). Experimental quantitative comparison of different control architectures for master-slave teleoperation. IEEE Trans. Control Syst. Technol. 12, 2–11. doi: 10.1109/TCST.2003.819586

CrossRef Full Text | Google Scholar

Argall, B. D., and Billard, A. G. (2010). A survey of tactile human–robot interactions. Robot. Auton. Syst. 58, 1159–1176. doi: 10.1016/j.robot.2010.07.002

CrossRef Full Text | Google Scholar

Asadnia, M., Kottapalli, A., Miao, J., Randles, A., Sabbagh, A., Kropelnicki, P., et al. (2013). High temperature characterization of PZT (0.52/0.48) thin-film pressure sensors. J. Micromech. Microeng. 24:015017. doi: 10.1088/0960-1317/24/1/015017

CrossRef Full Text | Google Scholar

Asadnia, M., Kottapalli, A. G. P., Karavitaki, K. D., Warkiani, M. E., Miao, J., Corey, D. P., et al. (2016). From biological cilia to artificial flow sensors: biomimetic soft polymer nanosensors with high sensing performance. Sci. Rep. 6:32955. doi: 10.1038/srep32955

CrossRef Full Text | Google Scholar

Asadnia, M., Miao, J., Warkiani, M. E., Triantafyllou, M. S., and Kottapalli, A. G. P. (2015). Artificial fish skin of self-powered micro-electromechanical systems hair cells for sensing hydrodynamic flow phenomena. J. R. Soc. Interface 12:20150322. doi: 10.1098/rsif.2015.0322

CrossRef Full Text | Google Scholar

Billard, A., and Kragic, D. (2019). Trends and challenges in robot manipulation. Science 364:eaat8414. doi: 10.1126/science.aat8414

CrossRef Full Text | Google Scholar

Cabibihan, J.-J., Pattofatto, S., Jomâa, M., Benallal, A., and Carrozza, M. C. (2008). Towards Humanlike social touch for sociable robotics and prosthetics: comparisons on the compliance, conformance and hysteresis of synthetic and human fingertip skins. Int. J. Soc. Robot. 1, 29–40. doi: 10.1007/s12369-008-0008-9

CrossRef Full Text | Google Scholar

Chang, W.-Y., Fang, T.-H., Yeh, S.-H., and Lin, Y.-C. (2009). Flexible electronics sensors for tactile multi-touching. Sensors 9, 1188–1203. doi: 10.3390/s9021188

CrossRef Full Text | Google Scholar

Dahiya, R. (2019). “E-skin: from humanoids to humans,” in Proceedings of the IEEE. 107, 247–252.

Google Scholar

Dhillon, G. S., and Horch, K. W. (2005). Direct neural sensory feedback and control of a prosthetic arm. IEEE Trans. Neural. Syst. Rehabil. Eng. 13, 468–472. doi: 10.1109/TNSRE.2005.856072

CrossRef Full Text | Google Scholar

Fahn, C. S., and Sun, H. (2005). Development of a data glove with reducing sensors based on magnetic induction. IEEE Trans. Ind. Electron. 52, 585–594. doi: 10.1109/TIE.2005.844259

CrossRef Full Text | Google Scholar

Flórez, J. A., and Velásquez, A. (2010). “Calibration of force sensing resistors (fsr) for static and dynamic applications,” in 2010 IEEE ANDESCON (Bogota: IEEE), 1–6.

Google Scholar

Gao, S., Arcos, V., and Nathan, A. (2016). Piezoelectric vs. capacitive based force sensing in capacitive touch panels. IEEE Access 4, 3769–3774. doi: 10.1109/ACCESS.2016.2591535

CrossRef Full Text | Google Scholar

Gibbard, J. (2013). Open Hand Project. Available online at: http://www.openhandproject.org/news.php

Google Scholar

Guthart, G. S., and Salisbury, J. K. (2000). “The Intuitive/sup TM/telesurgery system: overview and application,” in Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (San Francisco, CA), 618–621.

Google Scholar

Hagihghi, R., Razmjou, A., Orooji, Y., Warkiani, M. E., and Asadnia, M. (2020). A miniaturized piezoresistive flow sensor for real-time monitoring of intravenous infusion. J. Biomed. Mater. Res. B Appl. Biomater. 108, 568–576. doi: 10.1002/jbm.b.34412

CrossRef Full Text | Google Scholar

Hashtrudi-Zaad, K., and Salcudean, S. E. (2001). Analysis of control architectures for teleoperation systems with impedance/admittance master and slave manipulators. Int. J. Rob. Res. 20, 419–445. doi: 10.1177/02783640122067471

CrossRef Full Text | Google Scholar

Hwang, Y. H., Kang, S. R., Cha, S. W., and Choi, S. B. (2019). A robot-assisted cutting surgery of human-like tissues using a haptic master operated by magnetorheological clutches and brakes. Smart Mater. Struct. 28:065016. doi: 10.1088/1361-665X/ab15bc

CrossRef Full Text | Google Scholar

Introducing the Hero Arm. Available online at: https://openbionics.com/ (accessed October 01 2020).

Google Scholar

Karam, Z. A., Al-Kadhimi, A. M., and Saeed, E. A. (2018). “Design and implementation of a wireless robotic human hand motion-controlled using Arduino,” in 2018 International Conference on Advanced Science and Engineering (Duhok: ICOASE).

Google Scholar

Khan, H., Razmjou, A., Ebrahimi Warkiani, M., Kottapalli, A., and Asadnia, M. (2018). Sensitive and flexible polymeric strain sensor for accurate human motion monitoring. Sensors 18:418. doi: 10.3390/s18020418

CrossRef Full Text | Google Scholar

Kim, S.-H., Engel, J., Liu, C., and Jones, D. L. (2005). Texture classification using a polymer-based MEMS tactile sensor. J. Micromech. Microeng. 15:912. doi: 10.1088/0960-1317/15/5/003

CrossRef Full Text | Google Scholar

Konyo, M. (2016). “Remote transmission of multiple tactile properties,” in Pervasive Haptics (Tokyo: Springer), 285–303.

Google Scholar

Kottapalli, A. G. P., Asadnia, M., Miao, J., and Triantafyllou, M. (2015). Soft polymer membrane micro-sensor arrays inspired by the mechanosensory lateral line on the blind cavefish. J. Intell. Mater. Syst. Struct. 26, 38–46. doi: 10.1177/1045389X14521702

CrossRef Full Text | Google Scholar

Kottapalli, A. G. P., Asadnia, M., Miao, J. M., Barbastathis, G., and Triantafyllou, M. S. (2012). A flexible liquid crystal polymer MEMS pressure sensor array for fish-like underwater sensing. Smart Mater. Struct. 21:115030. doi: 10.1088/0964-1726/21/11/115030

CrossRef Full Text | Google Scholar

Langevin, G. (2014). Hand and Forarm. Available online at: http://inmoov.fr/hand-and-forarm/(accessed October 01, 2020).

Google Scholar

Lee, H.-K., Chung, J., Chang, S.-I., and Yoon, E. (2008). Normal and shear force measurement using a flexible polymer tactile sensor with embedded multiple capacitors. J. Microelectromech. Syst. 17, 934–942. doi: 10.1109/JMEMS.2008.921727

CrossRef Full Text | Google Scholar

Lee, J. K., Lee, H. J., Park, B. S., and Kim, K. (2012). Bridge-transported bilateral master–slave servo manipulator system for remote manipulation in spent nuclear fuel processing plant. J. Field Robot. 29, 138–160. doi: 10.1002/rob.20419

CrossRef Full Text | Google Scholar

Lee, M. A., Zhu, Y. K., Srinivasan, K., Shah, P., Savarese, S., Li, F. F., et al. (2019). “Making sense of vision and touch: self-supervised learning of multimodal representations for contact-rich tasks” in 2019 International Conference on Robotics and Automation (ICRA) (Montreal, QC), 8943–8950.

Google Scholar

Li, C., Inoue, Y., Liu, T., Shibata, K., and Oka, K. (2010). Additional design features of a master–slave control system with force sensing and energy recycling for upper limb rehabilitation robots. Instrument. Sci. Technol. 38, 385–410. doi: 10.1080/10739149.2010.508322

CrossRef Full Text | Google Scholar

Li, J., Zhou, N., Wang, S., Gao, Y., and Liu, D. (2012). Design of an integrated master–slave robotic system for minimally invasive surgery. Int. J. Med. Robot. Comp. Assist. Surg. 8, 77–84. doi: 10.1002/rcs.439

CrossRef Full Text | Google Scholar

Liu, G., Geng, X., Liu, L., and Wang, Y. (2019). Haptic based teleoperation with master-slave motion mapping and haptic rendering for space exploration. Chin. J. Aeronaut. 32, 723–736. doi: 10.1016/j.cja.2018.07.009

CrossRef Full Text | Google Scholar

Mick, S., Lapeyre, M., Rouanet, P., Halgand, C., Benois-Pineau, J., Paclet, F., et al. (2019). Reachy, a 3D-printed human-like robotic arm as a testbed for human-robot control strategies. Front. Neurorobot. 13:65. doi: 10.3389/fnbot.2019.00065

CrossRef Full Text | Google Scholar

Moshizi, S. A., Azadi, S., Belford, A., Razmjou, A., Wu, S., Han, Z. J., and Asadnia, M. (2020). Development of an ultra-sensitive and flexible piezoresistive flow sensor using vertical graphene nanosheets. Nano-Micro Lett. 12:1–18. doi: 10.1007/s40820-020-00446-w

CrossRef Full Text | Google Scholar

Mukhopadhyay, S. C. (2015). Wearable sensors for human activity monitoring: a review. IEEE Sens. J. 15, 1321–1330. doi: 10.1109/JSEN.2014.2370945

CrossRef Full Text | Google Scholar

Nag, A., Afasrimanesh, N., Feng, S., and Mukhopadhyay, S. C. (2018a). Strain induced graphite/PDMS sensors for biomedical applications. Sens. Actuat. A Phys. 271, 257–269. doi: 10.1016/j.sna.2018.01.044

CrossRef Full Text | Google Scholar

Nag, A., Menzies, B., and Mukhopadhyay, S. C. (2018b). Performance analysis of flexible printed sensors for robotic arm applications. Sens. Actuat. A Phys. 276, 226–236. doi: 10.1016/j.sna.2018.04.031

CrossRef Full Text | Google Scholar

Saadeh, M. Y., and Trabia, M. B. (2012). Identification of a force-sensing resistor for tactile applications. J. Intell. Mater. Syst. Struct. 24, 813–827. doi: 10.1177/1045389X12463462

CrossRef Full Text | Google Scholar

Sang, H., Wang, S., Li, J., He, C., Zhang, L.a, and Wang, X. (2011). Control design and implementation of a novel master–slave surgery robot system, MicroHand A. Int. J. Med. Robot. Comp. Assist. Surg. 7, 334–347. doi: 10.1002/rcs.403

CrossRef Full Text | Google Scholar

Sengupta, D., Kottapalli, A. G. P., Chen, S. H., Miao, J. M., Kwok, C. Y., Triantafyllou, M. S., et al. (2017). Characterization of single polyvinylidene fluoride (PVDF) nanofiber for flow sensing applications. AIP Adv. 7:105205. doi: 10.1063/1.4994968

CrossRef Full Text | Google Scholar

Syed, M. S., Rafeie, M., Henderson, R., Vandamme, D., Asadnia, M., Warkiani, M. E. J., et al. (2017). A 3D-printed mini-hydrocyclone for high throughput particle separation: application to primary harvesting of microalgae. Lab Chip 17, 2459–2469. doi: 10.1039/C7LC00294G

CrossRef Full Text | Google Scholar

Tadakuma, R., Asahara, Y., Kajimoto, H., Kawakami, N., and Tachi, S. (2005). Development of anthropomorphic multi-DOF master-slave arm for mutual telexistence. IEEE Trans. Vis. Comput. Graph. 11, 626–636. doi: 10.1109/TVCG.2005.99

CrossRef Full Text | Google Scholar

Taunyazov, T., Koh, H. F., Wu, Y., Cai, C. X., and Soh, H. (2019). “Towards effective tactile identification of textures using a hybrid touch approach,” in 2019 International Conference on Robotics and Automation (ICRA) (Montreal, QC), 4269–4275.

Google Scholar

van den Noort, J. C., Kortier, H. G., van Beek, N., Veeger, D., and Veltink, P. H. (2016). Measuring 3D hand and finger kinematics-a comparison between inertial sensing and an opto-electronic marker system. PLoS ONE 11:e0164889. doi: 10.1371/journal.pone.0164889

CrossRef Full Text | Google Scholar

Vogt, D. M., Park, Y. L., and Wood, R. J. (2013). Design and characterization of a soft multi-axis force sensor using embedded microfluidic channels. IEEE Sens. J. 13, 4056–4064. doi: 10.1109/JSEN.2013.2272320

CrossRef Full Text | Google Scholar

Vujaklija, I., and Farina, D. (2018). 3D printed upper limb prosthetics. Expert Rev. Med. Devices 15, 505–512. doi: 10.1080/17434440.2018.1494568

CrossRef Full Text | Google Scholar

Wang, T., Zhang, D., and Da, L. (2010). Remote-controlled vascular interventional surgery robot. Int. J. Med. Robot. Comp. Assist. Surg. 6, 194–201. doi: 10.1002/rcs.306

CrossRef Full Text | Google Scholar

Wang, X., Zhong, Y., Sun, Y., and Li, X. (2014). A flexible capacitive tactile sensing array for pressure measurement,” in 2014 IEEE International Conference on Robotics and Biomimetics (ROBIO 2014) (Bali: IEEE), 2344–2349.

Google Scholar

Wheatland, N., Wang, Y., Song, H., Neff, M., and Zordan, V. Jörg, S. (2015). “State of the art in hand and finger modeling and animation,” in Computer Graphics Forum. 34, 735–760. doi: 10.1111/cgf.12595

CrossRef Full Text | Google Scholar

Xu, C., Kim, K. J., Wang, Y., Yoon, S., Ren, G., Mandal, D., et al. (2013). “Flexible nanogenerator and nano-pressure sensor based on nanofiber web of PVDF and its copolymers,” in Soft Fibrillar Materials: Fabrication and Applications, eds XY. Liu, and J-L. Li. 233–264. doi: 10.1002/9783527648047.ch8

CrossRef Full Text | Google Scholar

Zhao, S., and Zhu, R. (2017). Electronic skin with multifunction sensors based on thermosensation. Adv. Mater. 29:1606151. doi: 10.1002/adma.201606151

CrossRef Full Text | Google Scholar