The TAB is a low-cost sensorised arm brace (Figure 1). It consists of an armband fitted with 8 force sensitive resistive (FSR) sensors uniformly distributed around the user's forearm. Its purpose is to monitor the normal interaction forces, as detailed in Stefanou et al. (2018b), capturing tactile signatures of the forearm.

A gripping device was developed for these experiments. It comprises two load cells under a handle (Figure 1) (Stefanou et al., 2018a) and has a resolution of 0.27N and sensitivity of 0.17N. An average error of 1.79% was calculated in the range of 0N and 9.81N. The acquisition of the TAB and gripping device data was done using an Arduino MEGA on which a custom made board was mounted to provide 16-bit analog inputs with a sampling rate of 89Hz. Calibration of the TAB FSR sensors was performed prior to the experiments where the force on each TAB sensor was recorded as a function of time. The two load cells were also calibrated using a set of known weights. The communication between the Arduino, where all sensor data were captured, and the computer was synchronous.

Experiments were performed with 20 healthy participants, 10 male and 10 female. Ethics approval from the University of the West of England Ethics Committee was acquired as well as informed written consent from all participants. The TAB sensing surfaces were placed three quarters of the way up the length of the forearm.

This section analyses the four grip types shown in Figure 2. The activity of the forearm muscles during each grip type determines the tactile signatures expected to be recorded by the TAB. The contribution of each forearm muscle to each grip type was determined based on its anatomical and physiological parameters. The hand configuration, the placement of the fingers, the wrist angle and the supination/pronation of the forearm were taken into account when analysing the TAB sensor readings and the features that can distinguish the griping and relaxed states and the four grip types.

Previous studies have indicated that individual finger force contributions during 5-finger precision (trapezoid) gripping are proportional to each finger's strength (Radwin et al., 1992). The first three gripping configurations used are precision grips. The fourth grip type used, Figure 2D, is a power grip, with the thumb adducted. All four grip types have a prismatic shape and they were all performed with the forearm in the supination position resting on a 3D-printed support. The grips were selected according to the detailed taxonomy of human grasps presented by Feix et al. (2016). Table 1 details these four grips and the percentage force contribution of each finger as found in the studies by Radwin et al. (1992) and Kinoshita et al. (1996)¹. The grip force limits used during the experiment for each grip type are presented in Table 1.

The pose and involvement of each hand digit and the wrist in each grip type determine the expected magnitude of the forearm muscle contractions. The contribution of each muscle and its proximity to individual TAB sensors, with any damping effects caused by the soft tissue, determine the expected tactile signature of the forearm. A detailed diagram of the forearm cross-section anatomy, just below the elbow, was used to associate each TAB sensor with the muscles in its proximity (Figure 3, each sensor annotated as S1-S8). The digit flexor muscles, the flexor digitorum superficialis (FDS) and flexor digitorum profundus (FDP) (Nordin and Frankel, 2001), flex the four fingers and the diagram provides details of their correspondences to the TAB sensors. The cross-section area of each muscle shown in Figure 3 has a direct correlation to its strength capacity (Cutts et al., 1991). The muscle compartments that control the middle finger joints are expected to produce the highest forces followed by the ones that control the index and ring fingers. This is important as each of the four parts of the FDP and FDS muscles has different proximities to the TAB sensors, and is thus expected to produce a different tactile signature. The interosseous membrane (Skahen et al., 1997) that divides the forearm into the anterior and posterior compartments is tensed when the forearm is in the supination position (McGinley and Kozin, 2001). This leads to effective damping that confines any muscle contraction effects (in particular in the inner forearm layer) to the compartment of its origin. This especially affects non-superficial muscle contraction effects on the TAB sensors.

To label the data with the state of the hand for each grip type, gripping or relaxed, the onsets and terminations of gripping were determined using the gripping force measurements. An algorithm was developed to determine gripping instances by checking whether the grip force threshold of 0.49N has been exceeded and whether any proximal grip/release events have occurred. A release event is identified only after a grip event and when the grip force is <0.69N. Another criterion for determining the release event is whether at the midpoint between the possible release point and the corresponding grip point the grip force is higher than a certain threshold (which varies between the grip types). All thresholds used in the algorithm were based on the sensitivity and accuracy of the gripping device and determined by trial and error. Each data point was labeled with a number that indicates the configuration of the hand (one of four grip types presented in Figure 2) and whether the hand was gripping or relaxed.

The force measurements around the forearm in the gripping and relaxed states for all participants are visualized in Figures 3–6, one for each grip type. At first glance, it is obvious that the data are not normally distributed and that generalizing for all participants may not be feasible. In order to further break down the data distribution and gain a better understanding of the results, bivariate histograms were produced (Figure 7). These visualize the frequency density of the individual sensor readings with respect to the grip force for each grip type.

Figures 3–6 show how the muscles engage during each grip type (indicated by red circles) and the response of each of the TAB sensors when the hand is gripping (orange histograms) and relaxed (blue histograms). The force frequency distribution of some sensors is significantly different in the relaxed and gripping states for particular grip types. For example, there is a clear distinction between the relaxed and gripping force distributions of S6 for Grip2, Grip3, and Grip4. A very strong bi-modal distribution is visible for Grip4 (Figure 6) while a clear shift of the median can be seen in Grip3 (Figure 5). Grip1 and Grip2, that differ only by the use of the middle finger have very similar tactile signatures when gripping, with the exception of the TAB sensors situated on the volar/ulnar side of the forearm; ie. S7 and S8. The distributions have longer tails and higher frequencies, around 0.8 which can be explained by the parts of the FDS and FDP muscles that flex the middle finger (annotated with “M” in the forearm cross-section diagrams), which are located in the proximity of those two sensors. This is one example that demonstrates that despite variations across participants, the results do agree with the physiological analysis performed and expected tactile signatures as tabulated in Table 2. It was expected that Grip1 and Grip2 would cause similar forces on all TAB sensors with the exception of S7 and S8. Comparisons with Grip3 where the remaining parts of the FDP and FDS muscles, located in the inner/ulnar part of the forearm, are engaged as well, indicate that not only are the S7 and S8 responses higher but also that S6, located at the ulnar side of the forearm, is sensitive to Grip3 as expected (Table 2). Overall, the results agree with the analysis performed and the expected outcomes.

Figure 7 shows the S6 readings against the grip force during Grip3 (scaled as indicated earlier). S6 was chosen because of its location in a region of the forearm where prominent force changes take place. This can be also seen in the histogram of S6 in Figure 5 (Grip 3). The relaxed state data are presented in Figure 7A and the gripping data in Figure 7B. While relaxed, the recorded grip force mainly ranges between 0 and 5% of the maximum recorded. There is no correlation between the force measured by S6 and the one measured by the gripping device. The variability of the forces recorded when the hand is relaxed could be due to the variability of TAB tightness on the participants' forearm, sensor hysteresis or increased blood flow in the forearm (as a result of gripping). This behavior was observed in all relaxed sensor data (S1–S8). In Figure 7B, the TAB readings correspond to gripping forces ranging between 20% and 60% of the maximum recorded force. In the 3D plot, a more normalized distribution is visible, in contrast to the histograms presented in Figures 3–6 which do not break down the gripping force. The visible diagonal indicates a linear correlation of the grip force with the TAB S6 force readings. This happens despite the variability in the relaxed contact forces recorded across participants (Figure 7A) caused mainly by TAB fit and forearm adipose tissue content. This could be a result of the general increase in the physiological cross-sectional area (PCSA) of the forearm and S6 location near the radius bone. However, it is also, more specifically, an indication of muscle activity in the proximity of the sensor.

Following initial data analysis, statistical analysis was used to determine whether the four grip configurations of the hand while relaxed or gripping are statistically different and thus separable. That would provide some indications on the possible predictive algorithm that could be used to classify the data and statistical evidence of the tactile signatures mapping to the muscle synergies involved in each grip type. The data were split in eight groups; with the hand relaxed or gripping in the 4 positions shown in Figure 2. The separability of all these states is important for the development of predictive algorithms that are transparent and based on physiological cues. For the creation of a generalized model, that can perform well with all TAB users, it is paramount that there are similar patterns across the participants.

The two-sample Kormogolov-Smirnov (KS) test, run in Matlab using the kstest2 function (MathWorks, 2018), was chosen to make the necessary comparisons as it does not assume Gaussian distribution. After testing each TAB sensor data normality, it was concluded that the data do not adhere to a Gaussian distribution (this is also evident in Figures 3–6). In this two-sample KS test, the hypothesis with regards to the distribution of a dataset is rejected by comparing the p-value with the significance level Alpha (default value of 0.05). D, the test statistic, is an indication of the “distance,” or difference, between the two samples' distributions when using the kstest2 function. When it is 0, the data follow the exact same distribution as the specified one, the higher the value the greater the difference between the two data distributions.

No statistically significant similarity was found when analysing and comparing the individual sensor datasets for each grip type across participants. This was attributed to the differences in the forearm shapes and their adipose tissue-to-fat content ratio, as well as experimental errors and confounding variables. The latter includes the tightness of the TAB on each individual, the spacing of the TAB sensors and the surface area covered by the eight sensors in relation to the total circumference of the forearm (the larger the circumference the lower the coverage and thus the lower the sensing resolution). Nonetheless, statistically significant differences were found between the relaxed and gripping states across all 20 participants. The KS test statistic value, D, calculated for the relaxed state datasets across participants (for the individual grip types) was under 0.1. Therefore, 0.1 was set as a critical test value to determine statistically different distributions. Thus, any statistical difference between two data samples would need to be confirmed by both the null hypothesis and the critical test value.

Given the evidence of linear separability in the data, PCA was performed, using the IBM SPSS software (IBM, 2019), to determine the extent to which the TAB data variability of each grip type across states can be expressed by linear components. Eight different components that describe the variance in the data were produced. Using the Keiser criterion only the ones with an eigenvalue greater than 1 were considered. The Varimax method was used and a maximum of 200 iterations were allowed for convergence. It is recommended that the components chosen describe 70–80% of the data variance. However, the principal components (PCs) explain 60.5, 68.7, 65.7, and 77.4% of the variance of Grip1-4, respectively. The lower values are due to the physiological differences between the participants. The results are in line with the grip analysis (section 2.3.1, Table 2). The PCs' highest correlations are observed with the sensors placed in the proximity of the digit flexors, S6, S7 and S8, and S3. S3 was found to be a good indicator of gripping but not of differentiating between the grip types (section 3.2.2).

During multiple participant experiments, the forearm tactile signature on the TAB mostly aligns with the muscle physiology and muscle synergies. Some observed incoherence in the data suggests that personalization of the TAB would greatly improve its accuracy. The variability between participants that affects the baseline readings and accentuates certain features over others can be attributed to:

Forearm shape – causes variations of the baseline readings

The tactile signatures of individual forearm muscle contractions can be used to create a map of each muscle contraction from the TAB response. This can be implemented in a motion intent recognition system to actuate an exoskeleton, instead of (or in combination with) training a machine learning model on particular motions. The advantage of this approach is that even unknown sensory inputs will provide a reliable control input increasing the robustness of the exoskeleton control.

Engineered features were created and ranked to improve the performance of the predictive algorithms for individual grip types as well as for the overall dataset. Both the temporal and spatial tactile patterns can provide characteristics of different hand movements despite the differences in the TAB response for each participant. It is, therefore, important to take into consideration the history of each sensor's readings (considering the change of force within about 0.5 s) as well as the spatial features which were generated from the differences between each of the TAB sensor force readings. The differences of the sensor readings could provide information on antagonistic muscle pairs or combinations of muscle contractions. Concatenating these features with the TAB sensor readings, a total of 44 features were established. The first eight are the TAB sensor readings [S₁, …,S₈], 9–16 their derivatives [dS₁, …,dS₈] and features 17-44 are the force reading differences between each pair of sensors; i.e., [S₁ − S₂, S₁ − S₃, …,S₇ − S₈].

The most important of these features were extracted for grip classification. Two approaches were taken to determine feature importance. In the first, the features were ranked using the Random Forests (RF) algorithm where the first 10 features were chosen for each classification scenario. The second approach transformed the engineered features with the principal components established using PCA. All individual sensor data, S1- S8, and some spatial features were found to be important. For Grip1 these included [S₇−S₃, S₈−S₂], for Grip2 [S₈−S₃, S₅−S₃], for Grip3 [S₆ − S₃, S₇ − S₆], and for Grip4 [S₄ − S₃, S₈ − S₄]. The temporal features, on the other hand, did not provide any useful information. The results are in line with our initial analysis as the spatial data that featured high on the importance list for individual grips mainly involved S3. S6, S7, and S8 were also key sensors in these spatial data. As seen earlier, S3 and S6 are good indicators of gripping due to the high variability in their data distributions during gripping. S8, S7, and S6 data are useful features in distinguishing between the four grip types. A number of machine learning techniques were employed to create models and make state predictions. The models were trained on 80% of the data and tested on the remaining 20% of participants' data.

Using the most important features calculated by the RF algorithm and the PCA-transformed data, supervised learning algorithms were implemented with a 10-fold cross-validation. There are multiple learning algorithms that can be used for classification purposes, both parametric and non-parametric. The k-nearest-neighbor (kNN) (Akhlaghi et al., 2016), logistic regression, random forests (RF) and support vector machines (SVM) (Wolf et al., 2013) were all implemented using the python scikit-learn library.

The prediction accuracy was higher when using the chosen engineered features in comparison with using solely the S1−S8 readings. Using all engineered features instead of only the ones found to be of importance has no significant effect on the results (h = 0). Being selective with the features decreases the computational time without compromising the accuracy. The results indicated that although the accuracy differences between the algorithms were not significant, the SVM had the best overall performance. When attempting to distinguish between all grip types and gripping/relaxed, a total of 8 classes, only about a quarter of the data were correctly classified. The kNN algorithm was not able to capture the complexity of the features as well as an ensemble method like RF. The SVM models, which used a Gaussian kernel, also resulted in higher prediction accuracy than kNNs and performed marginally better than RF in determining the state of the individual grips and classifying all grips and positions. The prediction accuracy using the principal components of the TAB data was the highest in the classification of Grip3. This could be due to the fact that the ratio of the maximum force allowed during the experiment, to the maximum that can be generated using that grip type is the highest for Grip3 (as evident in Table 1).

The high standard deviation of accuracy of all machine learning algorithms demonstrates a great variation of the TAB response within the participant pool. The training data and validation data prediction accuracies differed and only about a quarter of the data were correctly classified. Using the default parameters of the scikit-learn library, the SVM model generated with the features chosen using RF performed better than the one with the PCA transformed features. An exhaustive search over the gamma and C parameter values of the SVM was performed using grid search. This revealed that the model performs best when gamma is 1.0 and C is 1. Using the optimal parameter values, the SVM model's performance improved but it still failed to reach the desired accuracy as evident in Table 3.

The model did not perform well when tested on the data of the 4 “unseen” participants (20% of the data). This indicated that the variability across the population is not adequately captured with such a small number of participants to allow for generalization. The model was then trained on partial data from all participants. The data were shuffled prior to being split into the hold-out groups which meant that the training sample would most probably include some data from all 20 participants. This raised the model performance, a seen in Table 4. The most notable improvements were made with 8-way classification (4 gripping and 4 relaxed states) and on classifying the 4 relaxed hand configurations where the differences are subtle. The accuracy for both increased to over 95%. This indicates that using these learning algorithms the model would need to be personalized to the user.

The confusion matrix (Figure 10), shows the results of the 10-fold cross-validation of the SVM model which was trained using the principal component transformed features of partial unseen data from all participants. The labels relaxed1/gripping1 represent the relaxed/gripping states of the Grip1 hand configuration and the same applies for the other three grip types. It can be observed that the highest false negatives in the “one-vs.-rest” (training a single classifier per class) SVM approach were found when trying to distinguish between:

Grip3 and Grip4 which use all fingers; 1.6% of gripping3 was incorrectly classified as gripping4 and 1.5% of gripping4 as gripping3

Table 1 shows that the ring and little finger do not contribute as much force as the index or middle finger during Grip3 which contributes to its miss-classification as Grip2. Overall, Grip3 has the highest false positive rate with 4.7%. Relaxed4 is the one with the lowest false positive rate, with 0.9%. The relaxed states were, overall, classified slightly better than the gripping states possibly due to some difference in muscle engagement between the participants. In summary, the highest miss-classification of a state is for gripping3, 4.2%, the lowest for relaxed4 and relaxed2, 1.6%, with an average of 2.5% across all states.

The number of participants involved in this study was not sufficient to allow creation of a generalizable intent recognition model that can perform accurately (e.g., 90%) on the unseen participants. Nonetheless, the TAB system performance achieved an average classification accuracy of 99.96 ± 0.08% which is comparable to the state-of-the-art motion intent recognition systems not tested on weak muscle activations (Wolf et al., 2013; Xiao et al., 2014; Li et al., 2017).

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.