Original Research ARTICLE
Electroencephalographic Functional Connectivity With the Tacit Learning System Prosthetic Hand: A Case Series Using Motor Imagery
- 1Department of Hand Surgery, Graduate School of Medicine, Nagoya University, Nagoya, Japan
- 2Department of Health Sciences, Faculty of Medicine, Nagoya University, Nagoya, Japan
- 3Center of Brain Science (CBS), CBS–TOYOTA Collaboration Center, RIKEN, Nagoya, Japan
We previously created a prosthetic hand with a tacit learning system (TLS) that automatically supports the control of forearm pronosupination. This myoelectric prosthetic hand enables sensory feedback and flexible motor output, which allows users to move efficiently with minimal burden. In this study, we investigated whether electroencephalography can be used to analyze the influence of the auxiliary function of the TLS on brain function. Three male participants who had sustained below-elbow amputations and were myoelectric prosthesis users performed a series of physical movement trials with the TLS inactivated and activated. Trials were video recorded and a sequence of videos was prepared to represent each individual’s own use while the system was inactivated and activated. In a subsequent motor imagery phase during which electroencephalography (EEG) signals were collected, each participant was asked to watch both videos of themself while actively imagining the physical movement depicted. Differences in mean cortical current and amplitude envelope correlation (AEC) values between supplementary motor areas (SMA) and each vertex were calculated. For all participants, there were differences in the mean cortical current generated by the motor imagery tasks when the TLS inactivated and activated conditions were compared. The AEC values were higher during the movement imagery task with TLS activation, although their distribution on the cortex varied between the three individuals. In both S1 and other brain areas, AEC values increased in conditions with the TLS activated. Evidence from this case series indicates that, in addition to motor control, TLS may change sensory stimulus recognition.
Our movements are supported by conscious and unconscious motor control mechanisms (Steenbergen et al., 2010; D’Ostilio and Garraux, 2012; Hayashibe and Shimoda, 2014). Riding a bicycle, for example, requires an intricate interplay between the brain and muscles. A bicyclist must continuously adapt to widely ranging environmental changes while regulating balance, direction, and speed.
When we use an implement to extend the reach of our hands, the neural representation of our body is modified and the multisensory peripersonal space surrounding it is extended (Maravita and Iriki, 2004; Holmes, 2012; Juett and Kuipers, 2019). The tool is expeditiously integrated within our body schema. After upper limb amputation, the human control system generates neuromuscular adaptations to effectively utilize a tool such as a prosthetic. Compensatory mechanisms and sensorimotor integration are important for such adaptations (Claret et al., 2019). Wearing a prosthesis increases the perceived stump length and extends the peripersonal space boundaries so as to include it, such that the prosthesis partially replaces the missing limb (Canzoneri et al., 2013).
To reconstruct such natural motor control in robots, in addition to generating accurate motor output, implementing good sensory feedback is required in the internal processing system. However, the development of a technology that can perform these processes has been difficult to date because a feedback system with high precision is limited in its ability to flexibly respond to unanticipated environmental changes.
The Tacit Learning System (TLS; Shimoda and Kimura, 2010; Shimoda et al., 2013) is a closed-loop control system that allows adaptive movement in response to unexpected environmental changes. In its function, it contributes to a strategy of sensory feedback called sensory synergy (Alnajjar et al., 2015). The TLS is superior to conventional methods in terms of learning speed, cost, and robustness.
We integrated the TLS within a myoelectric prosthetic hand (Oyama et al., 2016) that automatically controls forearm pronosupination in accordance with the intention of the user. We demonstrated that this myoelectric prosthetic hand enabled sensory feedback and flexible motor output, allowing users to move naturally with little burden.
However, it is not easy to evaluate the internal and external benefits and costs involved in the use of a myoelectric prosthetic hand. A better motor output does not always correspond to less exertion. Therefore, we hypothesized that the effort to use a prosthetic hand, including the amount of neural activity required for motor control and sensory feedback, might be reflected in brain activity.
Imaging methods such as functional magnetic resonance imaging (fMRI) and near-infrared spectroscopy (NIRS) have been among the leading techniques for studying brain function in recent years. Both measure changes in blood flow within the brain; however, inherent disadvantages in the technologies, particularly in studies involving tool use, include the prohibition on metal materials in the former and low special resolution in the latter (Iwatsuki et al., 2020). The temporal resolutions of fMRI and NIRS extend from hundreds of milliseconds to seconds, insufficient to analyze human movement.
To overcome this problem, we employed magnetoencephalography during periods in which participants viewed previously recorded video of motor tasks, which is a feasible method for functional neuroimaging of the mirror neuron system (Wright and Jackson, 2007; Wang et al., 2008). In a preliminary study about the effect of TLS on brain function using magnetoencephalography, we observed stronger connections between the motor area and other cortical areas when the technology was used, as indicated by a significant increase in coherence values (Iwatsuki et al., 2019). Magnetoencephalography has high temporal resolution with reliable source estimation (Iwatsuki et al., 2019). It can safely measure the magnetic field generated by neuronal electrical activity. Therefore, highly accurate identification of source activities and connectivity measurements in regions of interest is possible (Iwatsuki et al., 2016). However, it is expensive and access is limited.
In contrast, electroencephalography (EEG) is readily available and can measure the same phenomena as magnetoencephalography based on the relationship between the magnetic field and electric current. Furthermore, recent advantages in source modeling and computing techniques have enabled us to estimate neural activity from EEG signals obtained with a limited number of electrodes. The technique of collecting EEG during video imaging has been established to evaluate hand function (Ikeda et al., 2019). Much is to be learned about how neural representations of the body are remodeled by amputations and the use of prosthetics. Therefore, in this study, we investigated the effect of the use of a TLS-equipped prosthetic hand on brain activity.
Materials and Methods
The study protocol was approved by the Ethics Committee of the Nagoya University School of Medicine. Each volunteer gave informed consent prior to their involvement. Participants were three adult males, each with a below the elbow amputation who used a myoelectric prosthetic lower arm and handpiece in his daily life. Participant 1 had sustained a left arm amputation and participants 2 and 3 right arm amputations.
Tacit Learning System (TLS)
The TLS-integrated prosthetic arm with a handpiece contains a closed-loop control system that supports user-initiated action resulting in functional motor synergy. It relies on the signal accumulation in its process of adaptation, during which a formative behavior is tuned into movement suiting a goal. The integrated algorithmic system is based on human kinesiology. It consists of three goniometric sensors (shoulder flexion, abduction and rotation) coupled with a handpiece equipped with hand grip and wrist rotation actuators. In this study, the system was attached to the participant’s own myoelectric prosthetic socket.
Physical Movement Phase
Participants wore the prosthetic hand equipped with the TLS and were asked to perform a physical movement task with the system inactivated and then repeat it with the system activated. Three bars placed on a table in front of the seated participant were to be moved from a horizontal to a vertical position and then returned to the original horizontal position. This sequence was to be repeated three times, which was considered one trial. Finger manipulation to grasp and release each bar was performed by means of conventional electromyography electrodes in the arm socket, while forearm pronosupination was supported by the TLS. Everyone was able to complete the task with only simple instructions. Participants were asked to repeat the task, performing approximately 20 subsequent trials in 10 min, until an expected level of performance where an optimal balance of energy to effort was achieved.
All movements were videotaped, and a series of two recordings lasting 30 s each were made of a selected trial undertaken each participant, one with the TLS inactivated and one with the TLS activated. The usability of the prosthetic hand under both conditions from the perspective of each participant was evaluated by means of Hand20, an assessment of upper limb function (Suzuki et al., 2010; Iwatsuki et al., 2014).
Motor Imagery Phase
In an EEG suite, each participant was seated directly in front of a large video monitor and asked to watch a sequence of the two previously recorded 30-s videos of himself while imagining performing the task under the condition presented. He was to actively perceive controlling the prosthesis in synchronization with what appeared on the monitor. The videos were presented six times, separated by a 20-s blank white interval screen.
EEG (Neurofax, Nihon Kohden Company, Tokyo, Japan) was taken during the motor imagery phase. EEG signals were recorded with 21 conventional passive gel electrodes based on the International 10-20 system. Continuous EEG signals were recorded using an initial bandpass filter between 3 and 120 Hz with a sampling frequency of 1,000 Hz. The impedance between electrodes was kept below 5 kΩ. Artifacts caused by heartbeat and ocular movements or blinks were rejected using electrocardiogram and Fp1 recording by signal space projection (Nolte and Hämäläinen, 2001).
EEG signals were collected during the 30-s motor imagery phases (without and with TLS activated). To avoid any possible brain reaction at the onset of the video, EEG signals from the first 5 s were discarded, yielding 25 s series for analyses. Brainstorm software (Tadel et al., 2011) was used for all analyses. The cortical surface of a standardized brain base on MNI/ICBN152 (Fonov et al., 2009) as divided into matrixes with 3,001 vertices and electrical current in each vertex was estimated by minimum norm estimation (Baillet et al., 2001). First, the mean cortical electrical current during each task was estimated. Then, the bilateral supplementary motor area (SMA) was set as a region of interest, and neural connectivity between mean neural signals in the SMA and those in other vertices were calculated. The amplitude envelope correlation (AEC) of orthogonalized signal pairs for the alpha frequency band (8–12 Hz) was used to estimate neural connectivity (Colclough et al., 2016). Each AEC value was expressed as a correlation coefficient between –1 and 1. The mean signal in the left and right SMA was calculated because the separation was ambiguous. Differences in the mean cortical current and AEC values between the SMA and each vertex for the motor imagery tasks were calculated for each participant. AEC values of the alpha frequency band between the primary somatosensory areas (S1) for arms and hands, as distinguished from other brain areas, were also calculated during the motor imagery task with the TLS inactivated and activated.
The characteristics of the three participants are shown in Table 1. Compared to when the TLS was inactivated, all Hand20 scores of the TLS activated condition substantially improved.
Differences in mean cortical current between the motor imagery tasks without and with TLS activation varied among the three participants (Figure 1). Yellow to red and blue indicate increased and decreased current (pAm) without and with TLS activation. Figure 1 and Table 2 show inter-individual variation in cortical current density and AEC values during the imagery task. The variation suggests that the strategy in neural processes to control the prosthesis was different among participants. Reorganization of the function might be made in various ways as reported in a previous study (Belardinelli et al., 2017).
Figure 1. Differences in mean cortical current during motor imagery tasks without and with TLS activation [(movement imagery with TLS activation)—(movement imagery without TLS activation)]. Yellow to red and blue indicates increased and decreased current (pAm) during the task with and without TLS activation, respectively. Mean cortical current density in the total cortex (Total), SMA, M1, and S1 areas was shown in the bar graph below. Black and gray bars indicate cortical current density during movement imagery without and with TLS activation, respectively. Each vertical bar indicates a standard deviation. TLS, Tacit Learning System; SMA, supplementary motor area; M1, primary motor cortex; S1, primary somatosensory cortex.
Table 2. Differences in mean AEC value between cortical areas during movement imagery tasks with and without tacit learning system (TLS) activation (movement imagery with TLS activation) − (movement imagery without TLS activation).
All participants had increased AEC values in the condition with TLS activation between both SMA or S1 and other brain areas. Differences between Amp and N-amp sides were not obvious in any participant (Figures 2, 3).
Figure 2. Neural connectivity expressed as amplitude envelope correlation (AEC) values for alpha frequency band between the SMA (black areas) and other brain areas without [TLS (−)] and with [TLS (+)] in the three participants. AEC values between SMA contralateral (upper row; Amp) and ipsilateral (lower row; N-amp) to the arm amputated and other areas are shown. All participants had increased AEC values in the condition with TLS activation between both SMA and other brain areas. Differences between Amp and N-amp sides were not obvious in participants. AEC, amplitude envelope correlation; SMA, supplementary motor areas; TLS, Tacit Learning System.
Figure 3. Neural connectivity expressed as AEC values for alpha frequency band between the primary sensorimotor area for arm and hand (S1, black areas) and other brain areas without [TLS (−)] and with [TLS (+)] in the three participants. AEC values between S1 contralateral (upper row; Amp) and ipsilateral (lower row; N-amp) to the arm amputated and other areas are shown. All participants had increased AEC values in the condition with TLS activation between both SI and other brain areas. Differences between Amp and N-amp sides were not obvious in participants. AEC, amplitude envelope correlation; TLS, Tacit Learning System.
A prosthetic system should ideally reinstate the bidirectional communication between the user’s brain and its end effector by restoring both motor and sensory functions lost after an amputation. Even without explicit feedback, grasping using a prosthesis partly relies on sensory information (Wilke et al., 2019). Furthermore, for smooth and efficient motor control, the brain needs to make fast corrections to resist possible perturbations, so both feedback corrections and feedforward adaptation are required (Yousif and Diedrichsen, 2012).
Functional connectivity may change in adaptation to the use of tools. Tools that easily induce brain adaptations and assist movements have a high user affinity. By means of motor imagery, in this study, we investigated changes in brain activity during the use of our newly developed prosthetic hand. These results are important to assess the prosthetic hand function and how the participants use their brains. Our results showed that the connectivity between the SMA and surrounding areas was increased by the support of the pronosupination movement with the TLS. This indicated that the SMA was associated with sequential movements, including arm and finger derived pronosupination and grasping.
In order to interact successfully with the environment, it is important to know precisely where our body parts are located in space and where the targets of our interactions are situated with respect to other people’s bodies and ours. Moreover, we need to maintain these topographical inputs updated as a function of ongoing and upcoming hand movements (Brozzoli et al., 2014). When we use tools, we alter our brain activity according to the type of tool to achieve adaptation. After achieving proficiency with a tool, the body image is expanded so that the user can perceive the tip of the tool as part of their body. This human adaptation is achieved by altering brain activity (Garbarini et al., 2015; D’Angelo et al., 2018).
In this study, we focused on activity in the SMA based on the findings of previous studies (Iwatsuki et al., 2019). The SMA plays a crucial role in domain-general sequence processes, contributing to the integration of sequential elements into higher-order representations regardless of the nature of such elements as motor, temporal, spatial, numerical, and linguistic (Kim et al., 2016). The SMA supports the sequence operations in a variety of cognitive domains that share an inherent sequence processing. These include action, time and spatial processing, numerical cognition, music and language processing, and working memory (Cona and Semenza, 2017). Furthermore, the SMA is involved in the planning and coordination of movement sequences (Schramm et al., 2018; Busan et al., 2019).
As shown in the movement imagery phase, neural connectivity between S1 for arms-hands and other brain areas increased during TSL activation. Increased neural connectivity was observed for S1 in both hemispheres. Although we could not refer to a specific neural activity changed by TLS, in terms of brain activity its effects were clearly observed. Participants 2 and 3 showed greater increases in neural connectivity between S1 contralateral to the arm amputated as compared to that ipsilateral and to other brain areas, but the laterality was not clear in participant 1. S1 receives information about motor output before the arrival of sensory feedback signals, suggesting that S1 executes the online processing of somatosensory signals via interactions with the anticipatory information (Umeda et al., 2019). This prosthetic system has the possibility to reinstate the bidirectional communication between the user’s brain and its end effector by restoring both motor and sensory functions lost after an amputation (Wilke et al., 2019).
This study had several limitations. It was conducted with only three participants. In addition, while following a method established elsewhere, the impact of TLS was ascertained solely from EEG results of the motor imagery phase. We attempted to record EEG during conditions with the TLS inactive and active for both the physical movement (bar relocation task) and motor imagery (video viewing) phases. However, during each condition in the physical movement phase, there was excessive motion noise. While EEG signals were collected continuously during the motor imagery phase, the two were not precisely synchronized. Last, we cannot show the differences in laterality. This could be attributed to the small sample size or because EEG does not have as high sensitivity and specificity as magnetoencephalography.
The increased AEC values observed during the motor imagery phase when the TLS was activated were similar to our previous results (Iwatsuki et al., 2019). But further, TLS use increased AEC values in both S1 and other brain areas. When we use tools, sensory-motor integration must occur. This is the process by which these two sensory and motor systems communicate and coordinate with each other. TLS may change not only motor control but also sensory stimulus recognition.
Data Availability Statement
The datasets generated for this study are available on request to the corresponding author.
The studies involving human participants were reviewed and approved by the Ethics Committee of the Nagoya University, School of Medicine. The patients/participants provided their written informed consent to participate in this study.
KI, SO, MH, SS, and HH performed the measurements. KI and MH evaluated the EEG results. KI, MH, and HY composed the manuscript. All authors discussed the results and contributed to the final manuscript.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Belardinelli, P., Laer, L., Ortiz, E., Braun, C., and Gharabaghi, A. (2017). Plasticity of premotor cortico-muscular coherence in severely impaired stroke patients with hand paralysis. NeuroImage Clin. 14, 726–733. doi: 10.1016/j.nicl.2017.03.005
Brozzoli, C., Ehrsson, H. H., and Farnè, A. (2014). Multisensory representation of the space near the hand: from perception to action and interindividual interactions. Neuroscientist 20, 122–135. doi: 10.1177/1073858413511153
Busan, P., Del Ben, G., Russo, L. R., Bernardini, S., Natarelli, G., Arcara, G., et al. (2019). Stuttering as a matter of delay in neural activation: a combined TMS/EEG study. Clin. Neurophysiol. 130, 61–76. doi: 10.1016/j.clinph.2018.10.005
Canzoneri, E., Marzolla, M., Amoresano, A., Verni, G., and Serino, A. (2013). Amputation and prosthesis implantation shape body and peripersonal space representations. Sci. Rep. 3:2844. doi: 10.1038/srep02844
Claret, C. R., Herget, G. W., Kouba, L., Wiest, D., Adler, J., von Tscharner, V., et al. (2019). Neuromuscular adaptations and sensorimotor integration following a unilateral transfemoral amputation. J. Neuroeng. Rehabil. 16:115. doi: 10.1186/s12984-019-0586-9
Colclough, G. L., Woolrich, M. W., Tewarie, P. K., Brookes, M. J., Quinn, A. J., and Smith, S. M. (2016). How reliable are MEG resting-state connectivity metrics? NeuroImage 138, 284–293. doi: 10.1016/j.neuroimage.2016.05.070
Cona, G., and Semenza, C. (2017). Supplementary motor area as key structure for domain-general sequence processing: a unified account. Neurosci. Biobehav. Rev. 72, 28–42. doi: 10.1016/j.neubiorev.2016.10.033
D’Angelo, M., di Pellegrino, G., Seriani, S., Gallina, P., and Frassinetti, F. (2018). The sense of agency shapes body schema and peripersonal space. Sci. Rep. 8:13847. doi: 10.1038/s41598-018-32238-z
Fonov, V. S., Evans, A. C., McKinstry, R. C., Almli, C. R., and Collins, D. L. (2009). Unbiased nonlinear average age-appropriate brain templates from birth to adulthood. NeuroImage 47:S102. doi: 10.1016/s1053-8119(09)70884-5
Garbarini, F., Fossataro, C., Berti, A., Gindri, P., Romano, D., Pia, L., et al. (2015). When your arm becomes mine: pathological embodiment of alien limbs using tools modulates own body representation. Neuropsychologia 70, 402–413. doi: 10.1016/j.neuropsychologia.2014.11.008
Hayashibe, M., and Shimoda, S. (2014). Synergetic motor control paradigm for optimizing energy efficiency of multijoint reaching via tacit learning. Front. Comput. Neurosci. 8:21. doi: 10.3389/fncom.2014.00021
Ikeda, Y., Nishimura, Y., and Higuchi, S. (2019). Effects of the differences in mental states on the mirror system activities when observing hand actions. J. Physiol. Anthropol. 38:1. doi: 10.1186/s40101-018-0192-8
Iwatsuki, K., Hoshiyama, M., Oyama, S., Shimoda, S., and Hirata, H. (2019). Magnetoencephalographic evaluation for the myoelectric hand prosthesis with tacit learning system. NeuroRehabilitation 44, 19–23. doi: 10.3233/nre-182514
Iwatsuki, K., Nishikawa, K., Chaki, M., Sato, A., Morita, A., and Hirata, H. (2014). Comparative responsiveness of the Hand 20 and the DASH-JSSH questionnaires to clinical changes after carpal tunnel release. J. Hand Surg. Eur. 39, 145–151. doi: 10.1177/1753193413485524
Iwatsuki, K., Yoneda, H., Onishi, T., Ishii, H., Kurimoto, S., Yamamoto, M., et al. (2020). Compatibility of magnetic resonance imaging in patients with orthopedic implants: manufacturer questionnaires. Nagoya J. Med. Sci. 82, 79–84.
Iwatsuki, K., Yoshida, A., Shinohara, T., Nakano, T., Uemura, J. I., Goto, S., et al. (2016). Recovery function of somatosensory evoked brain response in patients with carpal tunnel syndrome: a magnetoencephalographic study. Clin. Neurophysiol. 127, 2733–2738. doi: 10.1016/j.clinph.2016.05.273
Kim, H. Y., Yang, S. P., Park, G. L., Kim, E. J., and You, J. S. (2016). Best facilitated cortical activation during different stepping, treadmill and robot-assisted walking training paradigms and speeds: a functional near-infrared spectroscopy neuroimaging study. NeuroRehabilitation 38, 171–178. doi: 10.3233/nre-161307
Nolte, G., and Hämäläinen, M. S. (2001). Partial signal space projection for artefact removal in MEG measurements: a theoretical analysis. Phys. Med. Biol. 46, 2873–2887. doi: 10.1088/0031-9155/46/11/308
Oyama, S., Shimoda, S., Alnajjar, F. S., Iwatsuki, K., Hoshiyama, M., Tanaka, H., et al. (2016). Biomechanical reconstruction using the tacit learning system: intuitive control of prosthetic hand rotation. Front. Neurorobot. 10:19. doi: 10.3389/fnbot.2016.00019
Schramm, S., Sollmann, N., Ille, S., Meyer, B., and Krieg, S. M. (2018). Application of navigated transcranial magnetic stimulation to map the supplementary motor area in healthy subjects. J. Clin. Neurophysiol. doi: 10.1097/WNP.0000000000000530 [Epub ahead of print].
Steenbergen, B., van der Kamp, J., Verneau, M., Jongbloed-Pereboom, M., and Masters, R. S. (2010). Implicit and explicit learning: applications from basic research to sports for individuals with impaired movement dynamics. Disabil. Rehabil. 32, 1509–1516. doi: 10.3109/09638288.2010.497035
Suzuki, M., Kurimoto, S., Shinohara, T., Tatebe, M., Imaeda, T., and Hirata, H. (2010). Development and validation of an illustrated questionnaire to evaluate disabilities of the upper limb. J. Bone Joint Surg. Br. 92, 963–969. doi: 10.1302/0301-620x.92b7.23410
Tadel, F., Baillet, S., Mosher, J. C., Pantazis, D., and Leahy, R. M. (2011). Brainstorm: a user-friendly application for MEG/EEG analysis. Comput. Intell. Neurosci. 2011:879716. doi: 10.1155/2011/879716
Wilke, M. A., Niethammer, C., Meyer, B., Farina, D., and Dosen, S. (2019). Psychometric characterization of incidental feedback sources during grasping with a hand prosthesis. J. Neuroeng. Rehabil. 16:155. doi: 10.1186/s12984-019-0622-9
Keywords: tacit learning, electroencephalography, prosthesis, supplementary motor area, neuroplasticity, Hand20, brain function, neurorehabilitation
Citation: Iwatsuki K, Hoshiyama M, Oyama S, Yoneda H, Shimoda S and Hirata H (2020) Electroencephalographic Functional Connectivity With the Tacit Learning System Prosthetic Hand: A Case Series Using Motor Imagery. Front. Synaptic Neurosci. 12:7. doi: 10.3389/fnsyn.2020.00007
Received: 26 September 2019; Accepted: 07 February 2020;
Published: 28 February 2020.
Edited by:Martín Cammarota, Federal University of Rio Grande do Norte, Brazil
Reviewed by:Edgard Morya, Edmond and Lily Safra International Institute of Neuroscience of Natal (ELS-IINN), Brazil
Gustavo Balbinot, University Health Network (UHN), Canada
Copyright © 2020 Iwatsuki, Hoshiyama, Oyama, Yoneda, Shimoda and Hirata. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Katsuyuki Iwatsuki, email@example.com