Edited by: Sandro Mussa-Ivaldi, Northwestern University, United States
Reviewed by: Fernando Brunetti, Universidad Católica “Nuestra Señora de la Asunción”, Paraguay; Sean Kevin Meehan, University of Waterloo, Canada; Lucia Schiatti, Istituto Italiano di Tecnologia, Italy
This article was submitted to Neuroprosthetics, a section of the journal Frontiers in Neuroscience
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
Amyotrophic lateral sclerosis (ALS) is a neurodegenerative disorder causing a progressive motor weakness of all voluntary muscles, whose progression challenges communication modalities such as handwriting or speech. The current study investigated whether ALS subjects can use Eye-On-Line (EOL), a novel eye-operated communication device allowing, after training, to voluntarily control smooth-pursuit eye-movements (SPEM) so as to eye-write in cursive. To that aim, ALS participants (
Amyotrophic lateral sclerosis (ALS) is a neurodegenerative disorder causing a progressive motor weakness of all voluntary muscles, with notable exceptions such as extra-ocular muscles that remain durably spared (
As a matter of facts, classical eye-controlled ACD are based on the triad “saccade, fixation, selection” of predefined items (e.g., letters of the alphabet) displayed on a computer screen. These items constrain the choices of the user, who cannot generate figures or symbols of its own, thus limiting its creativity. This study evaluates whether ALS subjects can use a new eye-operated system (Eye-On-Line, EOL)
To our knowledge, using SPEM to eye-write in cursive with healthy participants was first described in a recent article showing the large variety of eye-written production that can be achieved (
Participants were recruited from the Department of Neurology at the Pitié-Salpêtrière Hospital with the following inclusion criteria: (1) probable or definite ALS disease according to the revised El Escorial criteria; (2) impaired handwriting but still intelligible speech; (3) absence of oculomotor impairment. Subjects with a history of epilepsia, with clinical evidence of oculomotor impairment, or with dementia were excluded. The study was carried out in accordance with the Declaration of Helsinki and was approved by all relevant ethics committees and national regulatory authorities (CPP n°14942 IDF V; ANSM 2014-A00392-45). All participants gave informed consent (written consent whenever it was possible). The study (recorded on Clinical Trials reference NCT02313402)
Between May 2014 and April 2015, twelve ALS subjects (3 females, 9 males) were enrolled: 8 participants with spinal ALS and 4 participants with a bulbar form. Their mean age was 56.8 years old, with an average disease duration of 43.3 months (SD: 42.4) and mean ALSFRS-R score of 35 (SD: 6.6).
Clinical characteristics of the ALS participants.
Patient No. | Age (years) | Sex | Disease duration (months) | Site of onset of ALS | ALSFRS score | Forced vital capacity |
---|---|---|---|---|---|---|
A | 62 | M | 33 | UL | 32 | 108 |
B | 66 | M | 61 | LL | 28 | 63 |
C | 51 | F | 31 | LL | 43 | 113 |
D | 61 | F | 25 | B | 31 | 24 |
E | 49 | M | 41 | B | 23 | Impracticable |
F | 50 | M | 32 | LL | 28 | 55 |
G | 66 | M | 36 | UL | 45 | 122 |
H | 71 | M | 34 | LL | 38 | 134 |
I | 74 | M | 8 | UL | 39 | 66 |
J | 20 | M | 11 | LL | 38 | 77 |
K | 59 | M | 37 | B | 37 | 114 |
L | 53 | F | 171 | LL | 38 | 93 |
Mean ( |
56.8 (14.2) | – | 43.3 (42.4) | - | 35 (6.6) | 88.1 (33.8) |
The EOL training program comprised six sessions (two sessions per week for 3 weeks with a minimum break of 1 day between sessions). Each session lasted at most 2 h, including breaks and pauses. ALS subjects initially (at inclusion during Session 1) underwent behavioral and neuropsychological assessments, including the hospital anxiety and depression scale (HAD,
Overall display settings. Participants sat on a comfortable chair with armrests and headrest, in front of a large video screen. To minimize measurement errors, head movements were restrained using a necklace ergonomic cushion.
Details about the principles underlying EOL can be found in
The stimulus against which subjects learned to generate SPEM consisted in 300 identical static disks (1° of visual angle), randomly distributed on a gray background (24 cd/m2). All disks had identical luminance, but the contrast alternated periodically, switching from light-to-dark and dark-to-light relative to the background at about 10 Hz. Moving the eyes while looking at this display elicits a retinal slip of the disks in a direction
After a 5-points eye-tracking calibration procedure, the movements of the right eye were recorded with a head-mounted infrared video-based eye-tracker (EyeTechSensor, sampling rate 60 Hz,
After the first contact with EOL in session 1, subjects’ eye-movements were repeatedly recorded during short episodes with the EOL display (
During a session, participants performed between 5 and 15 short interactive runs with advices and instructions to help them perceiving the illusory motion and to induce and maintain SPEM (e.g., by moving their head, or waving their hand in front of the EOL display). Whenever participants succeeded to perceive the illusory motion, they were guided to generate SPEM, either by maintaining it as long as possible, or by executing specific figures. Afterward, subjects performed 5–10 runs on their own, and were free to execute eye-movements as they wanted (
All eye-data were processed off-line with Matlab (version 2017a, The Mathworks). The different oculomotor tests (fixation, smooth-pursuit gain) were analyzed for each participant and each session. We used Linear Mixed Effects Model to analyze all the parameters derived from eye-movements, with subjects as crossed random effects (including by-subject random intercepts, see
For each recorded run of each subject, saccadic eye-movements were detected using both a velocity and an acceleration threshold (50°/s, 500°/s2, respectively). Data corresponding to blinks (pupil size equal to zero), were detected and removed from further analyses. Smooth pursuit was defined as any eye-trace comprising at least a cluster of 30 successive points (corresponding to 500 ms of recording) spaced by less than 50 pixels -to avoid including saccadic eye-movements-, and whose standard deviation was larger than 10 -to avoid including fixational eye-movements in the quantification of pursuit. The longest pursuit and the cumulated duration of pursuit (averaged and best runs) were determined for each run of each session and used throughout the study to evaluate the capability to generate smooth-pursuit at will.
To address the issue of using the EOL device to communicate, participants were requested to “eye-write” digits from 0 to 9. The generation of eye-written digits started no later than session 4 and was repeated until the end of the training program. The traces recorded in session 6 were collected from all subjects and processed to isolate each digit, stored as a separate graph. In addition to the raw traces, we generated smoothed versions of each digits, using a sliding averaged (width of the sliding window = 6 samples). We similarly, processed digits generated by twelve healthy controls matched for age and gender who followed the same training protocol. The eye-written digits of ALS and control subjects were shuffled, resulting in 440 digits (10 digits × 2 versions × 22 eye-writers). These digits were presented in random order on a computer screen to 20 naïve “readers” who were asked to identify each digit by entering their response with a keypad (0–9), and to respond at random whenever this appeared too difficult or impossible. The identification test lasted around 15 min. The correct identification rates were measured for ALS and control subjects with both the raw and smoothed traces.
The recorded eye-movements were highly variable within and across participants and sessions, both in the basic tests made at the beginning of each session (quality of fixation, ability to track a moving target), as well as for the EOL runs. These differences, observed for the ALS and control subjects, may reflect day-to-day modulations of cognitive states (attention, concentration, willingness to perform eye-movements). This intra and inter individual variability, together with the large data set collected over sessions, precluded conducting fine statistical analyzes of all eye-movement parameters at the group level on a session-by session basis. In the following, we focus on the mean duration of endogenously generated smooth-pursuit, and the longest pursuit realized during the different sessions. To assess the effect of training and to determine whether subjects made progress over sessions, we made statistical tests using the SPEM parameters of the early and late sessions. We then present the results of a separate experiment where “naïve” readers had to identify eye-written digits (see below), which is most relevant to evaluate the outcomes of the study and the feasibility of eye-writing for communication.
To analyze the effects of training on fixation and target tracking, we compared the mean results from sessions 2 and 3 (FH) to the sessions 5 and 6 (SH). The standard-deviations of the horizontal and vertical eye-positions in the fixation test averaged across participants are presented in
The average pursuit gains during target tracking (see section 2.5) measuring the quality of SPEM are reported in
Although these averaged data indicated that training had no net effect on the quality of fixation or of SPEM at the group level, individual results may differ, as indicated by the large standard deviations of the results.
Examples of eye-movements generated at will during the EOL training sessions are shown in
Duration of SPEM voluntarily generated SPEM using EOL, for each ALS subject during sessions 1–6, sorted from lowest to highest mean performance.
To get insights into the highest level that ALS participants can reach, we took the best runs for each session and each subject, corresponding to the longest cumulated duration of SPEM production in a session, expressed in percentage of the run duration. We then computed a linear regression on these data to assess the initial best performance (Intercept) and the learning rate (regression slopes) of each participant. The intercepts and the slopes computed in this way are plotted in
A high intercept indicates participants that were already performing well at the beginning of the training program, while low intercepts denote participants who performed poorly at start. Further, large slopes indicate a progression during training, while small (or negative) slopes indicate no progress over sessions. As it can be seen, seven ALS subjects initially performed well (producing SPEM for more than 70% of the time during a 30 s run, green symbols), and therefore could not improve much over sessions (ceiling effect resulting in small regression slopes). Three subjects were initially not performing well (low intercept), but made progress over sessions (steep learning slopes, Blue symbols). Finally, two subjects were initially not performing well (low intercept) and did not improve over session (small or negative slopes, Red symbols). Note, however, that task difficulty increased over sessions (from freely generating SPEM to voluntarily producing specific figures), such that the reported lack of improvement must be considered with caution.
This pattern of results suggests the existence of different profiles, as shown in
To evaluate the legibility of eye-written digits, we tested how well “naïve” readers (
Digits eye-written by ALS subjects during session 6.
Results of the digit identification experiment performed by 22 “naïve” readers. 440 digits from ALS and control subjects (10 digits × 2 Versions, × 22 subjects) were mixed and randomly presented in succession on a computer screen. Readers identified each digit (from 0 to 9), or responded at random when unable to read a digit.
We then evaluated whether the capability of ALS subjects to produce long SPEM during training was correlated with the recognition rate of the digits (
Correlation between the averaged duration of freely generated SPEM and the recognition of digits eye-written by ALS participants, either in their raw version (Blue dots) or in their smoothed version (Red squares). Subjects able to sustain SPEM for long durations are more likely to generate recognizable digits.
Averaged tiredness over the training program was 26.15 (SD 25.6); averaged motivation was 87.35 (SD 16.16). To determine the evolution of these features over time, we compared the results from the FH sessions (2, 3) to the SH sessions (5, 6). The results (
Results from neuropsychological tests are detailed in
No serious adverse event occurred during the study. Using EOL was associated with accentuated fatigue on VAS and one ALS subject dropped-out at fifth visit because of excessive tiredness and cervical pain (also related to a dropped head). Three subjects complained of itching eyes and one subject experienced vomiting after a session.
This study showed that SPEM resulting in recognizable cursive eye-writing was feasible in a small group of ALS patients, with a level of performance comparable to control subjects. As observed in healthy subjects, and despite adaptive training, the learning rates and the capability to eye-write widely differed across ALS patients and across control subjects, without being able, at this stage, to determine whether such variability relates to idiosyncratic differences in perceiving the motion illusion, in cognitive or psychological traits, or in differences in motor control. From this and other eye data sets, we could not see any trend for an effect of age or education. Understanding the origins of the observed inter-individual differences is in itself a scientific issue for which separate studies are needed. What the present study points to, is the observation that ALS and control subjects do not differ much in their ability to master SPEM with EOL, with different, albeit consistent, profiles. In particular, those participants able to initiate and maintain SPEM for long durations, in early or late sessions, were also those able to master eye-writing so as to produce legible digits (
In ALS participants, asthenia, motivation or anxiety are unlikely to account for inter-session variability in SPEM, as these remained stable across sessions. The limited sample size and missing data related to the ALS disability evaluation restricted the use of neuropsychological data to clinically assess underlying cognitive impairments. Nevertheless, we noticed that one ALS subject not improving its capability to generate SPEM was significantly cognitively impaired. Further, four of the five participants (A, B, J, L) with limited SPEM production at initial visits were diagnosed as apathetic versus none of the subjects immediately considered as good performers. Thus, the hypothesis of a frontal dysfunction as a limiting factor of EOL use can be raised, but should be confirmed on a larger population using more appropriate cognitive screening tools for patients with motor impairments, such as the Edinburgh Cognitive and Behavioral ALS screen (ECAS,
The observation of similar inter-individual differences in ALS and control participants suggests they cannot be exclusively related to the disease and its consequences, and are a general feature. That ALS and control participants exhibited similar performance in the digit recognition experiment indicates that eye-movement control was not deeply affected in ALS subjects, despite them being significantly impaired in other motor modalities (see
As the ALS participants had to be able to come on-site to follow the protocol on a short time period, they necessarily had moderate levels of disability, as stated by the mean ALSFRS score. This population was thus not representative of ALS patients in a compelling need for an ACD. However, a recent study (
As all video-based eye-trackers, EOL is safe and did not induce serious adverse events. As with eye-based ACDs, EOL requires efforts to master and control eye-movements in an unusual way, often inducing fatigue during the training phase. The use of a flickering background as a substrate on which SPEM can be generated did not specifically elicit complaints, since the contrast of the display was lowered once the motion illusion was well identified, and because the flickering frequency remained relatively low. A limitation worth mentioning is that the head-mounted eye-tracker used here had a sampling rate limited to 60 Hz, which appeared too low to provide good renderings of SPEM.
There are limitations to the use of EOL that must be addressed. First, the criteria used to include ALS participants in this study (see Study Design) already indicate that not all ALS subjects can use EOL, and that using EOL may become difficult or impossible as the disease evolves. In addition, this study only evaluated the production of digits, not the production of words or of on-line communication for significant durations, or in everyday life. The need for training, and the risk that training does not necessarily lead to mastering eye-writing, limits the possibility of routinely proposing EOL as a communication tool. In this regards, the lack of understanding of the origins of the observed inter-individual differences prevents predicting who could benefit from EOL.
To date, EOL should probably not be considered an alternative to existing ACDs, but rather as an adjunctive tool providing a new creative space. It could thus be added to classical ACDs using fixed items that subjects fixate and select in succession, as is the case when writing by selecting a letter from the alphabet. Nevertheless, in contrast with other systems, EOL brings both creativity and personal expression, allowing individuals to initiate eye-movements that reflect their own actions and states, which may ease communication with beneficial psychological consequences, both for ALS patients and caregivers.
In this study, no attempt was made to use sophisticated algorithms to recognize eye-written characters, although smoothing allowed improving the rendering of eye-written digits. We did develop algorithms that can reliably identify eye-written characters from trained healthy subjects, as well as the identity of the writers (
The study was carried out in accordance with the Declaration of Helsinki and was approved by all relevant ethics committees and national regulatory authorities (CPP n°14942 IDF V; ANSM2014-A00392-45). All participants gave informed consent (written consent whenever it was possible).
TL, JM, LL, YV, and JL designed and wrote the protocol. JM, JL, and MV-M ran the experiments. JM and JL analyzed the behavioral results. AF and LL ran and analyzed the neuropsychological tests. TL, GB, NLF, MdMA, PFP, and FS recruited and examined the patients. TL, JM, and JL wrote the manuscript. LL hosted the study and facilitated the recruitment of the patients.
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
We thank all the ALS participants and their caregivers for giving their time and support to this study.
The Supplementary Material for this article can be found online at: