Edited by: Chantal Milleret, CIRB, Collège de France, France
Reviewed by: Yasuo Terasawa, Nidek Co., Ltd., Japan; Lauren Ayton, Centre for Eye Research Australia, Australia
*Correspondence: Gislin Dagnelie
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution and reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
Current visual prosthesis designs intend to restore some hand-eye coordination to those who are otherwise blind, along with other domains of functional vision: visual information gathering (Geruschat et al.,
Sabbah et al. (
Considering the limitations of current prosthesis technology (Eiber et al.,
New prosthesis users, if their system uses an external camera, typically receive some form of camera alignment as part of their initial fitting or programming. Without such alignment, the camera will most likely be aimed in a direction different from that which the user will perceive as the source of light. Perceived percept locations are strongly bound to the electrode array’s position on the retina. The array’s field of view, based on input from the camera, must then be moved as close as possible to the perceived percept location to optimize localization accuracy. Camera alignment can be accomplished by physically moving the camera and/or changing what part of its field of view is sampled to drive stimulation. These modifications should ultimately minimize any apparent localization errors. If this is not performed, however, or not performed sufficiently well, it is unknown whether users will adapt to misalignments on their own.
Individuals with normal vision can quickly adapt to shifts in their vision, or even complete inversions of their visual fields, to correct localization and coordination errors introduced by wearing prism glasses. There is a possible reason, though, why one may not expect such adaptation from those using current visual prostheses. Held and Hein (
The camera alignment process for users of prostheses with external cameras, both current retinal and future intracranial designs, cannot be optimized without knowing whether percept localization is generally stable and whether users can adapt to misalignments. If localization is stable and users can adapt to misalignments, then no external alignment may be necessary at all. Conversely, if localization is not stable and users cannot adapt to misalignments, regular camera alignments will be required for optimal hand-camera coordination. This study examines the localization behavior of Argus II users to help guide future prosthesis designs and programming strategies.
Three Argus II users (S1, S2, and S3) involved in the Argus II Feasibility Study (ClinicalTrials.gov: NCT00407602) participated in this research. One male and one female were implanted in June 2007, and one male was implanted in June 2009. All subjects suffered from end-stage retinitis pigmentosa and had the prosthesis implanted in the right eye. This research was approved by the Johns Hopkins Institutional Review Board and adhered to the tenets of the Declaration of Helsinki. Subjects all provided their informed consent to participate before research began.
Subjects’ camera alignment positions (CAPs) were controlled through computer software. The electrode array for each subject covered approximately 17.9° × 10.8° of visual field. The image captured by the prosthesis camera spanned 66° × 49° or 49° × 38°, depending on external hardware models. The array’s field of view could be chosen as any approximate 18° × 11° area from the camera’s wider field of view. The subsection of the camera’s field of view mapped as the array’s field of view served as the source information for stimulation. The point mapped as the center of the array’s field of view was designated as the CAP. Once configured, the CAP would remain as a fixed parameter in the subject’s video processing unit, but could be changed by connecting the system to a computer with specialized software. The minimum step size for selecting CAP coordinates in horizontal or vertical dimensions was 0.27°. This setup did not allow for tilting of the camera with respect to glasses frame. Although the system permits rescaling of the array’s field of view (Sahel et al.,
Localization accuracy was primarily measured by asking subjects to touch solitary white square or circular targets that appeared on the black background of a touchscreen. The touchscreen covered an area of 37.5 cm × 30 cm and was positioned 36–38 cm away from the subject, depending on what was most comfortable for each subject. Head motion was not restricted, but subjects were encouraged to not move significantly closer or farther from the screen while they scanned for targets. Subjects were permitted to use both hands for touching the screen. Targets typically had 3 cm diameters, spanning about 4° of visual field, and appeared at random locations on the screen. Subjects completed up to 400 trials, broken into runs of 10–80 trials, as frequently as once per week for 3.5 years.
CAPs were intentionally misaligned from perceived percept locations to test whether subjects could adapt to such misalignments. Misalignments ranged 15°–40° away from optimal, and were maintained for approximately 1 year. Subjects were told that objects might appear to be in locations different from their true locations, and asked to report any detected misalignments. Any misalignments that were considered problematic by the subjects during daily use of the Argus II system were to be removed upon their report.
For the first 5–6 months of using misaligned CAPs, subjects’ localization accuracy and precision were tested as frequently as once per week. Localization tests during this period included auditory feedback, which informed the subject whether each response was correct, and if not, where the target was relative to the touched location on the screen. This feedback only specified direction, and did not specify distance. For example, if a subject touched anywhere below a target, the program’s feedback would be “It was higher.” Subjects did not have any opportunity to repeat trials, and the next target automatically appeared in a random location after the program finished providing feedback. If any improvement in accuracy was observed by the end of this period, CAPs were maintained and auditory feedback during testing was removed. Localization data were collected for another 5–6 months to determine whether any previously observed trend could continue without in-lab auditory feedback.
Prior to and following the period of localization tests with misaligned CAPs, subject CAPs were optimized each week during psychophysics testing sessions, or as frequently as subjects could be seen. Specifically, localization errors were used to calculate appropriate CAP adjustments to minimize errors, and subjects repeated testing with the new CAP. Localization and estimated optimal CAP data were collected over a total period of 3.5 years. Whenever subjects’ average responses were noticeably inaccurate, CAPs were adjusted and localization tests were repeated until average error was less than 1° or CAPs could not be further adjusted in the necessary directions.
When attempting to optimize a subject’s CAP, the number of percepts seeming to appear outside of the screen boundaries was minimized by implementing a margin near the screen’s border in which no targets could appear. Without such margins, any targets that appeared next to the screen border could have restricted the measurement of localization errors and, more importantly, provided unintended tactile feedback to the subject. Typical margins ranged from 4 to 23 cm, depending on each subject’s typical magnitude and direction of errors. Asymmetric margins were used when subjects had large localization errors in one direction, and the limits of the camera’s visual field would not allow for an eccentric enough CAP to correct the errors.
To determine whether subjects could adapt to misaligned percepts, localization errors, in degrees of visual field, were averaged for each trial run. Data analysis focused on the distance of each resulting error centroid from the origin, such that a centroid distance of 0° would imply perfect accuracy.
Linear regression, using the ordinary least squares method, of centroid distances vs. time was employed to identify any trends in localization accuracy. The significance of any effect of time was determined by bootstrap resampling of data pairs; see Henderson (
When trying to minimize localization errors and determine localization stability, pooled errors for each testing day were used to estimate optimal CAPs for each subject. Estimated optimal CAPs were calculated for each test trial, and CAP estimates for all trials in a day were averaged to estimate an optimal CAP to minimize errors for that day. Trial CAP estimates for each subject, grouped by day, were also analyzed directly through a one-way parametric bootstrap multivariate analysis of variance (MANOVA) using a multivariate Wald-type test statistic (Konietschke et al.,
The horizontal components of estimated optimal CAPs for subject S3 were additionally plotted against time to demonstrate the consistency of shifts in this dimension. The regression line for this plot and its confidence bands were calculated using the same methods as described above.
Data corresponding to intentional misalignments, as while adaptation was being tested, were excluded from analyses of localization stability. Further, trial runs with CAPs that were more than 5° away from optimal settings and not limited by the camera’s visual field boundaries were also excluded from analyses.
During the period in which subjects used misaligned CAPs and testing included auditory feedback, two of three subjects showed some significant improvement in accuracy. Improvement was very slow, averaging 0.02°/day. Subject S1 showed a total average decrease in error centroid distance of 6° during this period. S2’s decrease in centroid distance was not statistically significant, and only fell on average by 0.4°. S3 showed a significant decrease of 4°. Figure
When auditory feedback was removed, localization errors significantly increased over time for S1 and S2. S3 displayed a nonsignificant reduction in errors over time, but the expected error centroid distance for the last time point of the linear model of the feedback-ON period and its confidence interval were lower than any observed distance in the feedback-OFF period. Final error magnitudes were thus higher at the end of this observation period than before auditory feedback was removed. Comparing linear model expectations at the end of the feedback-OFF period with those at the end of the feedback-ON period, centroid distance significantly increased by 7° for S1, 4° for S2, and 4° for S3. Figure
Over the entire time that subjects used misaligned CAPs, none reported any problematic percepts. None of the subjects had difficulty using their systems or noticed any discrepancies between their visual percepts and their other senses. When asked to simultaneously view and hold an illuminated rod, subjects could detect changes in where they localized the light when different CAPs were set, but did not readily perceive any sensory discordance.
While little adaptation to misaligned CAPs was observed, CAPs required for proper alignment did fluctuate in all subjects. CAPs that provided optimal localization accuracy to subjects for a time eventually required adjustment to restore accuracy. MANOVA tests found significant effects of time:
Figure
Visual prostheses with extraocular cameras require calibration to optimize user hand-camera coordination. Camera input and/or processing can be changed to improve or degrade pointing accuracy. When users’ cameras were not properly configured, those in this study did not seem to fully appreciate the nature of the misalignments. Passive adaptation to misalignments, i.e., without specific instruction and coaching from someone such as a rehabilitation specialist, was possible, but only with very slow progress. Rates of adaptation seen here were about 4000 times slower than those for normally sighted subjects wearing prism glasses (Gibson,
Without consistent auditory feedback on in-lab localization errors, pointing accuracy deteriorated for all of our subjects. Error magnitudes increased immediately after auditory feedback was removed for S1 and S3, and only gradually increased for S2. This difference could once again be explained by S2’s less diligent approach: if S2 was paying relatively little attention to the feedback, one would not expect removing the feedback to have as great an effect on responses. The gradual yet distinct increase in S2’s errors after feedback was removed, however, does suggest that the feedback worked to maintain the subject’s accuracy, if not improve it. For S1 and S3, the immediate increases in error magnitude may reflect the feedback acting as a reminder for the subjects to attend more carefully to how they respond, alongside providing information necessary for adaptation.
One might expect daily activities to provide corrective feedback on camera misalignments, such as reaching for a white mug against a dark background and missing. Unfortunately, subjects in this study did not appear to encounter or register enough of that information to improve or maintain pointing accuracy. It is possible that rehabilitation specialists familiar with visual prostheses and camera misalignments could teach users to detect and adjust to misalignments in their home environments. Further, a variation of the localization test used in this study that provides more precise feedback and allows users to make multiple attempts for one target could promote faster adaptation. The results of this study are restricted to contexts that do not involve specific coaching or devices designed to actively train users on correcting localization errors.
Lacking the ability to readily and independently adapt to misaligned percepts, the flexible nature of how prosthetic visual input is integrated into the perception of egocentric space is a point of concern. If users consistently required the same CAP to maintain hand-camera coordination, prosthesis systems would only need to be properly configured once. If a CAP initially set to maximize pointing accuracy becomes less suitable over time, however, and users cannot independently adapt to emergent misalignments, more frequent system calibrations will be required.
Further research will be necessary to better understand what causes perceived percept locations, and thus optimal CAPs, to change over time. Some of the variation seen here may stem purely from the alignment and measurement processes used in this study; however, the consistent trends displayed over time by S2 and S3 suggest that at least part of this variability was intrinsic to the subjects. If variability originating from the subject could be explained by something as simple as how the eye rests in the orbit, prosthesis-integrated eye tracking mechanisms may be able to adjust CAPs automatically. If more complicated problems are involved, such as changing alignments of visual and proprioceptive percepts, more involved rehabilitation training or device programming may be needed to maintain optimal hand-camera coordination.
Visual prostheses are starting to restore modest levels of vision to those without any other available treatments. As the technology improves, prostheses may one day provide enough visual information for users to passively adapt to misaligned percepts on their own. At that point, camera alignment could simply be a matter of preference. Until that time, however, prostheses that use extraocular cameras will need more configuration to optimize hand-camera coordination. Users who consider accurate coordination very important, more so than the subjects tested in this study, should have their cameras aligned on a regular basis to get the most benefit from their prostheses. Alternatively, such users could also seek training that may help them to actively detect and correct camera misalignments, if they are sufficiently motivated. None of our three subjects expressed any problems that would have necessitated such alignments or training.
Materials used for the described experiments were provided by Second Sight Medical Products, Inc. without charge. Johns Hopkins University received payment from Second Sight Medical Products, Inc. for participation in the Argus II Feasibility Study.
MPB: designed experiments, collected data, analyzed data, wrote manuscript. GD: designed experiments, edited manuscript.
NIH T32 EY07143 to the Johns Hopkins Visual Neuroscience Training Program for MPB.
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Material presented in this article was previously included in poster presentations at the 2014 (Barry and Dagnelie,
camera alignment position.
1Patients implanted with IRIS-1 (49 electrodes) 6 month after implantation (short version), n.d. (video file), Available online at: