A Process-Model for Minimizing Adverse Effects when Using Head Mounted Display-Based Virtual Reality for Individuals with Autism

Abstract

Interest in the use of virtual reality technologies for individuals with autism spectrum disorders has been increasing for over two decades. Recently, research interest has been growing in the area of head mounted display-based virtual reality technologies, thanks to increased availability and affordability. Affordances and theorized benefits of headset-based virtual reality for individuals with autism spectrum disorders are quite promising. However, very little attention has been given in the literature to implementation safety and ethics. This is a particular concern in light of documented adverse effects associated with headset-based virtual reality. To approach this gap, this article details how the authors approached the issue of minimizing adverse effects with related and overlapping methods, but from two separate, independent research sites—one in the United States and one in the United Kingdom. A structured within- and across-case analysis of the two independent studies was conducted to identify central implementation processes and procedures. Analysis resulted in development of a model for minimizing potential adverse effects of headset-based virtual reality for this population. We assert that our model could provide clarity in terms of design and implementation of headset-based virtual reality for individuals with autism spectrum disorders, guide implementations of future researchers and practitioners, and contribute to minimizing and controlling for potential adverse effects.

Introduction

Researchers have been exploring the use of head-mounted displays (HMDs) and virtual reality (VR) in the area of autism spectrum disorders (ASD) since Strickland (1996) seminal work over two decades ago. Recently, interest and research activity have been increasing substantially due to the availability of affordable, consumer-grade HMD-based VR systems. ASD is a lifelong condition characterized by challenges with social communication/interaction and restricted, repetitive patterns of behavior (American Psychiatric Association, 2013; DSM-5 American Psychiatric Association, 2013). Manifestations of ASD present substantial heterogeneity across affected individuals (Masi et al., 2017). Early intervention has been shown to lead to improvements in core autism symptoms (Estes et al., 2015). However, access to and consistency of services in early intervention are known barriers, potentially leading to challenges in adulthood such as chronic underemployment, social isolation, and inability to live independently (Eaves and Ho, 2008; Taylor and Seltzer, 2011; Taylor and Mailick, 2014; Hedley et al., 2017). Prevalence of autism is increasing, with current estimates suggesting one in 54 children receiving an ASD diagnosis in the United States alone (Russell et al., 2014). As such, viable and effective therapeutic interventions are in high demand. Virtual reality has garnered significant attention as a potentially effective solution for delivering such interventions (Bellani et al., 2011; Parsons, 2016).

Virtual reality is considered to hold tremendous promise for individuals with ASD due to: (1) the suitability and natural affinity to computers and technology for this population; (2) the ability of HMD-based VR to simulate real-world situations and contexts in safe and predictable ways; and (3) the ability to control, shape, and tailor VR interventions to participant needs and local contexts (Parsons, 2016; Bozgeyikli et al., 2018). Therefore, VR appears to be intrinsically reinforcing for individuals with ASD, has technological affordances which align closely with instructional needs, and can broaden access to a range of services (Parsons et al., 2020). VR provides individuals with ASD the ability to encounter and practice skills in highly realistic and customizable contexts, for input stimuli to be intentionally manipulated, and to have real-world consequences mitigated or removed. Within this context, a patchwork of research studies have investigated VR-based interventions across multiple domains such as social/emotional skills (Moore et al., 2005; Ke and Im, 2013), safety skills (Self et al., 2007), street-crossing (Josman et al., 2011), daily living (Parsons et al., 2004; Mitchell et al., 2007; Jarrold et al., 2013), and communication (Kandalaft et al., 2013) to name but a few (Wang and Anagnostou, 2014; Mesa-Gresa et al., 2018). While reported research outcomes in this area suggest VR has promise for individuals with ASD, as a whole, research supporting connections between theorized benefits of VR HMDs and empirical evidence supporting effectiveness has been described as limited, piecemeal, and largely inconclusive (Parsons, 2016).

To date, the majority of research has focused on the use of desktop-based VR and virtual worlds. More recently, research interest in HMD-based VR for individuals with ASD has increased, in part due to increased consumer availability of HMDs. Researchers are particularly interested in implementing HMD-based VR for individuals with ASD, as specific affordances of the technology such as heightened sense of presence, immersion and ecological validity all lend themselves to a meaningful environment that potentially can enable opportunities for acquisition, maintenance, and generalization of skills (Freina and Ott, 2015). However, a widely-acknowledged concern related to using HMD-based VR is the potential that users will experience some degree of adverse effects (Palmisano et al., 2017). By adverse effects, we refer to negative effects including cybersickness (e.g., nausea, eye-strain, etc.), safety concerns (e.g., stumbling and falling), increased anxiety, sensory disturbances, etc. (cf. Kellmeyer, 2018). Although an abundance of calls for future research to investigate adverse effects can be found in the literature (e.g., Irish, 2013; Fletcher-Watson, 2014; Wong et al., 2015; Mesa-Gresa et al., 2018), actual research on adverse effects is largely absent. Research in the field has yet to meaningfully and systematically approach questions related to adverse effects for individuals with ASD when applying VR in general, and HMD-based VR specifically (Bradley and Newbutt, 2018; Malihi et al., 2020).

The majority of research on VR interventions for individuals with ASD has been principally concerned with establishing intervention effects. However, researchers’ desire to establish intervention effects problematically positions VR technology as the primary driver of intervention outcomes. To-date, no design heuristics or guidelines have been published in this area. Although a variety of heuristics and design principles exist for developing 2D computer interfaces for individuals with ASD (Benton et al., 2011; Khowaja and Salim, 2013), far less is known about how to design 3D interfaces and environments: “[…] there is no well-established literature on the best practices in designing VR user interface attributes for individuals with ASD yet” (Bozgeyikli et al., 2018, p. 22:5). With a dearth of guidance in terms of VR design best practices for individuals with ASD, researchers are broadly left to their own devices when confronting how to design effective interventions. The lack of practical, detailed, and considered processes to help minimize possible adverse effects in published research suggests a gap in the literature that could limit the full potential of HMD-based VR for individuals with ASD.

Alongside the lack of guidance for designing 3D interfaces and environments for individuals with ASD in general, there is also a lack of understanding regarding how these interfaces and environments might lead to potential adverse effects when experienced in HMDs specifically. That adverse effects have not been a particular focus within the field poses a potential issue around researchers adopting and using emergent technologies like HMD-based VR before their implications of use are understood or before evidence-based recommendations for implementation are established. Borrowing from Kellmeyer (2018), “very little systematic discussion of the neurophilosophical and ethical challenges from the clinical use of these new VR systems is available” (p. 2). Given the limited research in the field, a significant gap exists related to exploring and reporting potential adverse effects (Bradley and Newbutt, 2018). We argue that with rapid technological advancements in this area, researchers, practitioners, and other professionals working with people with ASD have an urgent responsibility to confront these issues (Parsons, 2016).

Thus far we have established the three following research gaps: (1) consideration of adverse effects of VR for individuals with ASD in published research is limited; (2) published design heuristics and principles related to VR for individuals with ASD are largely absent; and (3) there is a paucity of research that explicitly explores and reports potential adverse effects for this vulnerable population. The purpose of this article, therefore, is to approach these gaps by offering a set of preliminary implementation guidelines for HMD-based VR to minimize potential adverse effects for individuals with ASD. Our work is intended to guide researchers and practitioners alike, and therefore serves to bridge a research-to-practice gap in this area.

We turn to our own practice to assert our design expertise in this area, establish our positionality, and frame the issue from the perspective of expert recommendations and best practices, as informed by ethical guidelines. We, the authors, are among the first to have pursued formal research studies with HMD-based VR for individuals with ASD. Our research was performed in two separate, independent research groups. We have published and presented widely on VR for ASD and iteratively have refined our implementations (e.g., Schmidt et al., 2012; Newbutt et al., 2016; Newbutt et al., 2017; Glaser and Schmidt, 2018; Newbutt, 2019; Schmidt et al., 2019; Newbutt et al., 2020). Over the course of multiple studies and through working with stakeholders, we have developed principles and expert knowledge that inform our professional practice This has been achieved through a process of reflection-on-action that considers a range of critical issues and factors directly affecting the acceptance, perceived utility, implementation quality, and participant well-being of VR experiences for this population.

Materials and Methodology

To approach these questions, we analyzed two recently published research studies (Newbutt et al., 2020), associated artifacts, and documentation so as to extract and synthesize salient details of how researchers implemented HMD-based VR in their studies. The first and second authors were the lead researchers. Case study methodology was used to perform a structured within- and across-case analysis of implementation procedures with the goal of promoting analytic generalizations (Yin, 2017). A “case” was defined as one of the two independent research studies led by the first and second author.

The USA-based study led by the first author (Schmidt et al., 2019) utilized a design-based research (DBR) methodological approach that considered ease-of-use, the nature of participants’ user experience, feasibility, and relevance of a spherical, video-based virtual reality (SVVR) app as well as a fully immersive, collaborative 3D virtual environment, both of which were delivered using HMDs for a group of adults with ASD in a day program. The UK-based study led by the second author (Newbutt et al., 2020) sought to place children on the autism spectrum at the center of a study examining the potential of VR HMDs used in primary and secondary school classrooms; explored a range of VR experiences, how autistic users reported physical experiences, enjoyment, and potential of VR HMDs in their classrooms, while systematically exploring potential adverse effects.

Our work is presented as an instrumental, collective case study (Thomas, 2015). Our analytic process involved describing each case in detail and then presenting themes within the case, followed by thematic analysis across cases. As articulated in Figure 1, we adopted a three-phase approach. Phase 1 consisted of structured critical discussions in which we reviewed and summarized research details, methods, and findings, and established a basis for across-case comparison. In phase 2, artifacts were systematically discussed and reflected upon through formal meetings in which authors interrogated one another’s work to identify themes. Finally we synthesized lessons learned (Yin, 2017) into a framework that captured best practices.

FIGURE 1

Trustworthiness

We implemented a range of techniques to ensure the methodological rigor of our research from Lincoln and Guba (1985) recommendations for promoting trustworthiness. Most important to trustworthiness is establishing credibility (Shenton, 2004), which we sought to accomplish in the current case study in three ways. We aimed to promote transferability, which according to Tobin and Begley (2004), refers to the generalizability of research from one case to another. Thick description of our separate contexts is provided in the results of phase 1 below. By presenting similar findings from two studies, but in two separate settings, we provide a more inclusive overall picture. We also aimed to establish dependability and confirmability by maintaining an audit trail. We provide in the results section below a clear and logical description of our research process. We report our process in detail so that it can be repeated in future work. In this way, readers could adopt the research design we report here as a type of prototype model (Shenton, 2004). To allow the traceability of our work in a stage-wise process, we used software that logged all changes to our work over time. Our audit trail consists of versioned files that include step-by-step histories of the methods and findings presented in this paper.

Results

In the following sections, we present the outcomes of our within- and across-case analysis according to three separate phases of analysis. We present these phases in alignment with our research questions, beginning with detailed case descriptions, followed by thematic analysis and synthesis of lessons learned.

Results of Phase 1: Common Elements and Key Points of Differentiation Across Studies

In this section, we articulate each case’s details, background, and context related to: (a) equipment, participants, and virtual environments (Table 1); (b) research aims, data collection protocols (Table 2); and (c) key findings and limitations (Table 3). Study 1 study utilized a design-based research (DBR) methodological approach that considered ease-of-use, the nature of participants’ user experience, feasibility, and relevance of a spherical, video-based virtual reality (SVVR) app as well as a fully immersive, collaborative 3D virtual environment, both of which were delivered using HMDs for a group of adults with ASD in a day program. Study 2 sought to place children on the autism spectrum at the center of a study examining the potential of VR HMDs used in primary and secondary school classrooms; explored a range of VR experiences, how autistic users reported physical experiences, enjoyment, and potential of VR HMDs in their classrooms, while systematically exploring potential adverse effects. Detailed case descriptions are presented in Table 1. Cases are presented in a side-by-side tabular format to align each study’s research details, methods, findings, etc. for thematic analysis in Phase 2.

Results of Phase 2: Emergent Themes from Comparison of Studies

On the basis of the detailed case descriptions from Phase 1 (Tables 1–3), Phase 2 sought to present themes within and across cases. Results are framed within themes that emerged as lead researchers compared and analyzed cases. Over the process of thematic analysis, the researchers identified common segmentation points shared between cases. Seeking to sufficiently capture the complexity and highly contextual nature of our cases, the metaphor of stage production emerged. Stage production is particularly apt for characterizing our approach, as it seeks to gradually lead to a refined and reliably consistent performance that involves a range of stakeholders working together. As such, we characterize our implementations' segmentation points as “stages”, and articulate these as: (a) setting the stage (Table 4); (b) dress rehearsal (Table 5); (c) first preview (Table 6); and (d) opening night (acts 1 and 2; Table 7). Within each stage, we briefly detail the processes underwent by each research team and highlight common elements and key points of differentiation. In line with our instrumental, collective case study methodology, the approach of structuring processes across cases in a stage-wise manner (see Tables 4–7) established the foundation for further interrogation and analytic generalization in Phase 3.

Results of Phase 3: Lessons from Within- and Across-Case Comparison Related to Minimizing Adverse Effects of HMD-Based VR for Individuals with ASD

Results presented in this section represent the final outcome of the 3-phase distillation procedure. Having identified points of convergence and divergence across cases in phase 2, our focus transitioned in phase 3 to integrating the processes from both studies (Tables 4–7). Considerations of how to convey the lessons learned from prior analysis presented an opportunity to merge the processes and common elements from both cases into a representation of best practices. This required a translation of the specific details of each case into a set of generalizations that described each of the implementation stages. Attempts to develop textual descriptions of phases led to tensions, as text-based descriptions did not sufficiently represent the linear, step-wise nature of the implementation processes uncovered in our analysis. This led us to develop an initial linear visualization of our processes, which we subsequently refined using the approach shown in Figure 1 above. As a result, we articulated the stage-wise implementation framework as a linear process with descriptions and intent of each stage. Resonant in this process was the concept of gradual acclimation, which we discuss as follows.

Gradual Acclimation for Minimizing Adverse Effects

To minimize potential adverse effects gradual acclimation is needed (Figure 2). This is the process through which VR is continually optimized within and across implementation stages. We articulate gradual acclimation in two different areas of the process model: (1) contextual acclimation (Stages 1 and 2) and (2) technology acclimation (Stages 3 and 4). Contextual acclimation is concerned with researchers identifying the local variables that potentially could impact successful implementation. Contextual acclimation is primarily informed via input from stakeholders and facilitators (e.g., teachers, staff, program managers, collaborators, directors). Contextual acclimation is followed by technology acclimation, which is informed principally by participants when they begin by handling and experiencing the technology in a gradual manner in stages 3 and 4.

FIGURE 2

Gradual acclimation is, in part, predicated on ethical concerns. When working with a vulnerable population, researchers have a special obligation and greater responsibility to actively take precautions to help minimize real or potential risks (American Educational Research Association, 2011; Behavior Analyst Certification Board, 2014). This extends to possible adverse effects associated with the use of HMD-based VR for individuals ASD. We therefore assert that the potential for adverse effects could be minimized through intentional procedures that promote gradual acclimation within and across all stages. This is captured in Figure 2 as a triangle labeled Potential for adverse effects, which illustrates how the potential for adverse effects could be reduced by utilizing the intentional and systematic processes through contextual acclimation and technology acclimation. Figure 2 represents our final stage-wise process model of gradual contextual and technology acclimation to minimize potential adverse effects of HMD-based VR for individuals with ASD.

Discussion

Auspicious claims championing the potential of VR for individuals with ASD are abundant in the literature of the past 25 years—from the earliest reports (e.g., Strickland, 1996; 1997) to the most recent (Miller et al., 2019; Ke et al., 2020). However, “While it is apparent that VR has great conceptual potential, research and development are early on the road to making it a reality” (Wang and Anagnostou, 2014, p. 2137). Contributing to this are the many gaps in this research area, including a lack of guidelines for implementing VR interventions with this population, a general lack of consideration of adverse effects of VR in the published research, and very little specific research that reports findings related to adverse effects. Given the known characteristics of people with ASD (Christensen et al., 2018), and compounded by common comorbidities (Deprey and Ozonoff, 2018) such as anxiety (van Steensel and Heeman, 2017) and sensory processing differences (Thye et al., 2018), the prospect of adverse effects related to VR usage for individuals with ASD presents salient risks. Approaching calls and concerns for the safe and ethical applied use of technology for individuals with autism (Parsons, 2016; Bradley and Newbutt, 2018; Schmidt et al., 2019), the current article approaches the urgent need to consider potential adverse effects of HMD-based VR use for individuals with ASD.

To confront this need, we present a stage-wise process model derived from our own research that illustrates how researchers and practitioners alike can seek to reduce potential adverse effects in the design and execution of their HMD-based VR implementations for individuals with ASD. To contextualize the embedded processes and promote recall, we adopt the metaphor of a stage production to describe the different stages of the process. Like stage production, VR implementation requires considerable attention to details, many of which are unknown until they are encountered. Successful VR implementation also requires intentional, continuous coordination with key stakeholders such as teachers, administrators, IT staff, etc. (Newbutt, 2019), similar to a stage production requiring coordination between theater and production staff (e.g., stage manager, stagehands). Further, and perhaps most importantly, planning opportunities for participants to encounter VR technology and practice using it before engaging in an actual intervention is as important to VR implementation as is memorization of lines and table-reads to actors’ performance in a stage production. Finally, in stage production, preparation and rehearsal are critically important to bringing the production to a high standard of quality. We posit that application of the stage-wise model presented here could likewise influence the quality of the VR experience for participants by controlling for potential adverse effects.

Stage-Wise Process Model

The stage-wise process model represents a distillation of how two separate research teams approached the challenge of controlling for potential adverse effects. Our intent here is to provide guidance in an approachable, actionable framework and in doing so to portray our model in a minimalist manner so as to be useful to a range of audiences. However, this introduces a risk that it could be interpreted as an oversimplification of what in many cases can be exceptionally complex processes. The reader is therefore cautioned not to interpret the model at face value, but instead to consult the specific information presented in Tables 4–7 for detailed examples of the specific procedures and activities that were applied by using each stage of our work. We provide further discussion of the stages in the following sections.

Stage 1: Setting the Stage

Stage 1 includes front-end analysis and subsequent design and development of the intervention. These are activities that are well documented in the learning and instructional design literature (Morrison et al., 2012; Dick et al., 2014). What differentiates our approach from such established instructional design processes, however, is intentional solicitation of collaborative and meaningful input from all stakeholders, beginning in Stage 1, and continuing across all stages of the process. Continual input from stakeholders ensures the voices of those who matter the most are heard (Parsons et al., 2020). We highlight that this goes beyond requirements of institutional review boards or ethics committees for human subjects research (Protection of Human Subjects, 2009). Catalyzing this were critiques rebuking research studies that: (1) do not incorporate the voices and perspectives of individuals with ASD, their support communities, and/or relevant stakeholders in the design, implementation, and/or execution of studies; and (2) do not pursue and incorporate the priorities and cultural values of autistic communities (Parsons, 2016; Parsons et al., 2020). Importantly, teams of researchers and stakeholders should explicitly acknowledge in this stage that VR use and/or intervention has inherent potential for adverse effects. As such, they should actively seek to identify factors that could promote reduction of adverse effects as early as this stage. For example, a team might discover that some participants find the fit of one headset to be disagreeable, which could prompt exploration of more adjustable straps or different headsets altogether.

Stage 2: Dress Rehearsal

In Stage 2, the VR technology used to deliver the intervention is tested in-situ. This testing allows teams to collectively examine the space where the technology will be used. When working in-situ, researchers could find themselves provided with a space that is not ideal for HMD-based VR application. For example, they could be assigned a room that is too small, a cluttered area, or spaces containing tables and chairs. Such a space can present collision and trip hazards, which could lead to adverse health effects. Further, space needs to be amenable to a positive sensory experience for participants. For example, fluorescent lights or loud noises could be severely distracting for individuals with ASD (Grandin, 2002). In addition to this, testing allows the team to set up, configure and test the technology outside of a controlled lab setting. This allows for discovery of a variety of potential error points, for example, connectivity challenges such as restricted networks and prohibitive firewall rules, logistical challenges such as setting up for multi user experiences, or computer problems, such as misconfigurations and unforeseen glitches. Importantly, dress rehearsal is not technology testing; it is preparing for social-behavioral human subjects research (American Psychiatric Association, 2013). As such, it provides opportunities for stakeholders to see and experience the application of the technology and provide feedback, with the ultimate goal of providing an error-free, enjoyable implementation by identifying and addressing vectors that potentially could lead to adverse effects such as confusion, frustration, etc.

Stage 3: First Preview

In Stage 3, focus shifts to technology usage by participants. In line with the principle of gradual acclimation, this stage provides participants the opportunity to examine the technology and learn about the intervention. Many individuals with ASD respond positively to order, structure, and predictability, with deviations potentially leading to increased stress and anxiety (Gotham et al., 2013; Lidstone et al., 2014; Factor et al., 2016). First Preview seeks to address this by providing structured opportunities to communicate the project to participants. Although research studies are often communicated to research participants through approved informed consent procedures (e.g., American Educational Research Association, 2011; British Educational Research Association, 2018), we have found in our work the language used to describe VR projects (as approved by institutional review boards or ethical committees) can be abstract and contextually dependent. Given that many individuals with ASD are quite literal in their thinking (Volden et al., 2009; Deliens et al., 2018), this can frustrate communication efforts. The purpose of providing the opportunity for examining the technology and previewing the VR materials on a monitor is to provide concrete and real examples of what they will be experiencing, thereby enhancing communication with our participants. In this manner, two key aspects are achieved: (1) we are ensuring our ethical obligation to our participants (i.e., clear and concise communication); and (2) vulnerable participants are able to make better-informed decisions about whether they choose to participate in a given research study and what to expect if they do agree to be involved. This again illustrates the importance of privileging the voices and perspectives of participants throughout the implementation process (Parsons and Cobb, 2013; Politis et al., 2019).

Stage 4: Opening Night Act 1

In Act 1 of Opening Night the equipment and experience is delivered in a limited manner so as to promote gradual acclimation. Examples include: limiting the amount of time using the equipment; using lower tech versions of the equipment; and/or using simplified versions of the equipment. First, limiting the amount of time that participants engage in the VR experience could reduce the probability of serious adverse effects and allows researchers to check frequently with participants regarding how they are feeling, as well as with their guardians, caregivers, educators, etc. For example, researchers might design a series of 2-min tutorials or short 360 videos and take breaks in between to evaluate participants’ well-being and willingness to proceed. Second, using lower tech versions of the technology could be less complicated and therefore provide superior usability (Parish-Morris et al., 2018; Schmidt et al., 2019). For example, Google Cardboard devices have only a single button and no straps, allowing for the HMD to be removed easily in the case that a participant is experiencing negative effects. Third and finally, using simplified versions of the equipment, software, etc. could provide authentic experiences while reducing potential frustration or confusion (Rojo et al., 2019). For example, a series of 360 videos might auto-play instead of requiring participants to operate controls, or short tutorials might be presented using an “on rails” design in which participants are limited to moving their avatar, making selections, or changing their view, but not controlling the path their avatar takes. Given that the local context necessarily circumscribes how this stage of implementation is designed, these examples are not meant to be prescriptive but instead to illustrate how gradual acclimation is realized in this stage. Importantly, researchers must take care to intentionally and often question participants and other advocates who are involved regarding their well-being. Given recognized communication differences in many individuals with ASD (Masi et al., 2017), soliciting responses is recommended over providing opportunities for self-initiated reporting. This requires careful discussion and agreement to be sure that, for example, non-verbal participants can express their willingness to continue or desire to end the experience and/or study.

Stage 4: Opening Night Act 2

Act 2 of Opening Night is when participants engage comprehensively in the VR experience using fully immersive headsets (i.e., Oculus Quest, HTC Vive). During this stage, researchers should remain vigilant in their observation of participants while they engage in the experience(s), scrutinizing for any signs of adverse effects. Assuming that the prior stages have been enacted in a considered manner, participants should have gradually acclimatized to the technology, and the likelihood that adverse effects would unfavorably impact participants in this stage should be diminished.

Efforts to minimize risk do not end at this point. Even though a VR intervention might be very feasible for certain applications initially, using the VR intervention might become frustrating over time or introduce new risks. Evidence of this is scant given the lack of longitudinal research in this area. However, emerging research suggests longitudinal exposure effects of VR with individuals with ASD (Glaser et al., 2020), with participants rating their perceptions of cybersickness higher in later sessions and as the VR environment increased in complexity. Therefore, after participants have begun a training program or intervention, vigilance in observation should be ongoing and continual adjustments should be made as needed.

Through the stages presented above and the synthesised data from a US and a UK-based study, we have addressed an established lack of explicit guidance (Newbutt, 2019). On the one hand, research has noted the potential for VR HMDs to induce some negative effects (Sharples et al., 2008; Chessa et al., 2019; Weech et al., 2019), while on the other, very limited advice is offered on how to practically overcome these challenges. This is especially true in the field of autism, in which we are working with individuals who can present with sensory concerns (American Psychiatric Association, 2013). Our results provide initial insights to both fill a gap in the literature and also promote safe HMD-based VR practices for individuals with ASD and their support communities.

Implications

In the current article, we have presented a stage-wise process model promoting gradual acclimation to minimize potential adverse effects of HMD-based VR for individuals with ASD. We do not assert that these procedures are novel of-themselves. Instead, the innovation of our approach rests in how we distilled specific processes from successful research studies which could be used by others potentially to guide their own implementations. Further, our process model intentionally embraces and embodies social disability models, foregrounding themes of dignity, agency, and empowerment (Braddock et al., 2013). By locating the perspectives and voices of individuals with ASD centrally, we seek to avoid the mistake of insufficiently considering neurophilosophical issues such as hyper- or hyporeactivity to sensory inputs that individuals with ASD may experience (Tavassoli et al., 2016; Uljarević et al., 2016). While the impetus for developing the stage-wise process model was to provide guidance for reducing potential adverse effects, benefits of using the process model could represent a generalizable set of guidelines that others could use to inform their VR implementations. Inclusion of stakeholders and participants across all stages of implementation could increase their confidence in potential benefits of using the technology, positively influence potential anxiety on the part of participants, and increase stakeholder confidence that the technology will not cause harm. Designers could map existing methods and processes to the various stages in the model and use this to inform and guide their designs. Ultimately, the process model we provide in Figure 2 and discuss above could provide some assurance that stakeholders are being supportive of their ethical responsibilities when working with a vulnerable population (Bell, 2008; Pittaway et al., 2010).

Limitations

The research presented in the current article represents an important first step in reporting how researchers approached the problem of potential adverse effects in their implementation procedures. However, researchers seeking to utilize and further develop our approach should be aware of its limitations. The purpose of the research presented here was not to validate the stage-wise process model. While we believe it is likely that following a stage-wise implementation process helped to minimize adverse effects, we do not currently have sufficient data to support a causal relationship between the stage-wise process model and reduction in adverse effects. Because this work drew from completed studies, it was not possible to establish whether our stage-wise implementation processes were specifically related to the general lack of adverse effects observed in both studies. In addition, despite our collaborative approach working with individuals with ASD, we have not yet returned to these communities with our final process model to solicit feedback and input—a direction for future research. Furthermore, to extend the work presented here and approach limitations identified above, future research should explore the empirical relationship between utilization of our process model and influence on adverse effects. Moreover, given the heterogenous nature of ASD, future research should address how to tailor the process model in consideration of factors such as necessary level of support, individual differences (i.e., reactivity to sensory inputs, levels of cognition, etc.), and extended exposure to HMD technologies.

Conclusion

The focus in the current article on adverse effects of HMD-based VR for individuals with ASD rests on the basis of ethical guidelines published in our respective research contexts, including but not limited to those of BERA (British Educational Research Association, 2018), AERA (American Educational Research Association, 2011), and BACB (Behavior Analyst Certification Board, 2014). We situate the respect and dignity of our participants at the forefront of our work and incorporate practices that promote inclusion and involvement of participants and stakeholders. Paramount to our work is an acute sensitivity to the vulnerability of our target population (Parsons, 2015). In light of this, we endeavor to honor our special obligation and the greater responsibility we have as researchers to actively take precautions to help minimize real or potential risks associated with the use of HMD-based VR for ASD (e.g., emotional and physical harm or intervention side-effects). These potential risks could represent environmental constraints that we as researchers have an ethical obligation to eliminate. To this end, we have developed methods and processes for minimizing adverse effects of HMD-based VR for individuals with ASD. Our sincere hope is that the work presented here can serve as a signpost for how researchers can ethically approach considerations of adverse effects in future HMD-based VR implementations, and we urge researchers to consider applying and empirically interrogating the model proposed here.

References

• 1

  American Educational Research Association. (2011). Code of ethics. Available at: https://www.aera.net/Portals/38/docs/About_AERA/CodeOfEthics(1).pdf

  • Google Scholar

• 2

  American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders. Washington, DC: American Psychiatric Association.

  • Google Scholar

• 3

  BordK. (2007). Wicked ID: conceptual framework for considering instructional design as a wicked problem. Cjlt/Rcat33 (1), n1. 10.21432/t2cg6h

  • CrossRef
  • Google Scholar

• 4

  Behavior Analyst Certification Board (2014). Professional and ethical compliance code for behavior analysts. Available at: https://www.bacb.com/wp-content/uploads/2020/05/BACB-Compliance-Code-english_190318.pdf

  • Google Scholar

• 5

  BellN. (2008). Ethics in child research: rights, reason and responsibilities. Child. Geogr.6 (1), 7–20. 10.1080/14733280701791827

  • CrossRef
  • Google Scholar

• 6

  BellaniM.FornasariL.ChittaroL.BrambillaP. (2011). Virtual reality in autism: state of the art. Epidemiol. Psychiatr. Sci.20, 235–238. 10.1017/s2045796011000448

  • CrossRef
  • Google Scholar

• 7

  BentonL.AshwinE.JohnsonH.GrawemeyerB.BrosnanM. (2011). IDEAS: An interface design experience for the autistic spectrum, 1759–1764. Berlin: Springer.

  • Google Scholar

• 8

  BozgeyikliL.RaijA.KatkooriS.AlqasemiR. (2018). A survey on virtual reality for individuals with autism spectrum disorder: design considerations. IEEE Trans. Learn. Technol.11 (2), 133–151. 10.1109/tlt.2017.2739747

  • CrossRef
  • Google Scholar

• 9

  BraddockD.HoehlJ.TanisS.AblowitzE.HafferL. (2013). The rights of people with cognitive disabilities to technology and information access. Inclusion1 (2), 95–102. 10.1352/2326-6988-01.02.95

  • CrossRef
  • Google Scholar

• 10

  BradleyR.NewbuttN. (2018). Autism and virtual reality head-mounted displays: a state of the art systematic review. J. Enabling Tech.12, 101–113. 10.1108/jet-01-2018-0004

  • CrossRef
  • Google Scholar

• 11

  British Educational Research Association. (2018). Ethical guidelines for educational research. Available at: https://www.bera.ac.uk/researchers-resources/publications/ethicalguidelines-for-educational-research-2018

  • Google Scholar

• 12

  ChessaM.MaielloG.BorsariA.BexP. J. (2019). The perceptual quality of the oculus rift for immersive virtual reality. Human-Comput. Interact.34 (1), 51–82. 10.1080/07370024.2016.1243478

  • CrossRef
  • Google Scholar

• 13

  ChristensenD. L.BraunK. V. N.BaioJ.BilderD.CharlesJ.ConstantinoJ. N.et al (2018). Prevalence and characteristics of autism spectrum disorder among children aged 8 Years - autism and developmental disabilities monitoring network, 11 sites, United States, 2012. MMWR Surveill. Summ.65 (13), 1. 10.15585/mmwr.ss6513a1

  • CrossRef
  • Google Scholar

• 14

  DeliensG.PapastamouF.RuytenbeekN.GeelhandP.KissineM. (2018). Selective pragmatic impairment in autism spectrum disorder: indirect requests versus irony. J. Autism Dev. Disord.48 (9), 2938–2952. 10.1007/s10803-018-3561-6

  • CrossRef
  • Google Scholar

• 15

  DepreyL.OzonoffS. (2018). Assessment of comorbid psychiatric conditions in autism spectrum disorder. Berlin: Springer.

  • Google Scholar

• 16

  DickW.CareyL.CareyJ. (2014). The systematic design of instruction. 8th Edn. London: Pearson.

  • Google Scholar

• 17

  DSM-5 American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders. New York, NY: American Psychiatric Publishing.

  • Google Scholar

• 18

  EavesL. C.HoH. H. (2008). Young adult outcome of autism spectrum disorders. J. Autism Dev. Disord.38, 739–747. 10.1007/s10803-007-0441-x

  • CrossRef
  • Google Scholar

• 19

  EstesA.MunsonJ.RogersS. J.GreensonJ.WinterJ.DawsonG. (2015). Long-term outcomes of early intervention in 6-year-Old children with autism spectrum disorder. J. Am. Acad. Child Adolesc. Psychiatry54 (7), 580–587. 10.1016/j.jaac.2015.04.005

  • CrossRef
  • Google Scholar

• 20

  FactorR. S.CondyE. E.FarleyJ. P.ScarpaA. (2016). Brief report: insistence on sameness, anxiety, and social motivation in children with autism spectrum disorder. J. Autism Dev. Disord.46 (7), 2548–2554. 10.1007/s10803-016-2781-x

  • CrossRef
  • Google Scholar

• 21

  Fletcher-WatsonS. (2014). A targeted review of computer-assisted learning for people with autism spectrum disorder: towards a consistent methodology. Rev. J. Autism Dev. Disord.1 (2), 87–100. 10.1007/s40489-013-0003-4

  • CrossRef
  • Google Scholar

• 22

  FreinaL.OttM. (2015). A literature review on immersive virtual reality in education: state of the art and perspectives. Virt. Real.1, 133. 10.1007/s10055-020-00489-9

  • CrossRef
  • Google Scholar

• 23

  GlaserN. J.SchmidtM. (2018). Usage considerations of 3D collaborative virtual learning environments to promote development and transfer of knowledge and skills for individuals with autism. Tech. Know. Learn.25 (25), 315–322. 10.1007/s10758-018-9369-9

  • CrossRef
  • Google Scholar

• 24

  GlaserN.SchmidtM.SchmidtC. (2020). Fear and loathing in VR: cybersickness evidence in headset-based VR training for adults with Autism Presented at the 2020 international convention of the association for educational communications and technology. Online.

  • Google Scholar

• 25

  GothamK.BishopS. L.HusV.HuertaM.LundS.BujaA.et al (2013). Exploring the relationship between anxiety and insistence on sameness in autism spectrum disorders. Autism Res.6 (1), 33–41. 10.1002/aur.1263

  • CrossRef
  • Google Scholar

• 26

  GrandinT. (2002). Teaching tips for children and adults with autism, 2. Fort Collins: Colorado/EUA, 5.

  • Google Scholar

• 27

  GrynszpanO.WeissP. L.Perez-DiazF.GalE. (2014). Innovative technology-based interventions for autism spectrum disorders: a meta-analysis. Autism18 (4), 346–361. 10.1177/1362361313476767

  • CrossRef
  • Google Scholar

• 28

  HedleyD.UljarevićM.CameronL.HalderS.RichdaleA.DissanayakeC. (2017). Employment programmes and interventions targeting adults with autism spectrum disorder: a systematic review of the literature. Autism21 (8), 929–941. 10.1177/1362361316661855

  • CrossRef
  • Google Scholar

• 29

  IrishJ. E. N. (2013). Can I sit here? A review of the literature supporting the use of single-user virtual environments to help adolescents with autism learn appropriate social communication skills. Comput. Hum. Behav.29 (5), A17–A24. 10.1016/j.chb.2012.12.031

  • CrossRef
  • Google Scholar

• 30

  JarroldW.MundyP.GwaltneyM.BailensonJ.HattN.McIntyreN.et al (2013). Attention in a virtual public speaking task in higher functioning children with autism. Autism Res.6, 393–410. 10.1002/aur.1302

  • CrossRef
  • Google Scholar

• 31

  JosmanN.Ben-ChaimH. M.FriedrichS. (2011). Effectiveness of virtual reality for teaching street-crossing skills to children and adolescents with autism. Int. J. Disabil. Hum. Dev. IJDHD7 (1), 49–56. 10.1515/ijdhd.2008.7.1.49

  • CrossRef
  • Google Scholar

• 32

  KandalaftM. R.DidehbaniN.KrawczykD. C.AllenT. T.ChapmanS. B. (2013). Virtual reality social cognition training for young adults with high-functioning autism. J. Autism Dev. Disord.43 (1), 34–44. 10.1007/s10803-012-1544-6

  • CrossRef
  • Google Scholar

• 33

  KeF.ImT. (2013). Virtual-reality-based social interaction training for children with high-functioning autism. J. Educ. Res.106 (6), 441–461. 10.1080/00220671.2013.832999

  • CrossRef
  • Google Scholar

• 34

  KeF.MoonJ.SokolikjZ. (2020). Virtual reality–based social skills training for children with autism spectrum disorder. J. Spec. Education Technol.14, 016264342094560. 10.1109/fie44824.2020.9273818

  • CrossRef
  • Google Scholar

• 35

  KellmeyerP. (2018). Neurophilosophical and ethical aspects of virtual reality Therapy in Neurology and Psychiatry. Camb Q. Healthc. Ethics27 (4), 610–627. 10.1017/s0963180118000129

  • CrossRef
  • Google Scholar

• 36

  KhowajaK.SalimS. S. (2013). A systematic review of strategies and computer-based intervention (CBI) for reading comprehension of children with autism. Res. Autism Spectr. Disord.7 (9), 1111–1121. 10.1016/j.rasd.2013.05.009

  • CrossRef
  • Google Scholar

• 37

  LidstoneJ.UljarevićM.SullivanJ.RodgersJ.McConachieH.FreestonM.et al (2014). Relations among restricted and repetitive behaviors, anxiety and sensory features in children with autism spectrum disorders. Res. Autism Spectr. Disord.8 (2), 82–92. 10.1016/j.rasd.2013.10.001

  • CrossRef
  • Google Scholar

• 38

  LinconY. S.GubaE. G. (1985). Naturalistic inquiry. Sage: Beverly Hills.

  • Google Scholar

• 39

  MalihiM.NguyenJ.CardyR. E.EldonS.PettaC.KushkiA. (2020). Short report: evaluating the safety and usability of head-mounted virtual reality compared to monitor-displayed video for children with autism spectrum disorder. Autism24, 1924–1929. 10.1177/1362361320934214

  • CrossRef
  • Google Scholar

• 40

  MasiA.DeMayoM. M.GlozierN.GuastellaA. J. (2017). An overview of autism spectrum disorder, heterogeneity and treatment Options. Neurosci. Bull.33 (2), 183–193. 10.1007/s12264-017-0100-y

  • CrossRef
  • Google Scholar

• 41

  Mesa-GresaP.Gil-GómezH.Lozano-QuilisJ.-A.Gil-GómezJ.-A. (2018). Effectiveness of virtual reality for children and adolescents with autism spectrum disorder: an evidence-based systematic review. Sensors18 (8), 114. 10.3390/s18082486

  • CrossRef
  • Google Scholar

• 42

  MillerI. T.WiederholdB. K.MillerC. S.WiederholdM. D. (2019). Virtual reality air travel training with children on the autism spectrum: a preliminary report. Cyberpsychol. Behav. Soc. Netw.23, 10–15. 10.1089/cyber.2019.0093

  • CrossRef
  • Google Scholar

• 43

  MitchellP.ParsonsS.LeonardA. (2007). Using virtual environments for teaching social understanding to 6 adolescents with autistic spectrum disorders. J. Autism Dev. Disord.37 (3), 589–600. 10.1007/s10803-006-0189-8

  • CrossRef
  • Google Scholar

• 44

  MooreD.Yufang ChengP.McGrathP. (2005). Collaborative virtual environment technology for people with autism. Focus Autism Other Dev. Disabl.20, 231–243. 10.1177/10883576050200040501

  • CrossRef
  • Google Scholar

• 45

  MorrisonG. R.RossS.KalmanH.KempJ. (2012). Designing effective instruction. 7th Edn. Hoboken: John Wiley.

  • Google Scholar

• 46

  NewbuttN.SungC.KuoH.-J.LeahyM. J.LinC.-C.TongB. (2016). Brief report: a Pilot study of the use of a virtual reality headset in autism populations. J. Autism Dev. Disord.46 (9), 3166–3176. 10.1007/s10803-016-2830-5

  • CrossRef
  • Google Scholar

• 47

  NewbuttN.SungC.KuoH-J.LeahyM. J. (2017). “The potential of wearable technology (head-mounted displays) and virtual reality to support people with autism: acceptance, challenges, and future applications,” in Recent advances in technologies for inclusive well-being: from worn to off-body sensing, virtual worlds, and games for serious applications. Editors BrooksA.BrahnamS.JainL. (Berlin: Springer), 221–241. Chapter 11.

  • Google Scholar

• 48

  NewbuttN.BradleyR.ConleyI. (2020). Using virtual reality head-mounted display in schools for pupils on the autism spectrum; views, experiences and future directions. Cyberpsychol. Behav. Soc. Netw.23 (1), 22–33. 10.1089/cyber.2019.0206

  • CrossRef
  • Google Scholar

• 49

  NewbuttN. (2019). “Assisting people with autism spectrum disorder through technology,” in Encyclopedia of education and information technologies. Editor TatnallA. (Switzerland: Springer).

  • Google Scholar

• 50

  PalmisanoS.MursicR.KimJ. (2017). Vection and cybersickness generated by head-and-display motion in the Oculus Rift. Displays46, 1–8. 10.1016/j.displa.2016.11.001

  • CrossRef
  • Google Scholar

• 51

  Parish-MorrisJ.SolórzanoR.RavindranV.SazawalV.TurnaciogluS.ZitterA.et al (2018). Immersive virtual reality to improve police interaction skills in adolescents and adults with autism spectrum disorder: preliminary results of a phase i feasibility and safety trial. Annu. Rev. CyberTherapy Telemed.14 (16), 50–56. 10.1002/aur.2352

  • CrossRef
  • Google Scholar

• 52

  ParsonsS.CobbS. (2013). EPSRC observatory for responsible innovation in ICT. Who chooses what I need? Child voice and user-involvement in the development of learning technologies for children with autism. Berlin: Springer.

  • Google Scholar

• 53

  ParsonsS.MitchellP.LeonardA. (2004). The use and understanding of virtual environments by adolescents with autistic spectrum disorders. J. Autism Dev. Disord.34 (4), 449–466. 10.1023/b:jadd.0000037421.98517.8d

  • CrossRef
  • Google Scholar

• 54

  ParsonsS.YuillN.GoodJ.BrosnanM. (2020). ‘Whose agenda? Who knows best? Whose voice?' Co-creating a technology research roadmap with autism stakeholders. Disabil. Soc.35 (2), 201–234. 10.1080/09687599.2019.1624152

  • CrossRef
  • Google Scholar

• 55

  ParsonsS. (2015). Learning to work together: designing a multi-user virtual reality game for social collaboration and perspective-taking for children with autism. Int. J. Child-Comput. Interact.6, 28–38. 10.1016/j.ijcci.2015.12.002

  • CrossRef
  • Google Scholar

• 56

  ParsonsS. (2016). Authenticity in virtual reality for assessment and intervention in autism: a conceptual review. Educ. Res. Rev.19, 138–157. 10.1016/j.edurev.2016.08.001

  • CrossRef
  • Google Scholar

• 57

  PittawayE.BartolomeiL.HugmanR. (2010). 'Stop stealing our stories’: the ethics of research with vulnerable groups. J. Hum. Rights Pract.2 (2), 229–251. 10.1093/jhuman/huq004

  • CrossRef
  • Google Scholar

• 58

  PolitisY.OliviaL.OliviaT. (2019). Empowering autistic adults through their involvement in the development of a virtual world. Aia5 (4), 303–317. 10.1108/aia-01-2019-0001

  • CrossRef
  • Google Scholar

• 59

  Protection of Human Subjects (2009). CFR45, 46.

  • Google Scholar

• 60

  RojoD.MayorJ.RuedaJ. J. G.RayaL. (2019). A virtual reality training application for adults with Asperger's syndrome. IEEE Comput. Grap. Appl.39 (2), 104–111. 10.1109/mcg.2018.2884272

  • CrossRef
  • Google Scholar

• 61

  PotelG.RodgersL. R.UkoumunneO. C.FordT. (2014). Prevalence of parent-reported ASD and ADHD in the UK: findings from the Millennium Cohort study. J. Autism Dev. Disord.44 (1), 31–40. 10.1007/s10803-013-1849-0

  • CrossRef
  • Google Scholar

• 62

  SchmidtM.LaffeyJ. M.SchmidtC. T.WangX.StichterJ. (2012). Developing methods for understanding social behavior in a 3D virtual learning environment. Comput. Hum. Behav.28 (2), 405–413. 10.1016/j.chb.2011.10.011

  • CrossRef
  • Google Scholar

• 63

  SchmidtM.SchmidtC.GlaserN.BeckD.LimM.PalmerH. (2019). Evaluation of a spherical video-based virtual reality intervention designed to teach adaptive skills for adults with autism: a preliminary report. Interact. Learn. Environ.11, 33. 10.1080/10494820.2019.1579236

  • CrossRef
  • Google Scholar

• 64

  SelfT.ScudderR. R.WehebaG.CrumrineD. (2007). A virtual approach to teaching safety skills to children with autism spectrum disorder. Top. Lang. Disord.27, 242–253. 10.1097/01.TLD.0000285358.33545.79

  • CrossRef
  • Google Scholar

• 65

  SharplesS.CobbS.MoodyA.WilsonJ. R. (2008). Virtual reality induced symptoms and effects (VRISE): comparison of head mounted display (HMD), desktop and projection display systems. Displays29 (2), 58–69. 10.1016/j.displa.2007.09.005

  • CrossRef
  • Google Scholar

• 66

  ShentonA. K. (2004). Strategies for ensuring trustworthiness in qualitative research projects. Efi22 (2), 63–75. 10.3233/efi-2004-22201

  • CrossRef
  • Google Scholar

• 67

  StricklandD. (1996). A virtual reality application with autistic children. Presence Teleop. Virt. Environ.5 (3), 319–329. 10.1162/pres.1996.5.3.319

  • CrossRef
  • Google Scholar

• 68

  StricklandD. (1997). Virtual reality for the treatment of autism. Stud. Health Technol. Inform.44, 81–86.

  • Google Scholar

• 69

  TavassoliT.BellesheimK.SiperP. M.WangA. T.HalpernD.GorensteinM.et al (2016). Measuring sensory reactivity in autism spectrum disorder: application and simplification of a clinician-administered sensory observation scale. J. Autism Dev. Disord.46 (1), 287–293. 10.1007/s10803-015-2578-3

  • CrossRef
  • Google Scholar

• 70

  TaylorJ. L.MailickM. R. (2014). A longitudinal examination of 10-year change in vocational and educational activities for adults with autism spectrum disorders. Dev. Psychol.50 (3), 699–708. 10.1037/a0034297

  • CrossRef
  • Google Scholar

• 71

  TaylorJ. L.SeltzerM. M. (2011). Employment and post-secondary educational activities for young adults with autism spectrum disorders during the transition to adulthood. J. Autism Dev. Disord.41 (5), 566–574. https://doi.org/10/czqgq4. 10.1007/s10803-010-1070-3

  • CrossRef
  • Google Scholar

• 72

  ThomasG. (2015). How to do your case study. New York City, NY:Sage.

  • Google Scholar

• 73

  ThyeM. D.BednarzH. M.HerringshawA. J.SartinE. B.KanaR. K. (2018). The impact of atypical sensory processing on social impairments in autism spectrum disorder. Dev. Cogn. Neurosci.29, 151–167. 10.1016/j.dcn.2017.04.010

  • CrossRef
  • Google Scholar

• 74

  TobinG. A.BegleyC. M. (2004). Methodological rigour within a qualitative framework. J. Adv. Nurs.48 (4), 388–396. 10.1111/j.1365-2648.2004.03207.x

  • CrossRef
  • Google Scholar

• 75

  UljarevićM.LaneA.KellyA.LeekamS. (2016). Sensory subtypes and anxiety in older children and adolescents with autism spectrum disorder. Autism Res.9 (10), 1073–1078. 10.1002/aur.1602

  • CrossRef
  • Google Scholar

• 76

  van SteenselF. J. A.HeemanE. J. (2017). Anxiety levels in children with autism spectrum disorder: a meta-analysis. J. Child Fam. Stud.26 (7), 1753–1767. 10.1007/s10826-017-0687-7

  • CrossRef
  • Google Scholar

• 77

  VoldenJ.CoolicanJ.GaronN.WhiteJ.BrysonS. (2009). Brief report: pragmatic language in autism spectrum disorder: relationships to measures of ability and disability. J. Autism Dev. Disord.39 (2), 388. 10.1007/s10803-008-0618-y

  • CrossRef
  • Google Scholar

• 78

  WangM.AnagnostouE. (2014). “Virtual reality as treatment Tool for children with autism,” in Comprehensive guide to autism. Editors PatelV. B.PreedyV. R.MartinC. R. (Berlin: Springer), 2125–2141.

  • Google Scholar

• 79

  WeechS.KennyS.Barnett-CowanM. (2019). Presence and cybersickness in virtual reality are negatively related: a review. Front. Psychol.10, 158. 10.3389/fpsyg.2019.00158

  • CrossRef
  • Google Scholar

• 80

  WongC.OdomS. L.HumeK. A.CoxA. W.FettigA.KucharczykS.et al (2015). Evidence-based practices for children, Youth, and young adults with autism spectrum disorder: a comprehensive review. J. Autism Dev. Disord.45 (7), 1951–1966. 10.1007/s10803-014-2351-z

  • CrossRef
  • Google Scholar

• 81

  YinR. (2017). Case study research and applications: design and methods. 6th Edn. New York City, NY: SAGE.

  • Google Scholar