Akgun Tokatli- Department of Visual Communication Design, Istanbul Medipol University, Istanbul, Turkey
The rapid advancement of virtual reality technology has significantly influenced technological, social, and business sectors. In Virtual Reality (VR) product design and development, the prioritisation and optimization of ergonomic criteria are crucial for shaping product decisions and fostering innovation. Ergonomics encompasses several sub-dimensions, making product design a complex, multi-criteria decision-making process. In this context, decisions are typically made based on collective insights and data gathered by the development team. The Analytic Hierarchy Process (AHP) is widely used to understand the prioritisation of multiple criteria. Study aims to analyse the impact and relationship between the prioritisation of ergonomic criteria by VR design and development teams on product value and innovation propositions and to present an approach for ergonomics-based product differentiation and innovation analysis. Through an extensive literature review and expert interviews, 11 specific ergonomic criteria were identified. These criteria were then assessed by practitioners using the AHP methodology, enabling pairwise comparisons to determine their relative importance. The analysis explored the differences between teams that conduct user research and those that do not. The study provides insights into ergonomic priorities in VR product design and development. The results were compared with the value propositions highlighted on the websites of 20 companies producing non-gaming VR products. As expected, the comparison showed that the ergonomic criteria emphasised in marketing coincided with those prioritised by the participants, especially among those who conducted user research. It shows that this scientifically proven agile approach can be used to find points of differentiation in the product development and innovation. Marketing value propositions can be used to identify which ergonomics criteria are prioritised by the market or by specific products. These criteria are compared by determining how the product team’s mental model prioritises them and a gap and alignment analysis can be performed according to the product purpose and findings support ergonomics-based decision making and innovation.
1 Introduction
Virtual Reality (VR) continues to develop with great speed and competition in technological, social and business fields. Besides technological, business and budget constraints, VR creates an environment and must meet the ergonomic needs of people. Product design and development is a multi-criteria problem. Virtual realities enable users to interact in real time with computer-generated environments in three dimensions (Segal et al., 2011). Designers and developers need to consider the wider aspects of users’ ergonomic needs. Easa (2021) highlighted that considerable work remains for researchers and manufacturers to resolve various human factors challenges in virtual reality. Oberdörfer et al. (2023) emphasized the importance of identifing potential risks and establishing design guidelines to ensure user safety. Thinking about VR ergonomics in a holistic way can help to reduce these potential risks. Germanakos et al. (2022) highlights that while most professionals prioritize technology in developing high-fidelity Immersive Reality simulations, the critical role of human factors in fostering a sense of presence is often overlooked.
The researchers are developing a conceptual model for the integration of human factors to improve VR performance (Maneuvrier et al., 2020). Analysing the criteria evaluation and decision-making processes of designers and developers will form the basis for analysing the impact of human factors on product value propositions and innovation in virtual reality projects.
It is expected that the prioritisation of the practitioners will be reflected in the product value propositions and be analytically compatible within the framework of the customer/human-centered design approach adopted by companies today.
This study aims to (1) identify which ergonomic criteria are prominent in the projects of designers and developers working on VR projects, (2) identify the ergonomic criteria are emphasized in the value propositions of VR products, and (3) determine whether there is a consistency between these criteria. This consistency will indicate that the comparison of practitioners’ multi-criteria prioritisation of product criteria with sectoral prioritisation can be used as an innovation analysis to identify gaps and opportunities for differentiation.
A VR product development project typically includes analysis, design, development, and testing stages. During the analysis and design stages, project teams synthesize data from multiple sources such as user needs, market conditions, competitor analysis, resource constraints, and past product data. This leads to internal insights among team members (analysts, product owners, designers, developers).
The ergonomic criteria provides a structured framework through which these insights can be interpreted. The AHP method enables an objective model for prioritization and weighting. By using this method, teams can clearly define where to focus their efforts in product differentiation and innovation. During the study process, creating a holistic ergonomic criteria framework for VR products was identified as a sub-objective.
The study includes four main stages:
1. Literature review and identification of ergonomic criteria
2. Prioritization of ergonomic criteria by professionals using AHP
3. Analysis of marketing value propositions and identification of emphasized criteria
4. Comparison of the two prioritization results
The study utilized three primary data sources: literature and standards, expert input via AHP surveys (Supplementary Appendix Table 12), and VR company websites (Supplementary Appendix Table 11).
After identifing the ergonomic criteria, in the second part of the study examines how VR designer and developers prioritise and apply 11 specific ergonomic criteria in their projects, identified through a literature review and interviews with designers. These criteria and their descriptions are as follows: Visual comfort and clarity, Head and neck ergonomics, Interface and interaction design, Physical balance and movement optimisation, Mental workload, Immersion and presence, Accessibility and adaptability, Performance and response time, Social interaction, Trust, Inclusion and ethics.
In the analysis of decision processes where a large number of criteria are active, the method of analysing criteria by comparing them with each other can be used. There are many studies that investigate the use and effectiveness of pairwise comparision and The Analytic Network Process (ANP) or AHP to prioritise the importance of criteria in scientific studies (von Solms, 2011; Ordoobadi, 2012; De Felice, 2012). When deriving a scale of priorities from pairwise comparisons, the number of times one criterion is more important than another is assessed. When all criteria are compared with each other, their relative importance can be found (Saaty, 1980; Saaty, 2005; Beynon, 2002). The outputs were analysed according to whether the participants had conducted user research in their processes. In the third part of the study, ergonomic criteria are matched with the marketing value propositions collected from the websites of twenty non-gaming virtual reality companies, and an analysis is conducted to identify which criteria are most emphasized. The final part of the study compares the prioritization of ergonomic criteria by professionals with the ergonomics-based value propositions emphasized by 20 non-gaming VR companies.
2 Theoritical background
In a multi-criteria decision study, it is important to determine the criteria in a way that allows for an integrated analysis of the problem. Each criterion should be defined, and participants should understand the differences between the criteria. Saaty (1980) suggests that to create a hierarchy with all its breakdowns, one should typically review the literature to enrich ideas and often work with others, utilizing a free brainstorming session to list all concepts related to the problem, regardless of their relationships or order.
The criteria collected from human factors and ergonomics issues have been researched in studies and systematic reviews in academic databases, and standards set by standard-setting organisations in ergonomics have also been used.
ISO defines ergonomics through Physical, Cognitive, Organizational, Environmental, Social, and Cultural dimensions. International Ergonomics Association (2023) identifies three main branches of ergonomics: physical, cognitive, and organizational ergonomics, as well as safety culture. Stramler (1993) highlights psychological, social, physical, and biological characteristics as key areas of human factors research, which play a critical role in determining the quality of the VR experience. ISO (2024), Ergonomic guidance on interactions in immersive environments, including augmented reality and virtual reality, defines general concepts but does not provide a guide on which ergonomic criteria play a more significant role depending on different fields of work or user needs. In VR applications designed for different users and needs, ergonomic prioritization will vary. As accessibility and inclusivity requirements increase, it is not possible to establish a one-size-fits-all ergonomic standard. Systematic reviews emphasize the importance of developing human factors-based design models. In their reviews, authors (Chen and Wu, 2023), discuss the ergonomic issues of virtual reality software and hardware in three main aspects: visual, physiological, and cognitive. Additionally, topics researched in VR and ergonomics (Kazemi and Lee, 2023) include mental workload, physical workload and muscle fatigue, physiological responses (such as stress), visual fatigue, postural stability, usability, cybersickness, and presence. Numerous methods have been challenged for their failure to fully address the intricate relationships between technical, experiential, and psychological factors in VR experiences. The absence of standardisation and the inconsistent use of terminology further complicate the field. To tackle these issues, researchers have proposed a some taxonomies (Hameed et al., 2024).
3 Materials and methods
3.1 Identification and categorization of ergonomic criteria
As a result of literature review, 24 concepts were compiled in order to identify the criteria with the research participants. These are; parallax adjustment, dynamic lighting, eye fatigue, distance perception, upper limp health, movement optimization, user positioning, usability, user interaction, navigation, application interface, cybersickness, balance perception, motion perception, cognitive fatigue, immersion, presence, system performance, user response time, system response time, privacy, security, accessibility, muscle fatigue.
A workshop was held with the three research participants who have academic backgrounds in VR design and ergonomics to improve the quality of criteria definition and categorisation work, as suggested in the methodology. The participants were given 24 concepts and asked to categorise them according to their similarities and differences within the framework of the study purpose and to code the classes. They were told that they could add a new criterion.
Coding and categorisation studies are often used in social research, especially in grounded theory. Depending on the desired level of abstraction, the concepts created for the categories can be regrouped under a higher level concept (Punch, 2005; Miles and Huberman, 1994).
In the first iteration, the concept of usability was not used as a criterion because it is understood as a high level category encompassing multiple ergonomic criteria. ISO measures usability with effectiveness, efficiency, and satisfaction, and expands the dimensions of usability. Authors also discuss the scope of usability as defined by ISO (Bevan et al. 2015). In the second iteration, it was decided to expand the concept set with inclusion, trust and social interaction. Also researcher give point to both positive and negative social implications potentials of vr (Stanney, 1995).
In this framework, the criteria defined according to the research purpose can be evaluated by different researchers with a more abstract (less and inclusive criteria/categories) or more specific (more and narrowed criteria) classification due to different purposes, contexts, design dependencies and constraints.
As a result, 11 criteria were identified. The definitions of the criteria were sent to the participants in the research forms and they were asked to evaluate them in order to unify the meaning.
3.2 Prioritization of ergonomic criteria via AHP method
The second part of the research is based on the ranking of the importance of the identified ergonomic criteria from the designers’ perspective using the Analytic Hierarchy Process (AHP) methodology (Saaty, 1980). AHP is an effective method used in multi-criteria decision making by organising complex decision problems in a hierarchical structure. There are many surdies on the use of AHP in scientific researches (Sun, 2023; Emrouznejad and Marra, 2017; Carlos and Vansnick 2008).
Through a questionnaire designed using AHP methodology, designers and developers will be asked to compare and prioritise the ergonomic criteria they are working on in their projects. This methodology will allow for a systematic analysis of the priority relationships between ergonomic criteria and provide in-depth insights into the ergonomic design decisions made by practitioners. It will also investigate whether user research and testing is carried out during the design and development process.
Halbig and Latoschik (2021) list references in their literature review that similar dimensions have been analysed with the help of questionnaires and the advantages and disadvantages of the questionnaire method in VR research. Among the disadvantages listed, AHP has been used against self-report biases, the tendency to select neutral responses, the tendency to select extreme options in a rating.
Asma (2006) argues that AHP’s structured pairwise comparisons enhance objectivity and reduce biases common in self-report methods, while emphasizing the importance of user interpretation and consistency checks. Belton and Stewart (2002) emphasize that the mathematical rigor of multicriteria decision models provides objective guidance by aligning decision-maker preferences with a coherent set of assumptions. The fact that the participants are experts and often work on similar projects helps to overcome the problems mentioned and the difficulty in remembering their experiences at the time of the research. Identifying participants who are related to the problem is the most important aspect of creating a hierarchy. Participants were purposively selected based on their experience in VR product design and their knowledge of user ergonomics. The organisations where the participants work are given in the appendix (Supplementary Appendix Table 13). All of the participants included in the study declared that they received training on user ergonomics and regularly conducted user testing. They have enough theoretical and practical knowledge to contribute to this research. The 12 participants define themselves as follows: six as designers (3D, Graphic, UX/UI), four as both designers and developers (Product Owners and R&D Specialists), and two as developers. Technological advancements in hardware, an aspect that developers and designers cannot change during the project process, directly influence the priorities and relative importance of ergonomic criteria. Participants report using multiple software tools in projects. Likewise, while they use different hardware in various contexts, state that they develop projects using Meta Quest during the study period.
Participants made 55 pairwise comparisons for 11 criteria.
As shown in Table 1 the scale for pairwise comparisons was determined as 1, 3 and 5. 1; equal importance, 3; moderate importance, 5; strong importance (Saaty and Ozdemir, 2003).
Table 1. Comparision scale.
To calculate the common judgement of the participants, the state variables were transformed as 4,2,0,-2,-4. The arithmetic mean was used to determine how far all the evaluations were from zero, which represents the consensus. The value will be plus or minus, indicating the direction of the judgement. Such approaches are used to determine the level of value assigned by a group (Saaty, 1980). Calculation example is shown in Table 2.
Table 2. Evaluation example.
Mean of this comparision is (0-2-2)/3 = −4/3 The value to be used in matrix is |-4/3| + 1 = 7/3 And the sample matrix is shown in Table 3.
Table 3. Consensus matrix example.
The −4 to 4 scale was used to calculate the number of participants. At the beginning of the study, it is “a priori” that all criteria are at the same level and that the mean evaluation of all participants is zero.
To calculate the research participant size n = (z2 s2)/d2 formula was used. N = sample size, Z = z-score of desired level of confidence d = estimated critical difference z2 = variance (Sauro and Lewis, 2016).
In order to determine the critical difference, for each comparison starting from 5 participants, the distance of the participants’ evaluations from the comparison mean was counted. It was determined that 60% of the comparisons made with 5, 7 and 9 participants were 1 or more away from the mean. So the critical difference was taken as 1. In 4 to −4 scale, the difference between the values in scale is 2. This difference supports this determination.
The confidence level for the study was set as 90%. Z = 1,645.
Starting from the 7th participant, the variance was calculated for all comparisons, the required number of participants was calculated on after the 2 participant added to the group. When the calculated number of participants and the number of participants were equal, the study was stopped and the data were started to be evaluated within the research framework.
As shown in Table 4, when the number of participant reached 12, the required number of participant was calculated as 12. With the data provided form 12 participants pairwise comparision matrix was formed. Matrix is in the appendix.
Table 4. Number of participants calculation.
3.3 Analysis of marketing value propositions from VR companies
In the third part of the research, it was determined whether the value propositions matched any of the 11 ergonomic criteria. The 20 non-gaming VR companies were selected based on their industry visibility and presence in sectors such as healthcare, education, and simulation (Supplementary Appendix Table 11). In order to analyse this, it was checked whether there were tags/words describing the criteria in the value proposition discourses. For each firm, three marketing discourses matched the criteria. The websites of the analysed companies and the dates of access to their websites are given in the appendix.
4 Results
4.1 Result 1. Consolidation of 24 ergonomic concepts into 11 criteria
As a result, 11 criteria were identified. Definitions were given in order to create a framework.
4.1.1 Visual comfort and clarity
Effective design and adjustment of elements such as parallax, dynamic lighting and focal points to reduce eye fatigue, improve focus and accurate distance perception is crucial in VR design.
Stanley et al. (2001) discussed the key considerations in designing networked virtual environments from a software development perspective, emphasising visual quality and real-time response. Visual quality is shaped by elements like image resolution, scene complexity and visual mode. Studies on visual fatigue in VR show varied results. Kazemi and Lee (2023) reveals some research indicates that VR can induce visual fatigue and affect oculomotor functions differently than real-world settings, impacting accommodation but not vergence. However, other studies report no significant differences in visual fatigue or adverse effects on binocular vision when comparing VR with natural environments.
4.1.2 Neck and head ergonomics
Optimal motion and navigation design to reduce unnecessary bending and reaching, optimise user positioning, avoid prolonged head turning and sudden movements are important to the physical health of users. Richter et al. (2016) presents that the development of visual fatigue affects not only the cognitive processes of the visual system but also the biomechanical performance of the neck and shoulder region. Caple (2008) highlighted the significant rise in computer usage, which has been linked to a higher incidence of neck and upper limb disorders, as well as various physical and psychosocial issues related to growing computer usage.
4.1.3 Interface and interaction design
Designing movement, interactions and interfaces to improve usability in terms of ease of use is a key factor in VR application design. A variety of studies have focused on interface and interaction design in virtual reality, aiming to improve usability through the design of movement, displacement methods, interactions, and interfaces.
Cao, (2009) examines the improvement of usability through designing movement and interactions. The VRID model, developed by Tanrıverdi and Jacop, (2001) assists designers in virtual reality interface design by addressing challenges in comprehensive design thinking and communication with developers. Souchet et al. (2022) discussed how inappropriate or poorly designed metaphors or interfaces for interactions can lead to significant mental workload and suboptimal task performance.
Studies are being conducted on enhancing user experience in VR environments with the support of AI-driven adaptive user interface design (Zhou et al., 2024). Ponce et al. (2024) emphasize the importance of balancing the computational demands of real-time VR rendering and interaction with the processing requirements of AI-driven data analysis and decision-making. They highlight that achieving this equilibrium is crucial for ensuring VR experiences remain both effective and ergonomically sustainable, particularly for extended use in industrial applications. This perspective reinforces the necessity of designing VR systems where hardware and software advancements collectively contribute to an optimized user experience.
4.1.4 Physical balance and movement optimisation
In VR design, careful coordination of user movement and design of visual feedback helps to avoid cybersickness and positively affects balance and motion perception in VR applications. Bourdin et al. (2019) shows that virtual reality can significantly influence motor performance through body ownership illusions without the participants’ conscious awareness. This finding is critical for VR design decisions, as it highlights the potential of VR to manipulate and enhance user interaction and experience through subtle modifications to visual feedback.
4.1.5 Mental workload
Optimising flow, usage and break times to prevent user cognitive fatigue reduces mental workload in VR application usage. Cognitive load is often defined in terms of the relationship between mental capacity and the experienced task demand (Thorp et al., 2024). Makransky et al. (2019) suggests that the heightened sensory experiences in VR might distract from cognitive processes essential for learning. Users’ task performance can be affected by virtual reality environments. Optimizing mental workload in VR designs is critical for several reasons. In immersive environments, managing mental workload can significantly improve user experience and operational efficiency. For example, studies have shown that optimized mental workload in VR can prevent symptoms related to simulator sickness and visual fatigue, enhancing user performance and comfort (Souchet et al., 2022). Similarly, other research has indicated that appropriate mental workload levels can enhance learning and training effectiveness in VR, making experiences both more effective and enjoyable (Tinco-Tupac et al., 2024).
Furthermore, the design of virtual control elements and interfaces must consider the mental workload to maintain a balance between user engagement and cognitive overload, which can influence the overall effectiveness of VR applications, especially in complex tasks such as learning and training scenarios (Hinricher et al., 2023). Thus, the importance of optimizing mental workload is not only fundamental to enhancing user interaction but also crucial in reducing negative outcomes like increased stress and reduced performance, ensuring that VR systems are both effective and safe for long-term use.
4.1.6 Immersion and presence
Ensuring smooth graphics and visual quality, designing natural and well-placed sounds and effects that immerse the user in the virtual environment, support a sense of realism and, where appropriate, establish a relationship with the physical world.
To improve the use of VR applications, immersion and presence are key criteria that designers and researchers are working on. Yan (2023) discusses the integration of VR technology in interior design to enhance user experience through improved immersion and presence. Technological advancements in computer graphics, 3D space modeling, rendering, and binocular stereo vision collectively contribute to a realistic and engaging virtual design space. Also system supports multi-user real-time tracking and virtual scene rendering, ensures that users can explore and interact with the virtual environment simultaneously, enhancing the sense of presence and engagement. Reddy et al. (2016) emphasizes that the synchronization of audio and visual scenes in VR is vital for optimizing immersion and presence. By creating a dynamic system that adapts to user movements and provides realistic sensory experiences, the study demonstrates how integrating multiple sensory modalities can significantly enhance the effectiveness and engagement of VR applications.
4.1.7 Accessibility and adaptability
Designing for different abilities, needs and skill levels is a fundemantal aspect of virtual reality application development.
Accessibility is recognised as a key ergonomic criterion, with ISO (2011) defining it is as the usability of products for people with the widest range of capabilities. Oculus Developer (2024) highlights the importance of meeting the needs of people with different abilities and emphasises that elements such as narrative and presentation, rules and objectives, character interactions, significantly improve the accessibility of virtual reality applications.
4.1.8 Performance and response time
Optimising design to minimise delay and slutteer based on system performance is crtical in virtual environments. Lippi et al. (2010) investigated the effects of delay on user performance, highlighting the importance of optimising system response times and the impact of delay between sensors and visualisation on performance in a virtual environment catching task. IEEE (2021) recomended frame rate in VR content must be synchronized to the refresh rate of VR HMD, and minimum frame rate is recommended at least 30 fps of images, 60 fps of graphics, and at least 90 fps of interactive content.
4.1.9 Social interaction
Creating narratives, flows and content that allow individuals to interact or not interact with each other and the environment according to their needs and goals is essential. Immersive VR technology offers a powerful tool for simulating social scenarios. Slater and Sanchez-Vives, (2016) provide an in-depth analysis of social and cultural experiences in the context of VR. In their study, Researchers also emphasise the dimension of social interaction with artificial agents such as robots, avatars and voice agents (Heyselaar et al., 2023).Syukur et al. (2024) highlight the value proposition of the metaverse for şocial interaction. Metaverse serves a suitable medium to bring together individuals with similar interests, even if they are geographically dispersed, through the support of immersive experinece sessations.
4.1.10 Trust
Ensuring user trust through effective orientation, robust privacy protection, transparency, security, open communication and appropriate permissions is critical to the adoption and wideespread use of technology. Trust is a key factor influencing the pace at which users accept technological advancements. Pal et al. (2023) highlights the importance of addressing security, privacy, and trust concerns within the Metaverse. Di Pietro and Cresci (2021) emphasise security and privacy in the Metaverse within the term of singularity in a user-centric approach.
4.1.11 Inclusion and ethics
In the context of VR design, inclusion and ethics means ensuring that cultural sentivities are respected and that issues such as bullying, harassment and other harmful behaviours are addressed responsibly. ISO (2010), which focuses on social responsibility emphasises that respect for society and environment is a critical success. Carneiro et al. (2023) and Muntean et al. (2019) discuses and explores ethical priciples and concerns specific to VR. Vanacker and Heider (2012) highlight the importance of ethical considerations in VR communities. MacArthur et al. (2024) emphasises that VR designers and developers need to consider the impact of social power dynamics and inequalities on user experiences.
4.2 Result 2. Expert prioritization of ergonomic criteria
Overall weighted results and the prioritised criteria obtained from tha matrix are shown in Table 5 and Figure 1.
Table 5. Overall prioritisation.
Figure 1. Overall prioritisation.
Results reflect that according to designers and developers, visual comfort, physical balance, system performance and immersion criteria have more impact on the VR projects they design than the others and they give more importance to these criteria in their project compared to the other criteria. The combined value of these criteria is equal to the sum of the other seven criteria. These four criteria occupy half of the practitioners attention. Interface and interaction design and neck and head ergonomics criteria have values close to what they would get if all criteria were evaluated equally. Mental workload, ethics and trust criteria are slightly less important than the expected average, while accessibility and social interaction criteria have significantly lower impact on the projects compared to the others.
In the study, participants were asked if they conducted design research at the beginning of the design process and if they conducted user testing during the design and development process. Five participants reported that they conducted design research, while seven participants reported that they did not. All participants said that they did user testing. The study analysed whether there was a difference between the data from those who conducted user research at the beginning of the project and those who did not. Comparision of the relative importance assigned to the criteria by participants with and without user research are shown in Table 6 and Figure 2. The responses were evaluated and two matrices were created, as shown in Supplementary Appendix Tables 8, 9.
Table 6. Comparision of the relative importance assigned to the criteria by participants with and without user research.
Figure 2. Comparison of the relative importance values of, conducted user research (UR+), not conducted research (UR-) and overall by criterion.
When comparing the importance levels of the critreria between designers who conduct user research and those who do not, it is observed that there is no significant change in the overall prioritisation for six criteria. Those criteria are; Visual comfort and Clarity, Neck and Head ergonomics, Inclusion and Ethics, Trust, Accessibility and Adaptibility, Social Interaction.
Practitioners who have the user research phase in their design process evaluated the impact of Physical balance and movement optimization and Interface and interaction design criteria lower than the general importance values, while those who did not evaluated these criteria close to the general values.
While the Performance and response time criterion was evaluated higher than the general importance value by those who conducted user research, those who did not conducted user research again rated this criterion as close to the general values.
Those who conducted design research considered the impact of the Immersion and presence criterion to be much higher than the overall relative importance, whereas those who did not conducted design research considered the impact of the Immersion criterion to be much lower. For Mental workload, the evaluations are the opposite. The impact is much lower for practitioners who do user research and much higher for those who do not.
4.3 Result 3. Analysis of marketing value propositions
In the third part of the study, the matching of ergonomics criteria with the marketing value propositions collected from the websites of twenty non-gaming vr companies was analysed. The criteria priorities obtained from this analysis were compared with the criteria priorities obtained by participant evaluation. The distribution of sixty discourses matched with ergonomics criteria and comparision with expected distribution according to general participant prioritisation are given in Table 7.
Table 7. Distubution of marketing discourses and comparision with expected distribution according to general participant prioritisation.
As a result of the analysis, the most important differences are that the neck and head ergonomics criterion was much lower than expected and the immersion and presence and accessibility criteria were higher than expected.
4.4 Result 4. Comparison between expert and market priorities
Due to the p = 0.086 value obtained from the chi-square test, it can be said that the distribution is marginally consistent with the expected distribution.
In addition, according to the expected values determined by the prioritisation determined by the participants who conducted user research, p = 0.168. When compared with the p value of general prioritisation, it can be said that the observed distribution is more compatible with the expected distribution obtained from prioritisation of participants who conduct user research.
As a result, user ergonomic needs prioritisation of the product development and design team for VR products, which is the subject of this study, and ergonomic criteria prominent in marketing discourses are corelated. User research conducted during the product development and design process strengthens this correlation.
5 Discussion
In product design and development processes, it is important to evaluate the holistic analysis of qualitative phenomena in a multi-dimensional and multi-participant method. This study provides a framework which can be seen as a scientifically proven agile approach, can be used by teams under time-to-market pressure during the development process. This approach helps to understand a team’s decision-making position within a set of criteria and to analyse the general trend of the market. Comparing the team’s mental model, analysed within the framework of the criteria and shaped by user research, past experience and different expertise, with the market trend helps to find points of differentiation in the product development and innovation process.
The Analytic Network Process (ANP) can be an effective method for relating such dimensions and interdependencies. It can be used to support experts in collaboratively exploring dimensions, relationships, networks, and priorities. Comparative analyses can be conducted by holding a variable such as software, hardware or content constant, or interdependencies between variables can be analysed. These approaches have the potential to make significant contribution to creating a more holistic and human-centered perspective for future technological developments. In this context, conducting comprehensive ergonomic analyses will be essential for designing user-centered AI-assisted UIs, ensuring their effectiveness in evolving technologies.
The results of the research are expected to provide researchers with valuable insights into how to prioritise ergonomic criteria in virtual reality design and development processes.
This study will provide a basis for investigating how the importance of ergonomic criteria in projects will change with the changing technologies over time and the widespread use of VR by people, and will initiate a discussion on what ergonomic criteria should be evaluated in VR designs and the evaluation of its conceptual framework. Different researchers may add new dimensions or make new suggestions by combining existing ones within a different conceptual framework. This study also aims to provide a reference point and checklist of ergonomic criteria from different dimensions that should be checked in every VR design study.
The next step would be to investigate in more depth the reasons behind the prioritisation of the criteria. Understanding the reasons behind these prioritisations in different design processes, such as whether or not user research is conducted, can provide significant benefits to relevant stakeholders. For example, as seen in the study, immersion and accessibility are prioritised in marketing discourses. Immersion and accessibility are the most popular concepts in the field of virtual reality. This may indicate that in the product development process, teams need to consider how they assess the balance and impact of user needs and customer stakeholder expectations on value proposition creation and marketing discourse. In the context of rapidly advancing technological fields, it is essential to develop new approaches and contribute to existing methodologies to identify critical dimensions, networks and interdependencies that need to be focussed on in product development and innovation studies.
Data availability statement
The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.
Author contributions
AT: Conceptualization, Data curation, Investigation, Methodology, Validation, Writing – original draft, Writing – review and editing.
Funding
The author(s) declare that no financial support was received for the research and/or publication of this article.
Conflict of interest
The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Generative AI statement
The authors declare that no Generative AI was used in the creation of this manuscript.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frvir.2025.1549963/full#supplementary-material
References
Asma, B. (2006). The analytic hierarchy process: a methodology for win-win management. J. King Abdulaziz University-Economics Adm. 20, 3–16. doi:10.4197/eco.20-1.1
Belton, V., and Stewart, T. (2002). Multiple criteria decision analysis: an integrated approach. Boston: Kluwer Academic Publishers, Boston. doi:10.1007/978-1-4615-1495-4
Bevan, N., Carter, J., and Harker, S. (2015). “ISO 9241-11 revised: what have we learnt about usability since 1998?,” in Human-computer interaction: design and evaluation. Editor M. Kurosu (Cham: Springer), 9169, 143–151. doi:10.1007/978-3-319-20901-2_13
Beynon, M. (2002). An investigation of the role of scale values in the DS/AHP method of multi-criteria decision making. J. Multi-Criteria Decis. Analysis 11 (4-5), 327–343. doi:10.1002/mcda.339
Bourdin, P., Martini, M., and Sanchez-Vives, M. V. (2019). Altered visual feedback from an embodied avatar unconsciously influences movement amplitude and muscle activity. Sci. Rep. 9, 19747. doi:10.1038/s41598-019-56034-5
Cao, Y. (2009). “Research on interaction design of virtual reality,” in 2009 IEEE 10th international conference on computer-aided industrial design and conceptual design. Wenzhou, China (IEEE), 1340–1344. doi:10.1109/CAIDCD.2009.5375379
Caple, D. (2008). Emerging challenges to the ergonomics domain. Ergonomics 51 (1), 49–54. doi:10.1080/00140130701800985
Carlos, A. B. C., and Vansnick, J.-C. (2008). A critical analysis of the eigenvalue method used to derive priorities in AHP. Eur. J. Operational Res. 187 (3), 1422–1428. doi:10.1016/j.ejor.2006.09.022
Carneiro, J. A. S., Cardoso, A., Santos, F. P., Milagre, S. T., Lamounier, E. A., and Oliveira, L. C. (2023). “Ethical aspects in a virtual reality environment,” in 2023 18th Iberian conference on Information Systems and Technologies (CISTI), Portugal (Aveiro), 1–4. doi:10.23919/CISTI58278.2023.10211445
Chen, Y., and Wu, Z. (2023). A review on ergonomics evaluations of virtual reality. Int. J. Human-Computer Interact. 74, 831–841. doi:10.3233/wor-205232
De Felice, F. (2012). Research and applications of AHP/ANP and MCDA for decision making in manufacturing. Int. J. Prod. Res. 50 (17), 4735–4737. doi:10.1080/00207543.2012.657963
Di Pietro, R., and Cresci, S. (2021). “Metaverse: security and privacy issues,” in 2021 third IEEE international conference on trust, privacy and security in intelligent systems and applications (TPS-ISA). Atlanta, GA, United States (IEEE), 281–288. doi:10.1109/TPSISA52974.2021.00032
Easa, S. (2021). Human factor considerations in virtual reality: adequate or inadequate? Ergonomics Int. J. 5, 267. doi:10.23880/eoij-16000267
Emrouznejad, A., and Marra, M. (2017). The state of the art development of AHP (1979-2017): a literature review with a social network analysis. Int. J. Prod. Res. 55 (22), 6653–6675. doi:10.1080/00207543.2017.1334976
Germanakos, P., Sotirakou, C., Mourlas, C. I., Richir, S., and Boomgaarden, H. (2022). Editorial: immersive reality and personalized user experiences. Front. Virtual Real. 3, 1015155. doi:10.3389/frvir.2022.1015155
Halbig, A., and Latoschik, M. E. (2021). A systematic review of physiological measurements, factors, methods, and applications in virtual reality. Front. Virtual Real. 2, 694567. doi:10.3389/frvir.2021.694567
Hameed, A., Möller, S., and Perkis, A. (2024). A holistic quality taxonomy for virtual reality experiences. Front. Virtual Real. 5, 1434016. doi:10.3389/frvir.2024.1434016
Heyselaar, E., Caruana, N., Shin, M., Schilbach, L., and Cross, E. S. (2023). Editorial: do we really interact with artificial agents as if they are human? Front. Virtual Real. 4, 1201385. doi:10.3389/frvir.2023.1201385
Hinricher, N., Schröer, C., and Backhaus, C. (2023). Design of control elements in virtual reality—investigation of factors influencing operating efficiency, user experience, presence, and workload. Appl. Sci. 13 (15), 8668. doi:10.3390/app13158668
IEEE (2021). 3079-2020-IEEE standard for head-mounted display (HMD)-Based virtual reality (VR) sickness reduction technology. IEEE, 1–74. doi:10.1109/IEEESTD.2021.9416950
International Ergonomics Association (IEA). (2021). Principles and Guidelines for Human Factors/Ergonomics (HF/E) Design and Management of Work Systems. Available online at: https://iea.cc/wp-content/uploads/2021/06/Principles-and-Guidelines_June2021.pdf
ISO (2024). “Ergonomics of human-system interaction,” in Part 820: ergonomic guidance on interactions in immersive environments, including augmented reality and virtual reality.
Kazemi, R., and Lee, S. C. (2023). Human factors/ergonomics (HFE) evaluation in the virtual reality environment: a systematic review. Int. J. Human-Computer Interact. 40, 4533–4549. doi:10.1080/10447318.2023.2227835
Lippi, V., Avizzano, C. A., Mottet, D., and Ruffaldi, E. (2010). “Effect of delay on dynamic targets tracking performance and behavior in virtual environment,” in 19th international symposium in robot and human interactive communication. Italy: Viareggio, 446–451. doi:10.1109/ROMAN.2010.5598607
MacArthur, C., Kukshinov, E., Harley, D., Pawar, T., Modi, N., and Nacke, L. E. (2024). Experiential disparities in social VR: uncovering power dynamics and inequality. Front. Virtual Real. 5, 1351794. doi:10.3389/frvir.2024.1351794
Makransky, G., Terkildsen, T. S., and Mayer, R. E. (2019). Adding immersive virtual reality to a science lab simulation causes more presence but less learning. Learn. Instr. 60, 225–236. doi:10.1016/j.learninstruc.2017.12.007
Maneuvrier, A., Decker Leslie, M., Ceyte, H., Fleury, P., and Renaud, P. (2020). Presence promotes performance on a virtual spatial cognition task: impact of human factors on virtual reality assessment. Front. Virtual Real. 1, 571713. doi:10.3389/frvir.2020.571713
Miles, M. B., and Huberman, A. M. (1994). Qualitative data analysis. 2nd ed. Thousand Oaks, CA: Sage.
Muntean, R., Hennessy, K., Denes, A., and Phuttitarn, L. (2019). “On research ethics and representation in virtual reality,” in 2019 IEEE conference on Virtual Reality and 3D User Interfaces (VR), Osaka, Japan (IEEE), 1493–1496. doi:10.1109/VR.2019.8798160
Oberdörfer, S., Banakou, D., and Bhargava, A. (2023). Editorial: the light and dark sides of virtual reality. Front. Virtual Real. 4, 1200156. doi:10.3389/frvir.2023.1200156
Oculus Developers (2024). Comfort and usability. Menlo Park, CA, United States: Oculus. Available online at: https://developer.oculus.com/resources/locomotion-comfort-usability/ (Accessed July 11, 2024).
Ordoobadi, S. M. (2012). Application of ANP methodology in evaluation of advanced technologies. J. Manuf. Technol. Manag. 23 (2), 229–252. doi:10.1108/17410381211202214
Pal, S., Vangala, A., Jadidi, Z., Hou, Z., and Das, A. K. (2023). “Security, privacy and trust for the metaverse of things,” in 2023 IEEE international conference on metaverse computing, networking and applications (MetaCom). Kyoto, Japan (IEEE), 146–150. doi:10.1109/MetaCom57706.2023.00039
Ponce, P., Anthony, B., Bradley, R., Maldonado-Romo, J., Méndez, J. I., Montesinos, L., et al. (2024). Developing a virtual reality and AI-based framework for advanced digital manufacturing and nearshoring opportunities in Mexico. Sci. Rep. 14, 11214. doi:10.1038/s41598-024-61514-4
Punch, F. K. (2005). Introduction to social research: quantitative and qualitative approaches. 2nd ed. London: Sage.
Reddy, S., Rao, G. S. S. S., and Hegde, R. M. (2016). “On the development of a dynamic virtual reality system using audio and visual scenes,” 2016 twenty second national conference on communication (NCC), Guwahati, India, 1–6. doi:10.1109/NCC.2016.7561204
Richter, H. O., Crenshaw, A. G., Domkin, D., and Elcadi, G. (2016). Near-infrared spectroscopy as a useful research tool to measure prefrontal cortex activity during visually demanding near work. IIE Trans. Occup. Ergonomics Hum. Factors 4 (2–3), 164–174. doi:10.1080/21577323.2015.1067258
Saaty, T. L. (2005). “Theory and applications of the analytic network process: decision making with benefits,” in Opportunities, costs, and risks. Pittsburgh: RWS Publications.
Saaty, T. L., and Ozdemir, M. S. (2003). Why the magic number seven plus or minus two. Math. Comput. Model. 38 (3–4), 233–244. doi:10.1016/S0895-7177(03)90083-5
Sauro, J., and Lewis, j. (2016). Quantifying the user experience: practical statistics for user research. Elsevier.
Segal, R., Bhatia, M., and Drapeau, M. (2011). Therapists’ perception of benefits and costs of using virtual reality treatments. Cyberpsychology, Behav. Soc. Netw. 14 (1-2), 29–34. doi:10.1089/cyber.2009.0398
Slater, M., and Sanchez-Vives, M. V. (2016). Enhancing our lives with immersive virtual reality. Front. Robotics AI 3, 74. doi:10.3389/frobt.2016.00074
Souchet, A. D., Lourdeaux, D., Pagani, A., and Rebenitsch, L. (2022). A narrative review of immersive virtual reality’s ergonomics and risks at the workplace: cybersickness, visual fatigue, muscular fatigue, acute stress, and mental overload. Virtual Real. 27, 19–50. doi:10.1007/s10055-022-00672-0
Stanley, C., Temkin, B., Dev, P., Heinrichs, W. L., Srivastava, S., and Waldron, K. (2001). “Networked stereoscopic virtual environment system,” in Proceedings 14th IEEE Symposium on Computer-Based Medical Systems (CBMS 2001), Bethesda, MD, United States (IEEE), 400–406. doi:10.1109/CBMS.2001.941753
Stanney, K. (1995). “Realizing the full potential of virtual reality: human factors issues that could stand in the way,” in Proceedings virtual reality annual international symposium 95. 11–15 March 1995, Research Triangle Park, NC, United States (IEEE), 28–34. doi:10.1109/VRAIS.1995.512476
Sun, J. (2023). “Application status and progress of analytic hierarchy process in design research: bibliometric analysis based on citespace,” in Usability and user experience. Editors T. Ahram, and C. Falcão (United States: AHFE International). doi:10.54941/ahfe1003223
Syukur, A., Andono, P. N., and Syarif, A. M. (2024). Gamelan land: a multiplayer virtual reality game based on a social presence approach. J. Metaverse 4 (1), 1–10. doi:10.57019/jmv.1334412
Tanrıverdi, V., and Jacop, R. (2001). “VRST 01,” in Proceedings of the ACM symposium on Virtual reality software and technology, 175–182. doi:10.1145/505008.505042
Thorp, S. O., Rimol, L. M., Lervik, S., Evensmoen, H. R., and Grassini, S. (2024). Comparative analysis of spatial ability in immersive and non-immersive virtual reality: the role of sense of presence, simulation sickness and cognitive load. Front. Virtual Real. 5, 1343872. doi:10.3389/frvir.2024.1343872
Tinco-Tupac, S. T., Torres-Gonzales, S. B., and Maraza-Quispe, B. (2024). Analysis of the mental workload generated by learning experiences through augmented reality and virtual reality in students of regular basic education. Rev. De. Gestão Soc. E Ambient. 18 (5), e05131. doi:10.24857/rgsa.v18n5-065
Vanacker, B., and Heider, D. (2012). Ethical harm in virtual communities. Convergence 18 (1), 71–84. doi:10.1177/1354856511419916
Von Solms, S. (2011). Validity of the AHP/ANP: comparing apples and oranges. Int. J. Anal. Hierarchy Process 3 (1), 1–26. doi:10.13033/ijahp.v3i1.60
Yan, P. (2023). “Application of VR virtual reality technology in 3D image interior design system,” in 2023 asia-europe conference on electronics, data processing and informatics (ACEDPI). 17–19 April 2023, Prague, Czech Republic (IEEE), 354–358. doi:10.1109/ACEDPI58926.2023.00075
Keywords: virtual reality (VR) design, VR product development, VR product innovation, user ergonomics in VR projects, human factors in VR projects, multi-criteria decision making, AHP, design research
Citation: Tokatli A (2025) A multi-criteria framework for the holistic ergonomic design evaluation of VR products. Front. Virtual Real. 6:1549963. doi: 10.3389/frvir.2025.1549963
Received: 22 December 2024; Accepted: 30 April 2025;
Published: 22 May 2025.
Edited by:
Dioselin Gonzalez, Independent Researcher, Seattle, WA, United StatesReviewed by:
Steven Christian, University of Nevada, Reno, United StatesIsabella Sierra, Universidade Federal do Paraná, Brazil
Copyright © 2025 Tokatli. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Akgun Tokatli, YXRva2F0bGlAbWVkaXBvbC5lZHUudHI=