Measuring User Satisfaction for the Natural Hazards Engineering Research Infrastructure Consortium

Introduction

The Natural Hazards Engineering Research Infrastructure (NHERI) is a National Science Foundation (NSF) sponsored consortium consisting of physical and simulation infrastructure to support multidisciplinary research broadly focused on natural hazard impacts and resilience. As depicted in Figure 1, NHERI consists of (a) a network coordination office (NCO) that offers user support, leads education and outreach activities, develops strategic national and international partnerships, and brings stakeholders together to translate NHERI research into practice and articulates grand challenges for natural hazard engineering research; (b) a community cyberinfrastructure that offers web-based software, and reconnaissance repositories and visualizations open to all NHERI users; (c) a simulation center focused on developing and deploying next-generation computational modeling and simulation tools for infrastructure and regional scale natural hazard simulations; (d) a facility offering diverse state-of-the-art reconnaissance equipment for natural hazard-based measurements; (e) a center focused on convergence across disciplinary-based research communities, specifically bringing together fieldwork-based disciplinary networks; and (f) seven experimental testing facilities that include earthquake, hurricane, tsunami, and other dynamic testing capabilities. The NHERI Council is composed of the principle investigators from each of the eleven NHERI Awards (NCO and ten infrastructures on right side of Figure 1). External to and in support of NHERI, the network independent advisory committee (NIAC) consists of representatives from the broad scientific and engineering communities served by NHERI, and the User Forum serves as the community voice in NHERI governance. The User Forum provides independent advice on community user satisfaction, priorities, and needs relating to the use and capabilities of NHERI.

FIGURE 1

Figure 1. Components of NHERI.

NHERI is a nation-wide consortium with thousands of users. It is critical to measure success of NHERI to promote continued funding and operation. Furthermore, it is essential to measure user satisfaction to promote constant procedural improvements, identify and reduce biases, and promote widespread use of NHERI throughout the natural hazards engineering community. There are many compounding benefits to measuring user satisfaction. First, measuring user satisfaction can inform important changes that may be needed for user retention. User retention ultimately can lead to the sustainability of NHERI, and has its own compounding effects. The continued funding of NHERI with diverse and satisfied users will lead to more use of NHERI, thereby resulting in more scientific advancements in the natural hazards research space. These scientific advances hold tremendous possibility, including the high potential to improve quality of life through more strategic investments and reduced consequences from natural hazards. Such benefits would be felt across the United States, and are often scalable for use around the world.

In measuring user satisfaction, it is always best to be objective rather than subjective. There are a number of metrics that offer a meaningful way to measure user satisfaction across any large consortium. In general, metrics for user satisfaction include user retention, number of users, ways in which users engage or use NHERI, and user-rated experience. Metrics can be measured using different methods, including direct user feedback through interviews or surveys, systematic assessment of the consortium from an external team of evaluators, and through user-data collected in real-time during use (e.g., number of users, tracking how a single user uses NHERI in multiple ways). The User Forum's evaluation follows the “utilization-focused evaluation” approach as developed by Michael Quinn Patton (Patton, 2008). This approach was derived with the intent to focus on the intended use of evaluation data by the intended users and aims to collect data that are meaningful in supporting decision-making processes. The User Forum evaluation also falls into the Rajeshkumar et al. (2013) user experience taxonomy classification of “user-oriented” survey methods collecting both quantitative and qualitative data.

NHERI is one of many major multi-user research facilities funded by the NSF. NHERI is one of two sites in Engineering, where the other is the National Nanotechnology Coordinated Infrastructure (NNCI). These major multi-user research facilities also exist in the geosciences, mathematical and physical sciences, and as research and development centers. NSF-sponsored major multi-user research facilities have a long-supported history of measuring user satisfaction, which proves critically important for the NSF's responsible use of U.S. taxpayer dollars. Despite the long-standing history, there is limited information documenting user satisfaction measurement processes, approaches, challenges, and other decisions, thus providing the most significant impetus for this paper. Given the history of NSF consortiums, NHERI leveraged user satisfaction processes executed for other consortiums. This started with the election of an external committee of users, the User Forum, and continued with member engagement, metric adoption and measurement approaches. The details of each of these are detailed throughout this paper.

Member Election

As the community voice within the governance of NHERI, up until the time of this submission, the User Forum has been composed of nine representatives who are nominated and elected by the NHERI community. User Forum elections are generally held annually to fill vacant roles as needed. The maximum and minimum number of committee members is not fixed, and can fluctuate based on committee needs. Candidates for vacant roles in the User Forum can be nominated by anyone within the NHERI community, and this opportunity is broadcasted to the NHERI community through NHERI email communications. General User Forum elections are held using the DesignSafe-CI website (https://www.designsafe-ci.org/) to ensure NHERI users have broad access to the elections. Members are elected to the User Forum for a specified term of 1–2 years, with the opportunity to be re-elected.

Members on the User Forum represent the scientific and engineering communities who use NHERI's resources and services for research and/or educational purposes, but who are not directly affiliated with NHERI awardee institutions. User Forum membership spans academia and industry, the full breadth of civil engineering disciplines, the social sciences, and widespread hazard expertise including earthquakes, windstorms, and water events. This broad representation within the User Forum ensures that the needs and concerns of the diverse community of NHERI users are understood by the User Forum and can be accurately expressed to NHERI governance.

Roles and Responsibilities

The User Forum provides the NHERI Council with independent advice on community user satisfaction, priorities, and needs relating to the use and capabilities of NHERI. The User Forum, therefore, needs to have a good understanding at a strategic level of the work of the NHERI consortium. This includes an awareness of the vision, values and mission, strategic and operational plans, and evaluation relating to NHERI's work. As depicted in Figure 2, the User Forum has historically been comprised of nine members representing the User Forum across NHERI activities, including three User Forum officers (chair, vice chair, and secretary) working directly with the Network Coordination Office (NCO) and participating on monthly NCO meetings, two User Forum representatives working with the Education and Community Outreach (ECO) team, two User Forum representatives working with the Facilities Scheduling committee, and one User Forum representative working with the Technology Transfer committee. Likewise, two members of the NCO participate in User Forum monthly meetings to maintain direct communication between the User Forum and NHERI governance. Additionally, one member leads the User Satisfaction Survey subcommittee, and is joined by the chair, vice chair, and one NCO representative to spearhead the survey work.

FIGURE 2

Figure 2. User forum member roles and responsibilities.

Members of the User Forum participate in annual elections of User Forum officers, an annual in-person meeting, and monthly teleconference meetings. As shown in Figure 2, officer roles of the User Forum include Chair, Vice Chair, and Secretary. The Chair is responsible for leading User Forum meetings, assisting in the development of assessment tools of user satisfaction, and communicating committee updates and user satisfaction to the NCO. The Vice Chair serves as support for the Chair, assuming Chair responsibilities when necessary, and also aids in disseminating and encouraging NHERI users to engage in user satisfaction assessment surveys. The Secretary organizes monthly teleconference meetings, generates meeting agendas and minutes, and is responsible for updates to the User Forum website¹. All officers work directly with the NCO, and all User Forum members are expected to serve as liaisons for the greater NHERI community, providing the NHERI Council with independent advice on user satisfaction, priorities, and needs relating to the use and capabilities of NHERI.

The User Forum holds an annual in-person meeting typically during the ECO's Summer Institute with conference call-in capabilities for members who cannot attend in person. During this day-long meeting, the first part of the day is typically dedicated to discussing user satisfaction survey reports, describing challenges for conducting user satisfaction, identifying areas of improvement to the metrics used to evaluate user satisfaction, and developing strategies on how to best represent NHERI users' feedback and serve the NHERI community. The second part of the day is reserved for meeting with managers from Experimental Facilities, the NCO, the ECO, and the Network Independent Advisory Committee. The focus of those meetings is typically the communication of user satisfaction and feedback to the different entities. Figure 3 provides a timeline of activities executed by the User Forum since its initiation. As shown, the User Forum initiated in February 2017. To initially engage with the NHERI user community to share the founding and purpose of the User Forum, presentations were given at two major conferences.

FIGURE 3

Figure 3. User forum timeline of activities in years 1 through 3.

Monthly meetings are scheduled using an online scheduler, such as Doodle, that allows selection of several time slots and dates. Monthly meetings are typically scheduled at least 1 month in advance. In the event that more than one User Forum member is unable to attend a single date/time, the choice with at least two Officers and majority of User Forum members is chosen. The meetings are organized via teleconference software, such as Zoom, that allows screen sharing and computer and phone call-in options. At least 1 week in advance, the agenda, the previous meeting's minutes, and call-in information are emailed to User Forum members and representatives of the NCO. Meetings typically last about 1 h and include report-outs and discussion from User Forum members serving on their respective subcommittees as described below. Additional time is utilized to discuss member and officer elections, in-person meeting planning, and for any other specific items, such as the development of this paper. Meeting minutes are emailed to all User Forum members and NCO representatives typically within 1 week of the meeting. Once approved, the meeting minutes are publicly available on the User Forum website.

The most significant responsibility of the User Forum is to provide the Council with community advice. The User Forum provides this advice through administering an annual user satisfaction survey. This process includes survey development, data collection, data analysis, and the finalization of results in a public report. As shown in Figure 3, three annual surveys have been conducted and published to date. The duration of the open survey time has expended each year as new challenges and approaches are investigated and taken. The details on these challenges and the user satisfaction measurement process are described in the next section.

A User Forum member (typically one of the User Forum officers) attends the biweekly NCO conference call. During those meetings, a time slot is reserved for the User Forum to communicate issues and updates of the User Forum without delay to the NCO. This opportunity is also utilized by the NCO to direct requests, questions, or points of discussion to the User Forum. The presence of the User Forum during the NCO calls ensures a direct and sustained communication line between the NCO and the User Forum as well as the integration of the User Forum into the Governance of NHERI.

Two User Forum members serve as representatives on the ECO committee. The ECO committee includes representatives from all of the NHERI Awardees, and plans and executes the NHERI Research Experiences for Undergraduates (REU) program executed at across NHERI facilities, and the NHERI Summer Institute in San Antonio, TX (a multi-day workshop designed for orienting new users to NHERI). The ECO committee also collects and disseminates research and education in progress to the larger natural hazards community, and provides connections between research and education to K-12, community college, and practicing communities. The ECO committee and User Forum representatives meet once per month via teleconference, and once per year in-person, typically during the NHERI Summer Institute. The User Forum specifically provides the ECO committee with advice and recommendations on planning the REU program, Summer Institute, and other matters of concern to the ECO committee, e.g., Research to Practice webinars.

Similarly, two User Forum members serve as representatives on the Facilities Scheduling Committee. This committee is comprised of representatives from each of the Experimental Facilities and the Facility Scheduling and Operations Coordinator and is charged with developing and implementing protocol to standardize the scheduling of NHERI projects. The User Forum members on this committee report on challenges, lessons learned, and feedback from users who have scheduled NHERI projects using the centralized management protocol. This information is used to improve scheduling protocol and improve the online scheduling system.

One User Forum member serves on the Technology Transfer Committee, which consists of practitioners, decision makers, and researchers. This Technology Transfer Committee is focused on strengthening the tie between NHERI researchers and the implementers of NHERI-developed technology. The User Forum member represents both research users and practitioners within the NHERI network.

Measuring User Satisfaction

The focus of the annual user satisfaction survey is to provide evaluation data to inform decision-making processes among project leads (Patton, 2008). To spearhead this major task, a user satisfaction subcommittee was formed within the User Forum. The user satisfaction subcommittee consists of the chair, vice chair, User Forum awardee institution, and an NCO representative. The remainder of this section explains the User Forum's approach to measuring user satisfaction, executing the annual user survey, reporting survey findings, connecting with facilities, assessing facility surveys, the associated challenges experienced to date, how the process has evolved over time, and a synthesis of the outcomes and relevance of the survey findings. User satisfaction surveys have been completed so far in 2017, 2018, and 2019. The public reports are available at https://www.designsafe-ci.org/community/user-forum/.

Defining User Experience for NHERI

The NHERI User Forum was unable to find documentation on the user satisfaction evaluation processes for the other NSF consortiums. Generally, such documentation in the literature is rare (Vermeeren et al., 2010), however there are published works on measuring user satisfaction. The literature more broadly refers to user satisfaction as user experience, and has experienced a recent increase in said measurement with the growth in human-centered design (Rajeshkumar et al., 2013). For example, Vermeeren et al. (2010) collected information on 96 different user experience evaluation methods being executed in academia and industry. These methods varied by quantitative vs. qualitative measurements, generalizability vs. application-specific, expert-based vs. broad-user, measurement location (lab, field, online), product development/use phase (beginning, during, after experience), amongst others. The authors concluded that there was widespread interest in measuring user experience, however, there were also widespread systematic development needs, including on (a) methods for early phases of development, (b) validated user experience metrics, (c) methods for social and collaborative evaluation, (d) establishing practicability and scientific quality, (e) multi-method approaches, and (f) generating a deeper understanding of user experience. The interested reader is referred to Vermeeren et al. (2010) to learn more about the various methods they observed through exploration of their online database.

ISO 9241-110 defines user experience as “a person's perceptions and responses that result from the use and/or anticipated use of a product, system or service” (ISO DIS 9241-210, 2010, clause 2.15). This characterization importantly establishes user experience as being subjective (Law et al., 2009). For NHERI, user subjectivity could stem from disciplinary background, how NHERI was used (e.g., workshops, proposals, projects), which part of NHERI was used, and user experience levels in academia, the proposal writing process, with the NSF, and with the NHERI facilities. Furthermore, the characterization of user experience also establishes it as something that occurs through time and not at a single point (Karapanos et al., 2009). For NHERI users, user experience can occur with regular NHERI email communications, intermittent use of the cyber infrastructure, during the proposals writing process, applying to and attending workshops, and during and after funded projects. Thus, clearly defining a NHERI user became an important challenge.

Defining a NHERI User and Making a Connection

Measuring NHERI user satisfaction is a key task for the User Forum, but has also represented a number of challenges. The first question that arose from this task has been who is the NHERI user. The User Forum has defined the NHERI user as any individuals interacting with NHERI facilities and/or NHERI affiliated data. This includes individuals who have reviewed NHERI information and communicated with NHERI facilities for the preparation of proposals, individuals who actively collect data using the NHERI facilities, as well as individuals who utilize NHERI cyberinfrastructure, existing NHERI data sets, or utilize NHERI data repositories for natural hazard related data storage, amongst others. The User Forum is actively reaching out to different user groups through mailing lists as well as personal contacts to engage the users in providing feedback. The main mechanism for the collection of data on user satisfaction is the annual user satisfaction survey. The User Forum is exploring pathways of most efficient data collection and investigates what information is the most useful information for the NHERI governance.

Three user satisfaction surveys have been carried out at the time of this paper submission. Historically, one questionnaire was initially developed by an external entity and distributed to all users. The external entity was also responsible for developing a summary report of the survey results. After that first year, it was determined that the annual user satisfaction survey was better executed in-house. During the second year of the User Forum, User Forum members took control of developing and executing the survey. This was made possible by an NSF supplement award to a User Forum member institution for student support to incorporate User Forum feedback into survey development, and process findings. Once administered by the User Forum, the survey was sent to all registered users of DesignSafe-CI and investigators of NHERI projects. Across the 3 years of data, user responses were somewhat consistent and allowed preliminary conclusions on positive user satisfaction; however, limiting challenges included low response rates and the fact that, depending on the user group, most respondents encountered questions that did not apply to their user group. In subsequent surveys, the User Forum refined questions based on responses in previous surveys, included different tracks for the respective user groups to ask questions more relevant to the user group, reviewed survey information provided by the individual NHERI facilities, and included in-person surveys of individuals selected based on their survey responses.

Annual User Survey and Changes to the Survey Over Time

In accordance with requirements set by the National Science Foundation (NSF), the User Forum conducts an annual user satisfaction survey administered using the online survey software, Qualtrics. Potential participants receive an invitation to participate in the user satisfaction survey along with a link to take the survey late spring or early summer. The structure and content of the survey have changed with each iteration of the survey. The most substantial changes to the survey occurred in 2018, based on feedback from the User Forum committee. The committee opted to expand the 2017 survey by refining the questions to obtain more details. Questions featured mixed response options that yielded both qualitative and quantitative data. In 2019, the User Forum conducted the user satisfaction survey for the third time (see the Appendix for a copy of the 2019 survey). Rather than make changes to the survey instrument as in previous years, in 2019 the User Forum instead made changes to the data collection strategy. Two separate, but nearly identical surveys were sent out to (1) all registered users of DesignSafe-CI, and (2) to NHERI workshop participants and investigators of NHERI projects. This latter list was provided to the User Forum from the NHERI facilities. Changes in the approach facilitated comparative analysis and allowed the User Forum to garner a clearer picture of user satisfaction. It is anticipated that the 2020 survey will include many identical questions to previous years, preserving the longitudinal nature of the work to date, as well as modifying other questions based on temporal updates.

Reporting

Each year, an annual report is prepared once data collection and analysis are complete. This report includes an executive summary of all findings, detailed item-level response overviews, and frequencies of response type. The executive report includes information on key findings, response rates, and has details on data collection such as an overview of the sampling strategy and the dates within which data collection occurred. Detailed item-level responses provide an overview of both quantitative and qualitative responses, with qualitative responses reported in full in the appendices.

Connecting With Facilities and Assessing Facility Surveys

Communication with facilities primarily occurs through online correspondence facilitated by members of the User Forum. For example, in 2019, facilities PIs were contacted before data collection to obtain a list of known facility site users; they were also asked to provide a copy of any site-specific survey instruments. This request was motivated by a desire to assess whether there might be ways to streamline overall data collection regarding user satisfaction. Once all facility surveys were collected, the User Forum conducted a thematic analysis and synthesized findings.

Associated Challenges

As previously noted, there are a variety of challenges with efforts to measure user satisfaction. The first challenge has been addressing low response rates to the annual user satisfaction survey. This challenge has continued over time, and response rates to the annual satisfaction survey have declined each year. In order to address this challenge, the User Forum has explored adding incentives to participation and more actively involving site PIs in the data collection process. The former method is somewhat problematic given limited resources, but the latter approach holds some potential for the 2020 survey administration. The User Forum has also considered supplementing the annual user satisfaction survey with a module of questions to add to existing facility-administered surveys.

As noted above, the User Forum has also experienced and addressed the challenges of defining NHERI site users. Initially, NHERI made use of an extensive list of associated NHERI site users through a contact list developed in collaboration with DesignSafe-CI. This list, with over two thousand unique potential participants, held no guarantee that each potential participant had used or visited a NHERI site during the evaluation period. To address this challenge, in 2019 the User Forum collaborated with NHERI site PIs to develop a separate contact list of potential participants. This list, though much smaller (N = 108), only included potential participants that were known to have used or visited an NHERI site in the last calendar year. This allowed for comparative analysis in satisfaction among potential NHERI site users (the extensive list) and known NHERI site users (the smaller, targeted list).

Given that the user satisfaction survey is distributed online via email, the User Forum has experienced typical challenges associated with this method of distribution. During data collection in the summer of 2019, the User Forum discovered that the mass-distributed emails were being flagged by the utilized email server as “potential spam.” Concerned that this would negatively affect response rates, the User Forum worked with the team that maintains the email server to address this issue. No workarounds were available, so the User Forum opted to end data collection at that time.

The User Forum has carefully considered each of the associated challenges discussed above. In each case, solutions were co-developed and adaptations were made accordingly.

Summary of 2019 User Satisfaction Survey Findings

Due to limited changes in user satisfaction results from year to year, this section provides a focused and comparative overview of results from the 2019 NHERI User Satisfaction Survey. Detailed accounting of the 2019 survey and past survey results, as well as item-level descriptions of responses, are provided in the public reports accessible through the User Forum website.

The 2019 NHERI User Satisfaction survey instrument was nearly identical to the 2018 version, including questions that featured mixed response options yielding both qualitative and quantitative data. Building on feedback regarding the 2018 survey, the committee sent the 2019 survey to two targeted populations. As mentioned above, the first population included known NHERI facility users. The second population included a broader NHERI user list, compiled with assistance from NHERI facility PIs and the DesignSafe-CI support staff.

The first notable difference between known NHERI site users and general NHERI users was where respondents were in the proposal writing process. Not surprisingly, most known NHERI site users indicated they were at some point in the proposal writing process and had prepared at least one proposal that used NHERI facilities and resources. On the other hand, more respondents from the general user list indicated they did not plan to submit or prepare a proposal and reported preparing no proposals. Additional differences between these two user groups were how they rated information about NHERI facilities and resources. On the whole, responses from the general NHERI user list rated information about NHERI facilities and resources more positively than known NHERI site users. As shown in Figure 4, general users more positively rated information as readily accessible and comprehensive. Even so, known NHERI site users also rated information about NHERI facilities and resources positively but more often indicated they disagreed, strongly disagreed, or could not rate questions regarding the accessibility and comprehensiveness of NHERI information. Substantive responses from the general NHERI user list to questions regarding data were more positive than responses from the known NHERI site user list. Data items referred to the process of uploading data, adding metadata, and accessing data.

FIGURE 4

Figure 4. Ratings of information of NHERI facilities and resources by user group. (A) is the known users, and (B) is the general users.

Both known NHERI site users and general NHERI users displayed no major differences across a number of measures of user satisfaction such as quality of experience using NHERI facilities, intended utilization of online NHERI resources, and satisfaction with feedback on written proposals. As shown in Figure 5, intended utilization of NHERI facilities is similar across both user groups. A majority of both the known NHERI site users (70.6%) and general users (61.5%) indicate they intend to both produce and upload original data and utilize available data, whereas 23.5% of known users and 23.1% of general users intend to produce and upload original data. The remaining respondents (5.9% of known users and 15.4% of general users) indicate they intend to use only the available uploaded data.

FIGURE 5

Figure 5. Intended use of online NHERI resources by user group. (A) is the known users, and (B) is the general users.

Similarly, known NHERI site users and general NHERI users report similar experiences seeking assistance from NHERI facilities in the proposal writing process. For example, as shown in Figure 6, of the six respondents who requested assistance in proposal writing, 100% of them received assistance and found the assistance they received helpful. Satisfaction with online support resources and tools and the available training for these resources and tools were similar, and positive, among both groups. Across both user groups, participants indicated they believe information regarding NHERI and NHERI in DesignSafe-CI are useful, and distributed at a useful rate and quantity. Similarly, responses across both user groups indicated participants intend to use the NHERI Science Plan to learn of major research challenges, to reference how their research fits within the Science Plan in their NSF proposal, and to expand their current research scope.

FIGURE 6

Figure 6. Satisfaction with proposal writing feedback by user group. (A) is the known users, and (B) is the general users.

Closing Remarks

To help fill a gap in the existing literature on consortium-based user satisfaction measurement processes, this paper provides the roles, responsibilities, approach to and challenges with measuring user satisfaction as experienced by the User Forum for the National Science Foundation NHERI Consortium. The User Forum serves as the community voice in NHERI governance and provides independent advice on community user satisfaction, priorities, and needs relating to the use and capabilities of NHERI. As the community of voice within the governance of NHERI, the User Forum has been composed of nine representatives who are nominated and elected by the NHERI community, and who work directly with NHERI governance. User Forum membership spans academia and industry, and attempts to represent the full spectrum of NHERI user expertise.

A key task of the User Forum is measuring NHERI user satisfaction; three user satisfaction surveys have been carried out to date. Survey questions and data collection strategy have evolved over time, and the User Forum has prepared three final reports that include an executive summary of all findings, detailed item-level response overviews, and frequencies of response type. A significant challenge has been addressing low response rates to the annual user satisfaction survey. The User Forum has explored adding incentives to participation and more actively involving facility PIs in the data collection process. Overall, based on the 2019 survey results, the general NHERI user list rated information about NHERI facilities and resources more positively than known NHERI site users, however satisfaction with online support resources and tools and the available training for these resources and tools were similar, and positive, among both groups. Both groups also intend to use the NHERI Science Plan to learn of major research challenges, to reference how their research fits within the Science Plan in their NSF proposal, and to expand their current research scope.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

Author Contributions

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

Funding

This work was partially supported by the National Science Foundation under CMMI Grant No. 1612144. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

The authors are grateful for previous User Forum members, including Russell Green, who helped establish the committee and fulfill its mission in the initial year of operation. The authors also acknowledge Antonio Bobet, who helped establish the committee, and has continued to be supportive of its mission through participation on monthly calls and annual in-person meetings.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fbuil.2020.00101/full#supplementary-material

Footnotes

1. ^http://www.designsafe-ci.org/community/user-forum/

References

ISO DIS 9241-210 (2010). Ergonomics of Human System Interaction – Part 210: Human-Centered Design for Interactive Systems (Formerly Known as 13407). Geneva: International Standardization Organization (ISO).

Google Scholar

Karapanos, E., Zimmerman, J., Forlizzi, J., and Martens, J. (2009). “User experience over time: an initial framework,” in Proceedings CHI '09. ACM (New York, NY), 729–738.

Google Scholar

Law, E., Roto, V., HAssenzahl, M., Vermeeren, A., and Kort, J. (2009). “Understanding, scoping, and defining user experience: a survey approach,” in Proceedings CHI '09. ACM SIGCHI Conference on Human Factors in Computing Systems (Boston, MA).

Google Scholar

Patton, M. Q. (2008). Utilization-Focused Evaluation. Thousand Oaks, CA: Sage.

Google Scholar

Rajeshkumar, S., Omar, R., and Mahmud, M. (2013). “Taxonomies of User Experience (UX) evaluation methods,” in 2013 International Conference on Research and Innovation in Information Systems (ICRIIS) (Kajan; Selangor: IEEE), 533–538.

Google Scholar

Vermeeren, A. P., Law, E. L. C., Roto, V., Obrist, M., Hoonhout, J., Väänänen-Vainio-Mattila, K., et al. (2010). “User experience evaluation methods: current state and development needs,” in Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries (Tallinn), 521–530.

Google Scholar