The NOTECHS+: A Short Scale Designed for Assessing the Non-technical Skills (and more) in the Aviation and the Emergency Personnel

Abstract

This research presents the development of a short scale named “NOTECHS+” to measure the Non-Technical Skills (i.e., NOTECHS: Cooperation, Leadership and Managerial skills, Decision-Making, and Situational Awareness), Resilience and Emotion Regulation, in a sector that comprises the aviation and the emergency personnel: the Helicopter Emergency Medical Service (HEMS). The design process of the scale was carried out starting from a review on the behavioral markers used to detect the NOTECHS. Moreover, 70 interviews with HEMS experts have been conducted with the aim of developing Resilience and Emotion Regulation items by considering the different professional profiles (e.g., pilots, nurses, physicians, etc.) which compose the HEMS. Through a pre-assessment procedure, a Q-Sort test was performed on a sample of students (n = 30) to test the logical principles, but also intelligibility and clarity, of the items developed. Once the instrument was defined, 211 participants from the HEMS sector were surveyed to test the theoretical model behind the NOTECHS+ instrument. First exploratory and then confirmatory analysis yielded results that suggested that the 18 items selected conform to a bi-factor model composed of three skill-dimensions: Social skills (i.e., Cooperation, and Leadership and Managerial skills), Cognitive skills (i.e., Decision-Making and Situational Awareness) and Emotional skills (i.e., Resilience and Emotional Regulation). Finally, the study ends with a discussion on the results obtained, including practical implications on assessment and training based on this novel instrument.

Introduction

Aviation-related industries are to be considered high-risk organizations where responsibilities and pressures are high, as well as the physical and psychological risks related to the incidence and probability of accidents (Dekker, 2010). The operative core of such organizations is represented by flight crews, which are usually small-sized command and control teams, in a certain way unique, since they are extremely exposed to both high-risk factors and a high degree of human interaction (Salas et al., 2001). For such reason, a correct risk identification and an accurate accident analysis based on human interaction are fundamental to manage the safety levels of these teams. Moreover, this is the basis of the HR aviation culture where—given the nature of complex systems, characterized by human-machine interaction but also by teams' human-human interaction—human error represents the cause of a considerable percentage of accidents (Flin et al., 2002; Shappell et al., 2017).

Another aviation-related field, also extremely exposed to human errors, is the emergency sector, such as the HEMS (Helicopter Emergency Medical Service) field. The Helicopter Emergency Medical Service is an exceptional example of encapsulation of the medical and aviation sectors, since professionals belonging to both these fields (e.g., physicians, pilots, flight technicians, etc.) are working together and undergo various kinds of pressures (e.g., time pressure, peer pressure, self-induced pressures etc.) that could lead to human error. When carrying out HEMS operations, usually team members have to make difficult decisions in a very short time, and the consequences of a wrong decision may be extremely grave. In this sector, Cooperation and Leadership skills are fundamental, since during emergency operations the cohesion of the team members can be strongly tested. Communication and teamwork skills—together with the ability to make good decisions—have been addressed in a multitude of studies in this sector. This has led to the development of an assessment system called: Non-Technical Skills (NOTECHS). This term was coined by Flin et al. (2003) to define those cognitive and social skills needed to carry out safe operations and thus are complementary to technical skills. Flin et al. (2010) describe the functional process of the NOTECHS system, based on a tool developed from behavioral, social, and cognitive markers specifically designed for the aviation sector. Over time, other versions of the same assessment system have been developed with the aim of meeting the needs of other fields, such as the medical and emergency fields.

The specific definition of the NOTECHS behavioral markers makes the instrument a very useful means to accurately assess for certain professional roles. However, the flight crew encompasses different profiles, especially in the HEMS sector, which includes both aviation and emergency personnel. This might be perceived as a limit, since comparative analyses between these different roles are not possible with a unique instrument. A second methodological limit is related to the fact that NOTECHS does not include a psychometric scale for a quick assessment, which is useful for training evaluation, and in a dynamic sector such as the one of helicopter emergency services. Finally, a theoretical consideration. The current NOTECHS structure does not include the assessment of other relevant non-technical competencies, as the coping strategies used to manage stressful situations and emotions, which play a fundamental role both for operators in the aviation sector and for emergency workers (Weick et al., 2008; Cicognani et al., 2009; Setti et al., 2016). Considering these issues as a possible development in the assessment field, the purpose of this study is to develop a short psychometric scale named “NOTECHS+,” which allows a quick measure of the NOTECHS and of some new skills. Beyond NOTECHS original constructs, the scale also takes into account other competences, named Emotional skills, such as Resilience and Emotional Regulation (i.e., respectively the capacity to recover from a stressful event and to manage the emotions experienced). Moreover, such a scale can be applicable to different profiles of the aviation sector, including pilots but also technicians and the emergency crews (e.g., nurses, mountain rescue operators, etc.).

To reach the previously mentioned purposes, we will start with a review on NOTECHS and the components related to the management of emotional stressors. The literature review is important not only to identify correctly the components analyzed and needed to define the items, but also to design those theoretical models that may be validated through the subsequent statistical analyses. For the analyses, we will use a mixed approach. First, the procedure will be essentially qualitative, since it will consist in a series of systematic interviews and of a Q-Sort test for the development of the items. Qualitative procedure is important for considering aspects of different professional profiles in both the aviation and emergency fields, which can emerge through interviews. After having developed the scale and having collected the first data, exploratory and confirmative analyses will be performed to retain the most reliable items and to validate the new theoretical structure of the NOTECHS+ model, since it considers the new Emotional skills component. The study ends with a discussion on the results obtained, including the practical implications on the emergency and aviation contexts on which it is applied.

The Origins of the NOTECHS System

The project that led to the development of the NOTECHS system was carried out by four teams of psychologists from different countries belonging to different organizations: the Netherlands (NDR), Germany (DLR), France (IMASSA) and United Kingdom (University of Aberdeen). The project was intended to develop a system of behavioral markers thanks to a research funded by the Directorate-General for Mobility and Transport of the European Commission (Flin et al., 1998; Van Avermaete and Kruijsen, 1998; Flin and Martin, 2001). In addition to the NOTECHS consortium, the project involved further research organizations (SOFREAVIA, DERA) and three airline companies: British Airways, Airbus and Alitalia. The NOTECHS system provides a systematic approach to assess the Non-Technical Skills of pilots flying during their actual activities or in a flight simulator environment. The set of elements is based on theoretical models identified from a literature analysis and is compared with the KLM SHAPE and the NASA UT LLC (Helmreich et al., 1997) systems, in order to confirm that the essential elements envisaged in the international standards had been included. As to evaluate the usability, reliability and validity of the NOTECHS system as an assessment tool, five aviation research centers and four aviation commercial organizations were involved in the project. Flin et al. (2003) designed an experimental study that included: 105 examiners, selected from 14 European airlines, who were invited to assess the pilots' NOTECHS in different scenarios. The results showed that 80% of the instructors provided coherent scores and 88% of them were satisfied with the consistency of the method. On average, their score differed less than one point on the five-point scale established by the same experts that had developed the scenarios. This is considered an acceptable accuracy level since 98% of the instructors were satisfied with the NOTECHS system.

The structure of NOTECHS is designed considering relevant components in the work and the organizational domain. They are composed of four components, two social and two cognitive, which are: Cooperation, Leadership and managerial skills (or simply Leadership), Situational Awareness and Decision Making. In turn, these components are subdivided into elements that correspond to behavioral markers. The components and the elements of the NOTECHS system are measured on a five-point scale (“very good, good, acceptable, poor, very poor”). Besides, an overall evaluation “acceptable/not acceptable” is required. This structure was chosen after having analyzed other systems and upon the advice of subject matter experts such as examiners and instructors.

The first NOTECHS component is Cooperation, and is defined as the skill needed to work efficiently in a team for the same purpose. Cooperation comprises four elements: team building and maintaining, supporting others, conflict solving. The second social skill component concerns the role of Leadership, defined as the capacity of leading a group of people or an organization in a reliable way (Yukl, 2005). The behavioral markers of the NOTECHS related to Leadership are: the use of authority and assertiveness, maintaining standards, planning and coordinating, workload and resources management.

The other two components, which are defined as Cognitive Skills, are Decision-Making skills and Situational Awareness. In the psychology domain, Situational Awareness comprises the constructs of perception and attention, since it involves a continuous monitoring of the environment and of any changes in it (Lauria et al., 2019). It is a dynamic and multidimensional construct that includes the ability to anticipate critical events. The first element of this NOTECHS construct is the aircraft system awareness, which includes the active knowledge of the modes and status of the aircraft systems, as well as of the energetic status. The second element is the environmental awareness, i.e., active knowledge of the current and future aircraft position (e.g., weather, traffic, terrain, etc.). The last behavioral element of Situational Awareness is defined as the sense of time available and ability to look forward when considering future and contingent conditions (i.e., is the anticipatory awareness of time).

Decision-Making skills are the fourth NOTECHS construct and they are especially relevant for the personnel working in high-risk sectors, under high-stress conditions and pressures originating from the need to make quick decisions in a very short time. Elements that were included into the Decision-Making component are: problem definition, diagnosis and option generation, risk assessment, option choice, and outcome review.

What NOTECHS Do Not Measure: the Emotional-Coping Strategies

As stated above, NOTECHS are formed by two social categories and two cognitive categories, which have four constructs each and several behavioral elements. However, recent studies pointed out the ever-increasing importance of the development of coping strategies for stressful events in the aviation sector. In an overview on NOTECHS system and its future development, Kodate et al. (2012) ask for a future integration of the Systems Theory (i.e., the transdisciplinary study of the abstract organization of phenomena) in line with the recent development of the resilience engineering research. Also, Saurin et al. (2013) suggest to implement the resilience construct into the NOTECHS assessment. Resilience engineering is defined as an emergent discipline that stresses the capacity of a system to safely adapt to changes by limiting the emotional-stressful factors when facing unexpected events (Kodate et al., 2012). It is considered as an emergent job-person characteristic, in which the attention moves beyond human error to the knowledge of the system's and people's ability to adapt to changes by anticipating, managing and recovering from changing conditions, while managing emotional stress.

A similar construct close to Resilience is the ability to regulate emotions experienced at work, especially in demanding situations. Emotion regulation consists in automatic or intentional strategies used to begin, maintain, change the emotions felt during tense events (Gross and Thompson, 2007). Research suggests that emotion management is one of the major predictors of safety in the workplace, especially in presence of various stressful conditions, as it happens in the aviation context (Sexton et al., 2000) and in the healthcare system (Buruck et al., 2016). Moreover, in an aviation sector particularly subject to emergencies, such as the HEMS sector, the identification of the coping strategies needed to manage emotions in critical conditions assume even a more relevant role. The results of a study carried out by Cicognani et al. (2009) on a sample of 764 emergency workers show the importance of using and recognizing coping strategies for individuals that work in extremely stressful conditions, such as the emergency context, as to avoid the development of burnout or even of a post-traumatic stress disorder (PTSD) after an incident.

Conceptualization and Development Procedure of the NOTECHS+ Inventory

Hereinafter, we will describe the phases that led to the conceptualization and the development of the NOTECHS+ scale, for the assessment of both of the NOTECHS and the management of emotional stressors. The study has been approved by the University of Verona, Research Ethics Committee. The next procedures were approved in accordance with the National legislation and the University guidelines. All the subjects involved, participated in the study on a voluntary basis and gave their written informed consent for both interview and the next survey studies.

The first step consisted in an analysis of the scientific literature concerning the NOTECHS assessment methods (Flin et al., 2003). By using research data engines such as PsycINFO and PubMed, studies that involved NOTECHS assessment, and that could contribute to the development of the items of the questionnaire, were collected. We analyzed the main characteristics of the identified studies, namely: number of participants, number of citations (Scopus), authors, year of publication, abstract, method, instruments, results, discussion. For each study or scale taken into consideration, the items, categories, elements, selected behaviors and technical indexes including factor loadings and measures of internal coherence were examined (see Table 1).

The markers considered in the various studies were then analyzed. Based on the similarity among them, we decided to take into account especially the markers from the study conducted by Flin et al. (2003) since they included all the constructs that could help develop the items of the scale. This procedure allows benefiting of all the various behavioral markers, subdivided into good and bad operational behaviors, for every single element of the four NOTECHS constructs. Some limitations related to the applicability of such assessment systems emerged. For instance, the study of Robertson et al. (2014) takes into consideration the same four NOTECHS components, but the examples of related operational behaviors resulted to be very specific for technical-assistant personnel. The instrument developed by Nergård et al. (2011), instead, refers mostly to the characteristics of the pilot, but does not consider other roles such as technicians. From all the scales presented in Table 1, we searched for all the possible behavioral markers that could define all the four NOTECHS skills, and that could be adapted to a larger segment of the aviation personnel and of emergency crews. First, we translated these practices into Italian, and we selected the markers that, according to us, could be adapted as items on the basis of their content. The following step was to turn them into a first-person sentence since the final questionnaire is a self-report instrument. After that, we maintained five items for each construct, since not all the items were applicable to all the operators, regardless of the role they played. For instance, as it concerns Leadership, the items “I assign the tasks to the team members, I check and correct them in an appropriate way” or “I give the team the appropriate time to complete the tasks” were discarded since they did not apply to operators that do not manage other people. Thus, our choices were made as to be exhaustive by taking into account the different roles in HEMS, reaching the overall number of 20 items (five for construct) for the classical NOTECHS categories considered.

Critical Incident Interviews and the Development of Resilience and Emotion Regulation Items

Since NOTECHS literature does not include investigations on coping strategies for managing emotional stressors, we have carried out 70 interviews on an Italian sample of pilots, hoist operators, doctors, nurses and mountain rescue operators working in the HEMS sector. Sample recruitment was managed by a trained organizational psychologist with experience in the aviation and emergency sectors. The purpose was to understand the role of these variables in such a strategic sector, such as HEMS, since it comprises both aviation and emergency personnel. Focus-group, panels and interviews were carried out by using the critical incident interview method to explore an event that had actually happened during rescue operations, and that was also emotionally critical to the person interviewed. Then we focused on the individual actions taken to face the emotional stressful situation and its consequences. Below we provide some extracts from the interviews where, starting from the description of a critical incident, we investigated the coping strategies used by the individuals:

“Our work is emotionally intense, even if I feel quite detached, everything is calculated–I re-elaborate things a little bit. Interventions on children are more stressful since we are parents as well – I do feel the responsibility.”

“When I am working, I do not feel strong emotions. We are not hard-hearted people, but we have to work. I have been doing this job for 30 years; you must concentrate on what you have to do, you do not have to be overtaken by emotions. When you get back to the base, you think of it, but we are lucky, since usually after one intervention we immediately have another one, so you do not think about it anymore.”

“Emotions always come late, when operations are concluded. In that very moment (when you work) you are clear-headed and concentrated. Pilot and technician are less in contact with emotionally stressful situations than the rest of the crew. The most important thing is to talk about it immediately. Emotions are not shared, only technical aspects are”

Here are some statements of how the concept of resilience is perceived:

“Once my head was more “behind my seat” (where doctor and nurse are sitting), while now I do not even see the face of the patient. This began after a rescue ended up very bad with a child. This thing works, I work better like this”

“We have a reduced RAM in which we try to keep only aeronautical issues, and we try to reset all the rest – as to start with a “white sheet”.”

From what resulted from the interviews, due to the nature of this job environment, HEMS sector cannot be devoid of emotions and of recovery and resilience strategies used to face emergencies. From a resilience perspective, adaptation to emergency is an important element for coping with highly emotional and unexpected events. The capacity to adjust and adapt comprises knowledge in terms of anticipation (what to expect), attention (what to look for), response (what to do) and finally learning. This requires the ability to rebuild the personal resources quickly after a crisis and this process is fostered by emotional self-regulation capacity (Fredrickson and Losada, 2005). Recovery from emergencies is promoted by reappraisal and emotional self-regulation processes, by broadening one's thoughts and actions, and it is connected to the resilience level of the individual. Emotional self-regulation, and in particular cognitive reappraisal, which involves reframing a situation in order to change its emotional impact, helps in this situation, even if during the crisis the suppression of panic feeling might be more strategic (Mauss et al., 2007). Therefore, in order to develop the items, we considered the extracts from these interviews, along with some sample of items derived from the most reliable scales (based on citations and adaptability to the HEMS context) validated in Italian for assessing Resilience and Emotion regulation levels. Emotion regulation items derive partially from the cognitive reappraisal and emotional suppression subscales of the Emotion Regulation Questionnaire (ERQ) by Gross and John (2003). Example items are: “When I want to feel less negative emotion (such as sadness or anger), I change what I'm thinking about” and “When I want to feel more positive.” The Italian version of the Connor-Davidson Resilience scale (CD-RISC 25) has been considered for the development of the Resilience subscale (Connor and Davidson, 2003). Based on these samples of items and on the contents extrapolated from the interviews, we developed other ten items (five for construct) that were added to the 20 selected previously, as to assess a total number of six constructs.

Development of the Items: Analysis of Intelligibility and Clarity, Q-Sort Test and Multi Correspondence Analysis (MCA)

To increase accuracy, we decided to perform a Q-sort test on a sample of students. The sample was composed of 30 Italian students of Psychology, the 70% of which was female. The sample average age was 28 with a standard deviation of 9.1. One third of the subjects already had a bachelor's degree, while three subjects a superior education qualification. We analyzed the levels of intelligibility and clarity of this set of items through a pilot-questionnaire. The Q-sort test is a research method used to investigate how people think with regard to an issue (inter-rater comparison). This method is used as a tool in assessment and it shows to be particularly useful when the researchers wish to understand and describe the variety of subjective points concerning the face validity of the instrument. The observer starts with the set of items (Q-set) for evaluating the psychological construct to which the items were referred. The observer shall decide the degree of similarity of each item with some possible typical behaviors. After having decided, he/she will order them basing on their similarity, classifying them into a number of pre-determined groups. In this case, we deemed it useful to use this instrument and therefore we prepared the necessary material. Each participant was given an envelope containing 30 cards—the Q-set - and six paper clips. They were asked to group the items in accordance with a principle of logic and similarity. We did not provide the names of the corresponding categories. At the end, we administered the questionnaire with the same set of items, in order to investigate the degree of intelligibility and clarity of every single item on a scale ranging from 0 (not at all) to 4 (very clear).

Before analyzing the Q-sort result, we first checked the average score of intelligibility and clarity of the various items, setting a cut-off value of 2.5. Thanks to the analysis on the degree of clarity, we found out that the item 12 “I can recognize new voices, unexpected situations and changes in the instruments” and item 26 “When I want to feel less negative emotions I change the way I see things” were critical since their average score was respectively 2.3 and 2.4. At the same time, thanks to the analysis on the degree of intelligibility, we found that item 7 and 26 were critical, since their average score was 2.4 and 2.3, respectively. Besides, the researcher in charge of the preparation of the scale and administration of the Q-sort and of the questionnaires reported that some participants showed to be puzzled when associating those items according to logical principles related more to the lexical structure of some items rather than to their content. For instance, the prosocial aspects of the Cooperation items could induce social desirability by leading the responder to acquiescence in responding (Ray and Pratt, 1979; Ray, 1983; Spector, 1992).

After that, a contingency matrix through the row and column profiles was created by using dichotomous variables in which rows correspond to the relative frequencies of the items for all the categories created by the students with the Q-Sort method. The multiple correspondence pattern showed that - with 73% of cumulative inertia - six categories could explain much of the variance. The general rules in MCA suggest that the number of categories retained should be higher of 70% of the inertia explained or correspond to the number right before the “elbow” of the eigenvalues by category number of scree plot (see Sourial et al., 2010). Multiple Correspondence Analysis is an exploratory descriptive method that uses the multivariate extension of the Correspondence Analysis for examining tables containing three or more variables. The present data analysis and the next ones were performed using the IBM SPSS 21.0 package, and the IBM AMOS 21.0 package (SPSS Inc., Chicago, IL) for the CFA analysis. Multiple Correspondence Analysis can be considered a generalization of PCA for categorical variables that reveal patterning in complex data sets without requiring an assumption of underlying normality (Ayele et al., 2015). Therefore, we deemed that six components were fully sufficient to define the six constructs (i.e., Cooperation, Leadership, Decision-Making, Situational Awareness, Resilience and Emotional Regulation).

Revision of the Items and Definition of the NOTECHS+ Scale

In light of the items' distribution that emerged from the MCA resulting from the Q-sort and from the levels of intelligibility and clarity, we decided to revise the set of items to enhance their clarity. First, the items belonging to the Cooperation construct were turned into reverse score (items scale have been worded in the opposite direction), in reference to the poor practice detected by Flin et al. (2003). This was done with the objective to avoid response biases associated with social desirability effect (e.g., acquiescence, straight-line responding, etc.). Item 7 “I succeed in motivating my crew thanks to recognition and I support the crew when necessary”—which was critical from the intelligibility point of view—was modified, since it comprised two different concepts in one single sentence. Then, since Situational Awareness items were often associated with other categories, we replaced the term “situation” with the term “conditions,” as it has a more precise meaning compared to the previous one. Furthermore, item 15 “I discuss time limitations and strategies in emergency situations” was revised since the term “I discuss” at the beginning of the sentence diverted from the contents of situational awareness. Within the construct of Decision-Making, item 18 “I tend to ask other crew members for new options before making a decision” and item 20 “I discuss the possible risks when making decisions, always considering crew limitations” resulted to be critical from the MCA since they involved more categories. Therefore, these items were modified to address the decision made, rather than the contents of cooperation. For what concerns resilience, item 22 “After being sick, after an accident or another serious situation I recover quickly” and item 24 “Even when I am under pressure I can keep concentrated” were slightly modified, stressing the contents related to resilience rather than those that could be related to emotion regulation. Item 26 “When I want to feel less negative emotions I change the way I see things” was modified due to its low levels of clarity and intelligibility, perhaps due to the double negative.

In conclusion, the questionnaire to test is made up of 30 items referring to the six constructs and three dimensions: Social skills (i.e., Leadership and Cooperation), Cognitive skills (i.e., Situational Awareness and Decision-Making) and Emotional skills (i.e., Emotional regulation and Resilience). Answers to every single item are given on a 5-point Likert scale (“I do not see myself in it at all; I do not see myself much in it; I see myself in it; I see myself a lot in it; I fully see myself in it”).

Analysis for the Scale Validation

After having defined the scale, we started the validation process by administering it to a sample of people working in the HEMS sector. The first goal was to test the reliability of the scale for each of the constructs. We considered the possibility to remove the items that were not reliable since we wished to create a reliable instrument, quick to administer and suitable to the dynamic context of HEMS. The second goal was to investigate the latent component associated by the instrument both in an Exploratory (PCA) and Confirmative (CFA). Since two new components had been added to the NOTECHS theoretical framework, we could expect that the latent factors could differ from the original NOTECHS configuration. For instance, we can expect a second latent factor which includes the extra components (i.e., Resilience and Emotional Regulation) added to the original NOTECHS, or three dimensions based on a skills-structure (i.e., Social skills, Cognitive skills, Emotional skills).

Sample and Descriptive Statistics

We surveyed 10 Italian HEMS bases. We used a simple random sampling method to select the potential participants who were invited to take part to a survey aiming at studying NOTECHS components at work. Participation was voluntary. Once that participants agreed, they were informed about the focus of the survey and choose to fill a questionnaire online or using paper and pencil. We selected 225 participants, out of whom 14 did not complete entirely the questionnaire, and therefore they were excluded from the statistics, formed the sample (N = 211). The descriptive analyses regarding the professional roles show that most subjects are nurses (33.2%), followed by rescue operators (28.9%) and doctors (27.5%). The role of pilot (6.6%) and technicians are less frequent (3.8%). In the group of participants, the 79% were males and 21% female, while one subject did not specify his/her gender. Age was subdivided into clusters: three at 15-year intervals, the last being “more than 60 years.” From a first descriptive analysis, most participants of the sample belong to the cluster 31–45 years (48.8%), followed by the cluster 46–60 (44.5%). Finally, subjects aged more than 60 and the youngest subjects aged 15–30, are respectively 6.2 and 0.5% of the sample. For what concerns educational qualifications, descriptive analysis shows that most subjects have a high school diploma (33.5%) followed by postgraduate speciality certificate (28.2%). The rest of the sample has a middle school leaving certificate (6.3%), a bachelor's degree (13.6%) or a Master's degree (1.5%), a 5-year degree (4.4%) or a master diploma (12.6%). None among the participants has a PhD qualification. Seniority ranges from 0 to a maximum of 44 years of work in the same organization (M = 11.78, SD = 8.44).

Items Reliability Scores

We analyzed the reliability of the items considering the Cronbach's Alphas values. The six Leadership items showed high reliability (α = 0.86), while the Cooperation component showed low values. Alpha improves after deleting three items which showed a poor adjustment in the Cooperation component, by increasing it to α = 0.71. For what concerns the Cognitive dimension, Situational Awareness and Decision-Making presented acceptable values (respectively α = 0.80 and α = 0.82). Resilience showed high reliability, while the five items of Emotion Regulation did not present acceptable values. Therefore, we removed three items and kept the items 28 and 29: the value increased up to α = 0.72. In accordance with the aim of developing a short scale, we retained 24 items that were next analyzed.

Principal Component Analysis (PCA)

Even if the scale was designed considering six components, we could not exclude the presence of different latent factors models until the structure of the inventory was tested. Principal Component Analysis was preferred over other reduction techniques because of the heterogeneity of possible theoretical models, leaving us with a lack of prior knowledge regarding the numbers of given components and items loaded in them. Analyses were performed with a Varimax rotation to allow for component intercorrelations with a one-, two-, three- four –five, and six components, in order to determine the most interpretable solution. Based on a visual inspection of the scree plot, the five and six components solutions seemed reasonable. Where the first five components explained the 58.62% of variance, the same five-component solution has several components loadings with extremely low values (r < 0.30), which makes the interpretability of the dimension constrained (Russell, 2002). For such a reason we performed a parallel analysis, a Monte-Carlo simulation of eigenvalues, in order to determine the optimal number of components to retain. In the current analysis, 1000 random datasets were generated (95% CI) with the same number of cases and variables as the original dataset. According to the comparison between mean eigenvalues obtained with the observed eigenvalues from study, we kept the six-component solution over the five-component solution. Besides such methodological aspects, the six-component solution was preferred especially for theoretical reasons, since every item of the resulted components was distributed according to the six constructs. Finally, in order to obtain a more stable solution, we removed the items that showed a correlation with the component inferior to 0.60, considering that most of these also loaded moderately in the other components identified. That resulted in a reduction from five to four items for the Resilience and the Leadership components, and for what concerns the Cognitive dimension, the components Decision-Making and Situational Awareness were reduced from five to four items and from five to two items respectively. In summary, the scale is composed of a total of 18 items, based on six constructs which can present four (i.e., Leadership, Decision-Making, Resilience) or two items each (i.e., Situational Awareness, Cooperation, and Emotion Regulation). Table 2 presents the most interpretable solution.

Confirmatory Component Analysis (CFA)

Confirmatory Factor Analysis (CFA) was performed on the relationships indicated in the PCA. Since there was not sufficient theoretical basis, we tested the same data for cross-validation purposes (see Fabrigar et al., 1999). We tested the validity of the solution found with the restrictions implied by the CFA (e.g., fixed cross-loadings, measurement errors, etc., see Worthington and Whittaker, 2006). Moreover, this operation was helpful to define the presence of a possible bifactor configuration, which could result in several theoretical models tested herein.

The first CFA performed on the six components found and based on a single factor configuration showed a quasi-acceptable fit indexes: χ²₍₁₂₄₎ = 195.342 (p < 0.001), TLI = 0.928, CFI = 0.948, RMSEA = 0.052 (Cheung and Rensvold, 2002). Considering this first results, based on several indicators (e.g., Parallel Analysis scree plot, Velicer MAP, and BIC), we investigated the presence of a possible bi-dimensional structure. The second model that we tested considered all the previous subscales together through a bifactor model to verify if they underlie a single construct (NOTECHS). The results of the bifactor analysis little improved but they were still improvable. Then, the third model focused on the presence of a single underlying construct: the NOTECHS, based on the classical four dimensions and the two new constructs comprised in a single latent factor. This resulted in an increment to an acceptable CFI value (Hu and Bentler, 1999) and in an acceptable small increment of RMSEA value [χ²₍₁₀₆₎ = 136.486 (p = 0.025), TLI = 0.964, CFI = 0.978, RMSEA = 0.037]. Finally, we explored the possibility of having three possible latent dimensions based on skills configuration: Social skills (i.e., Cooperation and Leadership), Cognitive skills (i.e., Decision-Making and Situational Awareness), and Emotional skills (i.e., Resilience and Emotional Regulation), which resulted in the best fit configuration found [χ²₍₁₀₃₎ = 108.374 (p = 0.339), TLI = 0.993, CFI = 0.990, RMSEA = 0.016]. In conclusion, the validated instrument is made up of 18 items referring to three dimensions (i.e., Social skills, Emotional skills, Cognitive skills) composed of six items each and the six constructs with respective items. For a graphical schema of such a model, see Figure 1.

Figure 1

Discussion

Accident investigations carried out in the 70's showed that pilots' Non-Technical Skills such as Leadership, Communication, Teamwork, and Decision-making, were clearly disregarded as significant factors for safe flight operations (Cooper et al., 1980). The NOTECHS system, as stated, comprises two categories, one social and one cognitive. However, in the aviation context—and especially in the emergency context—the emotional response has a relevant role in the activities accomplishment (Beilock and Carr, 2001), since it can affect the muscular tone, breathing, cardiac, and endocrine activity. Indeed, beyond the social and cognitive categories, this study added a new dimension related to Emotional skills.

Considering the methodological approach, the mixed method used allowed for the development of a systematic and a psychometric assessment procedure. To validate this new theoretical model, we tried to verify if the six components of the NOTECHS+ scale—i.e., Cooperation, Leadership, Situational Awareness, Decision-Making, Resilience, and Emotional Regulation—could also be explained with another macro-factor; but we found no satisfying model fit-indexes. We then conceptualized another bi-factorial model that explained the four original NOTECHS as belonging to one dimension and the additional dimension as belonging to the extra Emotional skills dimension, but again indexes' values were not satisfying. Finally, we assumed and analyzed a model in which the six components were explained by three latent dimensions: Cognitive, Social, and Emotional skills. This result, is relevant because it can contribute some practical implications in relation to the assessment and the training of such competences, which could be organized following this tripartite structure. Before considering the practical aspects, below we will introduce some insights derived from the literature in support for these three dimensions coming from the confirmative analysis.

The Social Skills: Cooperation, Leadership and Managerial Skills

Despite the fact that they belong to a unique latent dimension, the two social categories of the NOTECHS system—i.e., Cooperation, and Leadership—could seem overlapping, since both of them refer to managing group processes. However, there is a conceptual difference according to which Cooperation concerns reciprocal assistance and group climate at work (Costantini et al., 2019), whereas Leadership covers all the aspects related to initiative, coordination and goal definition (Flin et al., 2003). The studies developed by Flin (1996) on aircraft accidents showed that the best performing leaders analyse the situation, have a wide range of leadership styles and adapt their style to the situation that they experience. Among the various studies on Leadership, Zaccaro et al. (2001) reported that the leadership aspects that affect group performance are: active participation of the team leader and of all the other team members, definition of group's direction and the attempt to organize the team as to maximize team development, respect from other team members; awareness of one's own strengths and the willingness to respect the other team members and their role, encouraging open communication, including the discussion on the team's goals and on expectations about performances, which lead to commitment and consensus within the team. As a matter of fact, it was demonstrated that good leadership is important for safe performance in the workplace (Hofmann and Morgenson, 2004; Glendon et al., 2006).

For what concerns cooperation, Grice and Katz (2005) stressed the importance of cohesion in military and aviation psychology, stating that efficient teams have shared goals. Some studies found that team skills are identifiable and can be trained (Cannon-Bowers and Salas, 1998). Flin et al. (2003) showed that cooperation does not refer to job characteristics, such as quality or quantity of job outcomes, but that good cooperation originates from an open and active communication among the team members. Teamwork is achieved when the various members begin to cooperate and coordinate their efforts, by maintaining their attention focused on the task, thus mitigating interpersonal relationships dynamics inside the team (Baker and Salas, 1996). Attention to task-oriented cooperation underlines the factors that contribute to achieving the team's goals. This happens thanks to conditions such as the attribution of roles, shared mental models and feedback. These features are obtained from the skills developed in Crew Resource Management (CRM), which improves performance and reduces human error in teams, such as in aviation crews (Nullmeyer and Spiker, 2003).

The Cognitive Skills: Decision Making and Situational Awareness

The second bi-factorial dimension regards the cognitive skills of decision-making and situational awareness, which are both relevant for sectors like emergency (Harris et al., 2016) and aviation. For what concerns the latter, Flin et al. (2010) found that most serious aviation accidents involved situational awareness issues. An analysis of the most important aviation accidents was performed between 1989 and 1992. It was found that the lack of situational awareness was the first cause of accident in 88% of the accidents related to human error (Endsley and Robertson, 2000). Situational awareness helps to manage all the information coming from the environment, through a cognitive organization, and taking into consideration human limited resources (e.g., working memory). A well-known model of situational awareness was developed by Endsley (1995) and it is made up of levels that correspond to the three elements of Situational Awareness. The first concerns the perception of the elements of the current situation; the second level refers to the interpretation of information collected; finally, the third level concerns the anticipation of future conditions. As a matter of facts, Flin et al. (2010) affirm that this predictive component of situational awareness is very important in dynamic working environments. Emergency operators shall be able to think ahead and to anticipate events. Mitchell and Flin (2008) stated that in an operational working environment there is a continuous monitoring and re-assessment of the environment as to make appropriate decisions.

Since the results of an analysis of the aviation accidents occurred between 1983 and 1987 in the United States, researchers showed that in 47% of accidents the main contributing factors were teams' poor assessment and poor Decision-Making. In the studies on aircraft pilots, the Naturalistic Decision-Making model used (i.e., a model of dynamic decision-making applicable to high-risk working environments such as aviation, see Orasanu, 1995) presents a two-stage process: assessment of the situation (Stage 1) and application of a decision-making method to choose the actions to be taken (Stage 2). Researchers noted that pilots use one or more Decision-Making processes depending on the demands of the situation, but that they also use intuition, a regulation framework and analysis of the available options. Moreover, they showed that when pilots have little time available and are faced with high risk, they use quicker strategies and apply known rules (Orasanu and Fischer, 1997). Fatigue, instead, may affect flexibility, cause an increase in the propensity to errors and affect the ability to promote an update of the situation (Harrison and Horne, 1999). Moreover, the effort required in decision-making processes rapidly depletes personal resources, thus leaving the executive function less efficient when performing other tasks (Ceschi et al., 2014, 2017a,b, 2018). As information processing increases, greater cognitive resources are required for a competent functioning. In complex environments such as HEMS, a competent decision-maker requires a variety of cognitive skills to continuously search for information to improve safety (Houghton et al., 2000). An essential point of Decision Making in NOTECHS is the structure for processing information, based on the last advance in cognitive psychology research (Flin et al., 2003).

The Emotional Skills: Resilience and Emotional Regulation

Generally, in the aviation sector and even more in the HEMS sector, operators show levels of PTSD much higher than in the rest of the population. In particular, operators of the emergency sector show a prevalence of 14.6% on general population. Moreover, there is evidence on how the proximity, duration and intensity of the exposure to traumatic events (e.g., accidents involving children, car accidents, violent accidents) are the most significant predictors of disorders affecting front-line operators of the emergency sector (Benedek et al., 2007). As a matter of fact, exposure to critical incidents involving death or life-threatening injury is potentially an integral part of the job for emergency services personnel (Pietrantoni and Prati, 2008). In relation to the extraordinary conditions of the HEMS sector, the facts reported by Gillespie et al. (2007) show that resilience is a crucial protective factor against the development of mental disorders caused by the exposure to traumatic events of the medical personnel working in the emergency sector. Furthermore, the identification of resilience as a fundamental trait for emergency staff provided empirical evidence to what Smith and Roberts' (2003) maintained to be a protective and adaptive process. Accordingly, different disciplines focus on several aspects of resilience, resulting in diverse but interrelated definitions (Folke, 2006). Studies within the workplace have demonstrated that resilience is a significant negative mediator between the effects of job stressors and work-related psychological disorders (Bartone, 2006). Individuals with high levels of resilience adapt their coping strategies, and they even turn stressors into learning opportunities (Steinhardt and Dolbier, 2008). These studies see Resilience as a component of mediated-coping processes aiming at avoiding exhaustion through the interaction with other protective factors. The capacity to use Resilience when facing adverse events consists of a set of characteristics held by individuals. Such a conglomerate of abilities and capabilities permits the individuals to promptly direct their action, going beyond the potentially debilitating consequences of negative events. For instance, evidence shows that in burnout recovering, personal resources post-crisis growth is related to Resilience and fostered by Emotional Regulation (Fredrickson and Losada, 2005).

Furthermore, Emotional Regulation, and in particular emotional suppression, involves the necessary capacity to inhibit the first emotional reaction due to extreme situations and critical events (Mauss et al., 2007). According to Bandura (1997), stress reactions mostly depend on these coping capabilities (Prati et al., 2010) based on self-regulation processes and ability to broaden one's thoughts and actions, and they are connected to the resilience level of the individual (Ceschi et al., 2017c). Individuals with high levels of resilience adapt their coping strategies and often turn stressful factors into learning opportunities (Steinhardt and Dolbier, 2008). Moreover, through the presence of some working experiences, individuals could develop more Resilience and increase their Emotional Regulation, by managing more dynamic and intensive workplaces (Judge and Bono, 2000). In this framework, resilient individuals are those who are able to implement coping strategies, such as being focused on the problem, the capacity to take time before acting and to give and receive support from their workmates (Bartone, 2006).

Practical Implications, Potentials, and Limitations

The NOTECHS+ system offers a systematic approach for the assessment of non-technical competences of the professionals involved in aviation and in the emergency sectors. On the whole, the indexes of the instrument are reliable for all the components analyzed. The 18-item questionnaire is characterized by brevity, which is an essential feature in such a dynamic and emergency-related context such as HEMS. Moreover, since this sector comprises both the aviation and the emergency fields, as well as technical personnel such as pilots, doctors, nurses, technicians etc., the instrument has been designed with the aim of assessing all these different profiles.

Considering some practical aspects, besides assessment, due to its brevity, the NOTECHS+ developed in this study could be used to measure the baseline and the effects of the intervention, where an intervention can be used to understand the strengths and limits of the three components considered and to then provide trainings to strengthen the non-technical components needed to deal with high-stress environments. Moreover, training could be reorganized following this tripartite skills-structure. Systematic training on NOTECHS has been provided for years now, thanks to the Crew Resource Management (CRM) training courses. Crew Resource Management courses include modules on leadership and supervision, techniques for the development of metacognitive skills, the implementation of skills on task management, recognition of critical signals, etc. (Mitchell and Flin, 2008).

As it happens for NOTECHS, Resilience and Emotional Regulation may be included in trainings and further developed with specific interventions as to understand and improve these psychological mechanisms that enable situational awareness and coping mechanisms in situations of high stress. This is especially important in fields that involve teams that endure high-stress on a daily basis. The applicability of this scale can also be extended to other teams in the healthcare system and in the military, where unforeseen events happen regularly and where it is especially important to be trained to cope effectively and maintain situational awareness under stressful conditions.

Although these results are promising, we must acknowledge several limitations that could represent the focus of future research. First, to reduce participant burden and items reliability, our indicators were limited to a small set. Second, individual differences may have had an influence on the factors found, however we did not consider personality, personal inclinations, and socio-demographic characteristics such as age or gender, education level, work experiences, etc. which might have a role in factor emergence. Third, data collected refers to cross-sectional self-reported values. Further studies could include longitudinal designs along with the inclusion of the original NOTECHS behavioral markers. Finally, the sample was composed only of Italian participants. Future research could consider possible cross-culture effects by sampling participants from different countries.

References

• 1

  Antersijn P. Verhoef M. (1995). Assessment of non-technical skills: Is it possible? in Applications of Psychology to the Aviation System, eds McDonaldN.JohnstonN.Fuller.R. (Aldershot: Averbury), 243–250.

  • Google Scholar

• 2

  Ayele D. Zewotir T. Mwambi H. (2015). Multiple correspondence analysis as a tool for analysis of large health surveys in African settings. Afr. Health Sci.14:1036. 10.4314/ahs.v14i4.35

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 3

  Baker D. P. Salas E. (1996). Analyzing team performance: in the eye of the beholder?Mil. Psychol.8, 235–245. 10.1207/s15327876mp0803_7

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 4

  Bandura A. (1997). Self-Efficacy: The Exercise of Control. New York, NY: W.H. Freeman and Company.

  • Google Scholar

• 5

  Barbaranelli C. Capanna C. (2001). Efficacia personale e collettiva nelle associazioni di volon-tariato socio-assistenziale, in La Valutazione Dell'autoefficacia, ed CapraraG. V. (Trento: Erickson), 147–156.

  • Google Scholar

• 6

  Bartone P. T. (2006). Resilience under military operational stress: can leaders influence hardiness?Mil. Psychol.18, 131–148. 10.1207/s15327876mp1803s_10

  • CrossRef
  • Google Scholar

• 7

  Beilock S. L. Carr T. H. (2001). On the fragility of skilled performance: what governs choking under pressure?J. Exp. Psychol. Gen.130, 701–725. 10.1037/0096-3445.130.4.701

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 8

  Benedek D. M. Fullerton C. Ursano R. J. (2007). First responders: mental health consequences of natural and human-made disasters for public health and public safety workers. Annu. Rev. Public Health28, 55–68. 10.1146/annurev.publhealth.28.021406.144037

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 9

  Brannick M. T. Prince C. Salas E. (2002). The reliability of instructor evaluations of crew performance: good news and not so good news. Int. J. Aviat. Psychol.12, 241–261. 10.1207/S15327108IJAP1203_4

  • CrossRef
  • Google Scholar

• 10

  Burger K.-H. Neb H. Hoermann H.-J. (2003). Lufthansa's new basic performance of flight crew concept-A competence based marker system for defining pilots performance profile, in Proceedings of the 12th International Symposium on Aviation Psychology, ed Jensen.R. S. (Dayton, OH: Wright State University Press), 172–175.

  • Google Scholar

• 11

  Buruck G. Dörfel D. Kugler J. Brom S. S. (2016). Enhancing well-being at work: the role of emotion regulation skills as personal resources. J. Occup. Health Psychol.21, 480–493. 10.1037/ocp0000023

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 12

  Cannon-Bowers J. A. Salas E. (eds.). (1998). Making Decisions Under Stress: Implications for Individual and Team Training. Washington, DC: American Psychological Association.

  • Google Scholar

• 13

  Ceschi A. Costantini A. Phillips S. D. Sartori R. (2017a). The career decision-making competence: a new construct for the career realm. Eur. J. Train. Dev.41, 8–27. 10.1108/EJTD-07-2016-0047

  • CrossRef
  • Google Scholar

• 14

  Ceschi A. Costantini A. Sartori R. Weller J. Di Fabio A. (2018). Dimensions of decision-making: an evidence-based classification of heuristics and biases. Pers. Indiv. Differ.10.1016/j.paid.2018.07.033

  • CrossRef
  • Google Scholar

• 15

  Ceschi A. Demerouti E. Sartori R. Weller J. (2017b). Decision-making processes in the workplace: how exhaustion, lack of resources and job demands impair them and affect performance. Front. Psychol.8:313. 10.3389/fpsyg.2017.00313

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 16

  Ceschi A. Dorofeeva K. Sartori R. (2014). Studying teamwork and team climate by using a business simulation: How communication and innovation can improve group learning and decision-making performance. Eur. J. Train. Dev.38, 211–230. 10.1108/EJTD-01-2013-0004

  • CrossRef
  • Google Scholar

• 17

  Ceschi A. Fraccaroli F. Costantini A. Sartori R. (2017c). Turning bad into good: how resilience resources protect organizations from demanding work environments. J. Workplace Behav. Health32, 267–289. 10.1080/15555240.2017.1398659

  • CrossRef
  • Google Scholar

• 18

  Cheung G. W. Rensvold R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Struct. Equation Model.9, 233–255. 10.1207/S15328007SEM0902_5

  • CrossRef
  • Google Scholar

• 19

  Cicognani E. Pietrantoni L. Palestini L. Prati G. (2009). Emergency workers' quality of life: the protective role of sense of community, efficacy beliefs and coping strategies. Soc. Indic. Res.94, 449–463. 10.1007/s11205-009-9441-x

  • CrossRef
  • Google Scholar

• 20

  Connor K. M. Davidson J. R. (2003). Development of a new resilience scale: the Connor-Davidson resilience scale (CD-RISC). Depress. Anxiety18, 76–82. 10.1002/da.10113

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 21

  Cooper G. E. White M. D. Lauber J. K. (eds.). (1980). Resource Management on the Flightdeck: Proceedings of a NASA/Industry Workshop (NASA CP-2120).Moffett Field, CA: NASA-Ames Research Center.

  • Google Scholar

• 22

  Costantini A. Scalco A. Sartori R. Tur E. M. Ceschi A. (2019). Theories for computing prosocial behavior. Nonlinear Dynamics Psychol. Life Sci.23, 297–313.

  • Pubmed Abstract
  • Google Scholar

• 23

  Dekker S. (2010). Pilots, controllers and mechanics on trial: cases, concerns and countermeasures. Int. J. Appl. Aviat. Stud.10, 31–50.

  • Google Scholar

• 24

  Endsley M. R. (1995). Measurement of situation awareness in dynamic systems. Hum. Factors37, 65–84. 10.1518/001872095779049499

  • CrossRef
  • Google Scholar

• 25

  Endsley M. R. Robertson M. M. (2000). Training for situation awareness in individuals and teams, in Situation Awareness Analysis and Measurement, eds EndsleyM. R.GarlandD. J. (Mahwah, NJ: Erlbaum), 349–366.

  • Google Scholar

• 26

  Fabrigar L. R. Wegener D. T. MacCallum R. C. Strahan E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychol. Methods4, 272–299. 10.1037/1082-989X.4.3.272

  • CrossRef
  • Google Scholar

• 27

  Flin R. Goeters K. M. Hormann H. J. Martin L. (1998). A generic structure of non-technical skills for training and assessment, in Proceedings of the 23rd Conference of the European Association for Aviation Psychology (Vienna).

  • Google Scholar

• 28

  Flin R. Martin L. (2001). Behavioural markers for crew resource management: a review of current practice. Int. J. Aviat. Psychol.11, 95–118. 10.1207/S15327108IJAP1101_6

  • CrossRef
  • Google Scholar

• 29

  Flin R. Martin L. Goeters K. Hoermann J. Amalberti R. Valot C. et al . (2003). Development of the NOTECHS (Non-Technical Skills) system for assessing pilots' CRM skills. Human Factors Aerosp. Saf.3, 95–117.

  • Google Scholar

• 30

  Flin R. O'Connor P. Mearns K. (2002). Crew resource management: improving team work in high reliability industries. Team Perform. Manag. Int. J.8, 68–78. 10.1108/13527590210433366

  • CrossRef
  • Google Scholar

• 31

  Flin R. Patey R. Glavin R. Maran N. (2010). Anaesthetists' non-technical skills. Br. J. Anaesth.105, 38–44. 10.1093/bja/aeq134

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 32

  Flin R. H. (1996). Sitting in the Hot Seat: Leaders and Teams for Critical Incident Management. Chichester: J. Wiley.

  • Google Scholar

• 33

  Folke C. (2006). Resilience: the emergence of a perspective for social–ecological systems analyses. Glob. Environ. Change16, 253–267. 10.1016/j.gloenvcha.2006.04.002

  • CrossRef
  • Google Scholar

• 34

  Fredrickson B. L. Losada M. F. (2005). Positive affect and the complex dynamics of human flourishing. Am. Psychol.60, 678–686. 10.1037/0003-066X.60.7.678

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 35

  Gillespie B. M. Chaboyer W. Wallis M. Grimbeek P. (2007). Resilience in the operating room: developing and testing of a resilience model. J. Adv. Nurs.59, 427–438. 10.1111/j.1365-2648.2007.04340.x

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 36

  Glendon A. I. Clarke S. G. McKenna E. (2006). Humansafety and Risk Management, 2nd Edn.Boca Raton, FL: CRC Press.

  • Google Scholar

• 37

  Gontar P. Hoermann H. J. (2015). Interrater reliability at the top end: measures of pilots' nontechnical performance. Int. J. Aviat. Psychol.25, 171–190. 10.1080/10508414.2015.1162636

  • CrossRef
  • Google Scholar

• 38

  Grice R. L. Katz L. C. (2005). Cohesion in Military and Aviation Psychology: An Annotated Bibliography and Suggestions for U.S. Army Aviation. PsycEXTRA Dataset. 10.1037/e440342005-001

  • CrossRef
  • Google Scholar

• 39

  Gross J. J. John O. P. (2003). Individual differences in two emotion regulation processes: implications for affect, relationships, and well-being. J. Pers. Soc. Psychol.85, 348–362. 10.1037/0022-3514.85.2.348

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 40

  Gross J. J. Thompson R. A. (2007). Emotion regulation: conceptual foundations, in Handbook of Emotion Regulation, ed GrossJ. J. (New York, NY: Guilford Press), 3–24.

  • Google Scholar

• 41

  Harris K. R. Eccles D. W. Freeman C. Ward P. (2016). ‘Gun! Gun! Gun!': an exploration of law enforcement officers' decision-making and coping under stress during actual events. Ergonomics60:1112. 10.1080/00140139.2016.1260165

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 42

  Harrison Y. Horne J. A. (1999). One night of sleep loss impairs innovative thinking and flexible decision making. Organ. Behav. Hum. Decis. Process.78, 128–145. 10.1006/obhd.1999.2827

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 43

  Helmreich R. Butler R. Taggart W. Wilhelm J. (1997). The NASA/University of Texas/Federal Aviation Administration Line/LOS checklist: A Behavioral-Based Checklist for CRM Skills Assessment (Version 4.4)[Computer Software]. Austin, TX: NASA/University of Texas/Federal Aviation Administration Aerospace Group.

  • Google Scholar

• 44

  Hofmann D. Morgenson F. (2004). The role of leadership in safety, in The Psychology of Workplace Safety, eds BarlingJ.FroneM. (Washington, DC: APA Books).

  • Google Scholar

• 45

  Hörmann H. J. (2001). Cultural variation of perceptions of crew behaviour in multi-pilot aircraft. Trav. Hum.64, 247–268. 10.3917/th.643.0247

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 46

  Houghton S. M. Simon M. Aquino K. Goldberg C. B. (2000). No safety in numbers: persistence of biases and their effects on team risk perception and team decision making. Group Organ. Manag.25, 325–353. 10.1177/1059601100254002

  • CrossRef
  • Google Scholar

• 47

  Hu L. T. Bentler P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct. Equation Model. Multidiscipl. J.6, 1–55. 10.1080/10705519909540118

  • CrossRef
  • Google Scholar

• 48

  Judge T. A. Bono J. E. (2000). Five-factor model of personality and transformational leadership. J. Appl. Psychol.85:751. 10.1037/0021-9010.85.5.751

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 49

  Kodate N. Ross A. Anderson J. E. Flin R. (2012). Non-Technical Skills (NTS) forenhancing patient safety: achievementsand future directions. Jpn. J. Qual. Saf. Healthc.7, 360–370.

  • Google Scholar

• 50

  Lauria M. J. Ghobrial M. K. Hicks C. M. (2019). Force of habit: developing situation awareness in critical care transport. Air Med. J.38, 45–50. 10.1016/j.amj.2018.09.007

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 51

  Mauss I. B. Bunge S. A. Gross J. J. (2007). Automatic emotion regulation. Soc. Personal. Psychol. Compass1, 146–167. 10.1111/j.1751-9004.2007.00005.x

  • CrossRef
  • Google Scholar

• 52

  Mitchell L. Flin R. (2008). Non-technical skills of the operating theatre scrub nurse: literature review. J. Adv. Nurs.63, 15–24. 10.1111/j.1365-2648.2008.04695.x

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 53

  Nergård V. Hatlevik O. E. Martinussen M. Lervåg A. (2011). An airman's personal attitude: pilots' point of view. Aviation15, 101–111. 10.3846/16487788.2011.651789

  • CrossRef
  • Google Scholar

• 54

  Nullmeyer R. T. Spiker V. A. (2003). The importance of crew resource management behaviors in mission performance: implications for training evaluation. Mil. Psychol.15, 77–96. 10.1207/S15327876MP1501_06

  • CrossRef
  • Google Scholar

• 55

  O'Connor P. Hörmann H. J. Flin R. Lodge M. Goeters K. M. JARTEL Group T. (2002). Developing a method for evaluating crew resource management skills: a European perspective. Int. J. Aviat. Psychol.12, 263–285. 10.1207/S15327108IJAP1203_5

  • CrossRef
  • Google Scholar

• 56

  Orasanu J. (1995). Training for aviation decision making: the naturalistic decision making perspective, in Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 39, No. 20 (Sage CA: Los Angeles, CA: SAGE Publications), 1258–1262.

  • Google Scholar

• 57

  Orasanu J. Fischer U. (1997). Finding decisions in natural environments: the view from the cockpit. Naturalistic Decis. Making3, 343–357.

  • Google Scholar

• 58

  Pietrantoni L. Prati G. (2008). Resilience among first responders. Afr. Health Sci.8, 14–20.

  • Pubmed Abstract
  • Google Scholar

• 59

  Prati G. Pietrantoni L. Cicognani E. (2010). Self-efficacy moderates the relationship between stress appraisal and quality of life among rescue workers. Anxiety Stress Coping23, 463–470. 10.1080/10615800903431699

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 60

  Ray J. J. (1983). Reviving the problem of acquiescent response bias. J. Soc. Psychol.121, 81–96. 10.1080/00224545.1983.9924470

  • CrossRef
  • Google Scholar

• 61

  Ray J. J. Pratt G. J. (1979). Is the influence of acquiescence on “catchphrase” type attitude scale items not so mythical after all?Aust. J. Psychol.31, 73–78.

  • Google Scholar

• 62

  Robertson E. R. Hadi M. Morgan L. J. Pickering S. P. Collins G. New S. et al . (2014). Oxford NOTECHS II: a modified theatre team non-technical skills scoring system. PLoS ONE9:90320. 10.1371/journal.pone.0090320

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 63

  Ruff-Stahl H. K. Vogel D. Dmoch N. Krause A. Strobl A. Farsch D. et al . (2016). Measuring CRM aptitude: is NOTECHS a suitable tool for pilot selection?Int. J. Aviat. Aeronaut. Aerosp.3, 4–13. 10.15394/ijaaa.2016.1128

  • CrossRef
  • Google Scholar

• 64

  Russell D. W. (2002). In search of underlying dimensions: the use (and abuse) of factor analysis in personality and social psychology bulletin. Personal. Soc. Psychol. Bull.28, 1629–1646. 10.1177/014616702237645

  • CrossRef
  • Google Scholar

• 65

  Salas E. Burke C. S. Samman S. N. (2001). Understanding command and control teams operating in complex environments. Inf. Knowl. Syst. Manage.2, 311–323.

  • Google Scholar

• 66

  Saurin T. A. Wachs P. Henriqson É. (2013). RETRACTED: identification of non-technical skills from the resilience engineering perspective: a case study of an electricity distributor. Saf. Sci.1, 37–48. 10.1016/j.ssci.2012.06.010

  • CrossRef
  • Google Scholar

• 67

  Setti I. Lourel M. Argentero P. (2016). The role of affective commitment and perceived social support in protecting emergency workers against burnout and vicarious traumatization. Traumatology (Tallahass. Fla).22, 261–270. 10.1037/trm0000072

  • CrossRef
  • Google Scholar

• 68

  Sexton J. B. Thomas E. J. Helmreich R. L. (2000). Error, stress, and teamwork in medicine and aviation: cross sectional surveys. Br. Med. J.320, 745–749. 10.1136/bmj.320.7237.745

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 69

  Shappell S. Detwiler C. Holcomb K. Hackworth C. Boquet A. Wiegmann D. A. (2017). Human error and commercial aviation accidents: an analysis using the human factors analysis and classification system. Human Error Aviat.49, 73–88. 10.4324/9781315092898-5

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 70

  Smith A. Roberts K. (2003). Interventions for post-traumatic stress disorder and psychological distress in emergency ambulance personnel: a review of the literature. Emerg. Med. J.20, 75–78. 10.1136/emj.20.1.75

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 71

  Sourial N. Wolfson C. Zhu B. Quail J. Fletcher J. Karunananthan S. et al . (2010). Correspondence analysis is a useful tool to uncover the relationships among categorical variables. J Clin Epidemiol.63, 638–646. 10.1016/j.jclinepi.2009.08.008

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 72

  Spector P. (1992). Summated Rating Scale Construction. Newbury Park, CA: Sage.

  • Google Scholar

• 73

  Stamm B. H. (2005). The Professional Quality of Life Scale: Compassion Satisfaction, Burnout, and Compassion Fatigue/Secondary Trauma Scales. Latherville, MD: Sidran Press.

  • Google Scholar

• 74

  Steinhardt M. Dolbier C. (2008). Evaluation of a resilience intervention to enhance coping strategies and protective factors and decrease symptomatology. J. Am. Coll. Health56, 445–453. 10.3200/JACH.56.44.445-454

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 75

  Thomas M. J. (2004). Predictors of threat and error management: identification of core nontechnical skills and implications for training systems design. Int. J. Aviat. Psychol.14, 207–231. 10.1207/s15327108ijap1402_6

  • CrossRef
  • Google Scholar

• 76

  Van Avermaete J. A. G. Kruijsen E. (eds.). (1998). The Evaluation of Non-Technical Skills of Multipilot Aircrew in Relation to the JAR-FCL Requirements (Project rep.: CR-98443). Amsterdam: NLR.

  • Google Scholar

• 77

  Weick K. E. Sutcliffe K. M. Obstfeld D. (2008). Organizing for high reliability: processes of collective mindfulness. Crisis Manage.3, 81–123.

  • Google Scholar

• 78

  Worthington R. L. Whittaker T. A. (2006). Scale development research: a content analysis and recommendations for best practices. Couns. Psychol.34, 806–838. 10.1177/0011000006288127

  • CrossRef
  • Google Scholar

• 79

  Yukl G. (2005). Leadership in Organisation. Upper Saddle River, NJ: Prentice-Hall International.

  • Google Scholar

• 80

  Zaccaro S. J. Rittman A. L. Marks M. A. (2001). Team leadership. Leadersh. Q.12, 451–483. 10.1016/S1048-9843(01)00093-5

  • CrossRef
  • Google Scholar