Phish Derby: Shoring the Human Shield Through Gamified Phishing Attacks

Introduction

Despite significant and increasing organizational spending on cybersecurity technologies and associated efforts, successful threats abound. For example, while organizational leaders are expected to spend more than $150 billion US on cyber and related technologies and services in 2021 (Gartner, 2021), threats related to remote work, cloud adoption, healthcare, and other domains continue to flourish (CheckPoint, 2021). Thus, cyber “solutions” are not always what they appear, and throwing technology at the cyber problem will create rather than solve problems (Schneier, 2015).

An important realization has been that organizational cybersecurity efforts depend largely on the employees who reside within organizational walls. These individuals are central to the effectiveness of organizational actions to protect sensitive assets, and research has shown that they can be detrimental (e.g., sabotage and computer abuse) (Straub and Nance, 1990; Willison and Warkentin, 2013) as well as beneficial (e.g., protective motivated behaviors, precaution taking) (Boss et al., 2009; Posey et al., 2013; Burns et al., 2019) to their employers. Employee actions thus range from accidental errors to malicious acts of sabotage on the negative side and forced compliance to security championing on the positive side.

A specific, significant context where employees continue to affect their organizations is how these individuals respond to phishing attempts that come through corporate email systems. Online phishing is a common attack vector used by external actors to penetrate organizational networks, steal employee credentials, and commit other forms of harm. In fact, more than 90% of malicious software is delivered by email, with personalized phishing attacks (i.e., spear phishing) being the entry gate (Purplesec, 2021). Because of this massive potential for injury, organizations have focused on how best to reduce the risk stemming from employees who encounter and fall victim to phishing attacks. These efforts rely largely on simulated phishing campaigns wherein employees encounter emails that mimic real phishing attacks, and the resulting failure metrics are used to examine progress within an employee base.

Notwithstanding the importance of assessing the number of employees who fall victim to these mock attacks, it is important to note how employees can also have positive reactions to phishing attacks—reactions that alert organizational representatives to the potential threat (Canham et al., 2021). It is unfortunate that many of these positive reactions are often overshadowed by the failures (i.e., successful mock attacks) despite serving as an important warning signal or beacon to the organization that something could be wrong. At a time when cybersecurity remains a top priority for leadership, but funding for the requisite resources is unable to keep pace with the ever-evolving threat landscape, it would serve organizations’ interests to also provide significant focus on the positive spectrum of employees’ cyber behavior.

To increase our understanding of this phenomenon, which we refer to as the “protective steward phenomenon,” we gamified a series of simulated phishing campaigns to see how such an alteration would influence employee cyber behaviors. Gamification refers to the “use of computer games and features of games for non-game purposes” (Fleming et al., 2020, p. 2). These campaigns, collectively called a “Phish Derby” competition, allowed employees to compete against one another in their efforts to detect and create an alert when encountering simulated phishing emails.

Given the evidence showing how the gamification of learning-based exercises can increase participants’ engagement and overall learning (Marín et al., 2018; Groening and Binnewies, 2019), we explored whether and how gamification could be used to foster positive employee reactions and experiences with a form of training (i.e., simulated phishing campaigns) seen by some workers as a source for decreased productivity and increased levels of boredom, anxiety, stress, embarrassment, and even ostracism (Conley, 2021; Emm, 2021; Ferrell, 2021). At the very least, gamification could prove to increase user attentiveness during these activities, which could then possibly translate to better performance during real attacks. In addition, not only was correct identification of phishing attempts important, but given the need for organizations to be able to respond to threats as quickly as possible, employee response times (i.e., time difference between phish receipt and employee alert) were also tabulated. Therefore, our experiment with the Phish Derby and its associated results provides a more holistic view to positive employee behaviors regarding one of the most harmful attack vectors used against modern organizations—online phishing attacks.

Background on Phishing

Since online phishing and its variants, like business email compromise, continue to be successful attack vectors, especially during the COVID-19 pandemic when cyberattacks increased by 600% (Purplesec, 2021), it is no wonder that substantial scholarly attention has been given to phishing attack detection. Unfortunately, when compared to automated, technical-detection solutions, research on human-based detection efforts is more limited and focuses on how training techniques can be leveraged to enhance detection capabilities (Khonji et al., 2013; Zielinska et al., 2014; Wash and Cooper, 2018). Fortunately, research shows some promise in increasing human-detection capabilities via phishing training embedded directly into corporate email systems (Kumaraguru et al., 2007), but even then, employees might not even fully read or pay attention to the training (Caputo et al., 2013).

Complementing the research on human-detection capabilities, recent efforts have drawn attention to all potential employee behavioral responses to email phishing attacks (Canham et al., 2021). By analyzing the responses of more than 6,000 employees at a large U.S. university over the course of 20 phishing training campaigns and 19 months, this effort demonstrated that a small subset of users (6% of the total population of users) were responsible for repeated phishing training failures (i.e., “Repeat Clickers”) and a larger subset (33%) of users (“Protective Stewards”) were responsible for reporting these emails to the Information Security Office. Thus, more employees alert their organizations about potential attacks than succumb to phishing attacks. Unfortunately, this positive-oriented and more sizable employee subpopulation has received relatively limited attention when compared to its smaller and more detrimental counterpart—a concerning trend when so many information security offices are struggling to handle day-to-day operations with limited resources.

One potential way to continue to increase employees’ 1) ability to detect and 2) motivation to report phishing emails might be through the gamification of the mock phishing campaign experience. The addition of gaming elements to non-gaming situations in this and other cyber-related contexts has been explored (Francia et al., 2014; Gjertsen et al., 2017; Emm, 2021; Khando et al., 2021). For example, gamification has demonstrated promise in the education of normal users regarding password security (Scholefield and Shepherd, 2019), and gamified systems can increase motivation to comply with security policy and reduce mock phishing failures, significantly outperforming training provided via email (Silic and Lowry, 2020). Different variations of gamification capabilities have also been examined in the context of employees’ online self-disclosure (Dincelli and Chengalur-Smith, 2020) and corruption behaviors (Baxter et al., 2017). In addition, previous work on gamified systems have relied on both monetary and non-monetary rewards to incentivize participants (Lewis et al., 2016; Karac and Stabauer, 2017; Meixner et al., 2020; Ueyama et al., 2014). It is evident that gamification can be a useful tool in educating and motivating individuals in a variety of contexts.

Given this opportunity, we extended previous research efforts by exploring the factors surrounding employees who actively choose to alert their information security office when they suspect a rogue email in their inbox. In addition, we wanted to determine if employee response times could be incentivized through such gamification. Akin to the field of positive psychology (Seligman and Csikszentmihalyi, 2014), our goal here is to help motivate positive behaviors rather than correct negative actions and understand whether gamification is a fruitful avenue for this objective.

Possible Employee-Performance Factors

To better understand potential variance in employee performance during our Phish Derby, we relied on concepts found in two theoretical foundations. The first foundation is commonly referred to as the “Big Five” personality traits. These traits include extraversion, emotional stability, agreeableness, conscientiousness, and openness to experience (Norman, 1963; McCrae and Costa, 1987). Because so much has been written on these traits, we briefly discuss them here.

Openness is a trait aligned with intellectual curiosity, creativity, and a preference for novelty. Individuals high in conscientiousness tend to be organized, self-disciplined, and have a need for achievement, whereas individuals high in extraversion tend to be socially outgoing, energetic, and seek stimulation. Agreeableness refers to those who tend to be cooperative, helpful, and well-tempered. Finally, individuals exhibiting neuroticism tend to be prone to anxiety and stress, easily experience unpleasant emotions, and be insecure.

The Big 5 has been examined as an influential factor in studies of information security previously (Pattinson et al., 2012; Uebelacker and Quiel, 2014; Halevi et al., 2015; Welk et al., 2015; Lawson et al., 2017; Sudzina and Pavlicek, 2017); however, how these traits influence phishing susceptibility is not always obvious. For example, people high in conscientiousness might be less susceptible to phishing attempts (Lawson et al., 2017), but they might also be leveraged to help an attack become more likely to succeed (Halevi et al., 2015). Regarding phishing vulnerability, research on individuals high in extraversion has shown mixed results. Two studies have shown increased susceptibility to phishing (Welk et al., 2015; Lawson et al., 2017), while another study (Pattinson et al., 2012) showed a better ability to detect phishing emails. Despite these differences, we believe that one or more of the Big 5 components could play an important role in understanding potential differences in our participants’ performance, especially given our unique context of gamification and the inclusion of relatively difficult-to-detect phishing emails in our Phish Derby.

Goal orientation theory (GOT) serves as our second theoretical foundation. This theory explains the reasons why individuals are driven to certain outcomes in achievement-focused tasks. Generally, individuals approach and engage in achievement tasks because they desire to 1) learn (i.e., learning), 2) prove their performance abilities (i.e., prove performance), and/or 3) avoid negative judgments and perceptions of inferiority (i.e., avoid performance) (Brett et al., 1999; Kaplan and Maehr, 2007).

GOT has been used in examining individuals in numerous achievement-focused scenarios. For example, goal orientation concepts have been linked to academic performance, even mediating the relationship between intrinsic motivation and performance (Cerasoli and Ford, 2014). Learning orientations have been linked to expatriates’ academic and social adjustment outcomes (Gong and Fan, 2006), and both learning- and performance-orientation goals have shown relationships with team adaptability when facing adversity (Porter et al., 2010). Finally, research has shown that trait-forms of goal orientation explain employee job performance above and beyond cognitive ability and even the personality variables mentioned above (Payne et al., 2007). Determining whether and how these goals drive Phish Derby performance in general, and in comparison, with the “Big Five” personality traits should prove fruitful.

Gamified Approach

Gamification was achieved via our “Phish Derby” by having participants prove their ability to spot phishing attacks and earn points based upon the number of attacks they successfully reported, as well as how quickly those alerts were issued. To help increase the amount of variance in user responses, the research team utilized very difficult simulated phishing attacks. The KnowBe4 platform was used for the Phish Derby. Participants received monetary prizes (i.e., Amazon gift cards) at the end of the competition, and they also knew that the research team would debrief all who were interested in an online seminar. Potential participants were notified of the Phish Derby a week prior to its beginning through email communication. Participation in the Phish Derby was voluntary, and competitors were instructed that because this was a competition, the simulated phishing emails that they received would be more difficult than the regular training emails that they had received in the past. Information Security Office staff informed volunteers that performance during this Phish Derby would not negatively impact their training requirements (e.g., being required to complete additional training if they fell for a simulated phishing message sent as part of this Phish Derby).

A total of six simulated phishing email templates were utilized for the Phish Derby competition. These six were titled “LinkedIn–People Are Looking at your Profile,” “UPS Label Delivery,” “Test of the Notification System,” “Sarah Butler Sent You a Secure File,” “Knightro’s Halloween Costume,” and “COVID-19 Reported Cases in Your Area.” The “LinkedIn–People Are Looking at your Profile” template purported to notify the recipient that their profile had been viewed and included a hyperlink that falsely claimed to redirect to LinkedIn. “The UPS Label Delivery” template used the pretext of a UPS delivery notification with a hyperlink made that appeared to redirect to UPS. The “Test of the Notification System” template claimed to be a notification test and requested the receiver verify their contact information through a deceptive hyperlink. The “Sarah Butler Sent You a Secure File” template appeared to be a shared document from Sarah Butler, a fictitious university employee. The “Knightro’s Halloween Costume” used the pretext of an invitation to enroll in a university costume contest. The final template, “COVID-19 Reported Cases in Your Area,” used the pretext of discovering reported COVID-19 cases in the area through a deceptive linked portal.

Developing an objective metric of email difficulty is a challenge that the NIST Phish Scale seeks to address. This difficulty scale considers two factors in operationalizing phishing email difficulty: first, the number of phishing “cues,” and second, the email premise alignment with user role (Steves et al., 2020). These factors were derived from previous empirical work demonstrating their central role in phishing email detection (Greene et al., 2018). Cues refer to inconsistencies within, or characteristics of, the message that may alert the target that the message might be a phishing attempt. Examples of cues include spelling and grammatical errors, technical indicators (e.g., a hyperlink mismatch), odd language, and the use of time pressure. Premise alignment refers to the degree to which the message aligns with the recipient’s job role and alludes to the user’s context in evaluating the message. Prior research demonstrates the more highly the message premise aligns with the target’s job role (i.e., a past-due notice sent to the accounting department, or a resume sent to a human-resources department), the less likely people are to notice detection cues in the message (Greene et al., 2018). We applied the NIST Phish Scale of email difficulty to each of the six simulated phishing email templates that we employed in the Phish Derby, and the difficulty ratings for each template are summarized in Table 1.

TABLE 1

TABLE 1. Email template phish scale difficulty with click and report rates.

Experimental Method

In early October 2020, participants completed an initial survey that covered demographic and model variables used in our analyses. A total of 116 individuals took part in the initial survey, but attention-check items indicated that only 101 individuals should remain in the study. These individuals then received six simulated phishing emails to their work email address throughout the remainder of October. Participants explicitly agreed to not use any means (technical or otherwise) that would prohibit a fair competition. Any evidence suggesting use of such methods would result in immediate disqualification from the competition. Participants were instructed to report emails as potential phishing attacks by using an embedded “Phish Alert” button as provided by KnowBe4 or by forwarding the email as an attachment to the Security Incident Response Team (SIRT). All interaction with phishing emails (e.g., email receipt, reporting) took place in the 8:00 am—5:00 pm (participants’ local time) window. The mean age of our sample was 44.4 years, with 40% identifying as female. Twenty-five percent of our sample was in administrative positions, and 10% was in an IT/IS role.

Our research team collected the number of phishing alerts/reports received from participants as well as the timeliness with which those alerts/reports were received. All participants began the competition with 10 “Derby Bucks.” For every simulated phishing email not alerted within 4 h of receiving the email, 1 Derby Buck was subtracted from their total. If the competitor reported the email but only after falling victim to the phishing email, $0.75 was subtracted from their total. This option was available to highlight the fact that while succumbing to a phishing attack is a negative event, it is still of benefit to the organization to report it as soon as possible. No Derby Bucks were removed when participants accurately alerted SIRT within 4 h of receiving the phishing email. At the end of the Phish Derby, competitors received the following rewards based on the total amount of Derby Bucks remaining in their possession:

Competitors with at least 6.00 Derby Bucks were awarded a $5.00 Amazon gift card.

Competitors with at least 7.50 Derby Bucks were awarded a $7.00 Amazon gift card.

Competitors with at least 8.50 Derby Bucks were awarded a $10.00 Amazon gift card.

The dependent variable in our model was a performance score normalized for report timeliness against the average response time for each phishing campaign. This represents the importance of timeliness in reporting potential threats to SIRT. For example, if two participants correctly identified all six phishing campaigns, their initial performance score would equal 6.00, but because their average response times differed, the final performance scores would be adjusted relative to those response times. Thus, instead of both participants receiving the same 6.00 performance score, the faster responder might receive a 5.93 and the slower one a 5.78. Thus, the faster the response, the higher the score (assuming the same number of phishing campaigns was identified). This normalized score was not used in the assignment of Derby Bucks mentioned above due to university institutional review board (IRB) stipulations. Participants were not informed of their performance relative to other competitors either during or after the Phish Derby; however, they were informed of the overall Phish Derby detection and reporting performance after the competition had concluded.

Measures

In addition to the demographic variables, we used previously published and validated scales to capture our constructs related to our two theoretical foundations. The Big 5 Personality dimensions were assessed using the IPIP–NEO–60 scale (Maples-Keller et al., 2019). This scale employs 60 items to infer an individual’s placement along each of the five dimensions on the Five-Factor Personality Scale. Learning (5 items), prove performance (4 items), and avoid performance (4 items)—concepts from GOT—were measured using the 13-item goal orientation scale (Brett and VandeWalle, 1999).

Results

We performed a hierarchical regression analysis where we focused on participants’ demographic variables first and then assessed components related to the Big Five personality traits and GOT. Given that we operationalized these components with previously validated measures, each exhibited adequate internal consistency metrics (α ≥ 0.70). While our sample size is relatively small (n = 101), our statistical power (1-β > 0.99) did not prohibit us from discussing non-significant relationships. What is rather interesting is that such a relatively simple model produced a rather large amount of variance in participants’ performance (i.e., R2 = 52.0%). The mode for phishing emails reported across all participants was 4, and the average response times (in minutes) for the email templates were 97.0, 66.4, 121.3, 189.0, 37.8, and 77.2 for templates 1–6, respectively. The mean normalized performance score (possible range 0–6) was 2.55 during the Phish Derby. Table 2 displays the correlations among our variables, and Table 3 displays our statistical results.

TABLE 2

TABLE 2. Interconstruct correlations.

TABLE 3

TABLE 3. Results from hierarchical regression of normalized performance on demographics and predictors.

In addition to the large R² value, we see several notable relationships between demographic variables and Phish Derby performance. First, participants’ exposure to, and performance during, previous simulated phishing campaigns matter as demonstrated by the significance of the percentage of reports relative to phishing emails received by the employees before entering the Phish Derby. We also see that age becomes a significant variable in our analysis, and the positive beta indicates that older participants performed better in the Phish Derby than did their younger counterparts. On the other hand, the years of education had the opposite effect on performance; more years of education led to poorer performance. Finally, in our assessment of whether participants used PCs or Macs at work, or a mix of both, we found that individuals who use a mix performed worse than those using a single platform.

Outside the demographic variables, we see variables of significance within our two theoretical foundations. First, within the Big Five personality dimensions, two personality dimensions influenced performance: extraversion and agreeableness. Extraversion was negatively associated with phishing email reporting performance (β = −0.181, p = 0.066). This finding is interesting because previous research on those high on the extraversion dimension has been mixed. At least three studies have found increased susceptibility to phishing in those higher in extraversion (Welk et al., 2015; Lawson et al., 2017; Anawar et al., 2019), while another study (Pattinson et al., 2012) showed a better ability to detect phishing emails. Messages that utilize likability as a social influence principle have also been found to be more persuasive to people high in extraversion (Alkış and Temizel, 2015). This aspect of personality needs to be examined more because more extraverted individuals might make more attractive targets for criminals, by virtue of having more connections and thus having more connections to target if their account is compromised. Agreeableness has been positively associated with self-reported cybersecurity behaviors in previous research (McCormac et al., 2017; Shappie et al., 2020); however, we found that higher agreeableness was also associated with poorer phish reporting performance in the Phish Derby (β = −0.261, p = 0.016).

From a goal orientation perspective, we found that participants whose goal was overall learning performed significantly worse than those who identified their goal as performing well in the competition. In other words, those who are trying to better themselves at identifying phishing attacks performed at a lower level than those who cared little about overall learning as their main goal. Goals of “performance proving” and “performance to avoid disapproval” did not exhibit a significant relationship with overall performance.

Finally, participants’ intention to do well in the Phish Derby was close to becoming a significant component in the model but ultimately was not. In the case of the gamified Phish Derby, intentions were not significantly related to performance—perhaps a case of the “knowing-doing gap” (Workman et al., 2008).

Regarding performance on the various templates, most had relatively low click-rates, with the exception of the UPS Label Delivery email template (18% click-rate). This template received the second highest reporting rate (69%) suggesting that it was among the more interactive templates of the Phish Derby. These results are summarized in Tables 4, 5. While the “Secure File Delivery,” “COVID-19 in Your Area,” and “Test of the Notification System” were also highly reported, the “Secure File Delivery” received zero clicks. This may have been due to similar simulated phishing emails having been previously used in training. No significant interactions (in click-rates or report-rates) between the email template types and job role were observed. Overall, the click-rates were relatively low compared to previously observed campaigns (Canham et al., 2021). This might have been the result of a self-selected sample of participants with knowledge of their contest participation.

TABLE 4

TABLE 4. Click-rates by email template and job role.

TABLE 5

TABLE 5. Report-rates by email template and job role.

Phish Derby Participant Comments

In addition to our quantitative assessment, we wanted to determine whether the participants viewed the Phish Derby experience as a success. Fortunately, the comments from participants during the debrief regarding the Phish Derby were overwhelmingly positive, and a sample of them includes the following direct quotes:

“I enjoyed taking part in the phishing derby–seriously a great idea!”

“It was kind of scary though... I would usually delete, but during this I felt like maybe I should report more.”

“After I got caught on the first one, I was much more alert for the rest”

“This was a great way to heighten awareness and learn about different kinds of things to watch for.”

“I thought I was already aware so kinda wanted to test myself.”

“More cautious now.”

“When can we do it again?”

“This was a perfect strategy: educational and fun!”

“It was a great learning experience. Thank you.”

“(I) would love to see this again thank you”

“I appreciate the Derby tests and will stay vigilant!”

Discussion

We aimed to assess whether the gamification of mock phishing exercises would be successful and whether key factors explaining participant performance would emerge. Further, our goal was to highlight employees’ positive, phish-reporting behaviors rather than focus on failure (i.e., click) rates. Our results suggest that gamification can be a useful, interesting, and perhaps even exciting approach to employ in mock phishing exercises—exercises that are usually thought to be intrusive or a waste of time by many employees. Moreover, we were able to determine considerable differences in Phish Derby performance, indicating that some employees are or could become star performers or champions for organizations’ security teams in the quest to quickly identify phishing attempts once they clear technical filters.

From a theoretical standpoint, it was interesting to discover that both extraversion and agreeableness exhibited negative relationships with Phish Derby performance. In phishing-susceptibility research, findings relative to the extraversion personality trait have been somewhat mixed, but at least three studies have found that extraverts exhibit increased susceptibility to phishing emails (Welk et al., 2015; Lawson et al., 2017; Anawar et al., 2019). Our findings suggest that extraverts perform more poorly on the positive-oriented behaviors of reporting as well. On the other hand, agreeableness has shown positive associations with self-reported cybersecurity behaviors in previous research (McCormac et al., 2017; Shappie et al., 2020), which is at odds with our results. We do not fully understand why such is the case from our Phish Derby exercise, but it is possible that individuals who desire to be helpful also do not like to have external pressures to do so. Perhaps the additional pressure of the research team tracking and rewarding response times backfired with participants high in agreeableness. This aspect of our findings and rationale deserves future attention.

An additional finding that surprised us was that the first of the three goal orientations (i.e., learning orientation) was negatively related to participants’ performance. In fact, learning-oriented individuals exhibited higher average response times, meaning that they were slower to report suspected phishing attacks to the organizational representatives. Like those high in agreeableness discussed above, it is likely that learning-oriented individuals wish to devote a reasonable amount of time to delve into an issue, and quick-reaction contexts do not bode well for these people. Conversely, these individuals are likely those participants who attended and actively participated in the Phish Derby debrief to learn of overall response rates and to discuss the possible cues within each of the mock phishing templates. Unfortunately, we did not assess this possibility.

Another surprising finding was that more education was related to poorer performance in the Phish Derby. On the one hand, it could be that those with more education perceived themselves to be more capable than those with less education at identifying phishing threats, thereby presenting a situation of overconfidence. On the other hand, a potential explanation for this finding may be the more highly educated participants might also have had a higher workload and/or received substantially more emails on a daily basis than the less educated participants. This finding deserves more investigation in future studies.

We also found that employees who alternate between PC and Mac systems in their daily job tasks performed worse than those using a single platform. Our rationale leans toward multi-platform users having an increased cognitive load due to the switching between platforms, icon sets, and other platform-dependent idiosyncrasies. If they are subconsciously expending mental energy on the effective utilization of the various platforms, they may not have the same level of energy available to recognize more difficult phishing threats. Of course, this assertion deserves more attention and could make for a very interesting experiment.

Several practical implications also emerged from our Phish Derby exercise. First, some CISOs might wonder whether exposing employees to simulated phishing campaigns works, and whether they should pay attention to individual metrics. In the case of gamified competitions, the answer is “yes.” Competitors’ previous reporting behaviors as a percentage of the total number of phishing campaigns they previously received (made available in the KnowBe4 platform) significantly related to their overall performance (i.e., correct identification and timeliness of report) during the Phish Derby.

Second, anecdotal evidence suggests that some CISOs and other security administrators feel unable to get and maintain older employees’ interest and attention regarding security matters. We did not find this to be the case during the Phish Derby. In fact, we found that the more aged employees performed better than the younger employees. Perhaps the friendly competition provided by the Phish Derby and the focus on protective reporting—a positive focus—appealed more to older employees than would a focus on thwarting phishing failures—a negative focus. We believe that research in the positive psychology movement (Seligman and Csikszentmihalyi, 2014) could help provide additional rationale for why such is the case on this matter.

Third, the individuals who really desired to learn about how to improve their future performance on reporting phishing emails were the ones who performed worse in the Phish Derby. Those individuals who tend to join competitions to show themselves and others how good they are and those who try to do well in achievement tasks to avoid negative judgment did not perform differently than those who do not. Thus, a true learning goal orientation affects performance during gamified phishing competitions. This is not to say that these individuals should not be involved in gamified phishing exercises; rather, organizational leaders should aim to provide a meaningful and engaging debrief that is focused on the needs of these employees. These will be the individuals most interested in understanding the cues and contexts that made the phishing templates difficult to detect.

Finally, of the variables that failed to exhibit a significant relationship with Phish Derby performance, two of the most interesting ones were self-reported computer skills and perceived ability to detect phishing messages. These variables never approached statistical significance in our analyses; thus, they were not included in our findings table. But the lack of findings here indicates that individuals who believe that they can identify phishing emails better than others failed to perform any differently from those who did not. Such is the same with self-reported computer knowledge and skills.

Conclusion

We complemented the online phishing training exercises of employees at a large U.S. university with a month-long gamification experiment. The gamified experience focused on the positive reporting behaviors of participants rather than click rates alone. In addition, we assessed the speed with which participants reported the simulated phishing emails to the appropriate organizational representatives. As evidenced by the findings and the comments provided by Phish Derby participants during the debrief, we view the gamification effort a success. Moreover, despite low enrollment—partially due to the lack of extensive marketing channels between the Information Security office and employees—we believe this event provided the single-most positive experience between the security office and the university’s employees. We encourage CISOs who are looking to improve employee participation in phishing exercises to strongly consider adding gamification elements to their efforts, and we implore other researchers to explore gamification’s influence more fully in increasing and sustaining individuals’ motivation to serve as stewards of security.

Data Availability Statement

The raw data supporting the conclusion of this article will be made available by the authors, without undue reservation.

Ethics Statement

The studies involving human participants were reviewed and approved by the Institutional Review Board, University of Central Florida. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

MCa developed experimental materials, conducted data analysis and wrote portions of the introduction, method, results, discussion, and conclusion. CP developed experimental materials, conducted data analysis and wrote portions of the introduction, method, results, discussion, and conclusion. MCo developed experimental materials, ran the study protocol, and wrote portions of the introduction, method, results, discussion, and conclusion.

Funding

Funding for this research was made possible through the University of Central Florida’s Office of Research. Funding was also provided by the National Institute of Standards and Technology (NIST) under Financial Assistance Award Number: 60NANB20D189. The views and conclusion contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of NIST or the U.S. Government.

Conflict of Interest

Author MCa was employed by the company Beyond Layer Seven, LLC.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

The authors would like to thank the Information Security Office at the University of Central Florida for engaging in a collaborative partnership with faculty. Without such efforts, this research would not have been possible.

References

Alkış, N., and Temizel, T. T. (2015). The Impact of Individual Differences on Influence Strategies. Personal. Individual Differences 87, 147–152.

Google Scholar

Anawar, S., Kunasegaran, D. L., Mas’ud, M. Z., and Zakaria, N. A. (2019). Analysis of Phishing Susceptibility in a Workplace: a Big-Five Personality Perspectives. J. Eng. Sci. Technol. 14 (5), 2865–2882.

Google Scholar

Baxter, R. J., Holderness, D. K., and Wood, D. A. (2017). The Effects of Gamification on Corporate Compliance Training: A Partial Replication and Field Study of True Office Anti-corruption Training Programs. J. Forensic Account. Res. 2 (1), A20–A30. doi:10.2308/jfar-51725

CrossRef Full Text | Google Scholar

Boss, S. R., Kirsch, L. J., Angermeier, I., Shingler, R. A., and Boss, R. W. (2009). If Someone Is Watching, I'll Do what I'm Asked: Mandatoriness, Control, and Information Security. Eur. J. Inf. Syst. 18 (2), 151–164. doi:10.1057/ejis.2009.8

CrossRef Full Text | Google Scholar

Brett, J. F., and VandeWalle, D. (1999). Goal Orientation and Goal Content as Predictors of Performance in a Training Program. J. Appl. Psychol. 84 (6), 863–873. doi:10.1037/0021-9010.84.6.863

CrossRef Full Text | Google Scholar

Burns, A. J., Roberts, T. L., Posey, C., and Lowry, P. B. (2019). The Adaptive Roles of Positive and Negative Emotions in Organizational Insiders' Security-Based Precaution Taking. Inf. Syst. Res. 30 (4), 1228–1247. doi:10.1287/isre.2019.0860

CrossRef Full Text | Google Scholar

Canham, M., Posey, C., Strickland, D., and Constantino, M. (2021). Phishing for Long Tails: Examining Organizational Repeat Clickers and Protective Stewards. SAGE Open 11 (1), 2158244021990656. doi:10.1177/2158244021990656

CrossRef Full Text | Google Scholar

Caputo, D. D., Pfleeger, S. L., Freeman, J. D., and Johnson, M. E. (2013). Going Spear Phishing: Exploring Embedded Training and Awareness. IEEE Security & Privacy 12 (1), 28–38.

Google Scholar

Cerasoli, C. P., and Ford, M. T. (2014). Intrinsic Motivation, Performance, and the Mediating Role of Mastery Goal Orientation: A Test of Self-Determination Theory. J. Psychol. 148 (3), 267–286. doi:10.1080/00223980.2013.783778

CrossRef Full Text | Google Scholar

CheckPoint (2021). Biggest Cyber Security Challenges in 2021. Retrieved from https://www.checkpoint.com/cyber-hub/cyber-security/what-is-cybersecurity/biggest-cyber-security-challenges-in-2021/.

Google Scholar

Conley, C. (2021). Ethical Phishing –The Slippery Slope with Employee Deception. Retrieved from https://www.sans.org/blog/ethical-phishing-the-slippery-slope-with-employee-deception/.

Google Scholar

Dincelli, E., and Chengalur-Smith, I. (2020). Choose Your Own Training Adventure: Designing a Gamified SETA Artefact for Improving Information Security and Privacy through Interactive Storytelling. Eur. J. Inf. Syst. 29 (6), 669–687. doi:10.1080/0960085x.2020.1797546

CrossRef Full Text | Google Scholar

Emm, D. (2021). Gamification - Can it Be Applied to Security Awareness Training? Netw. Security 2021 (4), 16–18. doi:10.1016/s1353-4858(21)00040-4

CrossRef Full Text | Google Scholar

Ferrell, S. (2021). The Problem with Phishing Simulators. Retrieved from https://www.inky.com/blog/the-problem-with-phishing-simulators.

Google Scholar

Fleming, T., Sutcliffe, K., Lucassen, M., Pine, R., and Donkin, L. (2020). “Serious Games and Gamification in Clinical Psychology,” in Reference Module in Neuroscience and Bio Behavioral Psychology (Elsevier). doi:10.1016/b978-0-12-818697-8.00011-x

CrossRef Full Text | Google Scholar

Francia, G., Thornton, D., Trifas, M., and Bowden, T. (2014). “Gamification of Information Security Awareness Training,” in Emerging Trends in ICT Security (Elsevier), 85–97. doi:10.1016/b978-0-12-411474-6.00005-0

CrossRef Full Text | Google Scholar

Gartner (2021). Gartner Forecasts Worldwide Security and Risk Management Spending to Exceed $150 Billion in 2021. Retrieved from https://www.gartner.com/en/newsroom/press-releases/2021-05-17-gartner-forecasts-worldwide-security-and-risk-managem.

Google Scholar

Gjertsen, E. G. B., Gjære, E. A., Bartnes, M., and Flores, W. R. (2017). “Gamification of Information Security Awareness and Training,” in Paper Presented at the ICISSP (Setúbal, Portugal: SciTePress). doi:10.5220/0006128500590070

CrossRef Full Text | Google Scholar

Gong, Y., and Fan, J. (2006). Longitudinal Examination of the Role of Goal Orientation in Cross-Cultural Adjustment. J. Appl. Psychol. 91 (1), 176–184. doi:10.1037/0021-9010.91.1.176

CrossRef Full Text | Google Scholar

Greene, K. K., Steves, M., Theofanos, M., and Kostick, J. (2018). “User Context: an Explanatory Variable in Phishing Susceptibility,” in Paper Presented at the in Proc. 2018 Workshop Usable Security (NY, United States: Association for Computing Machinery). doi:10.14722/usec.2018.23016

CrossRef Full Text | Google Scholar

Groening, C., and Binnewies, C. (2019). "Achievement Unlocked!" - the Impact of Digital Achievements as a Gamification Element on Motivation and Performance. Comput. Hum. Behav. 97, 151–166. doi:10.1016/j.chb.2019.02.026

CrossRef Full Text | Google Scholar

Halevi, T., Memon, N., and Nov, O. (2015). Spear-phishing in the Wild: A Real-World Study of Personality, Phishing Self-Efficacy and Vulnerability to Spear-Phishing Attacks. Rochester, NY: SSRN.

Google Scholar

Kaplan, A., and Maehr, M. L. (2007). The Contributions and Prospects of Goal Orientation Theory. Educ. Psychol. Rev. 19 (2), 141–184. doi:10.1007/s10648-006-9012-5

CrossRef Full Text | Google Scholar

Karac, J., and Stabauer, M. (2017). “Gamification in E-Commerce-A Survey Based on the Octalysis Framework,” in International Conference on HCI in Business, Government, and Organizations (Springer), 41–54.

Google Scholar

Khando, K., Gao, S., Islam, S. M., and Salman, A. (2021). Enhancing Employees Information Security Awareness in Private and Public Organisations: A Systematic Literature Review. Comput. Security 106, 102267. doi:10.1016/j.cose.2021.102267

CrossRef Full Text | Google Scholar

Khonji, M., Iraqi, Y., and Jones, A. (2013). Phishing Detection: a Literature Survey. IEEE Commun. Surv. Tutorials 15 (4), 2091–2121. doi:10.1109/surv.2013.032213.00009

CrossRef Full Text | Google Scholar

Kumaraguru, P., Rhee, Y., Acquisti, A., Cranor, L. F., Hong, J., and Nunge, E. (2007). “Protecting People from Phishing: the Design and Evaluation of an Embedded Training Email System,” in Paper Presented at the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (NY, United States: Association for Computing Machinery).

Google Scholar

Lawson, P., Zielinska, O., Pearson, C., and Mayhorn, C. B. (2017). “Interaction of Personality and Persuasion Tactics in Email Phishing Attacks,” in Paper Presented at the Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Santa Monica, CA: Human Factors and Ergonomics Society). doi:10.1177/1541931213601815

CrossRef Full Text | Google Scholar

Lewis, Z. H., Swartz, M. C., and Lyons, E. J. (2016). What's the point?: a Review of Reward Systems Implemented in Gamification Interventions. Games Health J. 5 (2), 93–99. doi:10.1089/g4h.2015.0078

PubMed Abstract | CrossRef Full Text | Google Scholar

Maples-Keller, J. L., Williamson, R. L., Sleep, C. E., Carter, N. T., Campbell, W. K., and Miller, J. D. (2019). Using Item Response Theory to Develop a 60-Item Representation of the NEO PI-R Using the International Personality Item Pool: Development of the IPIP-NEO-60. J. Pers Assess. 101 (1), 4–15. doi:10.1080/00223891.2017.1381968

PubMed Abstract | CrossRef Full Text | Google Scholar

Marín, B., Frez, J., Cruz-Lemus, J., and Genero, M. (2018). An Empirical Investigation on the Benefits of Gamification in Programming Courses. ACM Trans. Comput. Edu. (Toce) 19 (1), 1–22.

Google Scholar

McCormac, A., Zwaans, T., Parsons, K., Calic, D., Butavicius, M., and Pattinson, M. (2017). Individual Differences and Information Security Awareness. Comput. Hum. Behav. 69, 151–156. doi:10.1016/j.chb.2016.11.065

CrossRef Full Text | Google Scholar

McCrae, R. R., and Costa, P. T. (1987). Validation of the Five-Factor Model of Personality across Instruments and Observers. J. Pers Soc. Psychol. 52 (1), 81–90. doi:10.1037//0022-3514.52.1.81

CrossRef Full Text | Google Scholar

Meixner, C., Baumann, H., and Wollesen, B. (2020). Personality Traits, Gamification and Features to Develop an App to Reduce Physical Inactivity. Information 11 (7), 367. doi:10.3390/info11070367

CrossRef Full Text | Google Scholar

Norman, W. T. (1963). Toward an Adequate Taxonomy of Personality Attributes: Replicated Factors Structure in Peer Nomination Personality Ratings. J. Abnorm Soc. Psychol. 66 (6), 574–583. doi:10.1037/h0040291

CrossRef Full Text | Google Scholar

Pattinson, M., Jerram, C., Parsons, K., McCormac, A., and Butavicius, M. (2012). Why Do Some People Manage Phishing E‐mails Better than Others? Bingley, United Kingdom: Information Management & Computer Security.

Google Scholar

Payne, S. C., Youngcourt, S. S., and Beaubien, J. M. (2007). A Meta-Analytic Examination of the Goal Orientation Nomological Net. J. Appl. Psychol. 92 (1), 128–150. doi:10.1037/0021-9010.92.1.128

CrossRef Full Text | Google Scholar

Porter, C. O., Webb, J. W., and Gogus, C. I. (2010). When Goal Orientations Collide: Effects of Learning and Performance Orientation on Team Adaptability in Response to Workload Imbalance. J. Appl. Psychol. 95 (5), 935–943. doi:10.1037/a0019637

CrossRef Full Text | Google Scholar

Posey, C., Roberts, T. L., Roberts, T. L., Lowry, P. B., Bennett, R. J., and Courtney, J. F. (2013). Insiders' protection of Organizational Information Assets: Development of a Systematics-Based Taxonomy and Theory of Diversity for protection-motivated Behaviors. Misq 37, 1189–1210. doi:10.25300/misq/2013/37.4.09

CrossRef Full Text | Google Scholar

Purplesec (2021). 2021 Cyber Security Statistics: The Ultimate List of Stats, Data & Trends. Retrieved from https://purplesec.us/resources/cyber-security-statistics/.

Google Scholar

Schneier, B. (2015). Secrets and Lies: Digital Security in a Networked World. John Wiley & Sons.

Google Scholar

Scholefield, S., and Shepherd, L. A. (2019). “Gamification Techniques for Raising Cyber Security Awareness,” in Paper Presented at the International Conference on Human-Computer Interaction (Springer). doi:10.1007/978-3-030-22351-9_13

CrossRef Full Text | Google Scholar

Seligman, M. E. P., and Csikszentmihalyi, M. (2014). “Positive Psychology: An Introduction,” in Flow and the Foundations of Positive Psychology (Springer), 279–298. doi:10.1007/978-94-017-9088-8_18

CrossRef Full Text | Google Scholar

Shappie, A. T., Dawson, C. A., and Debb, S. M. (2020). Personality as a Predictor of Cybersecurity Behavior. Psychol. Popular Media 9 (4), 475–480. doi:10.1037/ppm0000247

CrossRef Full Text | Google Scholar

Silic, M., and Lowry, P. B. (2020). Using Design-Science Based Gamification to Improve Organizational Security Training and Compliance. J. Manag. Inf. Syst. 37 (1), 129–161. doi:10.1080/07421222.2019.1705512

CrossRef Full Text | Google Scholar

Steves, M., Greene, K., and Theofanos, M. (2020). Categorizing Human Phishing Difficulty: a Phish Scale. J. Cybersecurity 6 (1), tyaa009. doi:10.1093/cybsec/tyaa009

CrossRef Full Text | Google Scholar

Straub, D. W., and Nance, W. D. (1990). Discovering and Disciplining Computer Abuse in Organizations: A Field Study. MIS Q. 14, 45–60. doi:10.2307/249307

CrossRef Full Text | Google Scholar

Sudzina, F., and Pavlicek, A. (2017). Propensity to Click on Suspicious Links: Impact of Gender, of Age, and of Personality Traits. Bled, Austria: BLED.

Google Scholar

Uebelacker, S., and Quiel, S. (2014). “The Social Engineering Personality Framework,” in Paper Presented at the 2014 Workshop on Socio-Technical Aspects in Security and Trust (Vienna, Austria: IEEE). doi:10.1109/stast.2014.12

CrossRef Full Text | Google Scholar

Ueyama, Y., Tamai, M., Arakawa, Y., and Yasumoto, K. (2014). “Gamification-based Incentive Mechanism for Participatory Sensing,” in 2014 IEEE International Conference on Pervasive Computing and Communication Workshops (PERCOM WORKSHOPS) (IEEE), 98–103. doi:10.1109/percomw.2014.6815172

CrossRef Full Text | Google Scholar

Wash, R., and Cooper, M. M. (2018). “Who Provides Phishing Training? Facts, Stories, and People like Me,” in Paper Presented at the Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (New York: ACM).

Google Scholar

Welk, A. K., Hong, K. W., Zielinska, O. A., Tembe, R., Murphy-Hill, E., and Mayhorn, C. B. (2015). Will the "Phisher-Men" Reel You in? Int. J. Cyber Behav. Psychol. Learn. (Ijcbpl) 5 (4), 1–17. doi:10.4018/ijcbpl.2015100101

CrossRef Full Text | Google Scholar

Willison, R., Warkentin, M., and Warkentin, M. (2013). Beyond Deterrence: An Expanded View of Employee Computer Abuse. Misq 37, 1–20. doi:10.25300/misq/2013/37.1.01

CrossRef Full Text | Google Scholar

Workman, M., Bommer, W. H., and Straub, D. (2008). Security Lapses and the Omission of Information Security Measures: A Threat Control Model and Empirical Test. Comput. Hum. Behav. 24 (6), 2799–2816. doi:10.1016/j.chb.2008.04.005

CrossRef Full Text | Google Scholar

Zielinska, O. A., Tembe, R., Hong, K. W., Ge, X., Murphy-Hill, E., and Mayhorn, C. B. (2014). “One Phish, Two Phish, How to Avoid the Internet Phish: Analysis of Training Strategies to Detect Phishing Emails,” in Paper Presented at the Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Santa Monica, CA: Human Factors and Ergonomics Society).

Google Scholar