Abstract
Identifying those who are likely to become perpetrators of bullying may help direct prevention strategies to those who are most at risk of engaging in peer victimization. Meta-analyses have suggested that antibullying interventions are modestly effective only for students in grades 7 or below. Such interventions are typically Tier I programs of a multi-tiered system of support provided to all students. Some research has examined the cognitive and psychosocial predictors of a type of bullying, relational and social aggression, most often used by those in middle school or above, with limited ability of various cognitive profiles to predict perpetration behaviors. Therefore, this article explores which intrapersonal factors, both cognitive and psychosocial, signify increased susceptibility to perpetrating relational and social aggression from middle childhood through adolescence, as a means of developing targeted preventive strategies for this population. A review of existing interventions for indirect bullying is provided, and recommendations are made for additional strategies for addressing these behaviors through Tier II and III efforts.
Introduction
Bullying in the United States of America (US) is the most common form of school violence and is a significant academic, emotional, behavioral, and health issue (Thomas et al., 2015). As of 2023, nearly 20% of high school students in the US reported being bullied at school or on school grounds (Centers for Disease Control and Prevention [CDA], 2024a). Data from the Youth Risk and Behavior Survey Data Summary and Trends Report: 2013–2023 and 2021–22 School Crime Supplement (National Center for Education Statistics and Bureau of Justice Statistics) suggest that the prevalence of bullying is 26.3% in middle school and 15.7% in high school, and higher among female students (21.8%) than among male students (16.7%; Centers for Disease Control and Prevention [CDA], 2024b).
Rates of bullying tend to be consistent and high for minoritized students, including biracial students, LGBTQ+ students, and students with disabilities, compared to their White, straight, non-disabled peers (Centers for Disease Control and Prevention [CDA], 2024a; Hong et al., 2020; Wells et al., 2019). At school, children with more externalizing and behavioral problems, as well as students who come from poverty and unsafe neighborhoods, tend to be bullied by their peers at higher rates (Iyanda, 2022). Research by Joo et al. (2023) found that among a sample of US adolescents, 42% reported that peers bullied them due to biases related to their multiple social identities, which contributed to an increased fear of school and school avoidance.
Despite the frequency of bullying among those in minoritized groups and in non-Western nations, most research has focused on White children and adolescents in the US and Europe. In particular, studies on indirect bullying (the focus of this article) are even less encompassing of various ages, racial and ethnic groups, linguistic groups, cultures, and nations. Therefore, it is recommended that readers note the underrepresentation of diverse groups of students and ecologies and critically evaluate both the information about the perpetration of indirect bullying and the results of studies designed to assess the effectiveness of interventions for those individuals. It is strongly recommended that future research expand understanding of cultural/contextual differences and broaden studies to include non-Western nations.
Three essential elements of bullying behavior
The first element of bullying, intentional aggression, is a fairly broad term that implies instrumental aggression is expressed through social, verbal, physical, or electronic (i.e., cyberbullying) behaviors (Cornell and Limber, 2015; Gladden et al., 2014). Cyberbullying includes milder forms of aggression, such as teasing and name-calling, but can also intersect with illegal activities, such as assault or hate crimes (Cornell and Limber, 2015). The second element, an imbalance of power between the perpetrator(s) and victim(s), is the main component that helps differentiate bullying from other types of aggression among peers. Some power imbalances are easy to identify, such as size differences; however, other forms, such as higher peer status, may not be as readily recognizable (Cornell and Cole, 2011; Cornell and Limber, 2015; Olweus, 2013). The third element, repetitive aggressive behavior, tends to be more noticeable and helps differentiate bullying from less severe forms of behavior. However, most acknowledge that a single occurrence of aggression that is severely detrimental to the victim or highly likely to be repeated can be recognized as bullying (Cornell and Limber, 2015; Gladden et al., 2014; Olweus, 2013).
Types of bullying
There are two general categories of bullying: direct bullying, in which the perpetrator overtly uses aggression in physical and verbal forms, and indirect bullying, in which the perpetrator(s) use more sophisticated, covert methods to antagonize their victim(s) discreetly (Björkqvist, 1994; Múzquiz et al., 2023). While direct bullying and its perpetrators tend to be immediately apparent to both observers and victims, indirect bullying tends to be more subtle. It also may be done anonymously, which may leave victims unaware of the bullying or the identity of the perpetrators until after the behavior has occurred (PACER’s National Bullying Prevention Center, 2019).
Direct forms of bullying—physical and verbal
Physical bullying occurs when the perpetrator(s) cause harm or damage to the body or possessions of their victim(s). Such behaviors may include hitting, shoving, or spitting at the victim or damage to property, such as anonymous theft and/or destruction of the victim’s possessions (PACER’s National Bullying Prevention Center, 2019). Verbal bullying occurs when the perpetrator(s) use their words to harm their victim (e.g., teasing, name-calling, taunting, threats of harm; Wang et al., 2009).
Indirect forms of bullying—relational and social
Those engaging in indirect bullying (or relational aggression) target the victim’s social status or reputation, whereas direct bullying (verbal and physical) focuses on the characteristics of an individual. Indirect bullying includes purposeful exclusion, spreading rumors, and intentionally embarrassing individuals in public settings, among other attacks on the victim’s social standing or reputation (U.S. Department of Health and Human Services, 2023). While relational bullying is often conceptualized as inherently indirect, it too may have more overt, direct applications, such as ignoring a friend or threatening to end a friendship (Archer and Coyne, 2005). In this paper, we will use Crothers et al. (2009) delineation of indirect bullying into two separate, yet distinct constructs: relational aggression and social aggression. Relational aggression includes behaviors executed within a dyad to threaten the loss of friendship. In contrast, social aggression involves manipulating a group of individuals to inflict social harm on a victim. Both of these types of bullying will be further described later in the article.
Indirect forms of bullying—cyberbullying
In addition to the traditional forms of bullying detailed above, research has recently turned its attention to the growing prevalence of cyberbullying. In cyberbullying, the perpetrator(s) use technological methods such as cell phones, the internet, social media, and other forms of electronic communication to antagonize their victim (Campbell et al., 2012; Jungert et al., 2021; Topcu and Erdur-Baker, 2012; U.S. Department of Health and Human Services, 2023).
Though cyberbullying resembles traditional bullying in many regards, it is worth noting that due to the permanence of content posted online and the severity of its potential to affect the victim(s) detrimentally, cyberbullying does not necessarily require repetition of behaviors to be counted as such (Brighi et al., 2019; Jungert et al., 2021; Smith et al., 2009). Furthermore, in cyberbullying, the power imbalance between the perpetrators and their victim(s) has the potential to stem from differences in the technologies available to the involved parties (Campbell et al., 2012; Jungert et al., 2021)
Who are likely to be bullies/victims
It is important to understand the likelihood that a student will perpetrate or become a victim of bullying to inform prevention and intervention targeting students. Bullying includes different roles for students (bullies, victims, bully-victims, and bystanders). Despite conventional wisdom suggesting that bullies are insecure, research has revealed that bullies tend to be popular and socially intelligent. Because bullies may enjoy high social status, educators may fail to recognize their aggressive behavior toward peers (Hymel and Swearer, 2015).
Research has uncovered that males are more likely to use direct forms of bullying, including verbal and physical peer aggression, while females tend to use indirect bullying behaviors (relational and social aggression; Cornell and Limber, 2015). Understanding which students are likely to use which forms of bullying may allow educators to identify the full range of peer victimization better. In this paper, we will examine the cognitive and personality correlates of indirect bullying perpetration, which may enable early identification of vulnerability to perpetration. Specifically, this article was developed to attempt to contribute to the literature on perpetrators of indirect bullying. While many studies focus on the victims of bullying, few focus on the perpetrators of this common type of peer victimization.
A more complete look at indirect bullying/relational aggression
Indirect bullying is a set of behaviors exhibited by a perpetrator designed to harm a victim’s emotional or psychological health through threats or damage to their interpersonal relationships, social status, or reputation (Bell et al., 2018; Robers et al., 2013). Such peer victimization may occur in person or through technology (e.g., email, social media, texting, and gaming).
Gender and indirect bullying
Within childhood studies, females have appeared to be more likely than males to use indirect bullying instead of traditional (direct) bullying, as indirect forms of aggression rely upon developing strong communication skills, understanding the subtle nuances within social relationships, and a well-developed network of relationships (Bell et al., 2018). Other studies found the opposite, however, with no gender differences in indirect bullying apparent until around 10–11 years of age, when girls engaged in increased relational aggression than boys (Smith et al., 2009).
Crapanzano et al. (2010) found that, among a sample of children in middle childhood, boys’ use of relational aggression may reflect different reasons than their perpetration of physical aggression, as their relational aggression patterns related to distinct categories of reactive aggression alone or combined reactive/proactive aggression. In contrast, relationally and physically aggressive behaviors appear to be related to similar causal factors for girls. However, such gender differences continue to be inconsistent in studies and deserve further investigation—including distinguishing between cisgender and non-cisgender males and females—before we reach firm conclusions.
Understanding different conceptualizations of relational aggression and indirect bullying
There have been different interpretations of the types and intentions of relational aggression. Voulgaridou and Kokkinos (2018) describe four subtypes of the behavior (i.e., proactive direct, proactive indirect, reactive direct, and reactive indirect) that are the result of the intersections of the functions (i.e., proactive—aggressive behaviors motivated by a directed goal and reactive—driven by anger or by feelings of low peer acceptance) and forms (i.e., direct—awareness of perpetrator and indirect—perpetrator or the perpetration is initially unknown to the victim). Another perspective is from Crothers et al. (2009), who differentiated these behaviors into two subcategories, relational and social aggression, by distinguishing them through confirmatory factor analysis on a rating scale they developed. In this article, we will use Crothers et al. (2009) conceptualization of indirect bullying. While Voulgaridou and Kokkinos (2018) conceptualization of indirect bullying is undoubtedly more nuanced, the two-factor explanation introduced by Crothers et al. (2009) is straightforward. It captures the full behavior for intervention purposes.
Relational aggression
Crothers et al. (2009) contend that in perpetuating relational aggression, the individual’s primary focus is to influence or directly control the person’s behavior within that one-to-one relationship. Relationally aggressive behaviors are psychologically more intimate than social aggression, which relies upon the group context to inflict emotional hurt. The intent to harm another may be expressed through the exploitation of a friendship, sarcasm, speaking to another in a cold or hostile tone, ignoring, staring, eye-rolling, and “mean” facial expressions. The victim’s self-esteem is typically damaged through these kinds of relationally aggressive behavior (Archer and Coyne, 2005; Crain et al., 2005; Remillard and Lamb, 2005; Simmons, 2002).
When a victim is undermined in this way, they often feel emotionally and socially vulnerable and appease the perpetrator to ingratiate themselves to reestablish the security of the friendship. It may be easier to defer to the perpetrator than cause a conflict that risks the sustainability of the relationship. As adolescents become more reliant upon peers for their sense of self, sense of belonging, and emotional and psychological stability, the potential loss of a friendship may be devastating (Crothers et al., 2005).
Social aggression
Crothers et al. (2009) hypothesize that social aggression is the intent to manipulate and damage another’s social status or group membership through either covert or overt means. Social aggression requires manipulating a social group as the vehicle of harm, such as gossiping, spreading rumors, and social isolation. It is more sophisticated than relational aggression since it requires knowledge of social dynamics and the ability to subtly influence or orchestrate others’ behavior to achieve desired outcomes.
Social aggression damages children’s social standing, which may be devastating, given that attachment to friends during middle childhood and adolescence predicts positive social, emotional, and behavioral outcomes (Schoeps et al., 2020). Poor attachment to friends may damage children’s social self-perceptions, which may be associated with conduct problems and emotional difficulties (Schoeps et al., 2020). Furthermore, because of the group mentality involved with social aggression (e.g., following a dominant individual’s lead of disliking a victim), escaping or quelling the abusive treatment may be difficult to achieve among one’s peers (Bell et al., 2018; Comstock et al., 2013).
Cognitive correlates of indirect bullying
Crothers et al. (2024) reviewed the limited literature regarding cognitive correlates of indirect bullying. Unfortunately, it has proven difficult to identify new studies to understand the relationship between these two constructs since the writing of that article. As previously explained in the Crothers et al. (2024) review, researchers use different terms in the study of indirect bullying (e.g., relational aggression, indirect aggression, social aggression), making it challenging to aggregate study results to explain the phenomenon. Additionally, we have seen a recent morphing of the term “indirect bullying” in the literature, with some researchers seemingly using the term to describe bystanders, but not observable perpetrators, of bullying behavior. All this is to say that additional literature is required to understand cognitive links to indirect bullying.
Social information processing and indirect bullying
As mentioned, most research on the cognitive contributors to indirect bullying has been devoted to the SIP model. In SIP theory, children’s social behavior is a result of the sequential steps of the processing of social information, including (1) the encoding of social cues, (2) the interpretation of social cues, (3) the clarification of goals, (4) response access or construction, (5) response decision, and (6) behavioral enactment (Crick and Dodge, 1994, 1996). Steeves (2023) found that late adolescent perpetrators of relational aggression conveyed no specific SIP response decision (Assertiveness, Passiveness, Overt Aggression, and Relational Aggression), while socially aggressive participants were significantly less likely to endorse response decisions that are more overtly aggressive or relationally aggressive. Steeves (2023) concluded that this finding is also consistent with research indicating that social aggression is more sophisticated than relational aggression, as it requires understanding social dynamics and roles to achieve one’s own goals through manipulation (Crothers et al., 2019).
Executive functioning
Executive functioning is the higher-order cognitive skills necessary for goal-directed behavior, mediated by the pre-frontal cortex and providing management and direction to lower-order brain functions (Goldstein et al., 2014; O’Toole et al., 2019; Stuss and Levine, 2002). Executive functions may be categorized as hot (involving emotion) or cold (involving cognition; Ward, 2020). Hot EFs include processing information related to reward, emotion, and motivation (Salehinejad et al., 2023). O’Toole et al. (2019) explain that cool executive functions (EF) are cognitive processes, such as inhibition, working memory, and planning, used to solve abstract, emotionally neutral problems (Zelazo and Müller, 2002).
In later childhood, cool EF (e.g., flexible thinking, planning, and goal setting) was associated with relational aggression among 8–12-year-old children (McQuade, 2017). In 9–13-year-old children, relational aggression predicted future relational aggression, with this effect moderated by strong EFs (McQuade, 2017). These children learned that relational aggression effectively exerted control over peers and helped them to gain social status, likely reinforcing their relationally aggressive behavior.
A recent investigation captured our attention because it helped us understand the possible connection between general cognitive ability and executive functioning, as brain-based constructs, and the expression of indirect aggression. Weinreich et al. (2023) conducted a study of elementary school students. They determined that the presence of a learning disorder (a disorder rooted in theoretical dysfunction of cognitive skills) is not directly related to risk for engagement in bullying. However, the relationship between learning disorder and bullying is indirect through the existence of externalized behavior problems.
This finding is consistent with Kishimoto et al. (2023), who also found that externalizing behavior is related to the perpetration of bullying and pointed to executive functioning constructs, like decreased behavioral inhibition (please see our later discussion of personality and behavioral attributes related to relational aggression). Because a lack of behavioral inhibition has been associated with overt bullying across childhood (Jenkins et al., 2018; Kishimoto et al., 2023; Noten et al., 2020), more research is needed to understand the relationship between inhibitory control and indirect aggression. The numerous studies linking EF to indirect bullying suggest that a strong relationship exists; however, the direction, form, and magnitude of these connections appear to be dependent upon the age, cognitive skills, disability status, and social skills of the individuals.
Theory of mind and indirect aggression
Theory of Mind (ToM), the ability of individuals to recognize and appreciate their own and others’ mental states to use this information to explain and predict behavior (Premack and Woodruff, 1978), has been proposed as a variable that may explain indirect aggression (Sutton et al., 1999). ToM development permits children to be aware of others’ relationships, possibly teaching them how to be the most hurtful to others through gossiping, manipulation, or social exclusion (Gomez-Garibello and Talwar, 2015). Indeed, studies have found positive correlations between all forms of aggression, including indirect aggression, and ToM skills in preadolescents (e.g., Gini, 2006; Kokkinos et al., 2016a).
Therefore, researchers found that indirect aggression was significantly positively associated with moral disengagement and negatively with ToM in a sample of Greek preadolescents. However, although boys with deficient ToM were more likely to disengage morally from their actions and use relational aggression, poor ToM was only directly connected to relational aggression for preadolescent girls. ToM partially mediated the relationship between moral disengagement and relational aggression in girls (Kokkinos et al., 2016a). As with EF, there are undoubtedly linkages between ToM and indirect bullying, but the strength, direction, and mediation of these relationships are dependent upon gender, age, and cognitive sophistication. Additional study is needed to understand the role of ToM in indirect bullying fully.
Elaborative and deep processing and indirect bullying
Some may speculate that there is a relationship between cognitive maturity and the perpetration of indirect bullying. In an investigation examining the fourth stage of cognitive development in childhood and adolescence (formal operational thought, as described by Piaget, 1967) and the perpetration of indirect bullying, researchers sought to clarify the role of indirect bullying at the highest level of cognitive sophistication during this developmental period.
The investigators examined the relationship between college students’ learning styles (measured by the Inventory of Learning Processes—Revised; Schmeck et al., 1991) and their use of two forms of indirect bullying (social and relational aggression). In this study, two dimensions of reflective processing (deep and elaborative processing) were found to be related to both forms of indirect bullying (Crothers et al., 2014).
Relational aggression and social aggression accounted for 81% of the variance in deep processing (thinking abstractly, reasoning hypothetically, and solving problems systematically), and 87% of the variance in elaborative processing (encoding new information through personal metaphor and vocabulary; Schmeck et al., 1991). These findings were consistent with the view that indirect forms of aggression require higher levels of social intelligence and social skills than direct bullying (e.g., Björkqvist et al., 2000; Crothers et al., 2014).
General cognitive abilities and indirect aggression
As the research on cognitive maturity and indirect bullying was established by Crothers et al. (2014), subsequent research by this group extended the inquiry to overall cognitive ability and this type of bullying. In particular, the researchers speculated that verbal acuity and working memory would be related to the perpetration of indirect bullying, as these abilities would represent the savvy required for this type of aggression.
Crothers et al. (2019) administered a standardized measure of intellectual functioning, a measure of indirect bullying, and a measure of prosocial skills. Specifically, the researchers sought to learn if performances on a theory-based, individually administered test of cognitive abilities (Woodcock-Johnson III Normative Update Tests of Cognitive Abilities; Woodcock et al., 2007) could be used to understand the utilization of self-reported relational aggression, social aggression, overt aggression, and prosocial skills among late adolescent females.
None of the cognitive abilities assessed by this measure explained a notable amount of variance in indirect bullying, whether relational or social aggression. These results suggest that measures of general cognitive abilities—not directly tied to social contexts—are not particularly useful in understanding the expression of indirect aggression.
Summary of cognitive predictors of indirect bullying/relational aggression
Although this review article focuses on relational aggression, findings from the overt bullying literature may inform our understanding of the association between cognitive abilities and indirect aggression. Higher IQ and executive functioning are associated with a reduced risk of perpetrating direct bullying (see Liu et al., 2017; Verlinden et al., 2014, for example, experimental studies). These research implications from the Crothers et al. (2024) study of cognitive correlates are that the perpetration of indirect aggression is better understood by flexibility in thinking, the ability to focus on particular elements of social behavior, inhibit some responses, and engage in perspective-taking, and other characteristics associated with cool EF, than by overall intellectual functioning.
Intervention for bullying
By 2017, while all 50 states in the US had enacted antibullying laws for their schools, no federal laws explicitly prohibited school bullying. Standard components among state regulations, laws, and policies include definitions of bullying, characteristics of bullying behaviors, and comprehensive school district policy requirements (Stopbullying.gov, 2023). Unfortunately, in their earlier analysis of more than 40 statewide antibullying laws, Kueny and Zirkel (2012) noted their limited impact, explaining that such laws failed to address cultural variables within schools and to provide accountability for enforcing them. In a study of the effects of the Iowa Safe Schools Law, bullying rates remained unchanged before and after the law’s passage (McGeough, 2022).
In response to an earlier meta-analysis by Ttofi and Farrington (2011), Yeager et al. (2015) argued that between-study analyses of age-related effects on the effectiveness of antibullying programs may be misleading. These researchers described using within-study analyses that compare the duplicate content delivered at different ages, reducing confounding variables and increasing the precision of age-related estimates. Using a three-level meta-analysis of within-study age moderation, these researchers found that although bullying appeared to be prevented mainly below seventh grade, in eighth grade, and thereafter, the effects of antibullying programs sharply declined to an average of 0 (Yeager et al., 2015). Yeager et al. (2015) analyses suggest that, for students in the tenth through twelfth grades, antibullying interventions are generally associated with iatrogenic, harmful effects.
Why state laws and antibullying programs are not strongly effective
McGeough (2022) examined the ineffectiveness of state laws in curbing student bullying. These include numerous concerns, from limiting the constitutional right to free speech to a lack of training for educators and school staff in how to engage with perpetrators and victims of bullying. In their meta-analysis of antibullying programs, Ttofi and Farrington (2011) suggest that schools’ failure to monitor student behavior carefully, the use of ineffective consequences, and the omission of parents in preventing and responding to bullying curtail the effectiveness of antibullying interventions.
In a qualitative analysis of teachers’ perceptions of the limitations of the effectiveness of antibullying programs, educators cited the complexities of bullying, from the behaviors occurring in areas in which educators were not (off school property or through social media) to the variation in the behaviors (manipulative, covert actions seen in relational and social aggression or the switching of modalities to evade adults’ notice). Teachers also called out the failure to adapt the developmental level of antibullying activities to students at each grade level. Some suggested that programming for older students should shift to a more active, problem-based approach in which students find solutions (Cunningham et al., 2015).
Educators also described implementation factors that limited the impact of antibullying programs, including top-down selection of programming (lacking teachers’ social validity), insufficient time to learn and implement the antibullying lessons, staff attrition, insufficient resources, and failure to ensure long-term maintenance. Most educators in the study reported not being provided with convincing evidence of the program’s effectiveness and not seeing student behavior change (Cunningham et al., 2015).
Teachers reported that other lessons were moved to make time for the antibullying programming, and even then, there was not enough time to fully implement them. When students brought issues to their teachers, there was sometimes no time to help them talk to each other and work through their problems because of other educational responsibilities. Furthermore, some teachers were concerned about waning support from school boards and administrators or a lack of complete buy-in by colleagues, which created an uneven willingness to change and adopt new practices. Similarly, parental support was not uniform, with some parents actively resisting addressing bullying behavior in their children. Some students were unwilling to participate or cooperate with new policies or programming (Cunningham et al., 2015).
In line with the argument that states’ antibullying laws have not been effective, teachers reported that increased monitoring was not pursued as a preventive measure. Disciplinary measures or the tracking of those involved in bully-victim conflicts were not consistently and thoughtfully implemented. Relatedly, teachers reported that schools failed to apply programs consistently across schools and grades or sustain potentially successful programming. When implementing the antibullying program, some teachers modified the lessons, negatively affecting treatment integrity (Cunningham et al., 2015).
What should be done from a systems perspective?
Some antibullying researchers have suggested analyzing what shared components of schoolwide policies are working most and least effectively (Wampold, 2015). Funding for antibullying efforts and mental health should be a priority in all schools, as children who perpetrate peer victimization or are bullied have higher rates of depression and suicidality than those not involved in bullying. Additionally, actively monitoring the implementation of the policies must be a part of each state’s oversight of its antibullying law.
Those who are particularly vulnerable to being bullied, such as LGBTQ+ students, need to be protected through school climate interventions that promote social acceptance and the equality of academic and mental health initiatives while decreasing an atmosphere of hostility (Greene, 2006; Vreeman and Carroll, 2007). Effective antibullying initiatives that address broader community norms are needed to combat discriminatory practices. While antibullying practices have enjoyed some success in combating overt bullying, relational and social aggression have not been reduced to the same degree (Greene, 2006; McGeough, 2022).
Recommendation: targeted Tier II and Tier III intervention for existing and prospective perpetrators
In multiple meta-analyses on the effectiveness of whole-school antibullying intervention programs, the most recent research suggests that the positive effects are modest for children in grades 7 and below and are neutral to iatrogenic (harmful) for 8th-grade students and older. Multiple suggestions have been made in the previous pages to improve these antibullying programs. We will speak to our concerns about antibullying programs, in general, and then to the effects that such programs have upon the perpetration of relational and social aggression in children.
Multi-tiered systems of support
In multi-tiered systems of support (MTSS), antibullying programming is typically a Tier I intervention. As a brief review, the MTSS framework is a school-based intervention that provides the multiple degrees of support needed to address the range of learning, mental health, and emotional-behavioral concerns students may have. Regarding bullying prevention and intervention, MTSS delivery includes universal, selective, and targeted intervention services (Stopbullying.gov, 2021). After briefly explaining the universal prevention of bullying, we will turn our focus to Tiers II and III for students vulnerable to or already perpetrating relational and social aggression (Stopbullying.gov, 2021).
Tier I universal bullying prevention efforts reduce risk and increase the school community’s resilience. Tier I antibullying interventions often include strategies to improve the school’s overall social and emotional climate and to promote positive social behaviors and inclusiveness among the student body. Tier I interventions may include classroom meetings to reinforce positive behavior and also general strategies for responding to bullying (Stopbullying.gov, 2021).
Tier II antibullying strategies in MTSS are often called “selective prevention” and are designed for children who need intervention beyond the general strategies used with the entire school population. Such youth may be at risk for perpetration or victimization. Relevant to our review, for those who are vulnerable to bullying others, recommended selective prevention often incorporates reinforcing consequences for bullying, using teacher/counselor mediation to address interpersonal conflict, and encouraging peers to defend victims to remove the social benefits of bullying to perpetrators (Stopbullying.gov, 2021). We will review several Tier II intervention programs for relational aggression in the subsequent pages of this article.
Tier III antibullying strategies in MTSS are referred to as “indicated interventions” for students whose needs have not been adequately addressed at Tiers I and II. Tier III intervention has a greater intensity and specificity for the student’s specific needs. The review of Tier III interventions for perpetrators of relational and social aggression is surprisingly sparse. The spirit of Tier III interventions is to support mental health concerns, behavior issues, and academic performance by working with administrators, multiple teachers, school resource officers, family members, and other supportive adults in the student’s life (Stopbullying.gov, 2021). However, information from both the extant literature and clinical impressions from teachers suggests that there is a lack of targeted treatment for existing and potential perpetrators of indirect bullying (Cunningham et al., 2015).
Intrapersonal and familial contributions to perpetration of relational and social aggression
Temperament, personality, and personality psychology
Tackett et al. (2014) examined dispositional aspects of relational aggression, exploring the contribution of temperament, personality, and personality pathology. Relational aggression was more strongly associated with temperament traits than with personality traits. Specifically, relational aggression was associated with negative emotionality and limited self-regulation, which has a very similar profile to other youth externalizing problems (Tackett et al., 2013). The authors noted, however, that there were some divergent results: three subcomponents of extraversion were linked to relational aggression, but in contrasting ways: shyness and sociability were positively related to relational aggression, whereas positive emotions were negatively related to it. The authors concluded that such apparent contradictory results may suggest heterogeneity of youth who engage in relational aggression. Such variation is consistent with previous research indicating that relational aggression has been associated with, and without, social maladjustment (Crick et al., 2006; Rose et al., 2004).
Self-esteem
Self-esteem has been hypothesized to be related to the perpetration of relational and social aggression in two ways. One mechanism of this relationship may be that lower self-esteem, which leads a child to ward off feelings of inferiority or to increase their perceptions of power, may lead them to perpetrate relational aggression, which Weidmann et al. (2022) term the self-valorizing pathway (Ostrowsky, 2010). Two studies have found that lower self-esteem predicted later perpetration of relational aggression in children (Fanti and Henrich, 2015; Guerra et al., 2011), whereas another (Rose et al., 2017) did not.
Children with higher self-esteem may use relational aggression to protect their egos; Weidmann et al. (2022) describe this as the threatened egotism pathway (Baumeister et al., 2000). High self-esteem may also encourage the use of relational aggression as a goal-oriented behavior to obtain peer acceptance and bolster self-worth (Volk et al., 2014), a phenomenon Weidmann et al. (2022) describe as the instrumental pathway. Although theoretically sound, longitudinal studies have not found that perpetration of relational aggression predicted lower self-esteem later in childhood (Özdemir and Stattin, 2011; Rose et al., 2017). Still, other research found that girls, not boys, later experienced higher self-esteem after engaging in relationally aggressive behavior (Pollastri et al., 2010; Weidmann et al., 2022).
Weidmann et al. (2022) surveyed a sample of 674 Mexican-origin California youth in the fifth, seventh, ninth, and eleventh grades regarding their global and domain-specific self-esteem and the perpetration and victimization of relational aggression. However, we will restrict our discussion to only relational aggression perpetration. These researchers found that the perpetration of relational aggression is associated with later lower domain-specific opposite-sex relationships’ self-esteem at the between-person level. Participants also demonstrated that lower self-esteem in the domain of honesty-trustworthiness is associated with a later onset of perpetration. The findings of this research suggest that there are relationships between self-esteem (domain-specific) and the perpetration of relational aggression, although in various directions.
Psychological maltreatment
Another study also contributed to the understanding of the perpetration of indirect bullying by explaining that psychological maltreatment in childhood predicts the use of relational aggression in adolescence. The mechanism that permits adolescents (in this case, Chinese adolescents aged nine to 19) to engage in this behavior without self-blame is moral disengagement, a process by which individuals reduce the cognitive tension they experience when engaging in immoral behaviors (Wang et al., 2017). Therefore, moral disengagement, as the lack of self-regulation (EF), mediated the relationship between psychological maltreatment (as a victim) and indirect bullying (as a perpetrator), with gender moderating this process. Boys who demonstrated higher moral disengagement in this study also engaged in higher levels of relational aggression, rationalizing their use of this behavior (Ding et al., 2022).
Moral disengagement
In another investigation, indirect aggression was positively associated with moral disengagement among Greek preadolescents, which, in turn, mediated the effects of callous-unemotional traits and the behavioral activation system on relational aggression (Kokkinos et al., 2016b). Gini et al. (2020) conceptualize moral disengagement as cognitive mechanisms that allow the moralization of actions that would otherwise be considered immoral (thereby reducing cognitive dissonance). Additional research (see Bjärehed et al., 2021) suggests that further implications of this relationship are undoubtedly present.
Psychopathic traits
Psychopathic traits and indirect aggression have conceptual overlap, as the manipulation of others involved in relational aggression also characterizes children who exhibit higher levels of narcissism-grandiosity traits (e.g., Colins et al., 2014). Given the conceptual similarities and the fact that psychopathic traits have some malleability during childhood (Bégin et al., 2021), researchers have explored whether psychopathic traits are a risk factor for relational aggression.
Psychopathy comprises three dimensions: an affective component (callous-unemotional traits, i.e., lack of empathy and guilt), an interpersonal component (narcissism and grandiosity), and behavioral patterns (impulsivity and irresponsibility; Dong et al., 2014). A 5-year longitudinal study of children entering preadolescence revealed that, after controlling for demographic confounding variables and conduct problems, only narcissism predicted being classified as a high or consistent user of indirect aggression. Callous-unemotional and impulsive-irresponsible traits were not predictors of increasing or stable use of indirect aggression over time (Boutin et al., 2023). Narcissism-grandiosity has also been found to be related to indirect aggression among preadolescents in other investigations (Bukowski et al., 2009; Onishi et al., 2012; Reijntjes et al., 2016).
Family/parental contributions to relational aggression
Studies have shown that genetic inheritance and environmental factors predict the perpetration of relational aggression. In middle childhood, genetic and sibling environmental factors have been shown to predict the use of social aggression in a sample of identical twins raised within the same family (Slawinski et al., 2019). Social learning theory has been used to explain how parenting behavior influences the development of aggression (Bandura, 1973), suggesting that children learn, observe, and imitate aggressive behavior from their parents.
In a meta-analysis, Kawabata et al. (2011) evaluated 48 studies and uncovered that the warmth and emotional responsiveness of positive parenting were inversely related to children’s relationally aggressive behavior. In contrast, uninvolved, harsh, or psychologically controlling parenting predicted an increased use of relational aggression in children (Kawabata et al., 2011). Kawabata et al. (2011) theorized that children who receive positive parenting are less likely to engage in relational aggression as they acquire social competence and emotional regulation within a safe parent-child relationship.
Several meta-analytic investigations have focused on the role of parental psychological control in childhood relational aggression, given its conceptual similarities to relational aggression, as both include emotional and social manipulation. Parental psychological control involves manipulation of the parent-child relationship (e.g., withdrawal of love), criticism, and personal control (e.g., possessiveness and overprotectiveness). A parent who threatens rejection if their child fails to meet their expectations has been likened to a child who threatens to end a friendship if their friend does not comply with their wishes.
Kuppens et al. (2013) meta-analysis of twenty-three studies revealed that parental psychological control is associated with youths’ use of relational aggression. While this relationship was weak, explaining only three percent of the variance in youth relational aggression, it is comparable to the degree to which parenting is linked to other problems in childhood (e.g., McLeod et al., 2007). The relationship between parental psychological control and the use of relational aggression was more substantial for adolescents than for elementary-aged children. Kawabata’s meta-analysis found that paternal psychologically controlling parenting, but not maternal psychologically controlling parenting, was predictive of the perpetration of relational aggression.
Child/adolescent interventions to reduce relational aggression
Research regarding the efficacy of intervention and prevention programs to reduce relational aggression is not nearly as common as studies of physical bullying. It is reflected in our inability to locate any meta-analyses for prevention or intervention efforts for relational aggression in the literature. The Office of the Assistant Secretary for Planning and Evaluation (Blase and Fixsen, 2013) published core intervention components to identify and operationalize what makes programs effective. These include a clear description of the context of a program, its core components, the active requirements to operationalize the essential elements so that they can be taught, learned, and implemented in typical settings, and a practical strategy for measuring behaviors and practices reflecting the program’s principles, values, active ingredients, and activities. Finally, when outcomes are not achieved, assessing whether treatment fidelity has been established is critical to determining whether the lack of effectiveness is due to inadequate implementation or flaws in the programming (Blase and Fixsen, 2013).
While Leff et al. (2010) review of relational aggression programs identified nine programs with promising results, including several focused on middle childhood, nearly all have not been subject to rigorous program evaluation, which has traditionally been referred to as the use of randomized controlled trials (RCTs). In educational settings, RCTs are extremely difficult to execute. Lee-Easton et al. (2022) refer to hierarchies of evidence that use other designs as credible evidence of an intervention’s merit. These authors also describe an approach called “best available evidence” to evaluate interventions, recognizing that high-quality RCTs or meta-analyses may not be available for an intervention, as is the case for treating perpetration of indirect bullying. Rigorous evidence may include a range of research designs, and Lee-Easton et al. (2022) advocate for “accepted principles of research quality” that encompass numerous perspectives on acceptable research designs.
With these principles in mind, the programs that will now be described include the Friend to Friend program (Leff et al., 2009), Making Choices and MC Plus programs (Fraser et al., 2004; 2005), Second Step (Van Shoiack-Edstrom et al., 2002), Social Aggression Prevention Program (Capella and Weinstein, 2006), and Sisters of Nia (Belgrave et al., 2004). While we will not review all of them, we wanted to provide a representative sample of this important work. Leff et al. (2010) drew several conclusions from their review of relational aggression programs, including that programs should be culturally sensitive and adopt a systemic approach to prevention that focuses on the school and community contexts.
Friend to friend program (F2F)
In the F2F Program, interventions are designed to reduce relational and physical aggression, improve problem-solving skills, and increase prosocial behaviors for urban Black 3rd–5th-grade girls (Leff et al., 2015). Ecological-developmental-systems theory (Bronfenbrenner and Morris, 2006) and SIP (Crick and Dodge) were incorporated into the F2F Program, reflecting behavior strongly influenced by their interaction within a social environment. The program includes a leadership component in which participants co-facilitate F2F classroom lessons for their peers, following the hypothesis that demonstrating new problem-solving skills could modify the negative reputations of relationally aggressive children and improve their relationships with peers and teachers.
Social learning theory (Bandura, 1973) is also incorporated through behavioral practice and role-play to enhance social cognitive problem-solving. The 20-session curriculum teaches girls problem-solving strategies using detective analogies and various teaching modalities, including videos, cartoons, and role-plays. Leff et al. (2015) found that Black girls who received F2F exhibited decreased relational aggression and increased social problem skills in comparison to similar girls who were randomly assigned to a homework and study skills intervention. These gains were maintained at a 1-year follow-up.
Making choices: social problem-solving skills for children (MC)
The MC program (Fraser et al., 2005) is a Tier I intervention for third-grade students based on the SIP model (Crick and Dodge, 1994). The MC program comprises approximately twenty classroom lessons in which students learn to identify and regulate intense emotions and use self-talk to distinguish between harmful responses and responses more likely to maintain friendships. Smokowski et al. (2004) found that, compared with a control group, third-grade students who participated in MC demonstrated higher social contact scores and significantly lower overt aggression. Additionally, high-risk children demonstrated greater improvements than low-risk students.
MC plus program
Fraser et al. (2005) evaluated a version of the MC called MC Plus, which included teacher and family enhancements. These enhancements involved using behavioral management strategies of providing group-level incentives, creating heterogeneous assigned groups, and using the Good Behavior Game (GBG) (Evertson et al., 1983). Caregivers of the MC plus condition received newsletters regarding the MC lessons, were encouraged to engage in a home-based exercise, and were invited to attend five 90-min informational sessions about the MC curriculum (Fraser et al., 2005). Compared to the routine condition, children who received MC and MC Plus had lower social and overt aggression and greater social competence, and students who received the MC Plus intervention exhibited a lower hostile attribution bias and increased cognitive concentration.
Social Aggression Prevention Program
The Social Aggression Prevention Program (SAPP) is a school-based group intervention for fifth-grade girls that uses discussion, role-playing, modeling, games, and collaboration to enhance knowledge of relational aggression, increase self-awareness and empathy, promote assertive communication, and provide strategies for social problem-solving (Capella and Weinstein, 2006). Compared with a control group, participants demonstrated significant gains in social problem-solving, and teachers observed increases in prosocial behavior among girls who had higher levels of relational aggression prior to the intervention.
Sisters of Nia
Sisters of Nia is described by the creators (Belgrave et al., 2004) as a cultural intervention that seeks to address relational aggression among Black girls by enhancing cultural values and beliefs. The fifteen-session small-group intervention involves building trust, increasing knowledge of African culture, promoting hair care and body image, exploring how Black persons are portrayed in American media, emphasizing the value of education, and establishing short- and long-term personal goals. Results of the pilot study indicated that Black girls aged 11–13 who received the intervention demonstrated decreases in relational aggression and increases in ethnic identity.
Self-affirmation manipulation
Armitage and Rowe (2017) described an intervention in which children aged 11–16, drawn from a community sample in England, received a brief reading about the harm caused by relational aggression. The children were subsequently asked to engage in a self-affirmation manipulation by recalling past acts of kindness. These participants demonstrated a small but significant reduction in relational aggression over 1 month, whereas those in the control condition had slight increases in their use of relational aggression. The researchers concluded that the self-affirmation manipulation enabled participants to process information about the harms of relational aggression more effectively, thereby counteracting the tendency to avoid information threatening one’s self-concept.
Family interventions
Kawabata et al. (2011) concluded that the small effect sizes for the relationship between parenting and relational aggression negate a recommendation for the use of targeted interventions for the parents of children vulnerable to using relational aggression. However, the effect sizes for the link between parenting and relational aggression are substantial enough to justify the use of universal interventions that focus on the home environment. Unfortunately, we could not identify any intervention that used family intervention or parenting education as a standalone treatment for reducing relational aggression. However, some Tier II intervention programs included parents.
Limitations and future directions
In the final sections of this paper, we will use the term “relational aggression” to refer to all indirect forms of bullying. First, it is important to recognize that most of these programs have not yet been researched enough to be considered evidence-based practice. For this reason, practitioners and researchers should be cautious and conduct additional investigations (particularly regarding the ecology of the targeted group) before confidently recommending any of these programs. Nevertheless, Leff et al. (2010) identified several school-based relational aggression programs that achieved promising results with preadolescent children. Whereas most bullying prevention programs are universal, most of the relational aggression programs identified by Leff et al. (2010) can be characterized as Tier II interventions, with the primary intervention format consisting of small-group sessions, and many targeting students who had been identified as using relational aggression. A common characteristic of the programs identified by Leff et al. (2010) is a foundation in the SIP model, with sessions designed to target specific SIP deficits.
Leff et al. (2010) recommended that relational aggression programs provide perpetrators with healthy opportunities to pursue social recognition, and it may be argued that the structured group format granted relationally aggressive children an alternative way to connect with other children and satisfy a need for social status. It is possible that Tier I bullying prevention programs have not been enormously successful due to their focus on universal interventions and their failure to consider a primary motivation for perpetrators of relational aggression, the desire for social status, which may potentially be more achievable in the group setting of a Tier II intervention.
There are also emerging risk factors that we have not yet found sufficiently documented in the professional literature: specifically, the role of technology in the use of indirect bullying. There is much anecdotal information describing indirect bullying being perpetrated through technology use (e.g., texting, emailing, social media sites). However, preliminary research has distinguished indirect bullying from bullying being perpetrated through technology (i.e., cyberbullying; Lattanzio, 2018). Lattanzio (2018) used exploratory factor analysis to differentiate cyberbullying and indirect bullying in a sample of early college students. Undoubtedly, indirect bullying is perpetrated through technology. However, the boundary between indirect bullying and cyberbullying remains diffuse at this time.
Discussion
Our review of the research literature on the intrapersonal correlates of relational aggression may be interpreted in light of the success achieved by many Tier II relational aggression prevention programs. Relational aggression may protect one’s self-esteem by demonstrating power. Narcissism-grandiosity is the one psychopathic trait that has consistently been found to predict relational aggression. Interestingly, Armitage and Rowe’s (2017) finding regarding the effectiveness of engaging in self-affirmation immediately following reading psychoeducation about the harmful effects of relational aggression may address the tendency of relationally aggressive children to “tune out” such prosocial messages in order to protect their self-concept. In summary, these findings suggest motivations for relational aggression, such as demonstrating status and preserving self-concept through relational victimization.
Small group interventions
Small group interventions may offer relationally aggressive children an alternative, more prosocial context for achieving status. The group’s guided/didactic instruction, involving perspective-taking and role-playing of reciprocal interactions, may provide relationally aggressive children with a new, more mature way of relating that they find personally satisfying. Relationally aggressive children may not have experienced relationships involving mutual reciprocity. Indeed, our review of the research literature revealed that parental contributions to children’s use of relational aggression are small but significant. Along with Leff et al. (2010), we recommend that relational aggression programming include consultation to support parents in pursuing a different way of relating to their child.
Lack of Tier III interventions
We were not able to identify Tier III interventions designed explicitly for relationally aggressive children from the extant literature base. Using the SIP framework (Crick and Dodge, 1994) for relationally aggressive children who exhibit deficits in interpreting social cues, professional helpers can promote children’s encoding skills via facial recognition practice, asking students to hypothesize the feelings of the person exhibiting the facial display, and contemplating why they may be feeling that way. To reduce hostile attribution bias, relationally aggressive children can be encouraged to expand their attention to central stimuli regarding the overall context (e.g., “What were other children doing?,” “What happened next?”). Relationally aggressive children may be encouraged to think flexibly by being asked to generate alternative hypotheses to perceiving a peer as aggressive (e.g., “What do you think might have been going on for them?”)
Emotion regulation interventions
For deficits in the emotional regulation of perpetrators, professional helpers should affirm the first emotion expressed or implied by the child, which is often anger, but gradually help the child identify and express the more vulnerable feelings that often underlie anger, which include sadness, hurt, or jealousy. Identification of such vulnerable feelings often generates anxiety and requires self-soothing, so the professional helper should assess whether the child spontaneously engages in such self-soothing or whether it must be provided to the child by the helper through empathic responses (e.g., “I know you are upset, that hurts to think you are losing you friend, I want to help you with this.”). Children can be taught the primary principle of emotion theory: each emotion is associated with a desire/need.
Narcissism interventions
For relationally aggressive children who show little concern for others, the professional helper can appeal to their self-interest by helping them identify the advantages of prosocial responses, while also taking a longer-term approach to helping them internalize others’ feelings and concerns. Such an approach is consistent with Boutin et al. (2023) recommendation that interventions for preadolescents with psychopathic traits should help them to learn to consider the wellbeing of others and care about how others feel, not just learn to identify what they feel. Indeed, children who use relational aggression may have sufficient cognitive empathy, meaning they can take the perspective of others but lack emotional empathy, indicating that they are insensitive to the feelings of others (Crothers and Kolbert, 2008).
Despite several of the relational aggression prevention programs described in this paper appearing to have achieved adequate empirical support to conclude that they are effective, future research is needed to establish the mechanisms of change for these programs. The most significant area of need appears to be the development and evaluation of Tier III interventions for indirect bullying perpetrators. In summary, the research literature on the correlates of indirect bullying/relational aggression and intervention efforts may have implications for bullying prevention programs. We argue that more targeted interventions for perpetrators and their caregivers are necessary for effective bullying prevention.
Statements
Author contributions
LC: Conceptualization, Formal analysis, Project administration, Supervision, Writing – original draft, Writing – review & editing. JK: Writing – original draft, Writing – review & editing. AS: Writing – original draft, Writing – review & editing. JC: Writing – original draft, Writing – review & editing. KP: Writing – original draft, Writing – review & editing. AV: Writing – original draft, Writing – review & editing. AZ: Writing – original draft, Writing – review & editing.
Funding
The author(s) declared that financial support was not received for this work and/or its publication.
Conflict of interest
The author(s) declared that this work was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Generative AI statement
The author(s) declared that generative AI was not used in the creation of this manuscript.
Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
1
Archer J. Coyne S. M. (2005). An integrated review of indirect, relational, and social aggression.Personal. Soc. Psychol. Rev.9212–230. 10.1207/s15327957pspr0903_2
2
Armitage C. J. Rowe R. (2017). Evidence that self-affirmation reduces relational aggression: A proof of concept trial.Psychol. Violence7489–497. 10.1037/vio0000062
3
Bandura A. (1973). Aggression: A social learning approach.Hoboken, NJ: Prentice Hall, 10.2307/1227918
4
Baumeister R. F. Bushman B. J. Campbell W. K. (2000). Self-esteem, narcissism, and aggression: Does violence result from low self-esteem. Or from threatened egotism?Curr. Direct. Psychol. Sci.926–29. 10.1111/1467-8721.00053
5
Bégin V. Fontaine N. M. G. Vitaro F. Boivin M. Tremblay R. E. Côté S. M. (2021). Perinatal and early-life factors associated with stable and unstable trajectories of psychopathic traits across childhood.Psychol. Med.53379–387. 10.1017/S0033291721001586
6
Belgrave F. Z. Reed M. C. Plybon L. E. Butler D. S. Allison K. W. Davis T. (2004). An evaluation of sisters of Nia: A cultural program for African American girls.J. Black Psychol.30329–343. 10.1177/0095798404266063
7
Bell G. R. Crothers L. M. Hughes T. L. Kanyongo G. Y. Kolbert J. B. Parys K. (2018). Callous-unemotional traits, relational and social aggression, and interpersonal maturity in a sample of behaviorally disordered adolescents.J. Appl. School Psychol.3465–85. 10.1080/15377903.2017.1345814
8
Bjärehed M. Thornberg R. Wänström L. Gini G. (2021). Moral disengagement and verbal bullying in early adolescence: A three-year longitudinal study. J. Sch. Psychol.84, 63–73. 10.1016/j.jsp.2020.08.006
9
Björkqvist K. (1994). Sex differences in physical, verbal, and indirect aggression: A review of recent research.Sex Roles30177–188. 10.1007/BF01420988
10
Björkqvist K. Österman K. Kaukiainen A. (2000). Social intelligence – empathy = aggression?Aggress. Violent Behav.5191–200. 10.1016/S1359-1789(98)00029-9
11
Blase K. Fixsen D. (2013). Core intervention components: Identifying and operationalizing what makes programs work.Washington, DC: Office of the Assistant Secretary for Planning and Evaluation, Office of Human Services Policy, U.S. Department of Health and Human Services.
12
Boutin S. Bégin V. Déry M. (2023). Impacts of psychopathic traits dimensions on the development of indirect aggression from childhood to adolescence.Dev. Psychol.591716–1726. 10.1037/dev0001582
13
Brighi A. Menin D. Skrzypiec G. Guarini A. (2019). Young, bullying, and connected. Common pathways to cyberbullying and problematic internet use in adolescence.Front. Psychol.10:1467. 10.3389/fpsyg.2019.01467
14
Bronfenbrenner U. Morris P. A. (2006). “The bioecological model of human development,” in Handbook of child psychology: Theoretical models of human development, 6th Edn, edsLernerR. M.DamonW. (John Wiley & Sons, Inc.), 793–828.
15
Bukowski W. M. Schwartzmann A. Santo J. Bagwell C. Adams R. (2009). Reactivity and distortions in the self: Narcissism, types of aggression, and the functioning of hypothalamic-pituitary adrenal axis during early adolescence.Dev. Psychopathol.211249–1262. 10.1017/S0954579409990149
16
Campbell M. Spears B. Slee P. Butler D. Kift S. (2012). Victims’ perceptions of traditional and cyberbullying, and the psychosocial correlates of their victimisation.Emot. Behav. Difficulty17389–401. 10.1080/13632752.2012.704316
17
Capella E. Weinstein R. (2006). The prevention of social aggression among girls.Soc. Dev.15434–462. 10.1111/j.1467-9507.2006.00350.x
18
Centers for Disease Control and Prevention [CDA] (2024a). Bullying. youth violence prevention.Atlanta, GA: CDA.
19
Centers for Disease Control and Prevention [CDA] (2024b). Youth risk behavior survey data summary & trends report: 2013–2023.Washington, D.C: U.S. Department of Health and Human Services.
20
Colins O. F. Andershed H. Frogner L. Lopez-Romero L. Veen V. Andershed A. K. (2014). A new measure to assess psychopathic personality in children: The child problematic traits inventory.J. Psychopathol. Behav. Assess.364–21. 10.1007/s10862-013-9385-y
21
Comstock L. A. Crothers L. M. Schreiber J. B. Schmitt A. J. Field J. E. Hughes T. L. et al (2013). Relational, social, and overt aggression among aggressive and nonaggressive female adolescents.J. Child Adolesc. Trauma6106–117. 10.1080/19361521.2013.755652
22
Cornell D. Cole J. (2011). “Assessment of bullying,” in The handbook of school violence and school safety: International research and practice, 2nd Edn, edsJimersonS. R.NickersonA. B.MayerM. J.FurlongM. J. (England: Routledge), 289–303.
23
Cornell D. Limber S. P. (2015). Law and policy on the concept of bullying at school.Am. Psychol.70333–343. 10.1037/a0038558
24
Crain M. M. Finch C. L. Foster S. L. (2005). The relevance of the social information processing model for understanding relational aggression in girls.Merrill-Palmer Quar.51213–249. 10.1353/mpq.2005.0010
25
Crapanzano A. M. Frick P. J. Terranova A. M. (2010). Patterns of physical and relational aggression in a school-based sample of boys and girls.J. Abnormal Child Psychol.38433–445. 10.1007/s10802-009-9376-3
26
Crick N. R. Dodge K. A. (1994). A review and reformulation of social-information processing mechanisms of children’s social adjustment.Psychol. Bull.11574–101. 10.1037/0033-2909.115.1.74
27
Crick N. R. Dodge K. A. (1996). Social information-processing mechanisms in reactive and proactive aggression.Child Dev.67993–1002. 10.1111/j.1467-8624.1996.tb01778.x
28
Crick N. R. Ostrov J. M. Werner N. E. (2006). A longitudinal study of relational aggression, physical aggression, and children’s social-psychological adjustment.J. Abnormal Child Psychol.34131–142. 10.1007/s10802-005-9009-4
29
Crothers L. Kolbert J. B. (2008). Tackling a problematic behavior management issue: Teachers’ intervention in childhood bullying problems.Intervent. School Clin.43132–139. 10.1177/1053451207311606
30
Crothers L. M. Field J. E. Kolbert J. B. (2005). Navigating power, control, and being nice: Aggression in adolescent girls’ friendships.J. Counsel. Dev.83349–354. 10.1002/j.1556-6678.2005.tb00354.x
31
Crothers L. M. Kolbert J. B. Kanyongo G. Y. Field J. E. Schmitt A. J. (2014). Relational and social aggression and reflective processing in a university sample.J. Aggress. Maltreatment Trauma23670–681. 10.1080/10926771.2014.920451
32
Crothers L. M. Kolbert J. B. Schmitt A. J. (2024). “Exploring the cognitive predictors of indirect bullying with implications for prevention and intervention,” in Handbook of school violence, bullying and safety, edsHongJ. S.ChanH. C.FungA. L. C.LeeJ. (Cheltenham: Edward Elgar), 481–496. 10.4337/9781035301362.00048
33
Crothers L. M. Kolbert J. B. Schmitt A. J. Wells D. S. Meidl C. Berbary C. et al (2019). Cognitive predictors of relational and social bullying, overt aggression, and interpersonal maturity in a late adolescent female sample.Intern. J. Bullying Prevent.1136–146. 10.1007/s42380-018-0001-2
34
Crothers L. M. Schreiber J. B. Field J. E. Kolbert J. B. (2009). Development and measurement through confirmatory factor analysis of the Young Adult Social Behavior Scale (YASB): An assessment of relational aggression in adolescence and young adulthood.J. Psychoeducat. Assess.2717–28. 10.1177/0734282908319664
35
Cunningham C. E. Rimas H. Mielko S. Mapp C. Cunningham L. Buchanan D. et al (2015). What limits the effectiveness of antibullying programs? A thematic analysis of the perspective of teachers.J. School Viol.15460–482. 10.1080/15388220.2015.1095100
36
Ding W. Wang Q. Hu X. Li Z. Guo D. Hong Q. et al (2022). Childhood psychological maltreatment and relational aggression among Chinese adolescents: The mediating role of moral disengagement and the moderating role of gender.J. Fam. Viol.381521–1534. 10.1007/s10896-022-00475-6
37
Dong L. Wu H. Waldman I. D. (2014). Measurement and structural invariance of the antisocial process screening device.Psychol. Assess.26598–608. 10.1037/a0035139
38
Evertson C. M. Emmer E. T. Sanford J. P. Clements B. S. (1983). Improving classroom management: Good Behavior Game manual.Element. School J.83173–188. 10.1086/461354
39
Fanti K. A. Henrich C. C. (2015). Effects of self-esteem and narcissism on bullying and victimization during early adolescence.J. Early Adolesc.355–29. 10.1177/0272431613519498
40
Fraser M. Day S. Galinsky M. Hodges V. Smokowski P. (2004). Conduct problems and peer rejection in childhood: A randomized trial of the making choices and strong families programs.Res. Soc. Work Pract.14313–324. 10.1177/1049731503257884
41
Fraser M. Galinsky M. Smokowski P. Day S. Terzian M. Rose R. et al (2005). Social information processing skills training to promote social cognition and prevent aggressive behavior in the third grades.J. Consult. Clin. Psychol.731045–1055. 10.1037/0022-006X.73.6.1045
42
Gini G. (2006). Social cognition and moral cognition in bullying: What’s wrong?Aggress. Behav.32528–539. 10.1002/ab.20153
43
Gini G. Thornberg R. Pozzoli T. (2020). Individual moral disengagement and bystander behavior in bullying: The role of moral distress and collective moral disengagement. Psychol. Violence10, 38–47. 10.1037/vio0000223
44
Gladden R. Vivolo-Kantor A. Hamburger M. Lumpkin C. (2014). Bullying surveillance among youths: Uniform definitions for public health and recommended data elements, Version 1.0.Atlanta, GA: National Center for Injury Prevention and Control, Centers for Disease Control and Prevention and U. S. Department of Education.
45
Goldstein S. E. Naglieri J. A. Princoptta D. Otero T. M. (2014). “Introduction: A history of executive functioning as a theoretical and clinical construct,” in Handbook of executive functioning, edsGoldsteinS.NaglieriJ. A. (Berlin: Springer), 3–13.
46
Gomez-Garibello C. Talwar V. (2015). Can you read my mind? Age as a moderator in the relationship between theory of mind and relational aggression.Intern. J. Behav. Dev.39552–559. 10.1177/016502541558080
47
Greene M. (2006). Bullying in schools: A plea for measure of human rights.J. Soc. Issues6263–79. 10.1111/j.1540-4560.2006.00439.x
48
Guerra N. G. Williams K. R. Sadek S. (2011). Understanding bullying and victimization during childhood and adolescence: A mixed methods study.Child Dev.82295–310. 10.1111/j.1467-8624.2010.01556.x
49
Hong J. H. Yan Y. González-Prendes A. A. Espelage D. L. Allen-Meares P. (2020). Correlates of school bullying victimization among Black/White biracial adolescents: Are they similar to their monoracial Black and White peers?Psychol. Schools58601–621. 10.1002/pits.22466
50
Hymel S. Swearer S. M. (2015). Four decades of research on school bullying: An introduction.Am. Psychol.70293–299. 10.1037/a0038928
51
Iyanda A. E. (2022). Bullying victimization of children with mental, emotional, and developmental or behavioral (MEDB) disorders in the United States.J. Child Adolesc. Trauma15221–233. 10.1007/s40653-021-00368-8
52
Jenkins L. N. Tennant J. E. Demaray M. K. (2018). Executive functioning and bullying participant roles: Differences for boys and girls.J. School Viol.17521–537. 10.1080/15388220.2018.1453822
53
Joo H. Kim S. R. Gilfillan B. H. (2023). Exploring the relationships among bias-based bullying, fear, and school avoidance in a national sample of U.S. adolescents: The role of school counselors.Prof. School Counsel.27. 10.1177/2156759X231212889[Epub ahead of print].
54
Jungert T. Karataş P. Lotti N. O. Perrin S. (2021). Direct bullying and cyberbullying: Experimental study of bystanders’ motivation to defend victims and the role of anxiety and identification with the bully.Front. Psychol.11:616572. 10.3389/fpsyg.2020.616572
55
Kawabata Y. Lenneke R. A. A. Tseng W. van IJzendoorn M. H. Crick N. R. (2011). Maternal and paternal parenting styles associated with relational aggression in children and adolescents: A conceptual analysis and meta-analytic review. Dev. Rev. 31, 240–278. 10.1016/j.dr.2011.08.001
56
Kishimoto T. Ji X. Ding X. (2023). The multivariate associations among bullying experiences, executive function, and psychological disturbance.School Mental Health151158–1172. 10.1007/s12310-023-09612-y
57
Kokkinos C. M. Voulgaridou I. Mandrall M. Parousidou C. (2016a). Interactive links between relational aggression, theory of mind, and moral disengagement among early adolescents.Psychol. Schools53253–269. 10.1002/pits.21902
58
Kokkinos C. M. Voulgaridou I. Markos A. (2016b). Personality and relational aggression: Moral disengagement and friendship quality as mediators.Personal. Individ. Differ.9574–79. 10.1016/j.paid.2016.02.028
59
Kueny M. T. Zirkel P. A. (2012). An analysis of school anti-bullying laws in the United States.Middle School J.4322–31. 10.1080/00940771.2012.11461817
60
Kuppens S. Laurent L. Heyvaert M. Onghena P. (2013). Associations between parental psychological control and relational aggression in children and adolescents: A multilevel and sequential meta-analysis.Dev. Psychol.491697–1712. 10.1037/a0030740
61
Lattanzio L. (2018). Cyberbullying, relational, and social aggression.Doctoral dissertation, Pittsburgh, PA: Duquesne University.
62
Lee-Easton M. J. Magura S. Maranda M. J. (2022). Utilization of evidence-based intervention criteria in U.S. federal grant funding announcements for behavioral healthcare.Inquiry: A J. Med. Care Organ. Provis. Financ.59:469580221126295. 10.1177/00469580221126295
63
Leff S. S. Gullan R. L. Paskewich B. S. Abdul-Kabir S. Jawad A. F. Grossman M. et al (2009). An initial evaluation of a culturally-adapted social problem solving and relational aggression prevention program for urban African American relationally aggressive girls.J. Prevent. Intervent. Commun.37260–274. 10.1080/10852350903196274
64
Leff S. S. Paskewich B. S. Waasdorp T. E. Waanders C. Bevans K. B. Jawad A. F. (2015). Friend to friend: A randomized trial for urban African American relationally aggressive girls.Psychol. Viol.5433–443. 10.1037/a0039724
65
Leff S. S. Waasdorp T. E. Crick N. R. (2010). A review of existing relational aggression programs: Strengths, limitations, and future directions.School Psychol. Rev.39508–535. 10.1080/02796015.2010.12087739
66
Liu T. L. Guo N. W. Hsiao R. C. Hu H. F. Yen C. F. (2017). Relationships of bullying involvement with intelligence, attention, and executive function in children and adolescents with attention-deficit/hyperactivity disorder.Res. Dev. Disabil.7059–66. 10.1016/j.ridd.2017.08.004
67
McGeough B. (2022). An analysis of statewide antibullying laws employing the Iowa Safe Schools Law as a case study.Child Adolesc. Soc. Work J.3997–106. 10.1007/s10560-020-00700-5
68
McLeod B. D. Wood J. J. Weisz J. R. (2007). Examining the association between parenting and childhood anxiety: A meta-analysis.Clin. Psychol. Rev.27155–172. 10.1016/j.cpr.2006.09.002
69
McQuade J. D. (2017). Peer victimization and changes in physical and relational aggression: The moderating role of executive functioning abilities.Aggress. Behav.43503–512. 10.1002/ab.21708
70
Múzquiz J. Pérez-García A. M. Bermúdez J. (2023). Relationship between direct and relational bullying and emotional well-being among adolescents: The role of self-compassion.Curr. Psychol.42115874–115882. 10.1007/s12144-022-02924-3
71
Noten M. M. Van der Heijden K. B. Huijbregts S. C. Van Goozen S. H. Swaab H. (2020). Associations between empathy, inhibitory control, and physical aggression in toddlerhood.Dev. Psychobiol.62871–881. 10.1002/dev.21951
72
Olweus D. (2013). School bullying: Development and some important challenges.Ann. Rev. Clin. Psychol.9751–780. 10.1146/annurev-clinpsy-050212-185516
73
Onishi A. Kawabata Y. Kurokawa M. Yoshida T. (2012). A mediating model of relational aggression, narcissistic orientations, guilt feelings, and perceived classroom norms.School Psychol. Intern.33367–390. 10.1177/0143034311421433
74
Ostrowsky M. K. (2010). Are violent people more likely to have low self-esteem or high self-esteem?Aggress. Violent Behav.1569–75. 10.1016/j.avb.2009.08.004
75
O’Toole S. E. Tsermentseli S. Humayun S. Monks C. P. (2019). Cool and hot executive functions at 5 years old as predictors of physical and relational aggression between 5 and 6 years old.Intern. J. Behav. Dev.43157–165. 10.1177/0165025418798498
76
Özdemir M. Stattin H. (2011). Bullies, victims, and bully victims: A longitudinal examination of the effects of bullying-victimization experiences on youth well-being.J. Aggress. Conflict Peace Res.397–102. 10.1108/17596591111132918
77
PACER’s National Bullying Prevention Center. (2019). How is “direct bullying” different from “indirect bullying”?Available online at: https://www.pacer.org/bullying/info/questions-answered/direct-vs-indirect/
78
Piaget J. (1967). Psychology, interdisciplinary relations and the system sciences. Psychol. Wychow.10, 150–171.
79
Pollastri A. R. Cardemil E. V. O’Donnell E. H. (2010). Self-esteem in pure bullies and bully/victims: A longitudinal analysis.J. Interp. Viol.251489–1502. 10.1177/0886260509354579
80
Premack D. Woodruff G. (1978). Does the chimpanzee have theory of mind?Behav. Brain Sci.1515–526. 10.1017/S0140525X00076512
81
Reijntjes A. Vermande M. Thomaes S. Goossens F. Olthof T. Aleva L. et al (2016). Narcissism, bullying and social dominance in youth: A longitudinal analysis.J. Abnormal Child Psychol.4463–74. 10.1007/s10802-015-9974-1
82
Remillard A. M. Lamb S. (2005). Adolescent girls’ coping with relational aggression.Sex Roles: A J. Res.53221–229. 10.1007/s11199-005-5680-8
83
Robers S. Kemp J. Truman J. (2013). Indicators of school crime and safety: 2012 (NCES 2013-036/NCJ 241446).Washington, D.C.: National Center for Education Statistics, U.S. Department of Education, and Bureau of Justice Statistics, Office of Justice Programs, U.S. Department of Justice.
84
Rose A. J. Swenson L. P. Waller E. M. (2004). Overt and relational aggression and perceived popularity: Developmental differences in current and prospective relation.Dev. Psychol.40378–387. 10.1037/0012-1649.40.3.378
85
Rose C. A. Slaten C. D. Preast J. L. (2017). Bully perpetration and self-esteem: Examining the relation over time.Behav. Disord.42159–169. 10.1177/0198742917715733
86
Salehinejad M. A. Ghanavati E. Rashid M. H. A. Nitsche M. A. (2023). Hot and cold executive functions in the brain: A prefrontal-cingular network.Brain Neurosci. Adv.5:23982128211007769. 10.1177/23982128211007769
87
Schmeck R. R. Geisler-Brenstein E. Cercy S. P. (1991). Self-concept and learning: The revised Inventory of Learning Processes.Educ. Psychol.11343–362. 10.1080/0144341910110310
88
Schoeps K. Mónaco E. Cotoli A. Montoya-Castilla I. (2020). The impact of peer attachment on prosocial behavior, emotional difficulties and conduct problems in adolescence. The mediating role of empathy.PLoS One15:e0227627. 10.1371/journal.pone.0227627
89
Simmons R. (2002). Odd girl out: The hidden culture of aggression in girls.San Francisco, CA: HarperOne.
90
Slawinski B. L. Klump K. L. Burt S. A. (2019). The etiology of social aggression: A nuclear twin family study.Psychol. Med.49162–169. 10.1017/S0033291718000697
91
Smith R. L. Rose A. J. Schwartz-Mette R. A. (2009). Relational and overt aggression in childhood and adolescence: Clarifying mean-level gender differences and associations with peer acceptance.Soc. Dev.19243–269. 10.1111/j.1467-9507.2009.00541.x
92
Smokowski P. R. Fraser M. W. Day S. H. Galinsky M. J. Bacallao M. L. (2004). School-based skills training to prevent childhood aggression: Using the making choices program as a universal prevention initiative.J. Prim. Prevent.25233–251. 10.1086/674055
93
Steeves T. A. (2023). Indirect aggression and victimization: Investigating instrument psychometrics, gender differences, and its relationship to social information processing.Pittsburgh, PA: Duquesne University.
94
Stopbullying.gov (2021). Prevention and intervention: Multi-tiered approaches to bullying.Washington, D.C: U.S. Department of Health and Human Services.
95
Stopbullying.gov (2023). Laws, policies & regulations.Washington, D.C: U.S. Department of Health and Human Services.
96
Stuss D. T. Levine B. (2002). Adult clinical neuropsychology: Lessons from studies of the frontal lobes.Ann. Rev. Psychol.53401–433. 10.1146/annurev.psych.53.100901.135220
97
Sutton J. Smith P. K. Swettenham J. (1999). Bullying and ‘theory of mind’: A critique of the ‘social skills deficit’ view of anti-social behavior.Soc. Dev.8117–127. 10.1111/1467-9507.00083
98
Tackett J. L. Daoud S. L. S. B. De Bolle M. Burt S. A. (2013). Is relational aggression Part of the externalizing spectrum? A bifactor model of youth antisocial behavior.Aggress. Behav.39149–159. 10.1002/ab.21466
99
Tackett J. L. Kushner S. C. Herzhoff K. Smack A. J. Reardon K. W. (2014). Viewing relational aggression through multiple lenses: Temperament, personality, and personality pathology.Dev. Psychopathol.26863–877. 10.1017/S0954579414000443
100
Thomas H. J. Connor J. P. Scott J. G. (2015). Integrating traditional bullying and cyberbullying: Challenges of definition and measurement in adolescents - A review.Educ. Psychol. Rev.27135–152. 10.1007/s10648-014-9261-7
101
Topcu Ç Erdur-Baker Ö (2012). Affective and cognitive empathy as mediators of gender differences in cyber and traditional bullying.School Psychol. Intern.33550–561. 10.1177/0143034312446882
102
Ttofi M. M. Farrington D. P. (2011). Effectiveness of school-based programs to reduce bullying: A systematic and meta-analytic review.J. Exp. Criminol.727–56. 10.1007/s11292-010-9109-1
103
U.S. Department of Health and Human Services (2023). Facts about bullying.Washington, D.C: U.S. Department of Health and Human Services.
104
Van Shoiack-Edstrom L. Frey K. S. Beland K. (2002). Changing adolescents’ attitudes about relational and physical aggression: An early evaluation of a school-based intervention.School Psychol. Rev.31201–216. 10.1080/02796015.2002.12086151
105
Verlinden M. Veenstra R. Ghassabian A. Jansen P. W. Hofman A. Jaddoe V. W. et al (2014). Executive functioning and non-verbal intelligence as predictors of bullying in early elementary school.J. f Abnormal Child Psychol.42953–966. 10.1007/s10802-013-9832-y
106
Volk A. A. Dane A. V. Marini Z. A. (2014). What is bullying? A theoretical redefinition.Dev. Rev.34327–343. 10.1016/j.dr.2014.09.001
107
Voulgaridou L. Kokkinos C. M. (2018). The relational aggression scale (RAS): Psychometric properties of a newly developed measure of relational aggression.Rev. Eur. Psychol. Appl.6811–22. 10.1016/j.erap.2017.12.001
108
Vreeman R. C. Carroll A. E. (2007). A systematic review of school-based interventions to prevent bullying.Arch. Pediatr. Adolescent Med.16178–88. 10.1001/archpedi.161.1.78
109
Wampold B. E. (2015). How important are the common factors in psychotherapy? An update.World Psychiatry14270–277. 10.1002/wps.20238
110
Wang C. Ryoo J. H. Swearer S. M. Turner R. Goldberg T. S. (2017). Longitudinal relationships between bullying and moral disengagement among adolescents.J. Youth .Adolesc.461304–1317. 10.1007/s10964-016-0577-0
111
Wang J. Iannotti R. J. Nansel T. R. (2009). School bullying among adolescents in the United States: Physical, verbal, relational, and cyber.J. Adolesc. Health45368–375. 10.1016/j.jadohealth.2009.03.021
112
Ward J. (2020). The student’s guide to cognitive neuroscience. Taylor & Francis. Available online at: https://scholar.google.com/scholar_lookup?title=The+Student%E2%80%99s+Guide+to+Cognitive+Neuroscience&author=J+Ward&publication_year=2020
113
Weidmann R. Atherton O. E. Robins R. W. (2022). Bidirectional associations between self-esteem and relational aggression from 5th to 11th grade.Eur. J. Personal.38123–140. 10.1177/08902070221141581
114
Weinreich L. Haberstroh S. Schulte-Körne G. Moll K. (2023). The relationship between bullying, learning disorders and psychiatric comorbidity.BMC Psychiatry23:116. 10.1186/s12888-023-04603-4
115
Wells M. Mitchell K. J. Jones L. M. Turner H. A. (2019). Peer harassment among youths with different disabilities: Impact of harassment online, in person, and in mixed online and in-person incidents.Children Schools4117–24. 10.1093/cs/cdy025
116
Woodcock R. W. McGrew K. S. Mather N. (2007). Woodcock-Johnson III normative update tests of achievement.Suite: Riverside Publishing.
117
Yeager D. S. Fong C. J. Lee H. Y. Espelage D. L. (2015). Declines in efficacy of anti-bullying programs among older adolescents: Theory and a three-level meta-analysis.J. Appl. Dev. Psychol.3736–51. 10.1016/j.appdev.2014.11.005
118
Zelazo P. D. Müller U. (2002). “Executive function in typical and atypical development,” in Handbook of childhood cognitive development, ed.GoswamiU. (England: Blackwell), 445–470.
Summary
Keywords
indirect bullying, intervention, perpetrators, relational aggression, Tier II and Tier III intervention
Citation
Crothers LM, Kolbert JB, Schmitt AJ, Cowley J, Perfetto K, Vafiadis A and Zawodny A (2026) How do we identify potential perpetrators of indirect bullying and how do we help them? A review of the characteristics that are associated with perpetration and can be targeted through prevention and intervention. Front. Psychol. 16:1595958. doi: 10.3389/fpsyg.2025.1595958
Received
18 March 2025
Revised
29 November 2025
Accepted
02 December 2025
Published
07 January 2026
Volume
16 - 2025
Edited by
Muhammad Waseem, Lincoln Medical Center, United States
Reviewed by
Nuno Crokidakis, Fluminense Federal University, Brazil
Sakroni Sakroni, Politeknik Kesejahteraan Sosial Bandung, Indonesia
Updates
Copyright
© 2026 Crothers, Kolbert, Schmitt, Cowley, Perfetto, Vafiadis and Zawodny.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Laura M. Crothers, crothersl@duq.edu
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.