Psychometric Properties of the Competencies Compound Inventory for the Twenty-First Century

Introduction

The world is evolving rapidly, implying that the jobs of tomorrow, the socio-economic problems and the technologies that people will have to interact with, will no longer exist (WEF, 2020). However, in the context of all these problems and issues, humanity must be able to face them successfully to continue to evolve in this world. If education, as it is thought, consists of a transmission of knowledge, which must be applied, the consequences of this rapid change imply this knowledge must be used to create new adapted ones (Trilling and Fadel, 2009; Fadel et al., 2015).

For this, a new set of skills and competencies will be necessary, and this will allow us to face the twenty-first century. It is therefore necessary to rethink education to transmit, on the one hand, this traditional knowledge, but above all, to give the necessary skills to individuals, not only to apply this knowledge but to exploit it (Trilling and Fadel, 2009; Fadel et al., 2015), on the other hand. The competencies for the twenty-first century are therefore defined as all the skills and competencies needed by individuals to face, with adaptability and consciousness, by themselves and together, the technological, societal, and economic challenges that cannot be anticipated or thought through in the present because of their fast evolution and uncertain nature. Many models propose not only defining these competencies but also listing them [e.g., P21 Framework for Twenty-first Century Learning (Lai et al., 2017); Framework for Twenty-first Century Learning from the Partnership for Twenty-first Century Skills (P21, 2009); Twenty-first Century Skills System from ATC21S (Griffin and Care, 2014); twenty-first century skills and competences for new millennium learners from Organisation for Economic Co-operation and Development, OECD (Ananiadou and Claro, 2009), etc.]. Among these models, the “Four-Dimensional Education” model from the Center of Curriculum Redesign (CCR), which is developed by Fadel et al. (2015), stands out by proposing a framework that organizes competencies of the twenty-first century, defines them in a clear and usable way, and provides levers for action for all education stakeholders.

The Four-Dimensional Education Framework

The Center of Curriculum Redesign framework proposes to divide education, as it has to be from now, for the twenty-first century, into four main dimensions: knowledge, skills, character, and meta-learning (Fadel et al., 2015).

In the CCR model that is used here, 12 competencies are grouped to form 3 dimensions: skills (4 of the 12 competencies), character (6 of the 12 competencies), and meta-learning (2 of the 12 competencies). A fourth dimension is also present in the model, the “Knowledge” dimension. It includes all the “classical” knowledge, which is generally transmitted through the school curriculum (literature, science, mathematics, etc.). If this knowledge dimension has been taught for decades or even centuries, it is also widely evaluated, especially in a numerical way (i.e., through grades), which makes it possible, among other things, to situate the student among his or her age peers and to assess his or her level at a given moment and to propose lessons adapted to his or her understanding and evolution. The idea of this research is to propose a unique tool measuring the 3 of these 4 dimensions that are not taught in traditional education and, therefore, do not benefit from a systematic common/unique evaluation.

Skills Dimension

Skills refer to the way a person uses what he or she has learned (Bialik et al., 2015b). In general, a skill is defined as “an ability or proficiency acquired through training and practice” (VandenBos, 2015).

In the CCR frameworks, these skills refer to what used to be identified as social skills (i.e., “a set of learned abilities that enable an individual to interact competently and appropriately in a given social context,” VandenBos, 2015) because they encompassed these 4 key competencies: critical thinking, communication, collaboration, and creativity. They are also known as the 4C.

Creativity refers to the interaction between aptitude, process, and the environment by which an individual or group produces a perceptible product that is both novel and useful as defined within a social context (Bialik et al., 2015b; Fadel et al., 2015). The person will be able to come up with ideas and implement actions that are both new and useful. When faced with an obstacle or an uncertain situation, a creative person can propose multiple ideas, adopt a variety of perspectives, and adjust previous actions or ideas.

Critical thinking designs the mental processes, strategies, and representations people use to solve problems, make decisions, and learn new concepts (Bialik et al., 2015b; Fadel et al., 2015). It also refers to the ability to critically evaluate information and claims one is confronted with (Bialik et al., 2015b; Fadel et al., 2015). The individual has to be able to solve problems, make decisions, and learn new things using logic and reasoning. A person who thinks critically considers alternative and opposing perspectives and is able to identify the strengths and weaknesses of each solution or draw conclusions. This involves organizing information, knowing what questions to ask, and making sense of confusing ideas.

Communication is the ability to listen to and understand information and ideas presented through spoken words and sentences (and other media,; Bialik et al., 2015b; Fadel et al., 2015). It also relates to the aim to possess adequate ability to pass along or give information through public speaking, design, presentations, and use of media (Bialik et al., 2015b; Fadel et al., 2015). A skilled communicator is able to listen actively, understand others, and express clearly and precisely knowledge and ideas. This person is able to adapt their communication style according to the audience and deliver the message using various methods, such as verbal, non-verbal, written, or digital.

Collaboration could be defined as the coordinated and synchronous activity that is the result of a continued attempt to construct and maintain a shared conception of a problem (Bialik et al., 2015b; Fadel et al., 2015). A skilled collaborator will be a solid part of a group activity or/and project with the willingness to create and maintain a shared understanding of a problem. A collaborative person can share and take responsibility, give and receive feedback, and face a conflict, if needed. This kind of person works with empathy and others can rely on them.

Characters Dimension

Characters design how people behave and engage in the world (Bialik et al., 2015a). The notion of a character or character trait is often confused, intermingled, and used instead of terms like personality, temperament, or mood. It will be more generally apprehended as the set of personality traits and attributes which include, among others, the set of social, moral, belief, and ethical characteristics of individuals (Allport, 1921; VandenBos, 2015). It relates to half of the 12 competencies of the CCR framework: mindfulness, curiosity, courage, resilience, ethics, and leadership.

Mindfulness describes a present-oriented state of conscious awareness, in which the individual is aware of multiple perspectives (Bialik et al., 2015a; Fadel et al., 2015). There is an element of openness to novelty in which the individual actively constructs categories and distinctions. The person is able to be in the present moment and aware of their own state and the state of the world. A mindful person can have multiple perspectives, be aware of and express emotions appropriately, and understand the world in its complexity. Novelty is welcomed with calmness, happiness, and openness to it.

Curiosity refers to an interest in ideas and a love for learning, understanding, and intellectual exploration; an inquisitive, playful mindset; being drawn to thinking and playing with ideas (Bialik et al., 2015a; Fadel et al., 2015). Doing reflective activities or investigations are among the favorite activities that attract a curious person. Drawn to inquiry and new ideas, this person has an open and playful mindset.

Courage is the ability to voice opinions, needs, and feelings, aiming to exert social influence; the capacity to assert one's own will to accomplish goals in the face of opposition or consequences, such as speaking out, taking a stand, and confronting others if needed (Bialik et al., 2015a; Fadel et al., 2015). One will be able to accomplish aims and goals no matter what potential constraints or obstacles there are. Courageous people do not hesitate to express their opinions, needs, and feelings regardless of the potential consequences. A courageous person can speak up, take a stand, mobilize, and confront others when necessary.

Resilience refers to the ability to deal appropriately with the ambiguity, changes, and challenges that different perspectives and experiences can present and to maintain one's identity and/or develop personally, or as a result (Bialik et al., 2015a; Fadel et al., 2015). When facing obstacles, a resilient person will see things through with patience, flexibility, and a positive mindset.

Ethics refers to a system of moral principles that affect how people make decisions and lead their lives with concern for what is good for individuals and society (i.e., it will then include moral dilemmas and decisions, rights, and responsibilities, “good or bad,” and “right or wrong” dualities; Bialik et al., 2015a; Fadel et al., 2015). An individual can make decisions based on a strong system of moral principles, such as respect, equality, honesty, and justice. An ethical person is concerned with what is good for him or herself, other individuals, and society. Ethical people consider the consequences of their own actions and try to make decisions that will leave a positive mark on the world.

Leadership is defined as a relational and ethical process of people aiming to accomplish positive change together (Bialik et al., 2015a; Fadel et al., 2015). This type of person can set goals for themselves and inspire others to follow them to accomplish a positive change. A leader collaborates and manages ethically all the personal, financial, and material resources. Leaders have a strategic mindset and a clear vision, which are needed to accomplish goals and face challenges. A leader serves as an inspiration for their community.

Meta-Learning Dimension

Meta-learning concerns reflection on oneself and the fact that a person constantly adapts, continues to grow, and learn, reaching their goals, and purposes (Bialik and Fadel, 2015). It is then divided into two main competencies: metacognition and growth-mindset.

Metacognition refers to both the ability to recognize one's knowledge, skills, attitudes/values, and way of learning and to set goals and adapt learning strategies based on outcomes (Bialik and Fadel, 2015; Fadel et al., 2015). A person would then be capable of recognizing their own knowledge, skills, behaviors, and ways of learning. A metacognitive person can reflect on and adapt their own learning strategies and adjust goals accordingly. In general, a highly meta-cognitive person demonstrates flexibility in choices, decisions, and actions, thanks to in-depth self-knowledge and the ability to self-regulate.

A growth-mindset is believing that one can change and can learn, grow, and improve one's personal future as much as seeing progress as contingent on effort and obstacles (Bialik and Fadel, 2015; Fadel et al., 2015). With this mindset, a person will be able to internalize that they have the power to effect change in themselves, others, and the world. A person with a growth-mindset suggests that one can always learn, grow, and improve and is able to see progress from the efforts they put in. These people are always capable of seeing failure, drawbacks, and feedback as a chance to grow and improve themselves.

From the CCR to CCI: A Three-Dimensional Model-Based Assessment

The first dimension of the CCR-model, knowledge, is all about what a person knows and understands (Bialik and Fadel, 2018). Even if current knowledge areas covered in curricular subjects might still need to be carefully redesigned, to include modern educational subjects, interdisciplinary courses, and some of the traditional school subjects may need to be reshaped, this dimension remains the one that has been and still is, the most worked on and developed by educators around the world. In the same way, skills, characters, and meta-learning are private study objects and have been the focus of research in the social sciences and epistemology, in which this paper is rooted. The objective of this research will be to propose, through the analysis of psychometric qualities, a sensitive, faithful, and valid measurement tool, which can be used in the evaluation of each of the previously 12 competencies defined. The objective of this research will be to propose, through the analysis of psychometric qualities, a sensitive, faithful, and valid measurement tool, which can be used in the evaluation of each of the 12 competencies previously defined and cited.

A Proposal to Measure Twenty-First-Century Competencies

The importance of developing twenty-first century competencies has been underlined by actors from different backgrounds, many of whom emphasize the need to develop them through their evaluation.

Also, in addition to proposing a model that allows us to better understand and define them, the CCR has advanced the need to measure them (Bialik et al., 2016). As said before, the knowledge dimension is already widely measured, especially in the individual's school career, but the other dimensions (skills, characters, and meta-learning) are much less so, if at all (Bialik et al., 2016). Therefore, the CCR will propose measuring the latter by using an evaluation from several angles, including the use of measurement scales to measure the level of individuals for each of these skills (Bialik et al., 2016). In fact, they present tools to assess each of the 12 competencies wishing to be measured here. Yet, no tool of an individual or an actual knowledge can measure all of these competencies in a single, quick, robust, and self-reported way.

Thus, a specific tool measuring each of those 12 competencies, with sensitive, reliable, and strongly valid items, will help caregivers, educational professionals (particularly those who develop programs to develop each of them), kids, and themselves, to evaluate first, then monitor, and finally progress on these indispensable competencies.

Item Construction and Psychometric Procedure

Based on the definitions of the 12 competencies proposed by Fadel et al. (2015), researchers specializing in the field constructed an initial pool of items in English to assess these competencies. Items were constructed to fit with children's comprehension and memory or use words that were not beyond their vocabulary level. They were designed to make them as short and explicit as possible, written in simple language, and contextualized with examples taken in their familiar environment. Then, these items were compared to the four scales: the Values in Action Inventory of Strength (and for youth version, VIA96-youth, Park and Peterson, 2006); the Global Assessment of Character Strengths (GACS-24; McGrath, 2017), which is presented as a reduced version of the VIA-96; the Motivated Strategies for Learning Questionnaire (MSLQ; Pintrich et al., 1991); and the Character Strengths Inventory for Children (CSI-C; Shoshani and Shwartz, 2018). Indeed, Bialik et al. (2016) proposed a set of measures for each competency to assess them. This research focused only on tools using self-reported measurements that were associated specifically with one of the CCR-model competencies. Also, there were many dimensions measured by the VIA-96 that overlapped with those measured by the scale that this study aimed to develop, especially for skills and characters. The MSLQ covered mostly the meta-learning dimensions. By comparing their items to the others, a set of items were removed. Finally, the remaining items were scored independently on a 0–3 scale. Items with a score of 3 were retained, and those with an average score above 2.5 were discussed and either modified or retained. The objective was to retain 4 items for each dimension. Then, the 48 items were submitted to native English speakers for a check of the language and their adaptability to a population of young people between 13 and 18 years old. After the first pre-test of these items on a sample of fewer than 100 people, the first results in terms of the global sensitivity of the scale and fidelity of each skill were encouraging; the validation procedure for the scale was then launched. The analysis started with the sensitivity of each item and each competency. It continued by checking the three dimensions for sensitivity, and finally the scale as a whole. Second, the internal consistency of the three dimensions and the whole scale was assessed. Moreover, the validity of this scale in terms of structure was tested, analyzing its hierarchical structure; a global factor of twenty-first century competencies, divided into 3 dimensions: skills, character, and meta-learning.

Study Purposes

The purpose of this study is to assess the psychometric qualities of a twenty-first century competencies tool, in particular, the CCI21 presented in this paper. To this end, two concatenated studies were conducted: a first study measuring the sensitivity and factorial validity of the CCI21 scale to iterate the scale and find a better fit, and a second study measuring the sensitivity, reliability, and validity of the revised version of the CCI21 scale after study 1.

In particular, it can be hypothesized, on the one hand, that the scale will measure twenty-first century competencies and that it does not measure social desirability (Crowne and Marlowe, 1960) or it does not measure all the character strengths of the VIA (or GACS for the short version) that were not associated with one of the twelve competencies. On the hand, it can be hypothesized that there will be a link between CCI competencies and the one from the VIA/GACS that previous authors (Bialik et al., 2016) or ourselves have made in correspondence. Also, these matched character strengths are used as convergent validity criteria while both SDS and other character strengths are used as divergent validity criteria. We will observe a positive moderate correlation between VIA/GACS correspondent strengths and CCI-21 competencies, no or weak correlation between CCI-21competencies, and both VIA/GACS non-correspondence competencies and SDS scores.

Based on this, it is expected that after both studies, the following assumptions have been made:

1. The CCI21 psychometric properties will reach moderate to high scores in sensitivity, reliability, and validity indices:

a. Item sensitivity, as well as dimension and whole scale sensitivity of the CCI21, will reach normal levels in both skewness and kurtosis indices.

b. Dimension and whole scale reliability of the CCI21 will reach good to excellent Cronbach's alpha scores.

c. The CCI21 competencies' scores will present a moderate to strong positive correlation with eleven of the character strengths of the VIA-96 (Peterson and Seligman, 2004) and the GACS-24 (McGrath, 2017) scales, identified as related to the CCI21 (Table 1), regarding the observed correlations between these two previously validated scales.

d. Competencies' scores will show null or weak correlations with the VIA-96 competencies that are unrelated, specifically, religion and spirituality, and with the social desirability scale (SDS).

2. The CCI21 global score will present no effect on age or gender.

The following table presents the crosswalk between the competencies of the present tool (CCI21) and the VIA-96 and GACS-24 character strengths (see Table 1).

TABLE 1

Table 1. Crosswalk between competencies compound inventory for the twenty-first century (CCI21) competencies and values in action inventory of strength (VIA-96)/global assessment of character strengths (GACS-24).

Study 1

Materials and Method

Participants

A total of 349 English-speaking school-age students (M = 15.33; SD = 2.57) from different countries (88% from South Africa, 12% from other countries such as England and France) were recruited to complete the protocol. The participants in this study were primarily South African school-aged students who took this study as part of their school curriculum. The study was presented to them as a study of their psychosocial competencies, the competencies that will enable them to cope with the current and future challenges of the twenty-first century. This scale was also proposed to a lesser extent to English-speaking education students in France, as part of a university course. Only participants who completed the entire study (i.e., filled out all the questionnaires) were included in this study. Of those recruited, 269 started the protocol and fully completed the scale for the twenty-first century competencies. The final sample (N = 269) consisted of 115 boys (M = 15.5; SD = 2.14), 134 girls (M = 15.1; SD = 2.91), and 7 who did not specify gender (M = 15; SD = 2.31). Twenty students did not specify their age.

Material

Socio-Demographic Questionnaire

Students were asked about their age, gender, country of study, nationality, school level, name of the school, type of school, primary language, level of English, level of studies for both parents, home conditions (own room, computer, and internet access), and their interests. These questions had to be answered by choosing from a predefined range or by inputting a text when needed.

Social Desirability Scale

To measure social desirability and prevent response bias, the Barger (2002) version of Marlow's Social Desirability Scale (SDS, Crowne and Marlowe, 1960) was used. It reported good criterion validity (Barger, 2002). To match students' comprehension, six items were adapted (e.g., I sometimes try to take revenge rather than forgive and forget), arriving at a final dichotomous scale consisting of 13 items (true vs. false).

The Competencies Compound Inventory for the Twenty-First Century

This scale proposes measuring twenty-first century competencies using a 5-point Likert scale and 48 items. Each set of 12 competencies belongs to one of these 3 dimensions: skills (CRE for creativity, CRI for critical thinking, COL for collaboration, COM for communication), characters (MIN for mindfulness, CUR for curiosity, COU for courage, RES for resilience, ETH for ethics, and LEA for leadership); and Meta-learning (MET for metacognition and GRO for growth-mindset). Every competency is measured with 4 items presented randomly.

Procedure

Participants needed to register through an online platform, the DreamShaper platform, before starting the testing procedure. After registering, participants were invited to read a detailed information notice where all types of data that would be collected and stored were clarified and a presentation of the study and its objectives was explained, and then they gave their consent to data collection through a consent form. The detail of this information notice and consent form can be sent on request. A parent or guardian had to provide explicit permission for students under 16 years old, in accordance with the legal and ethical procedures. As the study was conducted in France, European rules apply, which means that over 16 years old, participants can give their own consent for the collection of their data. Moreover, ethical advice, previous to the submission of an official ethical form, was asked to an ethical university committee, which responded that according to national laws, there was no need for ethical approval as no medical intervention was conducted on participants, and information collected could not provoke prejudice in the participants. The advice was mostly focused on two points: (1) providing an information notice and a consent form to the participants before starting the data collection, which was done through the platform; and (2) detailing data storage information and procedure of non-identification of participants. Data collected from participants were stored in a secured European server to comply with European orientations. A General Data Protection Regulation document was also created to detail safety measures in case of national control, as required in France by the French National Commission for Information Technology and Civil Liberties (CNIL). Data were anonymized by automatically assigning a number to every participant within the platform, which was non-traceable, as data exports did not include any personal identification, and analyses were only conducted with anonymized data.

Then, the study was conducted through the online platform. After having filled in the socio-demographic information, they completed the social desirability scale, and finally, the CCI21 scale.

Results

Sensitivity

Sensitivity was analyzed at three levels: items, competencies, and global.

Item Sensitivity

With means ranging from 2.59 (MC1) to 3.97 (CU3), participants responded predominantly to the right on the 5-point Likert scale. Kurtosis and skewness indices showed an abnormal rightward skew for the following items, with dimensions in parenthesis: CR1, CR3, CR4, CT1, CM1 (Skills); MF2, MF3 (Characters); and MC1, MC2 (Meta-learning) (see Table 2).

TABLE 2

Table 2. Skewness and Kurtosis scores for abnormal items of the CCI21.

Competencies Sensitivity

With means ranging from 3.27 (MET) to 3.87 (GRO), participants responded centered on the 5-point Likert scale, following a normal distribution. Kurtosis and skewness indices confirmed normal distribution with no abnormal scores for any of the competencies.

Global Sensitivity

This analysis is presented in the Table 3. It was checked for normality indexes (skewness and kurtosis) and the shape of the scale distribution (normal curve). Both demonstrate a slight trend to the right of the scale. The observed centrality indices are close to each other and close to the expected theoretical values. It is almost the same for the observed range, which is close to the theoretical value. Finally, the observed SD is slightly below the theoretical SD (0.67) contrary to what was expected.

TABLE 3

Table 3. Global score sensitivity checklist.

Reliability

Internal Consistency

The reliability of the CCI21 and its dimensions were measured using Cronbach's alpha, showing excellent reliability for the whole scale (α_CCI21 = 0.94), and good for each dimension (α_skills = 0.82; α_character = 0.89; α_{meta−learning} = 0.82). Competencies' reliability presented scores ranging from poor to acceptable [α (0.51, 0.77)].

Split-Half Reliability

A correlational approach was used to determine if there is a consistency between the odds items and even items of the scale (each part was composed of 24 items). A strong positive correlation between these two parts (r = 0.986) was found.

Validity

Factor Structure of the CCI21

Principal component analysis (PCA) was conducted on the whole 48-item inventory and on each dimension. An oblimin rotation was used because it was supposed that there might be correlations between the factors. For each analysis, the factor loadings above 1 as a criterion were used to determine the number of factors to be retained. This analysis was conducted to see if any items needed to be removed from the scale.

First, PCA was performed on the first 16 items to see if the four-dimensional structure of the Skills dimension was found. The PCA [KMO = 0.83, χ²₍₁₂₀₎ = 952 p < 0.001] leads to a four-factor solution (λ1 = 2.71, λ2 = 2.25, λ3 = 1.89, λ4 = 1.43) explaining 51.8% of the variance (Hayton et al., 2004). Analysis showed that not all the items match their factors; thus, after iteration, the item that contributed the least to the factor loadings and the competency reliability was deleted for each competency. The first factor was composed of a mix of items belonging to creativity (CR) and communication (CM). Yet, when items among the least sensitive and least saturating on this factor are removed (CR4 = 0.54 and CM4 = 0.41), two factors are observed. One gathers three items related to critical thinking, and the other gathers three items related to Communication. Factor 3 gathered the four Collaboration (CL) items, although CL2 (= 0.36) was removed based on the same criteria (lowest weight in the competency within the dimension matrix). Critical thinking (CT) items were divided between factors 3 and 4, but when deleting CT1 (= 0.84), even if it presented an increased weight, CT2, CT3, and CT4 formed one single factor. The final PCA [KMO = 0.82, χ²₍₆₆₎ = 660 p < 0.001] led us to a four-factor solution (λ1 = 1.97, λ2 = 1.92, λ3 = 1.87, λ4 = 1.41), explaining 59.7% of the variance (Hayton et al., 2004, see Table 4).

TABLE 4

Table 4. Principal component analysis (PCA) on Skills dimension items (12) for the CCI-21 36 items version.

For the next 24 items, the same analysis was conducted in order to find the six-factor model that would correspond to the Characters dimension. The PCA [KMO = 0.90, χ²₍₂₇₆₎ = 1,795 p < 0.001] led to a six-factor solution (λ1 = 3.22, λ2 = 2.56, λ3 = 2.20, λ4 = 2.20, λ5 = 2.09, λ6 = 1.21), explaining 56.2% of the variance (Hayton et al., 2004), although some items did not completely match each factor as expected. Thus, the same iteration process was followed for the skill analysis and the following items that presented either the lowest weights, or abnormal scores in item sensitivity analyses were removed: MF4 (Mindfulness item), CU3 (Curiosity item), CO4 (Courage item), RS1 (Resilience item), ET3 (Ethics item), and LD2 (Leadership item). The final PCA [KMO = 0.88, χ²₍₁₅₃₎ = 1,362 p < 0.001] led to a six-factor solution (λ1 = 2.47, λ2 = 2.18, λ3 = 1.88, λ4 = 1.91, λ5 = 1.77, λ6 = 1.41), explaining 56.2% of the variance (Hayton et al., 2004, see Table 5).

TABLE 5

Table 5. PCA on Characters dimension items (6) for the CCI-21 36 items version.

Finally, the same analysis was conducted for the last 8 items to find the two-factor solution for the Meta-learning dimension. The PCA [KMO = 0.82, χ²₍₂₈₎ = 594 p < 0.001] led to a two-factor solution (λ1 = 3.09, λ2 = 1.62), explaining 58.8% of the variance (Hayton et al., 2004, see Table 5). The same path as before was followed and indicated the need to delete items MC4 (Metacognition item) and GM2 (Growth Mindset item). The final PCA [KMO = 0.74, χ²₍₁₅₎ = 411 p < 0.001] led to a two-factor solution (λ1 = 2.38, λ2 = 1.57), explaining 65.8% of the variance (Hayton et al., 2004, see Table 6).

TABLE 6

Table 6. PCA on Meta-learning dimension items (6) for the CCI-21 36 items version.

Discussion of Study 1

Competency sensitivity presented normal scores, although item sensitivity and global sensitivity presented some issues. The sensitivity of the items suggests a need to review or delete certain items, which was confirmed by the results for global sensitivity. Minor changes would be needed to improve the mode and dispersion indexes. In the same way, reliability analyses presented excellent and good scores for scale and dimension, nevertheless, competencies' reliability presented some poor scores, suggesting that items might need minor changes. Considering this information, the PCA analyses were satisfactory once iterations on items were made for the dimensions “skills and characters”. Nevertheless, the dimension “meta-learning” presented one item that was more linked to growth-mindset competency than to its original competency, the metacognition. In this sense, to arrive at a fully fitted model, a proposal to modify this item can be formulated, and also a new study with a revised version of the scale could be undertaken. All the other dimensions, after iteration, presented a good fit, which led us to continue the criterion validity study with this renewed 36-item CCI scale. This study will be presented in the following section.

Study 2

Materials and Method

Participants

A total of 162 English-speaking students (M = 15.6; SD = 2.43; 66 boys, 88 girls, and 6 who did not specify gender) fully completed the protocol. One student did not provide their age.

Material

Besides the already described scales, such as the socio-demographic questionnaire (Demo) and the Social Desirability Scale (SDS, Crowne and Marlowe, 1960; Barger, 2002), the following scales were used:

The Competencies Compound Inventory for the Twenty-First-Century−36-Item Version

This scale is the 36-item version that was created after factor analyses of the 48-item scale. The CCI21-36 is a self-reporting scale that measures 12 of the twenty-first century competencies through 3 items per competency on a 5-point Likert scale, ranging from not like me at all to very much like me.

The Values in Action Inventory of Strength 96

The scale is a youth adapted version of 24-character strengths measured with 4 items on a 5-point Likert scale, presenting a pole of 96 items (VIA 96-Youth, Park and Peterson, 2006). Traits evaluated are as follows: appreciation of beauty and excellence, bravery, creativity, curiosity, fairness, forgiveness, gratitude, honesty, hope, humility, humor, judgment, kindness, leadership, love, love of learning, perseverance, perspective, prudence, self-regulation, social intelligence, spirituality, teamwork, and zest. This scale presented good validity across different countries, including European and American countries (McGrath, 2017). Reliability scores of global scores for our sample achieved excellent scores for Cronbach's alpha (α = 0.96).

Global Assessment of Character Strengths-24

Students' character strength was also measured by asking them to read each character definition and then establish, using a Likert 7-point scale, how accurately this trait describes them (GACS, McGrath, 2017). This scale has been validated for adult use, although the author was contacted to ask if the scale could be used on a youth population, establishing it as having vocabulary adapted for a youth student population.

Procedure

Within the online platform Dreamshaper, after completing the questionnaires Demo, SDS, and CCI21-36, in order to establish convergent and divergent validity, students were asked to complete the VIA-96 and the GACS-24 scales.

Data Analysis

Kurtosis and skewness indices were calculated to measure sensitivity. Cronbach's alpha was calculated to measure internal consistency. A split-half method was used to measure test reliability through correlation analysis. Then, confirmatory factor analysis were performed to verify the factor structure of each dimension, that is to say for Skills dimension a four-factor solution, for Characters a six-factor solution and for Meta-learning dimension a two-factor solution. Finally, correlational analyses were performed to test for convergent and divergent validity.

Results

Sensitivity

Global scores and competencies scores followed a normal distribution.

Reliability

Internal Consistency

The global scores for the CCI21-36 presented excellent internal consistency (α_CCI21−36 = 0.93). Similar results were found for each dimension (α_skills = 0.79, α_character = 0.87, α_{meta−learning} = 0.78).

Split-Half Reliability

A correlational analysis was undertaken to determine if there was consistency between the odd items and even items of the scale (each part is composed of 18 items). It found a strong positive correlation between those two equal parts (r = 0.94).

Validity

Structural Validity

For each dimension, confirmatory factor analysis was conducted to test whether the multiple-factor structure fitted our data better than a single-factor structure.

Confirmatory factor analysis was conducted to test whether a four-factor structure fitted our data better than a single-factor structure (see Table 7). The analysis confirmed our hypothesis and led us to adopt this four-factor solution with 3 items per factor.

TABLE 7

Table 7. Confirmatory factor analysis on CCI21-36.

Again, for the character dimension, the analysis continued with confirmatory factor analysis to confirm how the six-factor model matched the data better than the one-factor model (see Table 7 below). As expected, this multiple-factor solution was the one solution that fitted the best, with 3 items per factor.

Confirmatory factor analyses were then conducted to see if the two-factor model matched the data better than the one-factor model, but this was not confirmed (see Table 7 below). Yet, this result is consistent with the PCA performed in this study, as the item MC3 matches more with “growth-mindset” than with the factor identified as “metacognition.”

Convergent/Concurrent Validity

As the table above shows, correlations between corresponding/concurrent dimensions (the characters of the VIA-96 that are aligned with the competencies of the CCI21-36) showed significant moderate correlations as expected in comparison with the correlations between the VIA-96 and the GACS-24 (see Table 8). The only correlations that appear to be weaker are as follows: (a) the competency “critical thinking,” aligned with the character “judgment,” but that presents better correlations with the GACS-24 character “judgment;” and (b) the competency “collaboration,” aligned with the character “teamwork,” which presents weaker correlations between the CCI21-36 and both concurrent scales, than the correlations between VIA-96 and GACS-24. Almost all other correlations between CCI21-36 and the concurrent scales appear to be stronger than the correlations between the VIA-96 and the GACS-24 (see Table 8).

TABLE 8

Table 8. Comparative table of final correlation scores between VIA-96 and GACS-24 characters, VIA-96 characters and CCI21-36 competencies, and GACS-24 characters and CCI21-36 competencies.

Divergent Validity

As expected, global scores for the CCI21-36 and each competency score presented weak to moderately weak correlations with the social desirability scale [M r = 0.26; (0.10–0.37)], as well as the “spirituality” character from the VIA-96 [M r = 0.25; (0.12–0.35)] (see Table 9).

TABLE 9

Table 9. Correlations between CCI21-36 & competencies, SDS, and VIA-96 spirituality character.

It is also observed that the global score strongly correlates with each of the competencies scores from the CCI21-36 scale (M r = 0.69; (0.56–0.79)] (see Table 9).

Gender and Age-Related Differences

One-way ANOVA analysis showed no effect for gender or for age.

Discussion of Study 2

The results of sensitivity analyses showed us that the scale presents normal levels of sensitivity in all the evaluated forms, leading us to conclude that the scale is sensitive enough in terms of items, dimensions, and as a whole scale. Regarding reliability, the scale presented excellent internal consistency, confirmed by PCA analyses. Finally, validity analyses confirmed the structure of each dimension and also showed consistent moderate correlations with concurrent dimensions of scales that measured similar constructs. In the same way, when competencies from the scale were contrasted with scales or dimensions measuring different constructs, the scale presented weak to moderate correlations. Moreover, the global score correlated strongly with each of its items, allowing us to conclude that the CCI21-36 presents correct validity indexes.

General Discussion

The CCI-21 scale is a self-report questionnaire that measures twenty-first century competencies based on the Four-Dimensional Education framework from the Center for Curriculum Redesign (CCR). They propose dividing Education and twenty-first century competencies into four main dimensions: knowledge, skills, character, and meta-learning. The scale presented in this study aimed to measure the three last dimensions of this model consisting of the 12 competencies that it covers. This study focused on two main objectives: (1) to present CCI21 psychometric properties that will give moderate to high scores in terms of sensitivity, reliability, and validity indices; and (2) to not find an age or gender effect on the CCI21 global and competency-based score. As a whole, these objectives have been achieved, allowing us to propose a scale, not in 48 but in 36 items, measuring the skills of the twenty-first century.

Thereby, to reach these goals, for each of the previous psychometrics qualities, analyses were conducted at several levels. The first version of the scale presented 48 items and was administered to a sample of international English-speaking middle- to high-school students. Results from psychometric analyses confirmed that there were some issues regarding the scale, particularly sensitivity and factor analyses, that suggested that some of the items needed further revision or to be deleted to correspond to a good model fit. After iterations of the scale, CCI21 was reduced from 48 items to arrive at a 36-item scale that was tested on the sample of students who completed the whole protocol (a protocol that would permit convergent and divergent validity analyses). This sample produced satisfying results in terms of sensitivity, reliability, and validity. These results permitted us to confirm that the 36-item version presents good psychometric qualities, which was confirmed by correlational analyses.

In this context, the hypotheses laid out under hypothesis 1 were confirmed as all psychometric properties presented good scores in the evaluated indices: for 1a, both skewness and kurtosis indices presented normal levels for each of the competencies of the scale; for 1b, Cronbach's alpha scores were excellent; for 1c, even if correlations were mainly moderate or moderately strong, they corresponded to the correlations expected in this type of analysis, sometimes presenting even stronger correlations than expected; for 1d, correlations with the “spirituality” character and the social desirability scale were weak and moderately weak, as expected, being always weaker than the correlations presented by the global score of the CCI21-36 and its competencies. Finally, for hypothesis 2, the global score for the CCI21-36 showed no effect from age or gender.

This study encountered some difficulties inherent in the validation of the tool. First of all, the recruitment of a large sample of English-speaking school-age children, a population that is difficult to interview, and the fact that the research team was not based in an English-speaking country caused difficulties. Access to the sample was coupled with the absence of a test-retest, which was very difficult to conduct without a high percentage of lost subjects. Moreover, the length of the study (i.e., too long) did not favor this research. Finally, it was not possible to retest the sample, one of the reasons being the health crisis context related to the coronavirus disease 2019 (COVID-19). Indeed, the global health context in the year 2021 may have slowed down the conduct of this study, the priority being the maintenance of educational continuity and programs, which was not easy for the schools in this pandemic context. In addition to the difficulties encountered, the study has limitations.

First of all, even if the sample was representative of a population of English-speaking students, future studies might require a larger sample than would confirm these claims. In this sense, colleagues have to consider this study cautiously regarding variables that could diminish the replicability of the results. In line with this, differences in the sample of the study, such as cultural differences, level of English, or beliefs regarding social and emotional learning should be considered as potential variables that might impact the results.

Self-report scales always represent a limit in studies, since it is almost impossible to ensure that the results of the scale are incontestable, as social desirability might play an important role in the type of response given by participants, particularly when working with adolescents. Here it can be observed that there is a link, yet weak, between the SDS and CCI-21 scores. Then, the results of those with very high scores (i.e., more than 2.5 standard deviations) on the desirability scale must be interpreted with caution or could even be uninterpretable.

Moreover, the perspectives of parents, educators, or peers were not considered in the construction of the tool, but only the student perspective, which some educational actors might consider to be an issue in terms of the evaluation of these competencies. A way to improve accuracy in the results could be to complement this tool with other measurements (e.g., educators' evaluation, parents' evaluation, and peers' evaluation) to have cross regards and perspectives and be able to provide a full profile of the students regarding their twenty-first century competencies. Nevertheless, the present tool presents reliable properties that permit us to claim that self-reported competencies can be measured accurately with it, which might have positive implications for the educational sector, particularly in terms of social and emotional learning.

Future research could be conducted on the CCI-21 36 version. For this CCI-21, shorter versions (with 36 items), it should be found the same (non-)correspondence made between the character of strengths of the VIA/GACS (or global score of SDS) and competencies of the CCI-21 48-version. Then, correlations of the same order, with similar values will be expected. Yet, to make the evaluation better and more user friendly, we can do without VIA-96 (only keeping the GACS), if, again, the convergent validity procedure is considered necessary (it is recommended but not mandatory in our opinion). Indeed, redoing and rechecking the fidelity, using an analysis of the internal consistency (Cronbach's alpha), of stability over time (using a test-retest) and confirmatory factorial analysis, seems to be the minimum and the best to consider. Going further, other variables can also be related to this measure, such as emotional intelligence (Mayer and Salovey, 1997), or we could even take an interest in the predictive validity of the scale in terms of academic success or educational and professional orientation.

Finally, some recommendations can be made on the use of this assessment tool. It could serve to evaluate students' perception of their level of competencies for the twenty-first century during their development throughout their schooling, to check their evolution in time, or even to check which competencies might need improvement or more work. It may be useful to analyze both the overall score and the competency score of school-age youths who would be assessed on this test. The global level will allow professionals to apprehend the general level of the person in terms of twenty-first century competencies, but considering the imbalance of the different dimensions in the global score, a competency-based analysis will refine the analysis and the possibilities for intervention. Indeed, this will give a more precise mapping of these competencies, allowing educators to identify if the student presents a homogeneous profile on the whole for these competencies or is rather heterogeneous with competency assets (or “strengths,” high scores on particular competencies), or competencies to develop (or “weaknesses,” low scores on particular competencies). Such an assessment will not only allow the participant to be situated among his or her peers (with the help of calibration). This will also allow individual or collective intervention or remediation work to help them optimize each of their twenty-first century competencies. For the professional, this will also allow the construction of educational programs in accordance with the level of the student (or students) and individualized or personalized to the class group. It could also be of great use to guide annual reports on students, interviews with parents, and even to help identify upcoming issues in the classroom. The scale could also be useful to test the effectiveness of learning programs concerning twenty-first century competencies or social and emotional learning programs that are to be used in the classroom, and even as a measure to define improvements in actions that have already been carried out within the school. In this sense, the scale could be a good asset for educational actors, which is why the team is currently working on the English adaptation (actually to extend it) of the scale for children aged 10–12, and a study is being conducted in order to validate the French version of the scale.

Conclusion

The CCI21-36 is a valid tool to measure the twenty-first century competencies of 13-year-olds and older youths. The scale not only presents good psychometric properties but also has promising implications in the field of the social and emotional education of youths.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. Written informed consent to participate in this study was provided by the participants' legal guardian/next of kin.

Author Contributions

Both authors listed have made a substantial, direct, and intellectual contribution to the work and approved it for publication.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Allport, G. W.. (1921). Personality and character. Psychol. Bull. 18, 441.

Google Scholar

Ananiadou, K., and Claro, M. (2009). 21st Century Skills and Competences for New Millennium Learners in OECD Countries. Paris, France: The Organisation for Economic Co-operation and Development, OECD.

Google Scholar

Barger, S. D. (2002). The Marlowe-Crowne affair: Short forms, psychometric structure, and social desirability. J Pers Assess 79, 286–305. doi: 10.1207/S15327752JPA7902_11

PubMed Abstract | CrossRef Full Text | Google Scholar

Bialik, M., Bogan, M., Fadel, C., and Horvathova, M. (2015a). Character Education for the 21st Century: What Should Students Learn? Boston, MA: Center for Curriculum Redesign.

Google Scholar

Bialik, M., and Fadel, C. (2015). Meta-Learning for the 21st Century: What Should Students Learn? Boston, MA: Center for Curriculum.

Google Scholar

Bialik, M., and Fadel, C. (2018). Knowledge for the Age of Artificial Intelligence: What Should Students Learn? Boston, MA: Center for Curriculum.

PubMed Abstract | Google Scholar

Bialik, M., Fadel, C., Trilling, B., and Groff, J. S. (2015b). Skills for the 21st Century: What should Students Learn? Boston, MA: Center for Curriculum.

Google Scholar

Bialik, M., Martin, J., Mayo, M., and Trilling, B. (2016). Evolving Assessments for a 21st Century Education The Ellen Koshland Family Fund . Center for Curriculum Redesign. Available online at: http://curriculumredesign.org/wp-content/uploads/Evolving-Assessments-for-the-21st-Century-Report-Feb-15-Final-by-CCR-ARC.pdf

Google Scholar

Crowne, D. P., and Marlowe, D. (1960). A new scale of social desirability independent of psychopathology. J. Consult. Psychol. 24, 349–354. doi: 10.1037/h0047358

PubMed Abstract | CrossRef Full Text | Google Scholar

Fadel, C., Bialik, M., and Trilling, B. (2015). Four-Dimensional Education: The Competencies Learners Need To Succeed. In C. Fadel (Ed.), Boston, MA: Center for Curriculum Redesign.

Google Scholar

Griffin, P., and Care, E. (2014). “The ATC21S method,” in Griffin P., Care E. (Eds.), Assessment and Teaching of 21st Century Skills: Methods and Approach (Dordrecht: Springer), 3–33. doi: 10.1007/978-94-017-9395-7_1

CrossRef Full Text | Google Scholar

Hayton, J. C., Allen, D. G., and Scarpello, V. (2004). Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis. Organ. Res. Methods 7, 191–205.

Google Scholar

Lai, E., DiCerbo, K., and Foltz, P. (2017). Skills for Today: What We Know About Teaching and Assessing Collaboration. Pearson.

Google Scholar

Mayer, J. D., and Salovey, P. (1997). What is emotional intelligence. Emot. Dev. Emot. Intell. Educ. Implic. 3, 31.

Google Scholar

McGrath, R. E. (2017). Technical Report: The VIA Assessment Suite for Adults: Development and Evaluation. Cincinnati, OH: VIA Institute on Character.

Google Scholar

P21 Partnership for 21st Century Skills. (2009). A Framework for Twenty-First Century Learning. Available online at: https://static.battelleforkids.org/documents/p21/P21_Framework_DefinitionsBFK.pdf

Google Scholar

Park, N., and Peterson, C. (2006). Moral competence and character strengths among adolescents: the development and validation of the values in action inventory of strengths for youth. J. Adolesc. 29, 891–909. doi: 10.1016/j.adolescence.2006.04.011

PubMed Abstract | CrossRef Full Text | Google Scholar

Peterson, C., and Seligman, M. E. (2004). Character Strengths and Virtues: A Handbook and Classification (Vol. 1). Oxford, LO: Oxford University Press.

Google Scholar

Pintrich, P. R., Smith, D. A. F., Garcia, T., and McKeachie, W. J. (1991). A Manual for the Use of the Motivated Strategies for Learning Questionnaire (MSLQ). (Vol. 1). Washington, DC: ERIC, Educational Research and Improvement.

Google Scholar

Shoshani, A., and Shwartz, L. (2018). From character strengths to children's well-being: development and validation of the character strengths inventory for elementary school children. Front. Psychol. 9, 2123. doi: 10.3389/fpsyg.2018.02123

PubMed Abstract | CrossRef Full Text | Google Scholar

Trilling, B., and Fadel, C. (2009). 21st Century Skills, Enhanced Edition: Learning for Life in Our Times. San Francisco, CA: Jossey-Bass.

Google Scholar

VandenBos, G. R, (Ed.). (2015). APA Dictionary of Psychology. American Psychological Association. Available online at: https://dictionary.apa.org/

Google Scholar

WEF: World Economic Forum. (2020). The Global Risks Report 2020 Insight Report 15th Edition. https://www.weforum.org/reports/the-global-risks-report-2020/

Google Scholar