Can community and educational interventions designed from the ground-up promote social and emotional learning? Experimental and quasi-experimental impacts of a country-wide Portuguese initiative

1 Introduction

Evidence has shown that social and emotional learning (SEL) at a young age is a powerful predictor of a variety of outcomes earlier and later in life, such as relationships with others, citizenship behavior, academic performance (Sackett and Walmsley, 2014), higher sense of school belonging (OECD, 2021a), mental and physical health (Strickhouser et al., 2017), as well as overall job and life satisfaction (Judge et al., 2002; Scorza et al., 2016).

There is also growing evidence that this set of skills is malleable, as opposed to somewhat fixed traits of personality (Weissberg et al., 2015), and can be promoted through high-quality SEL intervention programs (Jones et al., 2019), such as Second Step (Committee for Children; www.cfchildren.org), Incredible Years (Webster-Stratton, 1981, 1982), or the PATHS program (Kusché and Greenberg, 1994). Most available social and emotional learning interventions are universal (e.g., the aforementioned Second Step, PATHS, or Slowly but Steadily), i.e., their approach is aimed at promoting both protective factors and key competences, targeting a large audience of children and youth (Alexandre and Barata, 2020). These programs can be designed for all ages, from kindergarten to university students, since many of these skills start developing very early in life (e.g., Domitrovich et al., 2017; Blewitt et al., 2018).

Most SEL interventions are conducted in a school context, but there is a growing number of programs conducted outside of school time (e.g.: Kremer et al., 2015). These programs can be designed and implemented locally, or developed to be implemented country-wide, such as the 21^st Century Community Learning Centers, Boys & Girls clubs, or 4-H Clubs being validated in the US (Durlak et al., 2010; Kremer et al., 2015). These community-based interventions are often designed from the ground-up, address local needs, and offer promising pathways to promoting social and emotional learning. However, they often lack rigorous monitoring and evaluation of SEL change oftentimes because such evaluation procedures require additional resources and time so that local providers may acquire and implement the technical skills needed for rigorous methods of program evaluation. Universal programs are a critical component of a multitiered system of supports, as they are likely to have the greatest reach and potential to prevent future problems. Evidence for the effectiveness of universal approaches to SEL is still lacking (Wallender et al., 2020) but is essential to inform efforts to promote the psychosocial functioning and mental wellbeing of students (Green et al., 2021).

To address these gaps in the literature, and with an aim to influence educational policymaking, the Portuguese Gulbenkian Foundation offered to co-fund 100 intervention approaches to SEL, named Gulbenkian Academies for Knowledge (henceforth referred to as Academies), which sought to promote skills of children and youth between 0 and 25 years old. These interventions took place within schools or communities across the country, over 3 cohorts and 4 years of implementation, through a variety of different methodologies and focus areas, including sports, STEM, arts, or technology.

Each of these Academies benefited from supervision from an external Monitoring and Evaluation (M&E) Team, which supported them in developing and implementing rigorous implementation monitoring and impact evaluation plans. This included regular group training sessions inspired in the Data Wise model (Boudett and Steele, 2007; Boudett et al., 2020) on important topics such as the design of a theory of change incorporating the main implementation dimensions (Weiss, 1995), rigorous impact evaluation, measurement of social and emotional skills, careful monitoring of the implementation process, and finally communication and dissemination. This amounted to a five-session training program for all teams, combined with individual tutoring based on each team’s needs.

This paper presents the impact results of 40 Academies which chose to implement rigorous experimental and quasi-experimental methods, and use a standardized measure of SEL, the OECD Survey of Social and Emotional Skills, to measure the impact of their SEL intervention. The impact evaluation of the Gulbenkian Academies for Knowledge has the potential to support considerable changes in the field of SEL interventions, nationally and internationally, by informing discussion of evidence-based SEL interventions, and offering a sustainable model of capacity building with long-lasting effect on practices of SEL professionals.

2 Importance of social and emotional learning

Social and emotional skills are a multidimensional construct that encompasses a set of intrapersonal and interpersonal competencies that are important for an individual’s global functioning, and to successfully interact with others (Domitrovich et al., 2017).

Despite there being some consensus on the conceptual domain and importance of these skills, there is a wide variety of theoretical frameworks attempting to define, organize and operationalize SEL (Kotsiou et al., 2022). The most widely cited approach is by the Collaborative for Academic, Social, and Emotional Learning (CASEL), which defines SEL as the “process through which young people and adults acquire and effectively apply the knowledge, skills, and attitudes to develop healthy identities, manage emotions and achieve personal and collective goals, feel and show empathy for others, establish and maintain supportive relationships, and make responsible and caring decisions” (CASEL, 2020, p. 5). This conceptual approach frames SEL as five broad, interrelated areas of competence: self-awareness, self-management, social awareness, relationship skills, and responsible decision-making (CASEL, 2020).

Using a combination of the CASEL, the Big Five model (Goldberg, 1990), and other conceptual frameworks, the OECD proposed a policy approach to this domain highlighting the malleable, learnable and context-dependent character of the skills (Kankaraš and Suarez-Alvarez, 2019). The OECD approach organized SEL in five dimensions: Collaboration, Task Performance, Emotional Regulation, Engagement with Others, and Open-mindedness. Each dimension then encompasses several individual skills, which are the focus of the SSES, as can be seen in Table 1.

Table 1

Table 1. Domains, definitions, and skills from OECD conceptual framework for social and emotional skills (OECD, 2021a).

Based on this approach, the OECD implemented the first large-scale international study on social and emotional skills for children and youth – the Study on Social and Emotional Skills (SSES; OECD, 2021a). This study aimed to describe how SEL develops in children and youth, and to map aspects of their daily settings – family, school, community – that could potentially promote or hinder the development of SEL. The study collected data in 10 cities around the world, with over 60,000 participants of 10 and 15 years of age, their parents, and teachers. The survey collected data on 15 different social and emotional skills, as well as on sociodemographic, family, school, and community contextual characteristics. Portugal was represented in this study by the Municipality of Sintra, contributing with over 3,000 participants, and thus constituting the sample for the initial Portuguese adaptation of the survey instrument.

The OECD study, and the resulting survey, stand as a valuable effort to develop a comprehensive measure to the assessment of a broad array of social and emotional skills, allowing for researchers and practitioners to further delve into the evidence-based promotion and evaluation of SEL (Castro et al., 2023). The data from the study further confirmed the positive association between social and emotional skills and school achievement in reading, mathematics, and arts; the maximum educational level students expect to attain; life satisfaction and psychological wellbeing; and social relations at school, both with teachers and peers (OECD, 2021a).

3 Diversity of SEL interventions

The emphasis given to social and emotional learning in the school context, and associations with other skills and well-being, as well as the notion that these competences are malleable and can be learned, has given rise to a significant number of programs to promote social and emotional competences.

Two of the most widely implemented, studied and replicated programs in the school context are the US-based Promoting Alternative Thinking Strategies (PATHS) and the Second Step programs. The PATHS program (Kusché and Greenberg, 1994) aims to promote self-control, emotion regulation, attention, communication, and problem solving in students from kindergarten to 6^th grade, to ultimately reduce behavioral problems and improve teaching and learning in the classroom. This program is designed to be implemented by teachers or school counselors over the school year (Domitrovich et al., 2019). Another famous program is the Second Step program (Committee for Children; www.cfchildren.org), which directly targets students’ empathy, emotional management, and problem solving skills, in order to “strengthen their ability to learn” (Low et al., 2016) with 24 in-class weekly 30 min sessions implemented by teachers from kindergarten to second grade.

The majority of SEL interventions are implemented in school settings – ideally using a whole-school approach, given the privileged characteristics of these contexts in accessing all children, and tapping into the many levers of a systemic intervention (Corcoran et al., 2018; Durlak et al., 2022). There is strong evidence for a whole-school approach in promoting social and emotional skills, in comparison to focusing on a single school level, agent, or group of students (Goldberg et al., 2019). Examples of whole-school approaches to SEL include the Positive Action program (in the US; Flay and Allred, 2010), focused on promoting positive self-concept and self-esteem in K-12 students, with in-class, extra-class and family components; or the RESCUR program (in Europe; Cefai et al., 2015), a resilience curriculum for early years and primary schools aimed at develop children’s relationships skills, growth mindset, and self-determination, by connecting families and school professionals in benefiting from the intervention (Cefai et al., 2018).

However, SEL interventions can go beyond the school context, and often target skills broader than those relevant for school success. For example, the Incredible Years program was developed by Webster-Stratton (1981, 1982) to impact skills much earlier in life. Incredible Years was first designed as a parental intervention for reducing behavioral problems and promoting SEL in children aged between 3 and 8 years old, and has since been adapted to teacher-and child-focused interventions, and widely implemented worldwide. Its aim is to reduce risk factors in parent and teacher practices, as well as early onset conduct problems and emotional difficulties in young children, via the promotion of the child’s social and emotional development (Fossum et al., 2017).

SEL programs can also be implemented after-school, and/or outside school grounds in a variety of community settings (Durlak et al., 2010; Kremer et al., 2015; Schwartz et al., 2020). The diversity of contexts as also broadened the scope and thematic areas of SEL programs, now including such diverse approaches to skill development as STEM, arts, sports, among others. The 21st Century Community Learning Centers (CCLCs), in place in the US since 1994, is a program aiming to “open up schools for broader use by their communities” (James-Burdumy et al., 2005, p.1). After school hours and during school break periods; CCLCs offer regular opportunities for students and families to improve academic performance, benefit from a safe environment and from cultural enrichment opportunities, enjoy recreational activities, develop socially, and benefit from various family services. Similarly, the 4-H Clubs (Kremer et al., 2015) aim to promote positive youth development by leading children and youth to design and implement community development projects in different areas (health, science, agriculture, civic engagement) with the guidance from adult mentors, encouraging participants to take on proactive leadership roles, through in-school and after-school programs, and school and community clubs.

In Portugal, some of the most widely replicated and validated interventions include the Slowly but Steadily program (Raimundo, 2007), which consists of 21 teacher-led, in-class 1 h sessions, with students between 1st and 6th grades (i.e., 6–12 years old); or the Positive Attitude program (Coelho and Figueira, 2011), consisting of 13 one-hour weekly sessions, implemented by the school psychologist in the classroom, with students from 7st to 9th grade (i.e., 13–15 years old). Both programs are based on the CASEL approach to social and emotional learning, and both target its domains of self-awareness, social awareness, self-control, interpersonal relationships, and responsible decision making, operationalizing them through different activities and program dosages to each skill (Raimundo et al., 2013; Cristóvão et al., 2017; Coelho et al., 2023).

4 Monitoring and evaluating SEL programs

Keeping up with the increase in the number of SEL interventions developed over the past decades, there has been a systematic concern for evaluating the impact of SEL programs. Some of these programs have provided evidence of impacts in a set of important domains in the lives of children and young people (Durlak et al., 2010, 2011; Boncu et al., 2017). Research has shown that promoting social and emotional skills positively affects academic achievement in reading, mathematics, and science (Corcoran et al., 2018), as well as in children’s school engagement (Santos, 2022). Additionally, SEL interventions consistently show results in decreasing bullying and externalizing behaviors, as well as in increasing overall mental health and well-being (e.g., Durlak et al., 2010; Domitrovich et al., 2017; van de Sande et al., 2019). These positive impacts are seen not only in the short term, but also medium and long term, for instance by being related to better jobs and higher income in adulthood (Chernyshenko et al., 2018).

For instance, the PATHS program reports an increase in participants’ emotional understanding, self-control, social problem solving, peer relations, and a decrease in externalizing symptoms (Domitrovich et al., 2019). The Incredible Years program has been showing an improvement in children’s social skills and social competence, and a decrease in disruptive behaviors, aggression, and internalizing problems (Fossum et al., 2017). As for Portuguese interventions, Slowly but Steadily has shown effects in participants’ peer relations and social competence; whereas Positive Attitude recently showed impacts on social awareness, self-control, relationship skills, and responsible decision making in a national sample of young participants (Coelho et al., 2023).

Despite the evidence of positive benefits, Tanner-Smith et al. (2018), in a review of meta-analyses, highlight that the effect sizes found in programs promoting social and emotional skills are lower than those found in other scientific areas (as stipulated by Cohen, 1988). More recent meta-analyses (e.g.: Cipriano et al., 2023) confirm the results already found elsewhere (e.g.: Durlak et al., 2011; Tanner-Smith et al., 2018) regarding these reduced effect sizes, as well as the fact that these interventions promoting social and emotional competences still demonstrate greater effects on beliefs and attitudes, social and emotional competences, and reduction of emotional stress, than on the reduction of externalizing behavior or on academic performance.

Considering that most meta-analyses and systematic reviews on SEL program implementation and evaluation include mostly studies from Anglo-Saxon countries (e.g., Corcoran et al., 2018), a recent systematic review by Fernández-Martín et al. (2021) analyses the efficacy of Ibero-American SEL interventions on increasing school performance and social and emotional skills in children of different school grades. In this study, which found similar results to those found in research in Ango-Saxon countries, 12 Portuguese studies evaluating the impact of SEL interventions were included (Fernández-Martín et al., 2021).

Despite Portugal being identified as one of the countries in the Ibero-American scene that has been most committed to implementing and evaluating SEL interventions in educational settings over the last decade (Fernández-Martín et al., 2021), the evidence of social and emotional learning or social and emotional skills focused programs is still scarce and scattered in Portugal. Cristóvão et al. (2017) found a total of 19 publications regarding SEL program evaluations in Portugal, over an eight-year period (between 2008 and 2016). Although an increase in the number of publications is to be expected up to the present date, there is still a very small proportion of SEL programs being evaluated in Portugal in relation to those which are implemented in schools and community services across the country (Cristóvão et al., 2017). Thus, the current state of research on SEL program implementation and evaluation corroborates the importance and efficacy of these programs and adds to the need to monitor and evaluate social and emotional skills-focused programs in the Portuguese setting.

5 Gulbenkian academies for knowledge

In 2018, the Portuguese Calouste Gulbenkian Foundation set out to implement a national mechanism for the development and support of innovative solutions for complex societal problems. In order to do so, the Foundation offered to co-fund intervention approaches to SEL, named Gulbenkian Academies for Knowledge (henceforth referred to as Academies). Academies could include a broad array of domains of intervention such as educational, science learning, health, civic participation, among others, but had to contribute to the common goal of developing social and emotional competencies of children and youth 0–25 years of age across the country.

Between 2018 and 2022, the Foundation opened three rounds of applications (2018, 2019, 2020) in order to select 100 community-or school-based projects. Each project could be implemented across 1, 2 or 3 years. Because some Academies chose to test their intervention only in their second year of funding, there were in total 4 cohorts of Academies, across four school years.

In terms of implementation strategy, Academies could choose to implement intervention previously validated approaches with proven results (such as the Incredible Years Program, Webster-Stratton, 1981, 1982), or choose to develop and implement pilot programs or innovative interventions, designed by each Academy from the ground-up. Following the OECD approach to SEL, the initiative chose to focus on the following main competencies: Adaptability, Self-regulation, Communication, Creative thinking, Critical Thinking, Resilience, and Problem Solving.

In addition to co-funding the intervention, the Foundation offered the selected programs the technical support of an external Monitoring and Evaluation Team, which assisted Academies in all stages of their evaluation processes. The M&E Team provided continuous training opportunities to all Academies using a training model, based on the Data Wise model (Boudett and Steele, 2007; Boudett et al., 2020), focused on aspects related to monitoring (how to design a Theory of Change, how to observe program implementation, how to use program implementation monitoring data to improve interventions), impact evaluation (how to conceptually align intervention and evaluation, how to select evaluation measures, how to constitute intervention and control groups, how to analyze and discuss results), and ethical aspects inherent to research in the field. The M&E team also granted regular ethics and data protection awareness training sessions to all Academies, and provide countless session of mentoring and individual consultancy. Examples of mentoring and consultancy included detailed revision of evaluation materials, field visits to the evaluation teams to provide in-situ training, regular calls to fine-tune the theory of change, but also leveraging connections and networking between Academies geographically distant.

In exchange for this support, the Foundation requested that all Academies attempt to use an experimental approach to the impact evaluation of their intervention, by randomly assigning participants to control and intervention groups and collecting pre-test and post-test to the highest standards of program evaluation. Mostly because of ethical concerns with randomized controlled trials, and also limitations imposed by Covid-19, many Academies were unable to implement true experiments, and proceeded to implement quasi-experimental trials, by matching chosen intervention groups with comparison groups that were “equal in expectation,” i.e., assumed to be equal in observables and non-observables (Murnane and Willett, 2011). Another large group of academies were unable to employ rigorous methods of impact evaluation and conducted observation studies, monitoring the growth of SEL from the beginning to the end of their intervention.

The Foundation also required the use of the SSES (OECD, 2021a; Castro et al., 2023) as a common metric of impact measurement across Academies. This meant Academies were requested to use SSES for pre-and post-test assessment of all participants in their evaluation. Because theories of change across Academies varied greatly, and the Foundation wanted to fund intervention approaches with a clear goal, Academies could choose a minimum of two SSES competencies to monitor across evaluation stages. Since each Academy would select the SSES subscales that best aligned with their Theory of Change, i.e., that evaluated the social and emotional skills targeted by their intervention, there is great variability in choice and number of skills to be evaluated. Moreover, no items from the Energy subscale could be used because this skill was not aligned with the theoretical scope of the Foundation work. Academies were also incentivized to complement their impact evaluation with other standardized measures of assessment closer to their theory of change.

All Academies were also recommended to involve at least 100 participants in their impact evaluation, in order to ensure some statistical power in their impact evaluation. Although this was not mandatory, it was strongly recommended, and most of the projects complied to this rule.

6 The present study

To address previous gaps in the literature, and with an aim to influence educational policymaking, The Portuguese Gulbenkian Academies for Knowledge supported the implementation and rigorous impact assessment of community and educational interventions aiming to promote SEL in participants 0 and 25 years of age. Of these, eight Academies implemented an experimental approach, while 32 Academies employed a quasi-experimental approach to their impact evaluation.

This paper presents the secondary-data analysis of the experimental and quasi-experimental impacts of 40 Academies on the OECD Survey of Social and Emotional Skills in order to address the following research question: Can community and educational interventions using diverse intervention approaches change social and emotional learning? Specifically, what were the experimental and quasi-experimental impacts of Academies in SEL?

7 Method

7.1 Participants and settings

The study sample included participants from 40 Academies on a standardized measure of SEL, the OECD Survey of Social and Emotional Skills (SSES). Eight Academies chose to implement an experimental approach (20%), by randomly assigning participants to control and intervention groups. Thirty-two academies employed a quasi-experimental approach (80%), by matching chosen intervention groups with comparison groups. The requirement to use the SSES as a common impact measure was implemented starting in the second cohort of Academies, because SSES was not available prior. Therefore, no data from Academies implementing in the first cohort were included (2018–2019). Moreover, due to the low quality and quantity of data from the 2nd edition (2019–2020), which was severely impacted by the outburst of the COVID-19 pandemic midyear, data from these 40 Academies which implemented the SSES Child/Youth Form generally came from the third and fourth cohorts (2020–2021 and 2021–2022, respectively). Academies which chose not to administer the SSES in any of its forms have been excluded from the present study. Finally, all Academies were requested to provide written consent to the use of their data for the purpose of this study; eight Academies never replied to this request and were therefore excluded from the study sample.

Supplementary Table S1 provides an overview of program characteristics for the 40 Academies in the study sample. Academies targeted very diverse domains of intervention in addition to education, such as arts, science, culture, technology, sports, health, and solidarity. While the majority developed in a school context, many included community involvement, and a few used also family outreach. These Academies were also of considerable geographical diversity, and while most targeted 6 to 12 years old, there was also quite a lot of variation in age groups.

Supplementary Table S2 presents implementation and evaluation data for the 40 Academies in the study sample. These data indicated that participants received on average between 6.4 and 46.3% of intervention sessions. Satisfaction as reported by direct participants (children and youth) was quite high. The last column lists the SEL skills chosen as targets by Academies. Some Academies chose a wide range of SEL Skills which may indicate a lack of focus and a potential for low impact given that some programs were of very short duration.

The experimental sample was comprised of 4,460 participants, 52% of which were female, and with ages ranging from 0 to 25 years old (M = 10.28, SD = 5.68). Mean school grade was the 5th grade (M = 4.99, SD = 4.23), and the majority of participants (93%) were Portuguese. Both parents of the participants were predominantly Portuguese (92% of mothers and of fathers), and their highest educational level was, on average, high school, although mothers scored higher (mother’s educational level M = 4.01, SD = 1.09, father’s educational level M = 3.77, SD = 1.22¹). Most families lived in an urban setting (63%) (Table 1).

In the experimental study sample, participants in the intervention group were statistically different from participants in the control group in 5 out of 23 descriptive characteristics, largely testifying to the success of the randomization process. In specific, participants in the intervention group were older, more likely to attend public school, more likely to have a Portuguese mother and father, and less likely to have siblings, than participants in the control group (Table 2).

Table 2

Table 2. Descriptive statistics for academies using an experimental approach to impact analysis.

The quasi-experimental sample was comprised of 14,274 participants, 50% of which were female, and with ages ranging from 3 to 25 years old (M = 10.92, SD = 3.98). Mean school grade was the 5th grade (M = 5.22, SD = 3.57), and the majority of participants (93%) were Portuguese. Both parents of the participants were predominantly Portuguese (88% of mothers and of fathers), and their highest educational level was, on average, high school, although mothers scored higher (mother’s educational level M = 3.80, SD = 1.15, father’s educational level M = 3.52, SD = 1.19²). Most families lived in an urban setting (68%) (Table 3).

Table 3

Table 3. Descriptive statistics for academies using a quasi-experimental approach to impact analysis.

In the quasi-experimental study sample, participants in the comparison group were statistically different from participants in the intervention group in 13 out of 23 descriptive characteristics, testifying to the partial success of the matching process. In specific, participants in the intervention group were younger, attended lower educational levels, were more likely to have a special education diagnosis, less likely to attend public school, and more likely to have repeated a year, than participants in the comparison group. Moreover, in terms of their family characteristics, participants in the intervention group had younger mothers, of lower educational levels, who were less likely to work; and fathers also of lower educational levels, who were less likely to work. Finally, intervention participants lived in households that were more likely to received some form of support by social services, with a larger number of siblings, and more likely to live with their parents.

7.2 Measures

7.2.1 SSES – child/youth form

The SSES – Child/Youth form (OECD, 2021a) is a self-report instrument composed of 120 items, answered in a scale of one (Totally disagree) to five (Totally agree), which allows the assessment of a set of 15 social and emotional skills by child or youth participants aged between eight and 17 years old. It includes the following 15 subscales, with eight items each: Optimism (OPT; e.g.: “I look at the bright side of life”), Responsibility (RES; e.g.: “I am a responsible person”), Curiosity (CUR; e.g.: “I like learning new things”), Self-control (SEL; e.g.: “I stop to think before acting”), Emotional control (EMO; e.g.: “I stay calm even in tense situations”), Cooperation (COO; e.g.: “I get along well with others”), Sociability (SOC; e.g.: “I make friends easily”), Assertiveness (ASS; e.g.: “I enjoy leading others”), Creativity (CRE; e.g.: “I have a good imagination”), Resilience/Stress resistance (STR; e.g.: “I am relaxed and handle stress well”), Persistence/Perseverance (PER; e.g.: “I make sure that I finish tasks”), Empathy (EMP; e.g.: “I know how to comfort others”), Tolerance (TOL; e.g.: “I like hearing about other cultures and religions”), Trust (TRU; e.g.: “I believe most people are kind”) and Energy (ENE; e.g.: “I am full of energy”). The survey could be administered in paper format or online format. Data from the global sample of SSES’s main study by OECD (2021) indicates Cronbach’s alpha’s internal consistency levels between 0.71 (Empathy) and 0.85 (Assertiveness). An ongoing validation study of the Portuguese adaptation of the Child/Youth form of the Survey on Social and Emotional Skills (SSES) shows that the measure has good internal consistency and sensitivity, while also being sensitive to change over time (Castro et al., 2023).

7.2.2 SSES – teacher form

The SSES – Teacher form (OECD, 2021a) is a teacher-report instrument composed of 45 items, answered in a scale of one (Totally disagree) to five (Totally agree), which allows the assessment of a set of 15 social and emotional skills of child or youth participants aged between eight and 17 years old. It includes the same 15 subscales as the Child/Youth form, with three items each: Optimism (OPT; e.g.: “This student is a happy person”), Responsibility (RES; e.g.: “This student always keeps his/her promises”), Curiosity (CUR; e.g.: “This student likes learning new things”), Self-control (SEL; e.g.: “This student can control his/her actions”), Emotional control (EMO; e.g.: “This student keeps his/her emotions under control”), Cooperation (COO; e.g.: “This student likes to help others”), Sociability (SOC; e.g.: “This student makes friends easily”), Assertiveness (ASS; e.g.: “This student is a leader”), Creativity (CRE; e.g.: “This student has a good imagination”), Resilience/Stress resistance (STR; e.g.: “This student worries about many things”), Persistence/Perseverance (PER; e.g.: “This student keeps working on a task until it is finished”), Empathy (EMP; e.g.: “This student can sense how others feel”), Tolerance (TOL; e.g.: “This student likes hearing about other cultures and religions”), Trust (TRU; e.g.: “This student believes that their friends will never betray them”), and Energy (ENE; e.g.: “This student is full of energy”). The survey could be administered in paper format or online format. Data from the global sample of SSES’s main study by OECD (2021) indicates Cronbach’s alpha’s internal consistency levels between 0.68 (Resilience/Stress resistance) and 0.93 (Persistence). Due to weaker psychometric properties in OECD’s field test results (OECD, 2021b), the subscale Trust was excluded from the teacher form in its administration within the Gulbenkian Academies, hence being absent from our data.

7.2.3 Sociodemographic questionnaire

To facilitate collection of sociodemographic information, the M&E team proposed a draft questionnaire mapping a set of characteristics of participants (i.e., child’s age, school grade, gender, nationality, whether the child has special educational needs, is in pre-school, attends public school, has failed a school year), their families (mom and dad’s nationality, age, completed schooling, and whether each one works, and is married) and household (how many sources of social aid, whether the child has siblings and how many, whether child lives with parents, and in an urban area). This questionnaire was adapted by each Academy to their evaluation needs and sample characteristics. Depending mostly on the age of the target group, this questionnaire could be answered by the participants, their parents or legal representatives, teachers or other parties.

7.3 Databases

This paper employed only secondary-data analysis directly collected by each Academy’s team with their participants. Based on training and supervision from the External Monitoring and Evaluation team, Academies used common data collection and management procedures, as well as ethical procedures, while also selecting the appropriate mechanisms to match the specific needs of its setting and sample. For example, all Academies were required, prior to assessment, to collect informed consent from each participant’s legal tutor, but had to prepare materials (paper versions or online versions of each measure; adjustment to age), and adequate locations (e.g., classrooms, community facilities) for data collection depending on the characteristics of their target group.

Data collection procedures could be managed and implemented by any adequately trained member of the Academy’s team, including teachers, social and youth workers, psychologists, researchers, among others, with supervision. Depending mostly on the age of the participants, data collection materials could be answered by the participants, or with the help of an adult.

In line with the ethical considerations guiding research and intervention practice, when collecting the data, Academies were instructed to bear in mind its delicate nature and the need to keep the privacy of children/youths and their families protected. The M&E team recommended that data should only be accessed by a reduced number of team members. Moreover, all Academies were instructed to collect oral assent prior to assessment, and debrief underaged participants of study goals and procedures. The M&E team granted regular ethics and data protection awareness training sessions to all Academies, and provide countless sessions of mentoring.

Data on the participants’ sociodemographic characteristics, their group assignment, and pre and post-teste SSES scores was then fully anonymized, with each participant being assigned an ID by their Academy’s team, and submitted by the Academies to the M&E Team for further cleaning and analysis. All data cleaning and analysis procedures ensured confidentiality. Additionally, regarding pre-test and post-test scores, there is a decrease in sample size across subscales due to missing data: respondents may only have participated in one of the data collection moments, with participant mortality being common at post-test.

Finally, all Academies whose data is included in this paper granted their approval for it to be processed and published for this purpose by the M&E Team via signed informed consent.

7.4 Data analysis

To evaluate the impact of the Academies on the targeted social and emotional skills, we used a multilevel regression model for each specific competency at the end of implementation (i.e., at post-test), comparing intervention group with control or comparison group, and controlling for: (a) the same competency at the start of each implementation (i.e., at pre-test), (b) participants’ sociodemographic characteristics, and (c) clustering by Academy.

For Academies with an experimental design (i.e., with a randomized control group), the inclusion of control variables in the model allows to increase the accuracy of the estimate (i.e., decrease the confidence interval), assuring a significant effect is indeed detected. For Academies with a quasi-experimental design (i.e., with a non-randomized comparison group), the inclusion of control variables in the multilevel models aimed to decrease some initial differences between groups that have not been controlled by experimental design, and thus isolate the real impact estimate (Murnane and Willett, 2011).

The use of a multilevel model allowed us to respect the nature of the data, distributing the variation in the post-test measurement of each skill between variation between Academies, and individual variation. A preliminary assessment using the Intraclass Correlation Coefficient (ICC) determined that, in all skills evaluated by more than one Academy, part of the variation was indeed attributed to differences between Academies. Intraclass Correlations varied between 1.70 and 57.60% for the SSES – Child/Youth form, and 0% (only one case) and 24.50% for the SSES - Teacher form. In cases where sample limitations (whether in sample size or due to imbalance) did not allow for a multilevel analysis, a multiple regression model was used.

In order to decrease the impact of missing data on the sample available for analysis, a multiple imputation (of 20 bases) was used for sociodemographic and pretest data. No multiple imputation of outcome data was performed.

As recommended by APA (Espírito-Santo and Daniel, 2015), results for each subscale are summed in terms of significance level (i.e., p < 0.05) and effect size (ES), i.e., the difference between the adjusted mean of participants from the intervention group and that of participants from the control or comparison group, expressed in a proportion of the standard deviation of that same subscale for the control or comparison group (Gormley et al., 2005; Wong et al., 2008). The effect size indicates, thus, the amplitude of the effect, regardless of the measure or method used; it does not depend on sample size, as the p value does, and it contributes to understanding the impact results, since it allows to examine the magnitude of differences (Tanner-Smith et al., 2018). Significant positive effect sizes are interpreted as evidence of impacts in favor of the intervention group; significant negative effect sizes are interpreted as evidence of impacts in favor of the control or comparison groups.

According to Cohen (1988), an ES is considered small if <0.2, with this value being common in interventions with children (Kraft, 2020); moderate if between 0.2 and 0.6, and large when >0.6. Sawilowsky (2009) then expands these, adding very small (<0.01), very large (>1.20) and huge (>2.00). There is, however, some debate regarding effect sizes in social sciences, and in educational sciences in particular, as well as regarding the type of measures and designs that influence these effect sizes (McCartney and Rosenthal, 2000; Durlak et al., 2022).

Data was processed with IBM SPSS Statistics (Statistical Package for the Social Sciences), version 28, and analyzed with Stata Statistical Software, version 17.

8 Results

8.1 Evidence of experimental impacts of the academies in SSES

Table 4 summarizes the experimental results of 8 Academies in 13 subscales of the SSES – Child/Youth form and 7 subscales of the SSES - Teacher form, presenting the number of Academies and participants in each model, the difference between the adjusted means for intervention versus control groups, level of statistical significance of the group difference, and effect size.

Table 4

Table 4. Experimental impacts for Academies, comparing intervention and control groups, using multilevel models controlling for the score on each subscale at pre-test, as well as participants’ sociodemographic characteristics^a and clustering for Academies (N Academies = 8; N Participants = 4,460).

Overall, experimental evidence of positive (i.e., in favor of the intervention group) and significant results of the Academies were found in four (out of 19) subscales of the SSES. There was also experimental evidence of negative (i.e., in favor of the control group) and significant results of the Academies in one subscale of the SSES. Effect sizes ranged from −0.211 (Tolerance, Child/Youth form) to 1.307 (Tolerance, Teacher form).

Specifically, in the SSES – Child/Youth form, we found significant positive impacts of the Academies on the Curiosity subscale (p = 0.001, d = 0.151), and significant negative impacts on the Tolerance subscale (p = 0.019, d = −0.211). In the SSES – Teacher form, we found significant positive impacts of the Academies on the Responsibility subscale (p = 0.024, d = 0.639), on the Curiosity subscale (p = 0.013, d = 0.149), and on the Tolerance subscale (p = 0.000, d = 1.307).

8.2 Evidence of quasi-experimental impacts of the academies in SSES

Table 5 summarizes the quasi-experimental results of the Academies in 14 subscales of the SSES – Child/Youth form and 14 subscales of the SSES - Teacher form, presenting the number of Academies and participants in each model, the difference between the adjusted means for intervention versus control groups, level of statistical significance of the group difference, and effect size.

Table 5

Table 5. Quasi-experimental impacts for academies, comparing intervention and comparison groups, using multilevel models controlling for the score on each subscale at pre-test, as well as participants’ sociodemographic characteristics^a and clustering for Academies (N Academies = 32; N Participants = 14,274).

Overall, quasi-experimental evidence of positive (i.e., in favor of the intervention group) and significant results of the Academies were found in 9 (out of 28) subscales of the SSES. There was also experimental evidence of negative (i.e., in favor of the control group) and significant results of the Academies in one subscale of the SSES. Effect sizes ranged from −0.145 (Curiosity, Teacher form) to 0.270 (Tolerance, Teacher form).

Specifically, in the SSES – Child/Youth form, we found one significant positive impact of the Academies on the Assertiveness subscale (p = 0.000, d = 0.186). In the SSES – Teacher form, we found significant positive impacts of the Academies on the Optimism subscale (p. = 0.000, d = 0.153), on the Emotional Control subscale (p = 0.015, d = 0.167), on the Self-control subscale (p = 0.002, d = 0.156), on the Cooperation subscale (p = 0.009, d = 0.110), on the Sociability subscale (p = 0.000, d = 0.198), on the Assertiveness subscale (p = 0.002, d = 0.189), on the Persistence/Perseverance subscale (p = 0.023, d = 0.102), and on the Tolerance subscale (p = 0.000, d = 0.207). We also found a one significant negative impact of the Academies on the Curiosity subscale (p = 0.001, d = −0.145).

9 Discussion

This paper aimed at testing the impact of a set of SEL focused interventions – the Gulbenkian Academies for Knowledge – on the social and emotional skills of their child and youth participants on a standardized measure of SEL, the OECD Survey of Social and Emotional Skills (SSES).

Experimental evidence of positive (i.e., in favor of the intervention group) and significant results of 8 Academies were found in 21% of the measured skills for both SSES versions (1 in a total of 13 subscales for the Child/Youth form, and 3 in a total of 7 for the Teacher form). Specifically, significant positive results were found for Curiosity as reported by the child and youth participants, and for Responsibility, Curiosity, and Tolerance from the perspective of teachers.

Quasi-experimental evidence depicts a more favorable picture of the results of the Academies. Quasi-experimental evidence of positive (i.e., in favor of the intervention group) and significant results of 32 Academies were found in 42% of the measured skills for both SSES versions (1 in a total of 14 subscales for the Child/Youth form, and 7 in a total of 14 for the Teacher form). Specifically, significant positive results were found for Assertiveness as reported by the child and youth participants, and for Optimism, Emotional Control, Self-control, Cooperation, Assertiveness, Persistence/Perseverance, and on Tolerance from the perspective of teachers.

Evidence of positive (i.e., in favor of the intervention group) and significant impacts of the Academies were consistent for the teacher and child/youth perspective, particularly for Curiosity in the experimental trials, and Assertiveness in the quasi-experimental evidence. Further interpretation of this pattern of results is important. Curiosity and Assertiveness may be more amenable to change, or it may be easier for teachers and other practitioners to target their interventions to these skills, and if so, maybe we should focus our SEL interventions in such skills. It is also possible that change in these two skills is easier to notice, and measure (Duckworth and Yeager, 2015). More research is needed to understand why change is observed so consistently in these two particular skills.

Globally, results meet what several meta-analyses and literature reviews on the effects of SEL interventions have been finding: these programs tend to generate small effect sizes, with not always significant but generally positive results (e.g., Clarke et al., 2015; Tanner-Smith et al., 2018). Small positive effect sizes in previous literature may be a function of the diversity of SEL interventions, i.e., in terms of domains, contexts, targeted age groups, and dimensions of implementation such as dosage and frequency of sessions. The diversity in program characteristics and implementation dimensions observed across Academies in this study mirrors the SEL field of intervention, where artistic, educational, and cultural approaches (among others) are proposed side-by-side as opportunities for changing children and youth’s SEL paths of development. This richness of proposed interventions is welcomed by community and educational institutions, including in the Academies. However, it is also likely that such diversity in approaches adds noise to program evaluation results, limiting their interpretation as to what exactly is promoting SEL change.

Also, literature suggests non-randomized evaluation designs (i.e., quasi-experimental studies) tend to overestimate effect sizes (Cheung and Slavin, 2016; Corcoran et al., 2018), which adds to need for caution when interpreting these results of the Academies employing quasi-experimental methods.

The triangulation of informants also stands as a strong point of this research, combining the voices of children/youth and teachers as direct and indirect participants (respectively) of these interventions. This ensures greater rigor, quality, and security in the results, while also allowing for a wider picture of how social and emotional skills develop in children and young people engaged in SEL interventions. For example, a closer look at effect sizes shows they tend to be larger for the Teacher form. This may be due to teachers’ expectations towards the interventions, and their overall positive feedback towards the programs and the subsequent perceived positive changes in their students. Research also mentions children are usually more critical towards their own social and emotional development after being exposed to explicit SEL content and acquiring knowledge on what these skills are, how they translate into daily behavior, and their own limitations in these competences (e.g., OECD, 2021b; Martinsone et al., 2022).

Impact results also prompt a reflection on how social and emotional skills develop: literature has shown that these skills develop at different paces, in a non-linear form, with oscillations throughout childhood and adolescence (OECD, 2021b).

9.1 Implications for research, practice and policy

The need to employ rigorous methods to evaluate the impact of interventions on social and emotional skills has been highlighted in the national (e.g., Cristóvão et al., 2017) and international literature (e.g., Durlak et al., 2011; Corcoran et al., 2018; Kankaraš and Suarez-Alvarez, 2019; Kankaraš et al., 2022). The growing recognition of the value and importance of social and emotional skills is accompanied by an insufficient knowledge on “what works” to improve them. Despite the investment in implementing SEL programs in Portugal getting increasingly valued, there is still the need to maximize the investment by measuring the quality of the large number of programs already being implemented, as well as their impacts in a rigorous manner (Cristóvão et al., 2017).

However, setting high standards is not enough. The Academies example demonstrated that, even when given support and some degree of pressure to implement rigorous methods of evaluation, education and community programs are not ready to implement such methods and opt for studies with weaker methodological rigor. The impact of the COVID19 epidemic cannot be underestimated in this choice. However, even programs implemented after the pandemic choosing quasi-experimental designs now face problems interpreting results, given the differences found between intervention and comparison groups in terms of their sociodemographic characteristics. It can be interpreted from these differences that these groups are not fully equivalent in expectation (Murnane and Willett, 2011), increasing limitations for the interpretation of the results of this study.

In educational and community settings, opting for quasi-experimental designs is frequent, mostly because selecting samples by convenience is less disruptive to the normal daily functioning of a school or community service, and raises fewer ethical concerns. Considering that only rigorous methods, such as randomized control trials, allow to effectively attribute the observed effects to the interventions being evaluated, researchers and policymakers must consider the importance of the training and capacity building of SEL professionals on basic program evaluation skills, as well as providing enough time and resources for the implementation of these skills, so that investment leads to an increase in evidence-based policies and practices for educational, social, and community interventions on SEL.

Challenges also remain in evaluating social and emotional skills. The Portuguese Gulbenkian Foundation opted by standardizing assessment by providing access to one common measure of impact across Academies - the SSES (OECD, 2021a; Castro et al., 2023). The benefits in promoting rigorous and quality impact measurement were considerable (Chernyshenko et al., 2018), but there were also some limitations.

First, the field of social and emotional skills still offers several conceptual challenges and methodological constraints, which are felt by professionals in the field. First, SEL domains and subdomains are very distinct, there is no common theory for each skill and there is often considerable confusion over terminology (Duckworth and Yeager, 2015). For example, different theoretical frameworks often use similar terms to describe distinct skills, and different terms to describe similar skills (Schoon, 2021). This means two programs which intend to measure their impact on resilience may, in fact, mean very distinct, incomparable constructs.

Second, there is no common metrics available to measure most skills. For instance, different instruments may measure different developmental stages of creative thinking during adolescence, and the current literature lacks validation studies which allow the horizontal comparison between these measures (Humphrey et al., 2011).

Third, SEL competencies have different developmental speeds, with some progressing more quickly than others. Cognitive self-regulation, for example, progresses fast during the pre-school years (i.e., ages 3 to 5), whereas adaptability skills may have smaller expected development during the same stage. The direct comparison of two programs with the same target group but which promote different skills, would favor the intervention targeting, in our example, cognitive self-regulation.

Fourth, these skills also develop at different paces during childhood and adolescence, making it inadequate to compare the impact of a program in a certain skill during childhood and another program targeting the same skill during the teenage years.

Fifth, SEL skills develop often in a non-linear manner (i.e., very rapidly in the early years, and then slower for a while; or the opposite), and in sudden leaps. This implies longer interventions, working on an evidence-based set of skills, may see more favorable results than shorter interventions, although the duration of an intervention itself also does not bring more impacts in a linear manner. These measurement issues cannot be addressed in our study due to data limitations, and present further challenges in the interpretation of results.

Finally, given that our outcome measure - SSES - is a report measure, it offers particular methodological constraints (e.g.: Murano et al., 2021). Because it is based on perceptions, results may be influenced by the fact that some skills may be easier to report than others, students and teachers may differ in their comparison terms, or there may be a considerable social desirability effect. The development of measures of direct observation, performance tasks, or task-oriented tools, is thus a priority for the field of SEL measurement, particularly to allow for methodological triangulation and practice improvement (Duckworth and Yeager, 2015).

All these challenges require an investment from the scientific community, in order for the evaluation process of programs targeting the development of social and emotional skills to be effective and rigorous.

9.2 Conclusion

The impact evaluation of the Gulbenkian Academies for Knowledge has the potential to support considerable changes in the field of SEL interventions, nationally and internationally. Internationally, it contributes to a deeper discussion of evidence-based SEL intervention developed locally and from the ground-up, and taking place in educational and community settings. Locally, the initiative offered an incentive and a model of capacity building at a national scale for hundreds of SEL professionals, creating a cascading, sustainable and long-lasting effect in practices. These teams are now more likely to apply the new program evaluation knowledge and skills to their daily practices, and bring about answers (and new questions) to the growing knowledge of how these skills develop in children and youth, and on what works to its effective promotion.

Data availability statement

The data analyzed in this study are subject to the following licenses/restrictions: Data was the property of the Academies, not ours. Requests to access these datasets should be directed to pgconhecimento@gulbenkian.pt.

Ethics statement

The studies involving humans were approved by Calouste Gulbenkian Foundation. The studies were conducted in accordance with the local legislation and institutional requirements. Written informed consent for participation in this study was provided by the participants’ legal guardians/next of kin.

Author contributions

MB: Conceptualization, Data curation, Formal analysis, Funding acquisition, Methodology, Project administration, Supervision, Writing – original draft, Writing – review & editing. JA: Conceptualization, Supervision, Writing – original draft. CCa: Data curation, Project administration, Writing – original draft. CCo: Data curation, Project administration, Writing – original draft.

Funding

The author(s) declare financial support was received for the research, authorship, and/or publication of this article. All authors were funded by the Calouste Gulbenkian Foundation for the monitoring and evaluation work developed with the Academies. MB was funded by the Portuguese Foundation for Science and Technology (FCT) through the CEEC institutional funding (CEECINST/00126/2021). CCa was funded by an FCT doctoral grant (UI/BD/154443/2022).

Acknowledgments

The authors would like to thank all Academies, their participants and families, as well as the Calouste Gulbenkian Foundation, for contributing to making this work possible. We would also like to thank all academies that agreed to share the anonymized data for this paper, especially the Academy associated with the Universidade Portucalense.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/feduc.2023.1287259/full#supplementary-material

Footnotes

1. ^Scores were obtained by categories related to the Portuguese schooling system: 0 = Cannot read or write; 1 = up to the 4th grade, 2 = up to the 6th grade, 3 = up to the 9th grade, 4 = up to the 12th grade, 5 = university degree.

2. ^Scores were obtained by categories related to the Portuguese schooling system: 0 = Cannot read or write; 1 = up to the 4th grade, 2 = up to the 6th grade, 3 = up to the 9th grade, 4 = up to the 12th grade, 5 = university degree.

References

Alexandre, J., and Barata, M. C. (2020). “Intervenção comunitária com crianças e jovens em risco” in A Prática Profissional da Psicologia da Justiça. eds. R. Barroso and D. Neto. Ordem dos Psicólogos Portugueses. ISBN 978-989-54623-5-3 (print). Available at: https://www.ordemdospsicologos.pt/ficheiros/documentos/pratica_profissional_da_psicologia_na_justica.pdf (Accessed January 7, 2024).

Google Scholar

Blewitt, C., Fuller-Tyszkiewicz, M., Nolan, A., Bergmeier, H., Vicary, D., Huang, T., et al. (2018). Social and emotional learning associated with universal curriculum-based interventions in early childhood education and care centers: a systematic review and meta-analysis. JAMA Netw. Open 1:e185727. doi: 10.1001/jamanetworkopen.2018.5727

PubMed Abstract | Crossref Full Text | Google Scholar

Boncu, A., Costea, I., and Minulescu, M. (2017). A meta-analytic study investigating the efficiency of socio-emotional learning programs on the development of children and adolescents. Rom. J. Appl. Psychol. 19, 35–41. doi: 10.24913/rjap.19.2.02

Crossref Full Text | Google Scholar

Boudett, K. P., City, E. A., and Murnane, R. J. (2020). Data wise revised and expanded edition: a step-by-step guide to using assessment results to improve teaching and learning. Cambridge, MA, USA: Harvard Education Press.

Google Scholar

Boudett, K. P., and Steele, J. L. (2007). Data wise in action: Stories of schools using data to improve teaching and learning. Cambridge, MA, USA: Harvard Education Press.

Google Scholar

CASEL , (2020). Casel’s sel framework: what are the core competence areas and where are they promoted? CASEL. Available at: https://casel.org/casel-sel-framework-11-2020/ (Accessed July 17, 2023).

Google Scholar

Castro, C., Barata, C., Alexandre, J., and Colaço, C. (2023). Validation of a community-based application of the Portuguese version of the survey on social and emotional skills–child/youth form. Front. Psychol. 14:1214032. doi: 10.3389/fpsyg.2023.1214032

PubMed Abstract | Crossref Full Text | Google Scholar

Cefai, C., Bartolo, P., Cavioni, V., and Downes, P., (2018). Strengthening social and emotional education as a core curriculum area across the EU: a review of the international evidence. NESET II report, Luxembourg: Publications Office of the European Union.

Google Scholar

Cefai, C., Miljević-Riđički, R., Bouillet, D., Pavin Ivanec, T., Matsopoulous, A., Gavogiannaki, M., et al. (2015). RESCUR surfing the waves. A resilience curriculum for early years and primary schools. A teachers’ guide. Malta: Centre for Resilience and Socio-Emotional Health, University of Malta.

Google Scholar

Chernyshenko, O. S., Kankaraš, M., and Drasgow, F., (2018). Social and emotional skills for student success and well-being: Conceptual framework for the OECD study on social and emotional skills. Paris, France: OECD education working papers.

Google Scholar

Cheung, A. C. K., and Slavin, R. E. (2016). How methodological features affect effect sizes in education. Educ. Res. 45, 283–292. doi: 10.3102/0013189X16656615

Crossref Full Text | Google Scholar

Cipriano, C., Strambler, M. J., Naples, L. H., Ha, C., Kirk, M., Wood, M., et al. (2023). The state of evidence for social and emotional learning: a contemporary meta-analysis of universal school-based SEL interventions. Child Dev. 94, 1181–1204. doi: 10.1111/cdev.13968

PubMed Abstract | Crossref Full Text | Google Scholar

Clarke, A. M., Morreale, S., Field, C. A., Hussein, Y., and Barry, M. M. (2015). What works in enhancing social and emotional skills development during childhood and adolescence? A review of the evidence on the effectiveness of school-based and out-of-school programmes in the UK. National University of Ireland, Galway, Ireland: WHO Collaborating Centre for Health Promotion Research. Available at: https://aran.library.nuigalway.ie/bitstream/handle/10379/4981/Review_of_Social_and_Emotional_Skills-Based_Intervention_Report_(WEB_VERSION).pdf?sequence=1 (Accessed January 7, 2024).

Google Scholar

Coelho, V., and Figueira, A. P. (2011). Project “positive attitude”: promoting school success through social and emotional abilities development – design for elementary and middle school students, in Portugal. Int. J. Psychol. 45, 185–192.

Google Scholar

Coelho, V., Marchante, M., and Brás, P. (2023). Positive attitude upper middle school social and emotional learning program: influences of implementation quality on program outcome. Front. Psychol. 14:2517. doi: 10.3389/fpsyg.2023.1172517

PubMed Abstract | Crossref Full Text | Google Scholar

Cohen, J. , (1988). Statistical power analysis for the behavioral sciences (2nd Edn.). Hillside, NJ: Lawrence Erlbaum Associates.

Google Scholar

Corcoran, R. P., Cheung, A. C. K., Kim, E., and Xie, C. (2018). Effective universal school-based social and emotional learning programs for improving academic achievement: a systematic review and meta-analysis of 50 years of research. Educ. Res. Rev. 25, 56–72. doi: 10.1016/j.edurev.2017.12.001

Crossref Full Text | Google Scholar

Cristóvão, A. M., Candeias, A. A., and Verdasca, J. (2017). Social and emotional learning and academic achievement in Portuguese schools: a bibliometric study. Front. Psychol. 8:1913. doi: 10.3389/fpsyg.2017.01913

PubMed Abstract | Crossref Full Text | Google Scholar

Domitrovich, C., Durlak, J., Staley, K., and Weissberg, R. (2017). Social-emotional competence: an essential factor for promoting positive adjustment and reducing risk in school children. Child Dev. 88, 408–416. doi: 10.1111/cdev.12739

PubMed Abstract | Crossref Full Text | Google Scholar

Domitrovich, C. E., Li, Y., Mathis, E. T., and Greenberg, M. T. (2019). Individual and organizational factors associated with teacher self-reported implementation of the PATHS curriculum. J. Sch. Psychol. 76, 168–185. doi: 10.1016/j.jsp.2019.07.015

PubMed Abstract | Crossref Full Text | Google Scholar

Duckworth, A. L., and Yeager, D. S. (2015). Measurement matters: assessing personal qualities other than cognitive ability for educational purposes. Educ. Res. 44, 237–251. doi: 10.3102/0013189X15584327

PubMed Abstract | Crossref Full Text | Google Scholar

Durlak, J. A., Mahoney, J. L., and Boyle, A. E. (2022). What we know, and what we need to find out about universal, school-based social and emotional learning programs for children and adolescents: a review of meta-analyses and directions for future research. Psychol. Bull. 148, 765–782. doi: 10.1037/bul0000383

Crossref Full Text | Google Scholar

Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., and Schellinger, K. B. (2011). Enhancing students’ social and emotional development promotes success in school: results of a meta-analysis. Child Dev. 82, 405–432. doi: 10.1111/j.1467-8624.2010.01564.x

PubMed Abstract | Crossref Full Text | Google Scholar

Durlak, J. A., Weissberg, R. P., and Pachan, M. (2010). A meta-analysis of after-school programs that seek to promote personal and social skills in children and adolescents. Am. J. Community Psychol. 45, 294–309. doi: 10.1007/s10464-010-9300-6

PubMed Abstract | Crossref Full Text | Google Scholar

Espírito-Santo, H., and Daniel, F. (2015). Calcular e apresentar tamanhos do efeito em trabalhos científicos: as limitações do p<0.05 na análise de diferenças de médias de dois grupos. Revista Portuguesa de Investigação Comportamental e Social 1, 3–16. doi: 10.7342/ismt.rpics.2015.1.1.14

Crossref Full Text | Google Scholar

Fernández-Martín, F.-D., Romero-Rodríguez, J.-M., Marín-Marín, J.-A., and Gómez-García, G. (2021). Social and emotional learning in the ibero-american context: a systematic review. Front. Psychol. 12:738501. doi: 10.3389/fpsyg.2021.738501

PubMed Abstract | Crossref Full Text | Google Scholar

Flay, B. R., and Allred, C. G. (2010). “The positive action program: improving academics, behavior, and character by teaching comprehensive skills for successful learning and living” in International research handbook on values education and student wellbeing (New York, NY: Springer Science + Business Media), 471–501.

Google Scholar

Fossum, S., Handegård, B. H., and Britt Drugli, M. (2017). The incredible years teacher classroom management programme in kindergartens: effects of a universal preventive effort. J. Child Fam. Stud. 26, 2215–2223. doi: 10.1007/s10826-017-0727-3

Crossref Full Text | Google Scholar

Goldberg, L. R. (1990). An alternative “description of personality”: the big-five factor structure. J. Pers. Soc. Psychol. 59, 1216–1229. doi: 10.1037/0022-3514.59.6.1216

Crossref Full Text | Google Scholar

Goldberg, J. M., Sklad, M., Elfrink, T. R., Schreurs, K. M. G., Bohlmeijer, E. T., and Clarke, A. M. (2019). Effectiveness of interventions adopting a whole school approach to enhancing social and emotional development: a meta-analysis. Eur. J. Psychol. Educ. 34, 755–782. doi: 10.1007/s10212-018-0406-9

Crossref Full Text | Google Scholar

Gormley, W., Gayer, T., Phillips, D., and Dawson, B. (2005). The effects of universal pre-K on cognitive development. Dev. Psychol. 41, 872–884. doi: 10.1037/0012-1649.41.6.872

PubMed Abstract | Crossref Full Text | Google Scholar

Green, A. L., Ferrante, S., Boaz, T. L., Kutash, K., and Wheeldon-Reece, B. (2021). Social and emotional learning during early adolescence: effectiveness of a classroom-based SEL program for middle school students. Psychol. Sch. 58, 1056–1069. doi: 10.1002/pits.22487

Crossref Full Text | Google Scholar

Humphrey, N., Kalambouka, A., Wigelsworth, M., Lendrum, A., Deighton, J., and Wolpert, M. (2011). Measures of social and emotional skills for children and young people: a systematic review. Educ. Psychol. Meas. 71, 617–637. doi: 10.1177/0013164410382896

Crossref Full Text | Google Scholar

James-Burdumy, S., Dynarski, M., Moore, M., Deke, J., Mansfield, W., and Pistorino, C., (2005). When schools stay open late: the National Evaluation of the 21st century community learning centers program: Final report. U.S. Department of Education, Institute of Education Sciences, National Center for education evaluation and regional assistance. Available at http://www.ed.gov/ies/ncee

Google Scholar

Jones, S. M., McGarrah, M. W., and Kahn, J. (2019). Social and emotional learning: a principled science of human development in context. Educ. Psychol. 54, 129–143. doi: 10.1080/00461520.2019.1625776

Crossref Full Text | Google Scholar

Judge, T. A., Bono, J. E., Ilies, R., and Gerhardt, M. W. (2002). Personality and leadership: a qualitative and quantitative review. J. Appl. Psychol. 87, 765–780. doi: 10.1037/0021-9010.87.4.765

Crossref Full Text | Google Scholar

Kankaraš, M., de Fruyt, F., and Primi, R. (2022). Editorial: theory and empirical practice in research on social and emotional skills. Front. Educ. 7:993878. doi: 10.3389/feduc.2022.993878

Crossref Full Text | Google Scholar

Kankaraš, M., and Suarez-Alvarez, J., (2019). Assessment framework of the OECD study on social and emotional skills (OECD education working papers no. 207), Paris, France: OECD education working papers. doi: 10.1787/5007adef-en

Crossref Full Text | Google Scholar

Kotsiou, A., Fajardo-Tovar, D. D., Cowhitt, T., Major, L., and Wegerif, R. (2022). A scoping review of future skills frameworks. Irish Educ. Stud. 41, 171–186. doi: 10.1080/03323315.2021.2022522

Crossref Full Text | Google Scholar

Kraft, M. A. (2020). Interpreting effect sizes of education interventions. Educ. Res. 49, 241–253. doi: 10.3102/0013189X20912798

Crossref Full Text | Google Scholar

Kremer, K. P., Maynard, B. R., Polanin, J. R., Vaughn, M. G., and Sarteschi, C. M. (2015). Effects of after-school programs with at-risk youth on attendance and externalizing behaviors: a systematic review and meta-analysis. J. Youth Adolesc. 44, 616–636. doi: 10.1007/s10964-014-0226-4

PubMed Abstract | Crossref Full Text | Google Scholar

Kusché, C. A., and Greenberg, M. T., (1994). The PATHS curriculum. Seattle, Washington, DC: Development Research and Programs.

Google Scholar

Low, S., Smolkowski, K., and Cook, C. (2016). What constitutes high-quality implementation of sel programs? A latent class analysis of second step® implementation. Prev. Sci. 17, 981–991. doi: 10.1007/s11121-016-0670-3

PubMed Abstract | Crossref Full Text | Google Scholar

Martinsone, B., Supe, I., Stokenberga, I., Damberga, I., Cefai, C., Camilleri, L., et al. (2022). Social emotional competence, learning outcomes, emotional and behavioral difficulties of preschool children: parent and teacher evaluations. Front. Psychol. 12:760782. doi: 10.3389/fpsyg.2021.760782

PubMed Abstract | Crossref Full Text | Google Scholar

McCartney, K., and Rosenthal, R. (2000). Effect size, practical importance, and social policy for children. Child Dev. 71, 173–180. doi: 10.1111/1467-8624.00131

Crossref Full Text | Google Scholar

Murano, D., Lipnevich, A. A., Walton, K. E., Burrus, J., Way, J. D., and Anguiano-Carrasco, C. (2021). Measuring social and emotional skills in elementary students: development of self-report Likert, situational judgment test, and forced choice items. Personal. Individ. Differ. 169:110012. doi: 10.1016/j.paid.2020.110012

Crossref Full Text | Google Scholar

Murnane, R., and Willett, J., (2011). Methods matter: Improving causal inference in educational and social science research. Oxford University Press, Oxford, New York.

Google Scholar

OECD (2018). Social and Emotional Skills: Well-being, connectedness and success. OECD Publishing. Available at: https://www.oecd.org/education/school/UPDATED%20Social%20and%20Emotional%20Skills%20-%20Well-being,%20connectedness%20and%20success.pdf%20(website).pdf (Accessed July 17, 2023).

Google Scholar

OECD (2021a). Beyond academic learning: first results from the survey of social and emotional skills 2019. Paris, France: OECD Publishing.

Google Scholar

OECD , (2021b). OECD survey on social and emotional skills: technical report. Paris, France: OECD Publishing.

Google Scholar

Raimundo, R. (2007). Programa de desenvolvimento de competências sócio-emocionais: Devagar se vai ao longe [Social and emotional learning program: Slowly but steadily] Unpublished manual.

Google Scholar

Raimundo, R., Marques-Pinto, A., and Lima, M. L. (2013). The effects of a social–emotional learning program on elementary school children: the role of pupils’ characteristics. Psychol. Sch. 50, 165–180. doi: 10.1002/pits.21667

Crossref Full Text | Google Scholar

Sackett, P. R., and Walmsley, P. T. (2014). Which personality attributes are most important in the workplace? Perspect. Psychol. Sci. 9, 538–551. doi: 10.1177/1745691614543972

Crossref Full Text | Google Scholar

Santos, A. C. , (2022). Health promotion and school engagement in youth: The influence of social and emotional competencies. Thesis presented for a double degree of doctor in education in the expertise of health education and in school psychology and human development, Faculdade de Motricidade Humana, Lisboa.

Google Scholar

Sawilowsky, S. (2009). New effect size rules of thumb. J. Mod. Appl. Stat. Methods 8, 597–599. doi: 10.22237/jmasm/1257035100

Crossref Full Text | Google Scholar

Schoon, I. (2021). Towards an integrative taxonomy of social-emotional competences. Front. Psychol. 12:515313. doi: 10.3389/fpsyg.2021.515313

PubMed Abstract | Crossref Full Text | Google Scholar

Schwartz, H. L., Hamilton, L. S., Faxon-Mills, S., Gomez, C. J., Huguet, A., Jaycox, L. H., et al. (2020). Early lessons from schools and out-of-school time programs implementing social and emotional learning. Santa Monica, CA: RAND Corporation. Available at: https://www.rand.org/pubs/research_reports/RRA379-1.html.

Google Scholar

Scorza, P., Araya, R., Wuermli, A. J., and Betancourt, T. S. (2016). Towards clarity in research on "non-cognitive" skills: linking executive functions, self-regulation, and economic development to advance life outcomes for children. Adolesc. Youth Glob. Hum. Dev. 58, 313–317. doi: 10.1159/000443711

PubMed Abstract | Crossref Full Text | Google Scholar

Strickhouser, J., Zell, E., and Krizan, Z. (2017). Does personality predict health and well-being? A metasynthesis. Health Psychol. 36, 797–810. doi: 10.1037/hea0000475

PubMed Abstract | Crossref Full Text | Google Scholar

Tanner-Smith, E. E., Durlak, J. A., and Marx, R. A. (2018). Empirically based mean effect size distributions for universal prevention programs targeting school-aged youth: a review of meta-analyses. Prev. Sci. 19, 1091–1101. doi: 10.1007/s11121-018-0942-1

PubMed Abstract | Crossref Full Text | Google Scholar

Van de Sande, M. C., Fekkes, M., Kocken, P. L., Diekstra, R. F., Reis, R., and Gravesteijn, C. (2019). Do universal social and emotional learning programs for secondary school students enhance the competencies they address? A systematic review. Psychol. Sch. 56, 1545–1567. doi: 10.1002/pits.22307

Crossref Full Text | Google Scholar

Wallender, J., Hiebel, A. L., PeQueen, C. V., and Kain, M. A. (2020). Effects of an explicit curriculum on social-emotional competency in elementary and middle school students. Delta Kappa Gamma Bullet., 32–43.

Google Scholar

Webster-Stratton, C. (1981). Modification of mothers’ behaviors and attitudes through videotape modeling group discussion program. Behav. Ther. 12, 634–642. doi: 10.1016/S0005-7894(81)80135-9

Crossref Full Text | Google Scholar

Webster-Stratton, C. (1982). Teaching mothers through videotape modeling to change their children’s behaviors. J. Pediatr. Psychol. 7, 279–294. doi: 10.1093/jpepsy/7.3.279

PubMed Abstract | Crossref Full Text | Google Scholar

Weiss, C. (1995). Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families. New approaches to evaluating community initiatives: Concepts, methods, and contexts, 1, 65–92.

Google Scholar

Weissberg, R., Durlak, J., Domitrovich, C., and Gullotta, T.P., (2015). Social and emotional learning: past, present, and future. Handbook for social and emotional learning: Research and practice 3–19.

Google Scholar

Wong, V. C., Cook, T. D., Barnett, W. S., and Jung, K. (2008). An effectiveness-based evaluation of five state pre-kindergarten programs. J. Policy Anal. Manage. 27, 122–154. doi: 10.1002/pam.2031

Crossref Full Text | Google Scholar