Despite the existence of several instruments to measure burnout, the Maslach Burnout Inventory (MBI; Maslach et al., 2016) is the most used (Ahola et al., 2017) and is commercially available. However, there are other options, some of them free, such as the Copenhagen Burnout Inventory (Kristensen et al., 2005); the Burnout Measure (Pines and Aronson, 1988; Malach-Pines, 2005); the Educator Burnout Inventory (Wang et al., 2003), the Oldenburg Burnout Inventory (OLBI; Bakker et al., 2004); the Shirom-Melamed Burnout Measure (Shirom and Melamed, 2006); the Bergen Burnout Indicator (Salmela-Aro et al., 2011); the Karolinska Exhaustion Scale (Saboonchi et al., 2013); and the Spanish Burnout Inventory (Gil-Monte et al., 2017).

OLBI seems to be the most prominent alternative to MBI (Demerouti et al., 2000). It was originally developed by Demerouti and Nachreiner (1998), who suggested two burnout dimensions, disengagement and exhaustion, applicable to professionals outside human services occupations. OLBI's versions vary across occupational groups and countries (see Table 1). In some countries—Brazil and Portugal (Campos et al., 2012), Sweden (Dahlin et al., 2007; Rudman et al., 2014), Slovenia (Kogoj et al., 2014), South Africa (Mokgele and Rothmann, 2014), Germany and Greece (Reis et al., 2015), and Malaysia (Mahadi et al., 2018)—OLBI has a version for students. OLBI does not contain any factor correspondent to what the MBI calls “professional efficacy”; this dimension received criticism in some studies (Bresó et al., 2007; Marôco et al., 2014), and, in the opinion of various authors, it is not a core burnout dimension (Bakker et al., 2004; Demerouti and Bakker, 2008) but can be interpreted as a possible burnout consequence (Koeske and Koeske, 1989) related to personality characteristics (Cordes and Dougherty, 1993).

The exhaustion subscale of OLBI has eight items which relate to feelings of emptiness, work overload, the need to rest, and physical, cognitive, and emotional exhaustion (Demerouti et al., 2003). Differently from the exhaustion concept presented in the MBI, the OLBI approach to exhaustion covers cognitive, physical, and affective aspects of exhaustion, which may facilitate the use of the instrument with workers of different kinds of activity (Demerouti et al., 2003; Bakker et al., 2004). The disengagement subscale has also eight items which refer to distancing oneself from the work, together with negative and cynical behaviors and attitudes in relation to one's job (Demerouti and Bakker, 2008). The OLBI's concept of disengagement differs from MBI's depersonalization in terms of the amplitude of the distancing, since OLBI's concept is broader: it may refer to distancing oneself from work in general or, more specifically, to distancing oneself from the content and object, along with experiencing negative attitudes (Demerouti et al., 2003). Thus, disengagement offers a less restricted view of the lack of interest in work. It is important to note that González-Romá et al. (2006) suggested that two of the three work engagement dimensions measured by the Utrecht Work Engagement Scale (UWES), vigor and dedication (the third dimension is absorption), can be paired with emotional exhaustion and cynicism (burnout dimensions). One dimension, named “identification,” involved dedication and cynicism; the other, named “energy,” comprised vigor and exhaustion factors, this indicates that OLBI's negatively- and positively-worded items can be markers for work engagement and burnout (Halbesleben and Demerouti, 2005).

Table 1 summarizes the different OLBI versions used with different samples found through a search of Embase, Scopus, PubMed, Web of Science, and Google Scholar using the terms: “OLBI,” “Oldenburg Burnout Inventory,” “adaptation,” “version,” “validity,” and “psychometric properties.” The OLBI's total number of items changed since its original structure of 25 items (Demerouti and Nachreiner, 1998) to 15 (Demerouti et al., 2001); today's English language version has 16 items (Bakker et al., 2004; Halbesleben and Demerouti, 2005). It has positively and negatively worded items—an equal number of each kind in the two dimensions—something that is considered an advantage (Price, 1997) since it can diminish acquiescence bias despite diminishing the internal consistency of the instruments (Salazar, 2015). OLBI has been translated into many languages, although not always evaluated in its psychometric properties (Table 1). Some studies use OLBI without taking into consideration the recommended steps to adapt an instrument for a country or culture different from the one for which it was originally developed (van de Vijver, 2016). There is a certain lack of use of adequate guidelines when translating and adapting the instrument for a new sample (International Test Commission, 2018). As can be observed in Table 1, the majority of the new OLBI versions have not evaluated their psychometric properties with the appropriate technique (confirmatory factor analysis [CFA]; Brown, 2015). In fact, some of them have avoided both exploratory and confirmatory factor analysis. Usually, the original two-factor structure is the one with better goodness-of-fit indices—even when compared with the other two-factor structures (e.g., with positive items in one group and negatively-worded items in the other), and with one- or four-factor structures (Demerouti et al., 2001). The measurement invariance/equivalence of the instrument across different groups is essential to properly establish comparisons (Davidov et al., 2014). OLBI invariance has been addressed by Demerouti et al. (2001) in its 15-items version for three different professions, they observed metric invariance (same factor's loadings). Demerouti and Nachreiner (1998) stated that three different groups of professionals obtained a similar OLBI (25-items) structure after a principal component analysis for each of the groups. Others researchers have obtained measurement invariance between countries (Demerouti et al., 2003) and between workers and students (Reis et al., 2015). These findings suggest that burnout is not exclusive to human services professions (Demerouti and Nachreiner, 1998; Demerouti et al., 2001) since various studies have tested burnout levels using OLBI in other occupations (e.g., executive directors, white-collar employees, construction workers). Altogether, few studies tested the measurement invariance of the groups with which they established comparisons. Finally, regarding the reliability of the scores, the internal consistency estimates were acceptable to good in most of the studies, while almost all studies reported only the Cronbach's α (see Table 1).

This study aims to describe the psychometric properties of an OLBI version developed simultaneously for workers from Brazil and Portugal, its validity evidence based on the internal structure (dimensionality, measurement invariance, reliability), and the validity evidence based on the relationship with other variables (work engagement); and to compare burnout across sexes and countries. Additionally, the study seeks to present a revision of OLBI's different versions since this is the first version of the instrument developed simultaneously for Portugal and Brazil, adapting an important instrument for understanding burnout in relation to sexes in the organizations. It will be structured by presenting some considerations about burnout among sexes, followed by burnout measured by OLBI.

Following the recommendations of The Standards for Educational and Psychological Testing (American Educational Research Association, 2014), this paper aims to assess two types of validity evidence for the Portuguese version (PT-BR and PT-PT) of the OLBI (Bakker et al., 2004)—one related to the internal structure, the other based on the relations to other variables (work engagement). Since various studies have successfully confirmed the original two-factor structure of OLBI (Halbesleben and Demerouti, 2005; Peterson et al., 2011; Subburaj and Vijayadurai, 2016), it was hypothesized that the tested OLBI version would present a good fit confirming its original dimensionality of two factors (H1). Burnout has been hypothesized by some authors as a higher-order dimension (Taris et al., 1999; Shirom and Melamed, 2006; Marôco et al., 2008). Thus, a possible second-order latent factor, burnout, was tested for OLBI (H2). Through the review of the different versions (Table 1), the majority of the studies showed acceptable to very good reliability of the scores' evidence in terms of internal consistency (e.g., Demerouti and Bakker, 2008; Innstrand, 2016; Subburaj and Vijayadurai, 2016). Consequently, it was assumed that OLBI would present acceptable internal consistency reliability estimates (H3). Some studies found evidence of measurement invariance for OLBI between occupations (Demerouti et al., 2001; Demerouti and Bakker, 2008; Innstrand, 2016) and sex (Foster, 2015), but none investigated measurement invariance among workers of different countries. H4 hypothesized that OLBI will present evidence of measurement invariance between sexes and countries.

Research has found that burnout levels can vary among sexes, with females usually presenting slightly more exhaustion than males (Purvanova and Muros, 2010; Innstrand et al., 2011), females being more likely to experience burnout (Dimou et al., 2016). However, others suggest that research does not allow one to conclude any sex-specific risks (Seidler et al., 2014; Adriaenssens et al., 2015) considering that the burnout differences can be related with the levels of workload as well as care-load (Bekker et al., 2005; Langballe et al., 2011). Burnout can also vary among countries also (Poghosyan et al., 2010; Alexandrova-Karamanova et al., 2016; Jovanović et al., 2016). North American countries have a tendency to present higher exhaustion and disengagement levels than European countries—differences that can be related to cultural aspects (Maslach et al., 2001). However, regarding the Portugal-Brazil comparison, no differences were reported in a previous study (Dias et al., 2010). Occupations can play a substantial role in burnout levels (e.g., emotional challenges of working in the teaching or caregiving role) (Maslach et al., 2001). Altogether, it was hypothesized that burnout's latent means differ between sexes and countries (H5).

Work engagement is known to be a construct with strong correlations with burnout (Demerouti et al., 2010; Petrović et al., 2017), since both can be considered indicators of well-being (Bakker et al., 2014). Thus, the divergent validity evidence based on the relation to other variables, work engagement, was assessed (H6).

A total sample composed of 1,172 participants was collected by combining two independent samples: one sample of Brazilian workers in various occupations (n = 604), and one of Portuguese workers in various occupations (n = 568). Both samples completed the OLBI and the Utrecht Work Engagement Scale (Schaufeli and Bakker, 2003). Participation was anonymous and voluntary. The average age of the total sample was 35.47 years (SD = 9.95), with 65% being female. Workers' occupations were according to the International Standard Classification of Occupations ISCO-08 (International Labour Office, 2012)—mainly professionals or administrative support—and 73% of the sample were, at least, college graduates (Table 2). Regarding children, 59% had none; 45% reported being married or cohabiting.

A non-probabilistic convenience sampling was used. The inclusion criteria were: (1) all participants were workers with a contract or formal ties with their employers, (2) had easy access to a PC, smartphone, or tablet to access the online platform where the instruments were deployed, and (3) were literate.

The OLBI was used to assess burnout, through the development of a version transculturally adapted both for Brazil and Portugal (Table 3). The OLBI is a self-report five-point rating scale (1 = “Strongly disagree”; to 5 = “Strongly agree”) with eight questions within each of the two dimensions, disengagement and exhaustion (Demerouti et al., 2001). The disengagement factor refers to distancing from work in terms of both object and content, and to the development of cynical and negative attitudes and behaviors in relation to one's job (Bakker et al., 2004). Exhaustion refers to feelings of physical fatigue, the need to rest, and feelings of overtaxing and emptiness in relation to work (Demerouti and Bakker, 2008). To develop the Portuguese version (Table 3) the English version of the OLBI was used (Bakker et al., 2004) following The ITC Guidelines for Translating and Adapting Tests (International Test Commission, 2018), adapting the items to the Portuguese language according to the Orthographic Agreement signed by both Portugal and Brazil in 2009. The items were discussed with Portuguese and Brazilian psychologists and methodologists to create a version of the items that gathered the consensus of specialists regarding cultural, semantic, and idiomatic equivalence in the two countries. Finally, a small pilot test was done with 15 workers from each country; this did not suggest any modifications and the Portuguese adapted OLBI's 16 items were understood. The final single version (for both countries) had no other changes.

Work engagement refers to a positive motivational state and is composed of vigor, dedication, and absorption. This construct was measured with UWES-9 in its transculturally adapted version to both Brazil and Portugal (Sinval et al., 2018). It is a self-report instrument scored on a seven-point rating scale (0 = “Never”; 6 = “Always”), with three questions in each of its three dimensions. The UWES has shown good divergent validity evidence with the OLBI, since work engagement and burnout are moderately and negatively related (Goering et al., 2017). It was chosen not only for its good psychometric qualities for both countries, but also because it showed measurement invariance between both countries and it is a short instrument that allows for a robust work engagement measure with only a few items (Schaufeli and Bakker, 2003). It is a well-spread measure across many countries (Sinval et al., 2018) and is actually the most used instrument to measure work engagement. However, studies that investigated the relations between burnout and work engagement have mainly used the MBI for burnout and the UWES for work engagement (Schaufeli and de Witte, 2017). This study used OLBI together with UWES, trying to enrich the discussion about the two concepts, rather than just discussing instruments. The UWES dimensions are vigor, referring to the energy and resilience that one has in work; dedication, referring to being enthusiastic, inspired, and proud of one's work; and absorption, referring to being immersed in one's work without the perception of time passing (Schaufeli et al., 2002). It is expected that high levels of work engagement correspond to highly energized workers (Schaufeli and Bakker, 2010).

Data were gathered from 2015 to 2017, in both countries, in an effort to have a larger sample, since web surveys present low response rates (Massey and Tourangeau, 2013). Both samples completed the OLBI, a brief sociodemographic questionnaire, and the UWES-9. All the collected data were obtained online using LimeSurvey software (LimeSurvey GmbH, 2017) running on the website of two major universities in each country. Nearly 35 percent of the disseminated questionnaires were completed in both countries. Participants were both contacted individually and through companies which answered positively to the invitation to participate in the study. Before filling out the survey, participants were informed about the study, assuring them that the study was a research study and that the company would not access individual data and that companies simply helped the researchers disseminate the study. Informed consent was obtained online from all participants.

To allow comparative studies, the same procedures were used in both countries. The study was approved by the Ethics Committee of the University of Porto (on 03-18-2015), Portugal, and the University of São Paulo (on 01-09-2014; CAAE no. 33301214.2.0000.5407), Brazil, and followed the usual rules for online surveys, namely, no access of participants' companies to individual results and no direct contact between participants and researchers [A few used the email to clarify some details about access to individual data, but it is not possible to identify whether they participated in the study].

A confirmatory factor analysis (CFA) was conducted to verify if the original two-factor structure proposed by Bakker et al. (2004), presented an adequate fit to the study sample. Only complete data cases were considered. As goodness-of-fit indices, SRMR (Standardized Root Mean Square Residual), RMSEA (root mean square error of approximation), NFI (Normed Fit Index), CFI (Comparative Fit Index), and the TLI (Tucker Lewis Index) were used. The fit of the model was considered good for TLI, CFI and TLI values above 0.95; SRMR below 0.08; and RMSEA values below 0.08 (Hoyle, 1995; Boomsma, 2000; McDonald and Ho, 2002; Byrne, 2010). All statistical analyses were performed with R (R Core Team, 2018) and RStudio (RStudio Team, 2018). The descriptive statistics were obtained with the skimr package (McNamara et al., 2018), the standard error of the mean (SEM) was calculated with the plotrix (Lemon, 2006) package and the coefficient of variation (CV) was estimated with the package sjstats (Lüdecke, 2019). To assess multivariate normality, Mardia's multivariate kurtosis (Mardia, 1970) was used; it was calculated using the psych package (Revelle, 2018). The lavaan package (Rosseel, 2012) was selected to conduct the CFA analyses using the Weighted Least Squares Means and Variances (WLSMV) estimation method (Muthén, 1983).

To test the proposed structure for OLBI, the cross-validity evidence was assessed to give information about how well the new structure will fit an independent sample of the same population (Cudeck and Browne, 1983). To do so, the sample was randomly split into two sub-samples through the package minDiff (Papenberg, 2018). The workers' age was used as criteria variable for which it was desired to minimize differences between subsamples (Papenberg, 2018). The subsamples were generated using 1,000 repetitions in order to minimize the differences, since the most equal group assignment was selected. Having two independent subsamples with similar properties, one subsample can be used as calibration subsample, and another as validation subsample (Chin and Todd, 1995).

The convergent validity evidence was analyzed using the average variance extracted (AVE) which was estimated as described in Marôco (2014) and Fornell and Larcker (1981). The constructs' convergent validity evidence was assumed for values of AVE ≥ 0.5 (Hair et al., 2009).

The discriminant validity evidence was checked (Fornell and Larcker, 1981; Marôco, 2014) to verify whether the items that represent a dimension were strongly correlated with other dimensions (Marôco, 2014): for two factors, x and y, if AVEx and AVEy ≥ ρ²_xy (squared correlation between the factors x and y), there is discriminant validity evidence. The Heterotrait-monotrait (HTMT) criterion (Henseler et al., 2015) was also used. Values above 0.85 were considered indicative of satisfactory discriminant validity evidence (Kline, 2016). The HTMT ratios of correlations were calculated using the semTools package (Jorgensen et al., 2018).

The reliability of the scores was assessed with various estimates of internal consistency as recommended (Irwing and Hughes, 2018): α_ordinal (Zumbo et al., 2007), and ω_ordinal (Bollen, 1980; Raykov, 2001) using the semTools package (Jorgensen et al., 2018), higher values were indicative of better internal consistency results. Also, the McDonald's hierarchical omega (ω_H; Zinbarg et al., 2005) was estimated; a higher value of ω_H indicates a stronger influence of the latent variable common to all factors, and that the observed scale scores generalize to scores for the common latent variable (Zinbarg et al., 2007). The omega hierarchical subscale (ω_HS) was calculated for each specific factor, it reflects the reliability of each subscale after controlling for the variance due to the general factor (Reise et al., 2013). Both the ω_H and the ω_HS were used for calculating the internal consistency of the bi-factor model. There is some discussion about the use of α_ordinal (Revelle and Condon, 2018) as so we reported other estimates. The α_ordinal was calculated based on the polychoric correlations. However, the ω_ordinal and ω_H accounts for both item covariances and item thresholds (Green and Yang, 2009). The ω_ordinal and the ω_H are different in the denominator, the first assumes a congeneric factor model where measurement errors aren't correlated (Bollen, 1980), the second uses the observed covariance matrix instead of the model-implied covariance matrix (McDonald, 1999; Jorgensen et al., 2018). The CR was calculated by summing the z scores of the item scores. The second-order factor reliability was also calculated using the omega coefficient (Jorgensen et al., 2018). The proportion of observed variance explained by the second-order factor after controlling for the uniqueness of the first-order factor (ω_{partial L1}); the proportion of the second-order factor explaining the variance of the first-order factor level (ω_L2); and the proportion of the second-order factor explaining the total score (ω_L1) were also calculated. The reliability estimates were calculated with the semTools package (Jorgensen et al., 2018).

The measurement invariance of the higher-order model was assessed using the lavaan package (Rosseel, 2012), the categorical items were considered into account through theta-parameterization (Millsap and Yun-Tein, 2004) to compare a group of seven different models based on the recommendations of Millsap and Yun-Tein (2004) and on the second-order models' invariance specificities (Chen et al., 2005): (a) configural invariance; (b) first-order factor loadings; (c) second-order structural loadings; (d) thresholds of measured variables; (e) intercepts of first-order factors; (f) disturbances of first-order factors; and (g) residual variances of observed variables. Mean scores for burnout latent variable were compared within the structural equation modeling framework; effect sizes (Cohen's d) were determined (Cohen, 1988). The raw means, SDs and score percentiles were calculated using the doBy package (Højsgaard and Halekoh, 2018).

Following the existence of full uniqueness measurement invariance, latent means can be compared. The results of the chi-square difference test suggest that weren't significant differences in burnout (Δχ²_scaled (1) = 1.110; p = 0.292; d = 0.067) among countries. There weren't also statistically significant differences among countries in relation to the burnout dimension (Δχ²_scaled (1) = 1.066; p = 0.302; d = 0.066). The quartiles, means, and SDs (raw values) for each sex within each country are presented in Table 8, these values are presented with the intent of providing population norms values.

Burnout can be conceptualized as being the opposite of work engagement (Halbesleben and Demerouti, 2005). The UWES-9 second-order latent factor model presented a good fit (χ(24)2 = 175.820; p < 0.001; n = 1,104; CFI = 0.999; CFI_scaled = 0.992; NFI = 0.999; TLI = 0.999; SRMR = 0.028; RMSEA = 0.076; P(rmsea ≤ 0.05) < 0.001; 90% CI ]0.065; 0.086[). All factor loadings were statistically significant as also one added residuals' correlation from item 1 and 2 (r = 0.76). The internal consistency reliability estimates were good both for the first-order model (ω_vigor = 0.84; ω_dedication = 0.92; ω_absorption = 0.88; ω_total = 0.96) and for the second-order (ω_{partial L1} = 0.96; ω_L1 = 0.94; ω_L2 = 0.97).

The first-order models of each instrument were used to establish correlations among the first-order latent variables (Table 9), and the second-order models of the instruments were used to analyze the correlation among the respective second-order latent variables (r_{burnout*workengagement} = −0.85). The obtained correlation between the latent OLBI and UWES dimensions was negative and moderate to high, demonstrating the divergent validity of the measures obtained with OLBI (burnout) and the UWES-9 (work engagement).

Burnout seems to be a second-order factor that loads on two first-order factors, disengagement, and exhaustion—which are burnout's core dimensions. This underlines the expansion of the burnout domain beyond the exhaustion affective component (Halbesleben et al., 2004). Female and male workers experience similar levels of burnout, the same happens among Portuguese and Brazilian workers. The obtained results demonstrate that the absence of differences in burnout between sexes and countries suggest that the work experiences in terms of stress are similar in both samples. The observed differences between sexes reported on other studies are small (Purvanova and Muros, 2010) and might be related to other factors—namely family and workload (Bekker et al., 2005; Langballe et al., 2011).

To have instruments with good validity evidence, it is mandatory to have confidence in the obtained measures. OLBI can help to establish comparisons between sexes, and countries. However, one should be aware of the different versions and of the quality of the evidence provided in each study. Also, OLBI can be useful and practical, since it is a freely available self-report psychometric instrument which can contribute to studies where the impact of companies' interventions are studied (Gíslason and Símonardóttir, 2018). In fact, there is evidence that occupational stress can be reduced with specific interventions (Ruotsalainen et al., 2015). The development of a family-friendly work environment should be approached, allowing one a focus on the importance of a balance between work and life (Lo, 2003; Rubino et al., 2013). Giving workers paid sick days, medical and family leave insurance programs, and greater control over their schedule (Appelbaum et al., 2014) are good suggestions to improve the balance between work and home activities. However, any change in organizational practices without a corresponding change in social attitudes will not be enough (Field and Bramwell, 1998). Prevention strategies should consider the social and individual level of those that will receive them (Maslach and Leiter, 2017).

This study used two convenience samples. It had no other psychological measures besides burnout and work engagement, which would allow better assessment of the validity evidence based on the relationship between other variables—namely, predictive, concurrent, and discriminant evidence (American Educational Research Association, 2014). For example, a concurrent burnout measure which would allow verification of the concurrent validity evidence between different burnout instruments, as some studies have done (Demerouti et al., 2003; Marôco and Campos, 2012) would have been useful. Another limitation is that culture-specific aspects of stress were not assessed, which could explain some of the observed differences.

Further studies using OLBI should test its concurrent validity evidence with other burnout instruments (American Educational Research Association, 2014), something that has been tested with success in the Portugal-Brazil version for students (Campos et al., 2012) and in the Greek version for workers (Demerouti et al., 2003). The Portugal-Brazil OLBI version should be tested in samples of specific occupational groups (e.g., Armed Forces Occupations, Craft and Related Trades Workers, Skilled Agricultural, Forestry and Fishery Workers, Plant and Machine Operators and Assemblers, and Elementary Occupations), of whom there was not a satisfactory number in the sample collected for this study. The same applies to lower academic levels, which showed unsatisfactory frequency in this study. Future research should also assess the family load together with the workload to understand the family-work interaction regarding burnout.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.