ORIGINAL RESEARCH article

Front. Psychol., 31 March 2023

Sec. Quantitative Psychology and Measurement

Volume 14 - 2023 | https://doi.org/10.3389/fpsyg.2023.1130497

Psychometric properties of the post-stroke depression scale in the sequelae stage

  • 1. School of Nursing, Wenzhou Medical University, Wenzhou, China

  • 2. Department of Wound Repair, First Affiliated Hospital of Wenzhou Medical University, Wenzhou, China

  • 3. Department of Respiratory and Critical Care Medicine, First Affiliated Hospital of Wenzhou Medical University, Wenzhou, China

  • 4. Department of Emergency, First Affiliated Hospital of Wenzhou Medical University, Wenzhou, Zhejiang, China

  • 5. Department of Rehabilitation Medicine, First Affiliated Hospital of Wenzhou Medical University, Wenzhou, Zhejiang, China

Article metrics

View details

4,4k

Views

1,1k

Downloads

Abstract

Aim:

To evaluate the psychometric properties of the Post-Stroke Depression Scale in the Sequelae Stage (PSDS-SS).

Background:

The incidence of the sequelae stage Post-Stroke Depression (PSD) is high, and the best screening tools are still lacking. Under this circumstances, our research team developed the PSDS-SS by Delphi method, but its psychometric properties need to be further verified.

Method:

This was a cross-sectional study. Seven hundred and sixteen stroke patients in the sequelae stage were enrolled by purpose sampling from May 2022 to September 2022. The exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) were used to verify the factor structure of the scale. The reliability of the scale was tested by Cronbach’s α coefficient, test–retest reliability and composite reliability. The validity of the scale was tested by criterion-related validity, convergent and discriminant validity.

Result:

Eight items were deleted through item analysis. The EFA ended up with a 5-factor scale including 24 items after removing one item with low factor loading. Finally, a 21-item model was established by confirmatory factor analysis, and all the fit indexes were acceptable. The reliability and validity of the total scale and each factor are acceptable.

Conclusion:

The PSDS-SS has a stable factor structure, and demonstrated good reliability and validity. And it would be an effective tool to assess PSD in the sequelae stage.

1. Introduction

Post stroke depression (PSD) is a common neuropsychiatric complication after stroke (Paolucci, 2017; Camargos et al., 2020). As a chronic recurrent disease, the incidence of PSD is about one-third in different periods of stroke (Hackett et al., 2014; Paolucci, 2017). PSD occurs in the early stage, rehabilitation stage and sequelae stage after stroke. The symptoms of PSD in these three different stages are quite different (Li et al., 2019). PSD in the sequelae stage refers to depression occurring 6 months after stroke (Wang, 2009; Zhang et al., 2022). The main symptoms of PSD in the sequelae stage are insomnia, more irritability than usual, easy to be fatigued, decreased sensory ability, loss of speech, suicidal ideation and easy to be crying (Li et al., 2019). PSD in the sequelae stage can seriously affect patients’ functional recovery, rehabilitation outcome and quality of life. Firstly, PSD in sequelae stage significantly worsens the physical function, affects the prognosis of patients, and eventually increases the mortality and disability rate (Maaijwee et al., 2016; Pan et al., 2016; McCurley et al., 2019; Virani et al., 2020). Secondly, it can also reduce the patients’ desire for active rehabilitation, thus affecting the rehabilitation effect of language, movement, walking and other aspects (Huang and Wang, 2016). Finally, PSD would make it difficult for patients to return to work and reduce their quality of life (Mutai et al., 2016; Stein et al., 2018).

Despite the high prevalence and adverse effects of PSD in the sequelae stage, it has not been effectively screened and managed (Lanctôt et al., 2020). The lack of specific screening tools maybe the main reason for this phenomenon. Thus, we developed the Post-Stroke Depression Scale in the Sequelae Stage (PSDS-SS) with 8 dimensions and 33 items through literature review, semi-structured interviews with PSD patients in the sequelae stage, research group discussion, healthcare professional panel evaluation and Delphi consultation. Notably, the theory of learned helplessness was used as the theoretical basis for scale development. Learned helplessness refers to the cognitive, motivational, and emotional deficits when people experiencing an event beyond their control (Alloy et al., 1984; Peterson and Seligman, 1984). The manifestation of learned helplessness is similar to depression. Thus, learned helplessness theory is one of the important theories to explain depression (Wang and Zhang, 2004). Based on the theory of learned helplessness, we built up the conceptual framework of this study as a guidance of scale development (Figure 1). In the conceptual framework, stroke was supposed to be a stimulating event, which incurred six factors of PSD symptoms, namely, motivation, emotion, cognition, behavior, somatization and sleep. And the item pool of the scale were constructed based on the six. The aim of this study was to evaluate the psychometric properties of the PSDS-SS, so as to provide a specific tool for the screening of the sequelae stage PSD.

Figure 1

2. Background

At present, the diagnostic and screening tools for the sequelae stage PSD include mental disease diagnostic systems, ordinary depression rating scales and specific PSD screening scale. But the above tools all have some limitations.

2.1. Mental disease diagnostic systems

There are three mental disease diagnostic systems used to diagnose the sequelae stage PSD: The diagnostic and statistical manual of mental disorders-V (DSM-V); International Classification of Diseases-10 (ICD-10); Chinese Classification and Diagnostic Criteria of Mental Disorders-3 (CCMD-3). The mental disease diagnostic systems used to screen for the sequelae stage PSD have the following problems: (1) Due to the different causes and symptoms of PSD and functional depression, DSM-V, ICD-10, and CCMD-3 do not specify the concept and classification of PSD. So the classification and diagnosis of functional depression are not applicable to PSD (Gainotti et al., 1997; Ma et al., 2021). (2) The need for diagnosis by a physician limits its use in nursing staff (Li et al., 2017). (3) The DSM-V is the standard for diagnosing severe functional depression, which make patients miss the best opportunity for non-pharmacological management.

2.2. Ordinary depression rating scale

The ordinary depression rating scales for the sequelae stage PSD can be divided into examiner-rating scale and self-rating scale. Examiner-rating scales mainly include: Hamilton depression Rating Scale (HAMD), Montgomery and Asberg Depression Rating Scale (MADRS). Self-rating scales mainly include: Beck Depression Inventory-II (BDI-II), Patient Health Questionnaire-9 (PHQ-9), Zung Self-rating Depression Scale (SDS), Center for Epidemiological Studies Depression Scale (CES-D), Hospital Anxiety and Depression Scale (HADS), Geriatric Depression Scale (GDS). However, the above scales are not the best tools for screening PSD in sequelae stage for the following reasons: (1) These scales are not designed for PSD and therefore lack specific items for PSD, resulting in a lack of specificity in screening results (de Coster et al., 2005; Kang et al., 2013; Leng et al., 2020). (2) Some scales need to be evaluated by professional raters, which limits their widespread use. (3) The reliability and validity of some scales in the measurement of the sequelae stage PSD need to be further verified (Roger and Johnson-Greene, 2009; Adams, 2011; Prisnie et al., 2016).

2.3. Specific PSD screening scale

The specific PSD screening scales used to screen PSD in the sequelae stage mainly include: The Post-Stroke Depression Rating Scale (PSDRS), Post-Stroke Depression Scale (PSDS). These scales has the following limitations for screening PSD in the sequelae stage: (1) The PSDRS scale cannot evaluate and classify the severity of PSD (Gainotti et al., 1997), and the screening results are greatly affected by age (Quaranta et al., 2008). In addition, its accuracy in the assessment of sequelae stage PSD is insufficient (Quaranta et al., 2008). (2) The reliability and validity of PSDS has not been verified in large samples, and whether its screening results are affected by other factors needs to be further tested (Yue et al., 2015).

In summary, due to the lack of specific screening tools for sequelae stage PSD at present, our research team developed a specific screening scale for the sequelae stage PSD by Delphi method based on the theory of learned helplessness. This study aimed to evaluate the psychometric properties of the PSDS-SS.

3. Methods

3.1. Participants

Seven hundred and sixteen stroke patients in the sequelae stage were enrolled by purpose sampling from March 2022 to September 2022 in a Grade A class 3 hospital in southeast China. The inclusion criteria were patients: (1) who met the diagnostic criteria for stroke; (2) whose vital signs were stable after stroke; (3) who were ill for more than 6 months, that is, in the sequelae stage of stroke; (4) who were able to communicate in language; (5) who gave informed consent. The exclusion criteria were patients: (1) who were suffering from subarachnoid hemorrhage; (2) with severe heart, liver, renal insufficiency, respiratory failure, malignant tumor and other serious diseases.

The sample size was determined according to the requirements of exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). EFA and CFA use two data sets independently. EFA requires that the number of participants should be 5 to 10 times the number of scale items (Ding et al., 2022). The PSDS-SS obtained by Delphi method has 33 items, and the required sample size should be 165–330. Considering the non-response bias, the sample size should be increased by 10% and the required sample size should be 182–363. A sample size of 350 was assumed for EFA. In addition, the sample size of the CFA needs to be larger than that of the EFA (Li et al., 2019). Thus, we assumed a sample of more than 350 for the CFA. Finally, 716 patients were enrolled in the study. They were randomly divided into EFA and CFA data sets, including 356 patients in EFA and 360 patients in CFA. Forty-five participants were selected to fill in the PHQ-9 scale to test the criterion-related validity. In addition, 40 samples were selected to fill in the PSDS-SS again after 2 weeks to evaluate the test–retest reliability.

3.2. Measure

3.2.1. General demographic questionnaire

The general demographic questionnaire included the following items: (1) general information of patients: age, sex, educational level, marital status, with child or not, ethnicity, religion and personal income after stroke; (2) disease information of the patients: months after stroke, number of strokes, stroke type, stroke site, presence of comorbidities (hypertension, diabetes, dyslipidemia, heart disease and coronary heart disease).

3.2.2. The post-stroke depression scale in the sequelae stage

The PSDS-SS was developed by our research team. It was initially constructed through literature review, semi-structured interviews with PSD patients in the sequelae stage, research group discussion and healthcare professional panel evaluation. Then, a two-round Delphi consultation was conducted. For the two-round Delphi consultation, 18 experts were responded in the first round, and 15 experts were responded in the second round. Their areas of expertise include applied psychology, psychiatric nursing, community and aged care, stroke nursing, stroke rehabilitation and depression treatment. There are 4 nurses, 3 doctors, 10 nursing teachers and 1 psychological worker in the first round, with a working experience of 13–40 years. There are 4 nurses, 2 doctors, 8 nursing teachers and 1 psychological worker in the second round, with a working experience of 13–40 years. The positive coefficient of the two-round Delphi consultation were 81.81 and 83.33%. The authority coefficient of the two-rounds Delphi were 0.90 and 0.89. The Kendall’s W of two-rounds Delphi were 0.135 (p < 0.01) and 0.223 (p < 0.01). The means for the two rounds ranged from 3.83 to 4.89 and 3.93 to 5.00 respectively; the full score ratios ranged from 0.28 to 0.89 and 0.47 to 1.00 respectively; the coefficient of variation ranged from 0.07 to 0.36 and 0.00 to 0.33, respectively. The item-level content validity index (I-CVI) ranged from 0.56 to 1.00 and 0.67 to 1.00 in the first and second rounds consultation. The inclusion criteria of items were mean ≥ 4.00, CV ≤ 0.20, full score ratio ≥ 0.50 and I-CVI ≥ 0.78. After removing items that did not meet the inclusion criteria, we obtained the PSDS-SS with 8 factors (motivation, emotion, cognition, behavior, somatization, sleep, helplessness, and guilt) and 33 items. All items of the PSDS-SS had an I-CVI >0.78, and the S -CVI/Ave was 0.941. The PSDS-SS used a 5-point Likert scale ranging from 1 to 5, corresponding to “none,” “occasionally,” “sometimes,” “often” and “always.” The total score of scale was the sum of all items, and the score of factor was the sum of items in each factor. The total score of scale was 33–165 points in total. A higher score indicated more severe depressive symptoms.

3.2.3. PHQ-9

PHQ was developed by Spitzer et al. in 1999 (Spitzer et al., 1999). PHQ-9 was localized in 2009 (Bian et al., 2009) to measure depression in participants who attending medical, gynecologic, and surgical outpatient department. ThePHQ-9 uses a 4-point Likert scale ranging from 0 to 3, corresponding to “none,” “few days,” “more than half” and “almost everyday” When PHQ-9 is used for screening PSD, the cut-off value is mostly ≥5 points (Yang, 2016). The Cronbach’s α of PHQ-9 was 0.78 in the stroke patients (Dajpratham et al., 2020).

3.3. Data collection

Patients who met the inclusion criteria were selected as potential participants through the electronic medical record system. Data were collected by one-to-one telephone interviews. After informed consent was obtained, the researcher read out the items of the general demographic questionnaire, the PSDS-SS and PHQ-9 to the participants and recorded their answers. Each interview took about 20 min to complete.

3.4. Data analysis

SPSS 24.0 was used for descriptive analysis (mean, standard deviation, frequency and percentage), item analysis and EFA. Amos 24.0 was used for CFA.

3.4.1. Item analysis

Item analysis includes critical ratio (CR), correlation analysis, Cronbach’s α test and factor analysis. Items are deleted when they do not meet two or more of the following criteria: (1) the CR was significant, and should be >3; (2) the correlation between item scores and total scores, and should be significant and >0.40; (3) the Cronbach’s α after deleting an item is smaller than the Cronbach’s α of the total scale before deleting it; (4) factor loading of the items were >0.45; (5) item commonalities were >0.20 (Liu et al., 2022).

3.4.2. EFA

The Kaiser-Meyer-Olkin (KMO) index and Bartlett’s test of sphericity were used to evaluated whether the sample was suitable for EFA. If the Bartlett’s test of sphericity is significant and KMO value is >0.60, the scale is suitable for EFA (Wynter et al., 2017; Forsell et al., 2019). In EFA, Kaiser eigenvalue >1 and scree plot were used to select factors. It is also required that the factor loading of the reserved item should be >0.45 (Şenyuva et al., 2020).

3.4.3. CFA

CFA used maximum likelihood method to estimate the parameters. Because of the violation of normal distribution and multivariate normality, the Bollen–Stine bootstraps were used to estimate chi-square as a model fit index (Komlenac et al., 2018). Three types of model fit indexes were employed to determine the model fit: absolute indexes, relative indexes and parsimony indexes. The absolute index included chi-square value/degrees of freedom (χ2/df), Standardized root mean square residual (SRMR), Root-mean-square error of approximation (RMSEA), Goodness-of-fit index (GFI) and Adjusted goodness-of-fit index (AGFI). It is considered acceptable that χ2/df < 5.00, SRMR <0.08, RMSEA ≤0.08, GFI and AGFI >0.80 (Hooper et al., 2008; Haigh et al., 2011; Duan et al., 2015; Huang et al., 2022). The relative indexes included Tueker-Lewisindex (TLI) and Comparative fit index (CFI). When TLI and CFI are >0.90, the model is considered to have a good fit, but some studies believe that TLI and CFI are also acceptable when them are closer to 0.90 (Brust et al., 2022). Parsimony-adjusted normed fit index (PNFI) and Parsimony comparative fit index (PCFI) were selected as parsimony indexes. The values PNFI and PCFI >0.50 indicates that the model is acceptable (Feizi and Heidari, 2020).

3.4.4. Criterion-related validity, convergent and discriminant validity

The correlation coefficient between PHQ-9 and The PSDS-SS was used to reflect the criterion-related validity. Pearson correlation was used for normal data, and Spearman correlation was used for non-normal data. A significant correlation ranged from 0.40 to 0.80 indicated good criterion-related validity of the scale. The convergent validity was represented by the Average Variance Extracted (AVE), which should be >0.50 (Liu et al., 2022). The discriminant validity is illustrated by comparing the square root of AVE of each factor with the correlation coefficient of this factor and any other factor. The square root of AVE larger than the correlation coefficient between each factor and other factors indicates that the discriminant validity is good (Mo-xi and Xin, 2019).

3.4.5. Reliability

The Cronbach’ α, Test–retest reliability and Composite reliability (CR) were used to evaluate the reliability of the scale. Cronbach’ α was used to evaluate the internal consistency reliability, which needed to be >0.70 (Liu et al., 2022). The retest reliability was calculated by Intraclass correlation coefficient (ICC). When ICC < 0.40, the scale reliability is poor. 0.40 ≤ ICC < 0.75 indicates acceptable reliability; ICC ≥ 0.75 indicates good reliability (Padilha et al., 2020). CR > 0.70 is considered to have an acceptable composite reliability (Hu et al., 2021).

3.5. Ethical considerations

This study was approved by the Hospital Ethics Committee (approval number: KY2022-070). The researchers explained the purpose and significance of the study to the participants before data collection. The patients provided informed consent in this study. The data of patients were kept confidential and were not accessible to any person other than the research team members.

4. Results

4.1. General demographic of the participants

This study finally obtained 716 valid samples, which were randomly divided into two datasets to run EFA (N = 356) and CFA (N = 360). The demographic information is shown in Table 1, and there were no differences in demographic characteristics between the EFA and CFA samples.

Table 1

CharacteristicEFA (N = 356)CFA (N = 360)StatisticValue of p
Age61.671 (11.151,29–92)61.042 (11.151,27–87)Z = −0.8700.384
Sex
Male231 (64.9)244 (67.8)χ2 = 0.6700.413
Female125 (35.1)116 (32.2)
Educational status
Illiteracy82 (23.0)81 (22.5)χ2 = 2.7950.593
Primary school142 (39.9)160 (44.4)
Junior high school88 (24.7)72 (20.0)
High school31 (8.7)34 (9.4)
University or above13 (3.7)13 (3.6)
Marital status
Married336 (94.4)335 (93.1)F = 1.3970.740
Unmarried8 (2.2)7 (1.9)
Divorced2 (0.6)4 (1.1)
Widowed10 (2.8)14 (3.9)
Children
Have children345 (96.9)348 (96.7)χ2 = 0.0340.853
No children11 (3.1)12 (3.3)
Nation
Han348 (97.8)356 (98.9)χ2 = 1.4020.236
Minority8 (2.2)4 (1.1)
Religion
Have132 (37.1)113 (31.4)F = 3.6070.106
Not223 (62.6)247 (68.6)
Income after stroke
Salary or pension132 (37.1)157 (43.6)F = 4.6610.157
Security fund28 (7.9)21 (5.8)
Family support196 (55.1)181 (50.3)
No fixed income0 (0)1 (0.3)
Months after stroke8.406 (7.691)7.723 (4.832)Z = −0.0320.974
Stroke frequency1.132 (0.406)1.144 (0.491)Z = −0.0510.959
Stroke type
Hemorrhagic stroke63 (17.7)53 (14.7)χ2 = 4.5860.105
Ischemic stroke278 (78.1)300 (83.3)
Lacunar infarction15 (4.2)7 (1.9)
Stroke site
Left hemisphere151 (42.4)143 (39.7)χ2 = 1.0980.783
Right hemisphere126 (35.4)136 (37.8)
Cerebellum or brain stem64 (18.0)69 (19.2)
Multiple site15 (4.2)12 (3.3)

General demographic questionnaire.

4.2. Item analysis

The CR for all items were >3.00 (p < 0.001). The correlation coefficient between items 3, 10, 11, 12, 19, 20, 22, 23, and 25 and the total score was <0.40. The Cronbach’s α for the total scale was 0.927, and the Cronbach’s α for the scale increased after the removal of items 10, 16, 19, and 23. In factor analysis, the commonality of items 10, 12, 16, 19, 20, 21, 22, 23 was <0.20, and the factor load was <0.45. Items 10, 12, 16, 19, 20, 21, 22, and 23 that did not meet two or more of the criteria which were considered to be deleted in this step. Finally, 25 items were retained (Table 2).

Table 2

ItemCR ≥ 3Correlation between item and total scoreCronbach ‘α after item deletionFactor analysisThe number of criteria not metItems retained
CommonalityFactor loading
1. I do not want to take the initiative to do things8.296**0.464**0.9260.2050.4530Selected
2. I do not want to go out7.986**0.463**0.9260.2120.4600Selected
3. I do not want to communicate with people6.598**0.397**0.9260.2520.5021Selected
4. I lost interest in things that I used to be interested in8.825**0.511**0.9250.2920.5400Selected
5. I feel down9.317**0.545**0.9250.3740.6110Selected
6. I feel hopeless7.963**0.539**0.9240.5370.7320Selected
7. I feel anxious8.593**0.561**0.9240.4420.6650Selected
8. I worry about the future9.114**0.574**0.9240.3980.6310Selected
9. I can hardly be happy8.256**0.521**0.9250.3070.5540Selected
10. My memory has deteriorated5.462**0.326**0.9290.0640.2534Deleted
11. My ability to think about problems has declined5.879**0.388**0.9260.2130.4621Selected
12. My mind has become dull4.998**0.313**0.9270.1340.3663Deleted
13. I have fewer social activities9.969**0.564**0.9250.3330.5770Selected
14. I cry or want to cry6.490**0.450**0.9250.3870.6220Selected
15. I think a lot9.411**0.584**0.9240.4060.6370Selected
16. I feel tired6.969**0.441**0.9280.1210.3493Deleted
17. I find it difficult to manage everyday life8.876**0.550**0.9240.4600.6790Selected
18. I am not satisfied with my current physical condition10.590**0.644**0.9230.4830.6950Selected
19. I woke up in the middle of the night6.976**0.397**0.9280.0850.2914Deleted
20, I do not fall asleep easily5.857**0.337**0.9270.1510.3883Deleted
21. I have trouble falling asleep after waking up in the middle of the night7.172**0.415**0.9260.1630.4032Deleted
22. I get less sleep6.179**0.343**0.9260.1530.3913Deleted
23. I dream a lot while I sleep4.782**0.277**0.9280.0740.2724Deleted
24. I think the recovery effect is much lower than I expected8.820**0.577**0.9240.4640.6810Selected
25. I do not think friends and family fully understand me5.536**0.377**0.9260.2980.5461Selected
26. I do not feel like my body is getting better11.289**0.710**0.9220.6490.8060Selected
27. I feel powerless about a lot of things11.471**0.746**0.9220.6040.7770Selected
28. I feel isolated6.418**0.445**0.9250.4200.6480Selected
29. I do not think I can change my current status10.988**0.695**0.9220.5710.756Selected
30. I think I’m so useless10.302**0.635**0.9230.5080.7130Selected
31. I consider myself a burden on the family9.537**0.558**0.9240.4000.6320Selected
32. I feel guilty for dragging down my family9.206**0.555**0.9240.4110.6410Selected
33. I do not want to live5.696**0.427**0.9260.4150.6440Selected

The result of item analysis (N = 356).

**p < 0.001; *p < 0.05.

4.3. EFA

All 25 items retained after item analysis were included in the EFA. The KMO of the 25-item scale was 0.916, the Chi-square of the Bartlett’s test of sphericity was 5734.828 (p < 0.001). The result indicated that the data were suitable for EFA. Principal component analysis and varimax rotation were used for analysis. The factors with eigenvalues >1 were extracted. After the first round EFA, 5 factors were extracted, and the cumulative explanatory variation was 66.030%. However, item15 had a factor loading of 0.368, which was <0.45 and was considered be deleted. Then the 24-items scale was tested for suitability of EFA. The KMO was 0.912, and the Chi-square of Bartlett’s test of sphericity was 5546.582 (p < 0.001), indicating that the data were suitable for EFA. There were five factors with eigenvalues >1. In the scree plot, the broken line became flat after the fifth factor (Figure 2). Finally, 5 factors were extracted, and the cumulative explanatory variation was 67.167%. The factor loading of each item on its corresponding factor was >0.45, ranging from 0.478 to 0.900 (Table 3). Factor 1 consists of 6 items named the Helplessness; Factor 2 consists of 5 items named Emotional; Factor 3 consists of 5 items named the Distress; Factor 4 consists of 5 items named Motivation; Factor 5 consists of 3 items named Guilt.

Figure 2

Table 3

ItemFactor loading
12345
180.7910.2380.0540.1280.196
260.7790.2720.2350.1640.243
270.7130.4170.1600.1830.156
240.6790.0790.3220.0370.302
290.6350.4310.1380.2450.148
170.5800.0790.3810.1430.249
90.1180.7520.1120.2380.032
50.2020.7490.1770.0480.169
70.1850.7020.3130.0990.161
80.2350.6650.1720.1210.223
60.3930.5200.4190.0930.176
280.2390.2240.7720.1300.069
250.2020.2190.6870.171−0.079
140.3590.0690.6180.0410.238
330.1470.2270.5960.2090.330
11−0.0080.1670.5720.2520.047
20.0040.0920.1590.8400.144
30.0500.1490.2260.7740.135
10.1720.1680.0510.764−0.039
40.3400.1320.2030.610−0.018
130.400−0.0260.1760.4780.320
310.2490.1970.1020.0750.900
320.2600.1880.1250.0640.893
300.4120.2490.1200.1710.691
Eigenvalue4.0963.2303.0212.9342.838
Variance explained by each factor (%)17.06813.45912.58812.22711.825
Cumulative explanatory variation17.06830.52743.11655.34267.167

The result of EFA (N = 356).

Extraction method: Principal component analysis. Rotation method: Varimax with Kaiser normalization, rotate six times.

4.4. CFA

The above 5 factors (24 items) scale was confirmed by CFA and Model 1 was obtained. The model fit indexed were χ2/df = 3.316, RMSEA = 0.080, SRMR = 0.092, GFI = 0.839, AGFI = 0.801, CFI = 0.860, TLI = 0.841, PNFI = 0.713, PCFI = 0.754, which indicated that the model fits poorly. In addition, CFA requires the factor loading of the items to be >0.40 (Nogales-González et al., 2015), so item 11 and 14 were removed. Item 13 was also removed for better model fitting though its factor loading is slightly higher than 0.40 (0.401). Then the 21-item scale was confirmed by Model 2. The χ2/df = 3.172, RMSEA = 0.078, SRMR = 0.071, GFI = 0.863, AGFI = 0.823, CFI = 0.897, TLI = 0.880, PNFI = 0.731, PCFI = 0.765 of model 2 were all within the acceptable range, which indicated that model 2 was acceptable (Figure 3).

Figure 3

4.5. Validity

The validity of the 5-factor 21-item scale generated by CFA was analyzed. PHQ-9 was used to test the criterion-related validity of the scale. The correlations between the five factors and PHQ-9 were 0.602, 0.490, 0.432, 0.489, and 0.604 (p < 0.05), respectively. The correlation coefficient between the total score and PHQ-9 was 0.731 (p < 0.05). All were in the range of 0.40–0.80.

Convergent validity was indicated by AVE (Table 4). The AVE for the five factors are 0.548, 0.427, 0.533, 0.432, and 0.804, respectively. The AVE of emotions and motivation did not reach 0.50, and the other factors reached the standard. Discriminant validity was expressed by comparing the square root of AVE with the correlation coefficients of a factor and other factors. In this study, the square root of AVE of each factor was larger than the correlation coefficient between that factor and the other factors, except that the square root of the AVE of factor 2 was smaller than its correlation coefficient with factor 1.

Table 4

DomainAVEFactor 1Factor 2Factor 3Factor 4Factor 5
F10.5480.740
F20.4270.727**0.653
F30.5330.359**0.417**0.730
F40.4320.358**0.441**0.213*0.657
F50.8040.492**0.471**0.336**0.132*0.897

Estimated correlations between domains and average variance extracted (AVE) of each domain (N = 360).

**p < 0.001; *p < 0.05. The bold values mean the AVE and the square root of AVE of each factor.

4.6. Reliability

The Cronbach’s α of the total scale, helplessness, emotion, distress, motivation and guilt were0.892, 0.874, 0.779, 0.754, 0.727, and 0.914, which were >0.70. The ICC of the total scale and the five factors were 0.941, 0.901, 0.804, 0.904, 0.934, and 0.906, all of which were >0.75. The CR of each factor were 0.877, 0.788, 0.772, 0.747, and 0.897, all of which were >0.70.

5. Discussion

Through item analysis, EFA, and CFA, a 21-item PSDS-SS was developed. It includes 5 factors: helplessness (6 items), emotion (5 items), distress (3 items), motivation (4 factors) and guilt (3 items). The above three items reduction procedures are rigorous and have been used by many researchers (Jin et al., 2018; Li et al., 2019). The PSDS-SS has good reliability and validity, and is considered as an effective measurement to evaluate the sequelae stage PSD.

5.1. Factor structure of the scale

The PSDS-SS consists of 5 factors, which are helplessness, emotion, distress, motivation and guilt. Helplessness refers to a sense of helplessness caused by patients feeling that their current life and physical condition are out of control. The emotion factor included depressed mood, despair and other emotional changes of patients. Distress refers to the patients’ pessimistic view of social interactions and themselves. Motivation refers to patients’ willingness and initiative to engage in purposeful activities in daily life. Guilt means that patients feel indebted to their family because they bring great pressure to their family.

5.2. Revision of factors and items

The factors of cognitive, sleep, somatization and behavior in the original scale were deleted. The cognitive was removed after EFA. The sleep was removed after item analysis. About 80% of stroke patients experience cognitive decline and 38.2% of them experience insomnia (Umarova, 2017; Baylan et al., 2020; Iddagoda et al., 2020; Buvarp et al., 2021). Therefore, the inclusion of cognitive and sleep items in the scale may reduce the specificity of the scale in screening PSD in the sequelae stage. Somatization was removed from EFA. Items 17 and 18 of somatization were moved to the helplessness. Items 17 and 18 mainly express patients’ despair related to daily life and physical conditions that they could not change, so it is reasonable to move them to the helplessness. Behavior factor was removed from EFA. Specifically, item13 “I have fewer social activities” from the behavior factor was moved to the motivation factor. This item expresses the reduced initiative of patients to participate in social activities, so it is suitable to be placed in the motivation factor. Items 11, 14, 25, 28, and 33 of the cognitive, the somatization, the helplessness and guilt factors were incorporated into a new factor after the EFA. Since these 5 items mainly expressed the trouble of patients about the social environment and themselves, the factor was named “Distress.”

5.3. Validity and reliability of the PSDS-SS

Structural validity (EFA and CFA), criterion-related validity, convergent and discriminant validity were used to test the validity of the scale. (1) Structural validity: A total of 5 factors were extracted from EFA. The factor loadings of the 24 items reached the standard. For the CFA model with 5 factors and 21 items, all model fit indicators were within the acceptable range. The results of EFA and CFA indicate that the scale has good structural validity. (2) Criterion-related validity: The correlation between each factor and PHQ-9, and the total scale and PHQ-9 were all up to standard, indicating the criterion-related validity was good. (3) Convergent validity: Except for the emotion and motivation factors, the AVE of the other factors reached the standard. The AVE is calculated by the factor loadings. When calculating the AVE, the factor loading higher than 0.50 is acceptable (Liu et al., 2022). The factor loadings of all items in the emotion factor were >0.50, and the factor loadings of items in the motivation factor were >0.50 except for item 4 (which was close to 0.50). Thus, the convergent validity was also acceptable for the emotion and motivation factors. (4) Discriminant validity: Except for the discriminant validity of factors 2 and 1, all the other factors met the criterion of discriminant validity. Overall, the discriminant validity of this scale was acceptable (Sousa et al., 2022).

The Cronbach’ α, test–retest reliability and composite reliability (CR) were used to evaluate the reliability of the scale. The Cronbach’ α of each factor and total scale were >0.70, indicating good internal consistency of the scale. The ICC of all factors and total scale were >0.75, indicating that the scale has excellent test–retest reliability and has stability and consistency across time. The CR of all factors was >0.70, indicating good composite reliability.

6. Strengths and limitations

Compared with the current tools used to evaluate the sequelae stage PSD, the PSDS-SS is more practical and targeted. First, compared with the mental disease diagnostic systems, the PSDS-SS is convenient and can be used for patient self-assessment. Secondly, compared with the ordinary depression rating scale, this PSDS-SS was developed from the target population, so it is specific for measuring sequelae stage PSD with better sensitivity and specificity. Third, this PSDS-SS focuses more on the specific course of disease compared with the specific PSD screening scale. The items in the helplessness factor reflected the specific symptom of PSD in the sequelae stage. Specifically, they are the items 18, 24, 26, 27, 29 which came from interviews with PSD patients in the sequelae stage. During the interview, patients expressed a sense of helplessness arising from the long-term disease state, which differs from other scales. The items in the helplessness factor mentioned above manifested specific symptoms of PSD patients in the sequelae stage, which are dissatisfied with physical function, rehabilitation prognosis and life but unable to do anything about it. It can more accurately reflect the characteristics of PSD in a specific stage.

Several limitations should be noted for this study. First, the subjects were from only one hospital, which might limit the universality of the results; further multi-Centre study should be taken to confirm the suitability of the scale. Second, this is a cross-sectional study, longitudinal studies should be conducted to evaluate the dynamic effectiveness of this scale.

7. Conclusion

The PSDS-SS has acceptable reliability and validity that can be used to screen the depressive symptoms of stroke patients in the sequelae stage, which is helpful for the effective identification and symptom management of PSD in the sequelae stage.

Funding

This work was supported by The Wenzhou Municipal Science and Technology Bureau (grant number: Y20220098); The Health Commission of Zhejiang Province (grant number: 2021KY780); and The Wenzhou Municipal Science and Technology Bureau (grant number: Y20220102).

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Statements

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving human participants were reviewed and approved by Research Review Board of the First Affiliated Hospital of Wenzhou Medical University. The patients/participants provided their written informed consent to participate in this study.

Author contributions

YZ: research design and manuscript writing. FL, LN, and HL: data analysis. SL, BG, YG, and YF: data collection. JL: research design and manuscript revising. All authors contributed to the article and approved the submitted version.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

References

  • 1

    AdamsK. B. (2011). Do the GDS and the GDS-15 adequately capture the range of depressive symptoms among older residents in congregate housing?Int. Psychogeriatr.23, 950960. doi: 10.1017/s1041610210002425

  • 2

    AlloyL. B.PetersonC.AbramsonL. Y.SeligmanM. E. (1984). Attributional style and the generality of learned helplessness. J. Pers. Soc. Psychol.46, 681687. doi: 10.1037//0022-3514.46.3.681

  • 3

    BaylanS.GriffithsS.GrantN.BroomfieldN. M.EvansJ. J.GardaniM. (2020). Incidence and prevalence of post-stroke insomnia: a systematic review and meta-analysis. Sleep Med. Rev.49:101222. doi: 10.1016/j.smrv.2019.101222

  • 4

    BianC.HeX.QianJ.WuW.LiC. (2009). The reliability and validity of a modified patient health questionnaire for screening depressive syndrome in general hospital outpatients. J. Tongji Univ. (Med. Sci.)30, 136140.

  • 5

    BrustM.GebhardtW. A.van der VoordeN. A. E.NumansM. E.Kiefte-de JongJ. C. (2022). The development and validation of scales to measure the presence of a teachable moment following a cardiovascular disease event. Prev. Med. Rep.28:101876. doi: 10.1016/j.pmedr.2022.101876

  • 6

    BuvarpD.RafstenL.AbzhandadzeT.SunnerhagenK. S. (2021). A prospective cohort study on longitudinal trajectories of cognitive function after stroke. Sci. Rep.11:17271. doi: 10.1038/s41598-021-96347-y

  • 7

    CamargosQ. M.SilvaB. C.SilvaD. G.ToscanoE. C. B.OliveiraB. D. S.BelloziP. M. Q.et al. (2020). Minocycline treatment prevents depression and anxiety-like behaviors and promotes neuroprotection after experimental ischemic stroke. Brain Res. Bull.155, 110. doi: 10.1016/j.brainresbull.2019.11.009

  • 8

    DajprathamP.PukrittayakameeP.AtsariyasingW.WannaritK.BoonhongJ.PongpirulK. (2020). The validity and reliability of the PHQ-9 in screening for post-stroke depression. BMC Psychiatry20:291. doi: 10.1186/s12888-020-02699-6

  • 9

    de CosterL.LeentjensA. F.LodderJ.VerheyF. R. (2005). The sensitivity of somatic symptoms in post-stroke depression: a discriminant analytic approach. Int. J. Geriatr. Psychiatry20, 358362. doi: 10.1002/gps.1290

  • 10

    DingW.ChenJ.LiuJ.LinB.LiS.LiF.et al. (2022). Development and validation of the health education adherence scale for stroke patients: a cross-sectional study. BMC Neurol.22:69. doi: 10.1186/s12883-022-02597-2

  • 11

    DuanJ.FuJ.GaoH.ChenC.FuJ.ShiX.et al. (2015). Factor analysis of the caregiver quality of life index-cancer (CQOLC) scale for Chinese cancer caregivers: a preliminary reliability and validity study of the CQOLC-Chinese version. PLoS One10:e0116438. doi: 10.1371/journal.pone.0116438

  • 12

    FeiziA.HeidariZ. (2020). Persian version of the brief older People's quality of life questionnaire (OPQOL-brief): the evaluation of the psychometric properties. Health Qual. Life Outcomes18:327. doi: 10.1186/s12955-020-01586-8

  • 13

    ForsellE.KraepelienM.BlomK.IsacssonN.JernelövS.SvanborgC.et al. (2019). Development of a very brief scale for detecting and measuring panic disorder using two items from the panic disorder severity scale-self report. J. Affect. Disord.257, 615622. doi: 10.1016/j.jad.2019.07.057

  • 14

    GainottiG.AzzoniA.RazzanoC.LanzillottaM.MarraC.GaspariniF. (1997). The post-stroke depression rating scale: a test specifically devised to investigate affective disorders of stroke patients. J. Clin. Exp. Neuropsychol.19, 340356. doi: 10.1080/01688639708403863

  • 15

    HackettM. L.KöhlerS.O'BrienJ. T.MeadG. E. (2014). Neuropsychiatric outcomes of stroke. Lancet Neurol.13, 525534. doi: 10.1016/s1474-4422(14)70016-x

  • 16

    HaighE. A.MooreM. T.KashdanT. B.FrescoD. M. (2011). Examination of the factor structure and concurrent validity of the Langer mindfulness/mindlessness scale. Assessment18, 1126. doi: 10.1177/1073191110386342

  • 17

    HooperD.CoughlanJ.MullenM. R. (2008). Structural equation modeling: guidelines for determining model fit. Electron. J. Bus. Res. Methods6, 53–60.

  • 18

    HuG.YuanC.RenH.HuJ.ShangM.WangK. (2021). Reliability and validity of an instrument to assess pediatric inpatients' experience of care in China. Transl. Pediatr.10, 22692280. doi: 10.21037/tp-21-130

  • 19

    HuangC.LushaoboS.ZengpingS.YiX.DongW. (2022). Factors influencing physician's behavioral intention to use traditional Chinese medicine to treat coronavirus disease 2019 based on the theory of planned behavior. J. Tradit. Chin. Med.42, 6336400. doi: 10.19852/j.cnki.jtcm.20220607.001

  • 20

    HuangB.WangG. (2016). A treatment-eldercare integrating program for elderly patients with stroke sequela. J. Nurs. Sci.31, 14. doi: 10.3870/j.issn.1001-4152.2016.13.001

  • 21

    IddagodaM. T.InderjeethC. A.ChanK.RaymondW. D. (2020). Post-stroke sleep disturbances and rehabilitation outcomes: a prospective cohort study. Intern. Med. J.50, 208213. doi: 10.1111/imj.14372

  • 22

    JinX.LiuG. G.GersteinH. C.LevineM. A. H.SteevesK.GuanH.et al. (2018). Item reduction and validation of the Chinese version of diabetes quality-of-life measure (DQOL). Health Qual. Life Outcomes16:78. doi: 10.1186/s12955-018-0905-z

  • 23

    KangH. J.StewartR.KimJ. M.JangJ. E.KimS. Y.BaeK. Y.et al. (2013). Comparative validity of depression assessment scales for screening poststroke depression. J. Affect. Disord.147, 186191. doi: 10.1016/j.jad.2012.10.035

  • 24

    KomlenacN.SillerH.BliemH. R.HochleitnerM. (2018). Validation of the internal structure of a German-language version of the gender role conflict scale – short form. Front. Psychol.9:1161. doi: 10.3389/fpsyg.2018.01161

  • 25

    LanctôtK. L.LindsayM. P.SmithE. E.SahlasD. J.FoleyN.GubitzG.et al. (2020). Canadian stroke best practice recommendations: mood, cognition and fatigue following stroke, 6th edition update 2019. Int. J. Stroke15, 668688. doi: 10.1177/1747493019847334

  • 26

    LengH.DingW.ZhangW.WangH. (2020). Progress in assessment of post-stoke depression. Chin. J. Stroke15, 795800. doi: 10.3969/j.issn.1673-5765.2020.07.019

  • 27

    LiJ.LiY.DingW.TuH.GuoJ.XuH.et al. (2019). Research status quo on symptom clusters in patients with post-stroke depression. Chin. Nurs. Res.33, 29772978. doi: 10.12102/j.issn.1009-6493.2019.17.013

  • 28

    LiJ.LiY.LiP.YeM. (2019). Early symptom measurement of post-stroke depression: development and validation of a new short version. J. Adv. Nurs.75, 482493. doi: 10.1111/jan.13885

  • 29

    LiJ.LuoY.LiY.WuR. (2017). Research progress on diagnosis and screening tools of post-stroke depression. Chin. Nurs. Res.31, 12981300. doi: 10.3969/j.issn.1009-6493.2017.11.006

  • 30

    LiuJ.ZhengZ.GeL.HuangY.YangQ.ChenY.et al. (2022). Reliability and validity of the mandarin version of the Trust in Nurses Scale. J. Nurs. Manag.30, 13661375. doi: 10.1111/jonm.13623

  • 31

    MaZ.WangT.ChenJ.ZhongS. (2021). Advances in pathogenesis, diagnosis and treatment of post-stroke depression. J. Logist. Univ. PAP (Med. Sci.)30, 6974. doi: 10.16548/j.2095-3720.2021.01.022

  • 32

    MaaijweeN. A.TendolkarI.Rutten-JacobsL. C.ArntzR. M.SchaapsmeerdersP.DorresteijnL. D.et al. (2016). Long-term depressive symptoms and anxiety after transient ischaemic attack or ischaemic stroke in young adults. Eur. J. Neurol.23, 12621268. doi: 10.1111/ene.13009

  • 33

    McCurleyJ. L.FunesC. J.ZaleE. L.LinA.JacoboM.JacobsJ. M.et al. (2019). Preventing chronic emotional distress in stroke survivors and their informal caregivers. Neurocrit. Care.30, 581589. doi: 10.1007/s12028-018-0641-6

  • 34

    Mo-xiS.XinY. M. (2019). Leveraging core capabilities and environmental dynamism for food traceability and firm performance in a food supply chain: a moderated mediation model. J. Integr. Agric.18, 18201837. doi: 10.1016/S2095-3119(19)62590-6

  • 35

    MutaiH.FurukawaT.NakanishiK.HaniharaT. (2016). Longitudinal functional changes, depression, and health-related quality of life among stroke survivors living at home after inpatient rehabilitation. Psychogeriatrics16, 185190. doi: 10.1111/psyg.12137

  • 36

    Nogales-GonzálezC.LosadaA.Romero-MorenoR. (2015). Confirmatory factor analysis of the Spanish version of the revised memory and behavior problems checklist. Int. Psychogeriatr.27, 683692. doi: 10.1017/s1041610214002476

  • 37

    PadilhaJ. F.da SilvaJ. B.SeidelE. J.DriussoP. (2020). Intra- and inter-rater reliability of post-void residual bladder volume with ultrasound. Int. Urogynecol. J.31, 973979. doi: 10.1007/s00192-019-04045-1

  • 38

    PanL.YinY.YangN. (2016). To explore the effect of narrative medicine intervention on depression of stroke patients in sequelae period. Chin. Med. Hum.2, 1215. doi: 10.3969/j.issn.2095-9753.2016.12.006

  • 39

    PaolucciS. (2017). Advances in antidepressants for treating post-stroke depression. Expert. Opin. Pharmacother.18, 10111017. doi: 10.1080/14656566.2017.1334765

  • 40

    PetersonC.SeligmanM. E. (1984). Causal explanations as a risk factor for depression: theory and evidence. Psychol. Rev.91, 347374. doi: 10.1037/0033-295X.91.3.347

  • 41

    PrisnieJ. C.FiestK. M.CouttsS. B.PattenS. B.AttaC. A.BlaikieL.et al. (2016). Validating screening tools for depression in stroke and transient ischemic attack patients. Int. J. Psychiatry Med.51, 262277. doi: 10.1177/0091217416652616

  • 42

    QuarantaD.MarraC.GainottiG. (2008). Mood disorders after stroke: diagnostic validation of the poststroke depression rating scale. Cerebrovasc. Dis.26, 237243. doi: 10.1159/000147450

  • 43

    RogerP. R.Johnson-GreeneD. (2009). Comparison of assessment measures for post-stroke depression. Clin. Neuropsychol.23, 780793. doi: 10.1080/13854040802691135

  • 44

    ŞenyuvaE.KayaH.CanG. (2020). A valid and reliable tool in assessing patient education: the patient education implementation scale. Int. J. Nurs. Pract.26:e12800. doi: 10.1111/ijn.12800

  • 45

    SousaF. P.RoldãoM. G.RebotimA. M.FigueiraA. R.BarbosaJ.FradiqueE.et al. (2022). The neonatal palliative care attitude scale: psychometric properties for Portuguese neonatal nurses. Palliat. Support. Care1-6, 16. doi: 10.1017/s1478951522000797

  • 46

    SpitzerR. L.KroenkeK.WilliamsJ. B. (1999). Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study. Primary care evaluation of mental disorders. Patient health questionnaire. JAMA282, 17371744. doi: 10.1001/jama.282.18.1737

  • 47

    SteinL. A.GoldmannE.ZamzamA.LucianoJ. M.MesséS. R.CucchiaraB. L.et al. (2018). Association between anxiety, depression, and post-traumatic stress disorder and outcomes after ischemic stroke. Front. Neurol.9:890. doi: 10.3389/fneur.2018.00890

  • 48

    UmarovaR. M. (2017). Adapting the concepts of brain and cognitive reserve to post-stroke cognitive deficits: implications for understanding neglect. Cortex97, 327338. doi: 10.1016/j.cortex.2016.12.006

  • 49

    ViraniS. S.AlonsoA.BenjaminE. J.BittencourtM. S.CallawayC. W.CarsonA. P.et al. (2020). Heart disease and stroke Statistics-2020 update: a report from the American Heart Association. Circulation141, e139e596. doi: 10.1161/cir.0000000000000757

  • 50

    WangX. (2009). Effects of systematic family health education on patients with stroke sequelae. Modern Nurse05, 8990. doi: 10.3969/j.issn.1006-6411-B.2009.05.062

  • 51

    WangC.ZhangN. (2004). Attribution theory and attribution training for depression. Chin. Ment. Health J.06, 423425+435. doi: 10.3321/j.issn:1000-6729.2004.06.020

  • 52

    WynterK.TranT. D.RoweH.FisherJ. (2017). Development and properties of a brief scale to assess intimate partner relationship in the postnatal period. J. Affect. Disord.215, 5661. doi: 10.1016/j.jad.2017.03.001

  • 53

    YangH. (2016). Value of 9 items patients' health questionnaire depression scale in screening of post-stroke depression. J. Clin. Med. Pract.20, 2830. doi: 10.7619/jcmp.201608009

  • 54

    YueY.LiuR.LuJ.WangX.ZhangS.WuA.et al. (2015). Reliability and validity of a new post-stroke depression scale in Chinese population. J. Affect. Disord.174, 317323. doi: 10.1016/j.jad.2014.11.031

  • 55

    ZhangS.LiuY.YangH.ZhaoZ.DengP.XiaoT.et al. (2022). Investigation and analysis of depression status of stroke patients in sequelae stage in Mianzhu area. Chin. J. Phys. Med. Rehabil.44, 171175. doi: 10.3760/cma.j.issn.0254-1424.2022.02.018

Summary

Keywords

post-stroke depression, exploratory factor analysis, confirmatory factor analysis, sequelae stage, measurement development

Citation

Zeng Y, Li F, Ning L, Fu Y, Ge Y, Gan B, Lin S, Lin H and Li J (2023) Psychometric properties of the post-stroke depression scale in the sequelae stage. Front. Psychol. 14:1130497. doi: 10.3389/fpsyg.2023.1130497

Received

23 December 2022

Accepted

15 March 2023

Published

31 March 2023

Volume

14 - 2023

Edited by

Andrea Greco, University of Bergamo, Italy

Reviewed by

Malahat Akbarfahimi, Iran University of Medical Sciences, Iran; Wentao Huang, Sun Yat-sen University Cancer Center (SYSUCC), China

Updates

Copyright

*Correspondence: Suichai Lin, ; Haiyun Lin, ; Jufang Li,

This article was submitted to Quantitative Psychology and Measurement, a section of the journal Frontiers in Psychology

Disclaimer

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

Outline

Figures

Cite article

Copy to clipboard


Export citation file


Share article

Article metrics