Introduction
The lack of replication of key effects in psychology has highlighted some fundamental problems with reporting of research findings and methods used (Asendorpf et al., 2013; Open Science Collaboration, 2015). Problems with replication have been attributed to sources of bias such as questionable research practices like HARK-ing (Kerr, 1998) or p-hacking (Simmons et al., 2011). Another potential source of bias is lack of precision in the conduct and methods used in psychological research, which likely introduces systematic error into data collected with the potential to affect results. A related issue is lack of accuracy in reporting study methods and findings. There is, therefore, increased recognition in the importance of transparency when reporting study outcomes to enable the scientific community to make fair, unbiased appraisals of the implications and worthiness of study findings. Lack of transparency hinders scientific progress as it may lead to erroneous conclusions regarding the implications of research findings, and may impede comparison and synthesis of findings across studies. As a result, researchers have become interested in research quality and the need for comprehensive, transparent reporting of findings (Asendorpf et al., 2013). This has resulted in calls for appropriate reporting standards and means to assess study quality (Cooper, 2011; Greenhalgh and Brown, 2017). In the present article we review the issue of study quality in psychology, and argue for valid and reliable means to assess study quality in psychology. Specifically, we contend that appropriate assessment checklists be developed for survey studies, given the prominence of surveys as a research method in the field.
Importance of Assessing Study Quality
Study quality is the degree to which researchers conducting the study have taken appropriate steps to maximize the validity of, and, minimize bias in, their findings (Khan et al., 2011). Studies of lower quality are more likely to have limitations and deficits which introduce error variance to data that can bias results and their interpretation. Studies of higher quality are less likely to include these errors, or more likely to provide clear and transparent reporting of errors and limitations, resulting in greater precision and validity of findings and their interpretation (Oxman and Guyatt, 1991; Moher et al., 1998). Study quality assessment came to prominence from the evidence-based medicine approach, which focussed on identifying, appraising, and synthesizing medical research (Guyatt et al., 1992). The ideas have since been applied to other disciplines, including the behavioral and social sciences (Michie et al., 2005; APA, 2006b). Assessment of study quality has several advantages, such as identifying the strengths and weaknesses in evidence, providing recommendations for interventions, policy, and practice, and improving research and publication standards (Greenhalgh, 2014; Greenhalgh and Brown, 2017). Moreover, in the context of evidence syntheses, study quality can be used to screen studies for inclusion, identify sources of bias in the results, and measure the impact of study quality on the results through subgroup and sensitivity analyses (Johnson et al., 2014).
Study quality assessment is typically performed with the use of a checklist or “tool,” containing a series of quality-related items. Recent reviews have identified a large number of tools (N = 193) used to assess study quality in the health and social sciences (Katrak et al., 2004). Tools have been adopted to appraise the quality of studies with specific designs such as experimental (e.g., Jadad et al., 1996), systematic reviews and meta-analyses (e.g., Oxman and Guyatt, 1991), and qualitative (e.g., Long and Godfrey, 2004) research. Generic tools, purported to be applicable to multiple study designs and across multiple disciplines, also exist (e.g., Glynn, 2006). However, most quality assessment tools have not been developed with sufficient attention to validity and reliability (Katrak et al., 2004; Moyer and Finney, 2005; Crowe and Sheppard, 2011; Johnson et al., 2014), and no quality assessment tool has been universally endorsed as fully sufficient to assess study quality (Alderson et al., 2003). Prominent criticisms of existing tools refer to the absence of validity and reliability checks in their development, as well as the absence of clear guidance on assessment procedures and scoring (Moyer and Finney, 2005; Crowe and Sheppard, 2011). Despite these limitations, quality assessment tools have been applied extensively across health and social sciences, especially in evidence syntheses.
In psychology, study quality assessment was not recognized as an integral component of the research process until relatively recently. Formal recommendations for conducting quality appraisal in meta-analyses in psychology initially appeared in the Meta-Analysis Reporting Standards (MARS) and the American Psychological Association publication manual (APA, 2006a; Appelbaum et al., 2018). Since the publication of these guidelines, awareness and application of quality appraisal has expanded rapidly, and, while still not fully accepted as standard practice, quality appraisal is frequently viewed as an essential component of evidence syntheses in psychology.
Quality Assessment in Psychology Survey Research
Many studies in psychology adopt survey methods. Surveys are used extensively across psychology disciplines to examine relations among psychological constructs measured through psychometric scaling, and to test hypotheses with respect to relations among constructs (Check and Schutt, 2012; Ponto, 2015). However, despite the increasing demand for quality appraisal and the pervasiveness of survey designs in psychology, there are no quality assessment tools developed specifically for survey research in psychology. Given the centrality of survey methods (Ponto, 2015), development of a dedicated, fit-for-purpose quality tool should be considered a priority.
The lack of tools to appraise study quality in survey research has led researchers to adapt tools from other disciplines, or to identify relevant quality criteria from scratch and develop their own tool. To illustrate, in their meta-analysis linking job satisfaction to health outcomes, Faragher et al. (2005) stated that “…a thorough search failed to identify criteria suitable for correlational studies. A measure of methodological rigor was thus developed specifically for this meta-analysis” (p. 107). More recently, Hoffmann et al. (2017) in a meta-analysis of cognitive mechanisms and travel mode choice stated: “No suitable quality assessment tool was found to assess such survey studies. We therefore applied three criteria that were highlighted across six previous studies recommending bias assessment in correlational studies” (p. 635). In the absence of quality appraisal tools, some meta-analyses, especially those including intervention studies, have implemented universal reporting guidelines as proxies for study quality appraisal (Begg et al., 1996; Jarlais et al., 2004; Von Elm et al., 2007; Moher et al., 2009). Although these universal reporting guidelines are well-accepted, they are not, strictly speaking, quality appraisal tools, and it is unclear if they are suitable for assessing study quality in psychology, including research adopting survey methods.
The application of different tools, or individual criteria, to assess research quality, has a number of drawbacks. First, applying different tools to the same body of evidence can produce different conclusions about the quality of the evidence. This would have serious implications within the context of a meta-analysis, as the effect size may vary as a function of the quality appraisal tool used. For example, Armijo-Olivo et al. (2012) compared the performance of two frequently-used quality appraisal tools, the Cochrane Collaboration Risk of Bias Tool (CCRBT; Higgins and Altman, 2008) and the Effective Public Health Practice Project Quality Assessment Tool (EPHPP; Jackson and Waters, 2005) in a systematic review of the effectiveness of knowledge translation interventions to improve the management of cancer pain, and found that both tools performed differently. Similarly, Jüni et al. (1999) applied 25 quality appraisal scales to the results of a meta-analysis comparing low-molecular-weight heparin with standard heparin for clot prevention in general surgery, and found that different quality scales produced different conclusions regarding the relative benefits of heparin treatments. For studies classed as high quality on some tools, there was little difference in outcome for two types of heparin, whereas for studies classed as high quality on others, one was found to be superior. Moreover, the overall effect size was positively associated with scores on some quality tools but inversely associated with scores on others. Second, the adapted quality assessment tools used by psychologists were not developed to evaluate research in psychology, and may consequently lack validity, and incompletely cover important study quality components.
Problems Arising from Quality Assessment Methods: An Illustration
To illustrate the longstanding problems resulting from the absence of a fit-for-purpose tool and the application of a variety of quality appraisal strategies, we provide examples from a brief summary of quality assessments from meta-analyses of psychological survey research (Table 1)1 We identified two prominent limitations of the tools: the quality criteria adopted and the scoring strategies employed.
Table 1
| Study | Quality tool used | Discipline | Number of quality criteria | Scoring Strategy | Type of scoring | Guide or explanation of criteria provided? | Quality classification system |
|---|---|---|---|---|---|---|---|
| Cuijpers et al., 2010 | Developed quality criteria from a review of empirically supported psychotherapies (Chambless and Hollon, 1998) and from methodological quality recommendations of the Cochrane Collaboration (Higgins and Green, 2006) | Clinical/counseling psychology | 8 | Checks of whether quality criteria were met | A sum of criteria met by the study | Explanation of criteria provided by authors | A study that met all quality criteria was classified as high quality, otherwise it was classified as lower quality |
| Faragher et al., 2005 | Developed quality criteria based on guidelines on research procedures in organizational psychology and expert consensus | Organizational/ industrial/ occupational psychology. | 10 | Each criterion was given a 0 score (rating) for unacceptable rigor or 1 for acceptable rigor | A summated rigor score computed (range 0–10) | Not indicated | A study that met all 10 criteria was classified as of acceptable rigor, otherwise it was classed as of unacceptable rigor |
| Godfrey et al., 2015 | Effective Public Health Practice Project Quality Assessment Tool (EPHPP; Jackson and Waters, 2005) | Clinical/counseling psychology; health psychology; applied psychology | 6 | Each criterion was given 1 point for a weak quality rating, 2 points for a moderate quality rating, and 3 points for a strong quality rating | Sum of scores divided by total number of applicable criteria | Tool is published with guide | Studies of weak quality had a rating of 3, while studies of moderate quality had a rating of 2, and studies of strong quality had a rating of 1. |
| Hagger et al., 2017 | Quality criteria adapted from the National Institutes of Health Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies (National Institutes of Health, 2014), and from other quality checklists used in cross-sectional survey designs (Jack et al., 2010; Husebø et al., 2013; Oluka et al., 2014). | Health psychology; social psychology | 16 | A score of 1 was assigned for each criterion met and a score of zero 0 for each criterion not met or when there was insufficient information provided to evaluate the criterion | Three types of scoring: weighted checklist score out of 10; Tertile division of checklist scores; Average checklist score | Explanation of criteria provided by authors | Tertile division of scores on the quality checklist resulted in studies above the upper tertile classified as high quality and studies below the lower tertile classified as low quality. Also, studies scoring an average of ≥6 were classified as high quality and studies scoring an average score of < 6 were classified as low quality |
| Hoffmann et al., 2017 | Criteria for correlational designs recommended in six previous studies (Gauthier, 2003, (Effective Public Health Practice Project [EPHPP], Jackson and Waters, 2005; Von Elm et al., 2007; Wong et al., 2008; Pace et al., 2012; National Heart, Lung, and Blood Institute, 2014) | Applied psychology; traffic psychology | 5 | A score of one (1) assigned for criteria met and a score of zero (0) assigned for criteria not met or with insufficient information provided. | Total mean score | Explanation of criteria provided by authors | Studies that received an overall score > 2 were rated as high quality, those receiving scores 1–2 were rated as medium quality, and those receiving a < 1 score were rated low quality |
| Pantelic et al., 2015 | Adapted version of the Cambridge Quality Checklists (CQC; Murray et al., 2009) | Cultural psychology; health psychology | 8 | Each criterion was assigned a numerical score between 0 and 6 | One hundred per cent score would indicate the maximum possible score across all correlations in a study | Tool is published with guide | Manuscript reported quality scores but did not formally classify studies according to quality |
| Protogerou et al., 2018 | Adapted version of a generic quality appraisal tool Glynn, 2006 | Health psychology; social psychology; applied psychology | 23 | Each quality criterion was checked as being present (yes = 1); absent (no = 2); unclear (3) or not applicable (4) | A ratio of the “yes” answers by the total applicable items, multiplied by 100 | Tool is published with guide | In line with the tool's guidelines, studies receiving a total score of < 75% were classified as of questionable quality, whereas studies with a total score of ≥75% were classified of acceptable quality. |
| Quon and Mcgrath, 2014 | Eight criteria to assess study quality | Health psychology | 8 | Not indicated in manuscript. | Not indicated | Not indicated in manuscript | High quality or low quality (cut-offs not indicated) |
| Santos et al., 2017 | A short, adapted version of the Joanna Brigs Institute critical appraisal checklist for studies reporting prevalence data (Joanna Briggs Institute, 2014) | Health psychology; sports psychology. | 5 | Each quality criterion was scored as yes, no, unclear or not applicable No corresponds to a limitation in the respective methodological category | The tool does not allow for numerical summative scoring Quality was used in sensitivity analysis implying summative scoring but no details provided | Tool is published with guide | Not clearly indicated |
| Young et al., 2014 | Checklist informed by the Strengthening of Reporting of Observational Studies in Epidemiology (STROBE: Von Elm et al., 2007) and Consolidated Standards for Reporting Trials (CONSORT: Moher et al., 2010) statements, augmented with items from two reviews (Rhodes et al., 2009; Plotnikoff et al., 2013); and a list of “strong model characteristics” (Noar and Zimmerman, 2005) | Health psychology; sports psychology | 11 | Each quality criterion was scored as present (Y), absent (N), unclear or inadequately described' (0) or not applicable (n/a) | Sum of scores of present quality criteria | Explanation of criteria provided by authors | Not clearly indicated |
Summary of quality assessment tool characteristics in studies reviewed.
Quality Criteria
The number of assessed quality criteria ranged between 5 and 23 across the meta-analyses. Also, the type and origin of quality criteria was highly variable. For instance, two meta-analyses (Faragher et al., 2005; Cuijpers et al., 2010) developed quality criteria specifically for their research, while seven meta-analyses (Young et al., 2014; Godfrey et al., 2015; Pantelic et al., 2015; Hagger et al., 2017; Hoffmann et al., 2017; Santos et al., 2017) applied adapted criteria from existing quality tools, reporting guidelines, and literature searches. One study indicated quality criteria without explaining how those were developed or chosen (Quon and Mcgrath, 2014). Although most studies appraised sampling and recruitment procedures, there was variability in the criteria adopted. For example, Hoffmann et al. (2017) appraised whether or not the sample size was sufficient to analyze data using structural equation modeling, while (Quon and Mcgrath, 2014) adopted an absolute total sample size (N = 1000) as their criterion for quality. Similarly, most studies assessed the “appropriateness” of statistical analyses, without clarifying what was considered “appropriate”.
Assessment and Scoring
There was substantive variability in the scoring strategies used to assess study quality across the meta-analyses. Some meta-analyses adopted numerical scoring systems calculating overall percentages, summary scores, and mean scores for the quality criteria adopted (e.g., Protogerou et al., 2018), while other studies did not employ numerical or overall scoring (e.g., Santos et al., 2017). In relation to this, most studies classified studies in terms of high (or “acceptable”) quality vs. low (or “questionable”) quality, while others did not categorize studies in terms of quality. Some studies indicated that quality assessment was informed by published manuals or guidelines on quality criteria, while other studies provided no information on the guidelines or definitions of criteria adopted.
Given the disparate quality appraisal strategies adopted by the meta-analyses, we contend, in line with Armijo-Olivo et al. (2012) and Jüni et al. (1999), that quality assessment outcomes are dependent on the specific tool applied, and that different tools might lead to different conclusions on quality. Moreover, it would be difficult to replicate the quality assessment procedures adopted in most of these meta-analyses, given the limited information provided. We also note that quality criteria relevant to psychological survey studies were missed in the quality assessment on some meta-analyses. For example, ethical requirements, such as consent and debriefing procedures, and response and attrition rates were not checked consistently.
Conclusion and Recommendations
Assessment of study quality is an important practice to promote greater precision, transparency, and evaluation of research in psychology. Assessing the quality of studies may permit researchers to draw effective conclusions and broader inferences with respect to results from primary studies, and when synthesizing research across studies, provide the opportunity to evaluate the general quality of research in a particular area. Given the prominence of survey research in psychology, the development of appropriate means to assess the quality of survey research would yield considerable benefits to researchers conducting, and data analysts evaluating, survey research. We argue that a fit-for-purpose quality appraisal tool for survey studies in psychology is needed. We would expect the development of such a tool to be guided by discipline-specific research standards and recommendations (BPS, 2004; APA, 2006b; Asendorpf et al., 2013). We would also expect the tool to be developed through established methods, such as expert consensus, to ensure satisfactory validity and reliability of the resulting tool (for examples and discussion of these strategies see Jones and Hunter, 1995; Jadad et al., 1996; Crowe and Sheppard, 2011; Jarde et al., 2013; Waggoner et al., 2016).
Statements
Author contributions
CP and MH conceived the ideas presented in the manuscript and drafted the manuscript.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Footnotes
1.^A comprehensive version of Table 1 with full details of study quality criteria is provided online: https://osf.io/wbj5z/?view_only=ffbb265cf43f498999ab69bc57c60eb5
References
1
AldersonP.GreenS.HigginsJ. P. T. (2003). Cochrane Reviewers' Handbook. Available Online at: http://www.cochrane.org/resources/handbook/hbook.htm (Accessed October 1, 2018).
2
APA (2006a). American Psychological Association Publication Manual.Washington, DC: American Psychological Society.
3
APA (2006b). Evidence-based practice in psychology. Am. Psychol.61, 271–285. 10.1037/0003-066X.61.4.271
4
AppelbaumM.CooperH.KlineR. B.Mayo-WilsonE.NezuA. M.RaoS. M. (2018). Journal article reporting standards for quantitative research in psychology: the APA publications and communications board task force report. Am. Psychol.73, 3–25. 10.1037/amp0000191
5
Armijo-OlivoS.StilesC. R.HagenN. A.BiondoP. D.CummingsG. G. (2012). Assessment of study quality for systematic reviews: a comparison of the cochrane collaboration risk of bias tool and the effective public health practice project quality assessment tool: methodological research. J. Eval. Clin. Pract.18, 12–18. 10.1111/j.1365-2753.2010.01516.x
6
AsendorpfJ. B.ConnerM.De FruytF.De HouwerJ.DenissenJ. J. A.FiedlerK.et al. (2013). Recommendations for increasing replicability in psychology. Eur. J. Pers.27, 108–119. 10.1002/per.1919
7
BeggC.ChoM.EastwoodS. (1996). Improving the quality of reporting of randomized controlled trials: the CONSORTstatement. JAMA276, 637–639. 10.1001/jama.1996.03540080059030
8
BPS (2004). Guidelines for Minimum Standards of Ethical Approval in Psychological Research.Leicester: British Psychological Society. Available online at: http://www.bps.org.uk/ (Accessed October 1, 2018).
9
ChamblessD. L.HollonS. D. (1998). Defining empirically supported therapies. J. Consult. Clin. Psychol.66, 7–18.
10
CheckJ.SchuttR. K. (2012). Survey research, in Research Methods in Education, eds CheckJ.SchuttR.K. (Thousand Oaks, CA: Sage), 159–185.
11
CooperH. (2011). Reporting Research in Psychology: How to Meet Journal Article Reporting Standards. Washington, DC: American Psychological Association.
12
CroweM.SheppardL. (2011). A review of critical appraisal tools show they lack rigor: alternative tool structure is proposed. J. Clin. Epidemiol.64, 79–89. 10.1016/j.jclinepi.2010.02.008
13
CuijpersP.Van StratenA.BohlmeijerE.HollonS. D.AnderssonG. (2010). The effects of psychotherapy for adult depression are overestimated: a meta-analysis of study quality and effect size. Psychol. Med.40, 211–223. 10.1017/S0033291709006114
14
FaragherE. B.CassM.CooperC. L. (2005). The relationship between job satisfaction and health: a meta-analysis. Occup. Environ. Med.62, 105–112. 10.1136/oem.2002.006734
15
GauthierB. (2003). Assessing Survey Research: A Principled Approach. Available online at http://www.circum.qc.ca/textes/assessing_survey_research.pdf (Accessed January 27, 2017).
16
GlynnL. (2006). A critical appraisal tool for library and information research. Library Hi Tech24, 387–399. 10.1108/07378830610692154
17
GodfreyK. M.GalloL. C.AfariN. (2015). Mindfulness-based interventions for binge eating: a systematic review and meta-analysis. J. Behav. Med.38, 348–362. 10.1007/s10865-014-9610-5
18
GreenhalghJ.BrownT. (2017). Quality assessment: Where do I begin?, in Doing a Systematic Review: A Student's Guide, eds BolandA.CherryM. G.DicksonR. (London: Sage), 61–83.
19
GreenhalghT. (2014). How to Read a Paper: The Basics of Evidence-Based Medicine. London, UK: Wiley.
20
GuyattG.CairnsJ.ChurchillD. (1992). Evidence-based medicine: a new approach to teaching the practice of medicine. JAMA268, 2420–2425. 10.1001/jama.1992.03490170092032
21
HaggerM. S.KochS.ChatzisarantisN. L. D.OrbellS. (2017). The common-sense model of self-regulation: meta-analysis and test of a process model. Psychol. Bull.143, 1117–1154. 10.1037/bul0000118
22
HigginsJ. P. T.AltmanD. G. (2008). Assessing risk of bias in included studies, in Cochrane Handbook for Systematic Reviews of Interventions, eds HigginsJ. P. T.GreenS. (Chichester: Wiley), 187–241.
23
HigginsJ. P. T.GreenS. (2006). Cochrane Handbook for Systematic Reviews of Interventions 4.2.6. Chichester: John Wiley & Sons, Ltd.
24
HoffmannC.AbrahamC.WhiteM. P.BallS.SkipponS. M. (2017). What cognitive mechanisms predict travel mode choice? A systematic review with meta-analysis. Transport Rev.37, 631–652. 10.1080/01441647.2017.1285819
25
HusebøA. M. L.DyrstadS. M.SøreideJ. A.BruE. (2013). Predicting exercise adherence in cancer patients and survivors: a systematic review and meta-analysis of motivational and behavioural factors. J. Clin. Nurs.22, 4–21. 10.1111/j.1365-2702.2012.04322.x
26
JackK.McLeanS. M.MoffettJ. K.GardinerE. (2010). Barriers to treatment adherence in physiotherapy outpatient clinics: a systematic review. Manual Ther.15, 220–228. 10.1016/j.math.2009.12.004
27
JacksonN.WatersE. (2005). Criteria for the systematic review of health promotion and public health interventions. Health Promot. Int.20, 367–374. 10.1093/heapro/dai022
28
JadadA. R.MooreR. A.CarrollD.JenkinsonC.ReynoldsD. J. M.GavaghanD. J.et al. (1996). Assessing the quality of reports of randomized clinical trials: is blinding necessary?Control. Clin. Trials17, 1–12. 10.1016/01972456(95)00134-4
29
JardeA.LosillaJ. M.VivesJ.RodrigoM. F. (2013). Q-Coh: a tool to screen the methodological quality of cohort studies in systematic reviews and meta-analyses. Int. J. Clin. Health Psychol.13, 138–146. 10.1016/S1697-2600(13)70017-6
30
JarlaisD. C. D.LylesC.CrepazN.GroupT. T. (2004). Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: the TREND statement. Am. J. Public Health94, 361–366. 10.2105/ajph.94.3.361
31
Joanna Briggs Institute (2014). Joanna Briggs Institute Reviewers' Manual: 2014 Edition. The Joanna Briggs Institute.
32
JohnsonB. T.LowR. E.MacdonaldH. V. (2014). Panning for the gold in health research: Incorporating studies' methodological quality in meta-analysis. Psychol. Health30, 135–152. 10.1080/08870446.2014.953533
33
JonesJ.HunterD. (1995). Qualitative research: consensus methods for medical and health services research. BMJ311, 376–380. 10.1136/bmj.311.7001.376
34
JüniP.WitschiA.BlochR.EggerM. (1999). The hazards of scoring the quality of clinical trials for meta-analysis. JAMA282, 1054–1060. 10.1001/jama.282.11.1054
35
KatrakP.BialocerkowskiA. E.Massy-WestroppN.KumarV. S.GrimmerK. A. (2004). A systematic review of the content of critical appraisal tools. BMC Med. Res. Methodol.4:22. 10.1186/1471-2288-4-22
36
KerrN. L. (1998). HARKing: hypothesizing after the results are known. Personal. Soc. Psychol. Rev.2, 196–217. 10.1207/s15327957pspr0203_4
37
KhanK.KunzR.KleijnenJ.AntesG. (2011). Systematic Reviews to Support Evidence-Based Medicine.London: Hodder Arnold.
38
LongA. F.GodfreyM. (2004). An evaluation tool to assess the quality of qualitative research studies. Int. J. Soc. Res. Methodol.7, 181–196. 10.1080/1364557032000045302
39
MichieS.JohnstonM.AbrahamC.LawtonR.ParkerD.WalkerA. (2005). Making psychological theory useful for implementing evidence based practice: a consensus approach. Qual. Safety Health Care14, 26–33. 10.1136/qshc.2004.011155
40
MoherD.HopewellS.SchulzK. F.MontoriV.GøtzscheP. C.DevereauxP. J.et al. (2010). CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 340:c869. 10.1136/bmj.c869
41
MoherD.LiberatiA.TetzlaffJ.AltmanD. G.The Prisma Group (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med.6:e1000097. 10.1371/journal.pmed.1000097
42
MoherD.PhamB.JonesA.CookD. J.JadadA. R.MoherM.et al. (1998). Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?Lancet352, 609–613. 10.1016/S0140-6736(98)01085-X
43
MoyerA.FinneyJ. W. (2005). Rating methodological quality: toward improved assessment and investigation. Account. Res.12, 299–313. 10.1080/08989620500440287
44
MurrayJ.FarringtonD. P.EisnerM. P. (2009). Drawing conclusions about causes from systematic reviews of risk factors: The Cambridge Quality Checklists. J. Exp. Criminol.5, 1–23. 10.1007/s11292-008-9066-0
45
National Heart Lung, and Blood Institute (NHLBI). (2014). Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies. Bethesda, MD: National Institutes of Health.
46
National Institutes of Health (2014). Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies. Available online at: https://www.nhlbi.nih.gov/health-pro/guidelines/in-develop/cardiovascular-risk-reduction/tools/cohort (Accessed November 14, 2016).
47
NoarS. M.ZimmermanR. S. (2005). Health Behavior Theory and cumulative knowledge regarding health behaviors: are we moving in the right direction?Health Educ. Res.20, 275–290. 10.1093/her/cyg113
48
OlukaO. C.NieS.SunY. (2014). Quality assessment of TPB-based questionnaires: a systematic review. PLoS ONE 9:e94419. 10.1371/journal.pone.0094419
49
Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science349:aac4716. 10.1126/science.aac4716
50
OxmanA. D.GuyattG. H. (1991). Validation of an index of the quality of review articles. J. Clin. Epidemiol.44, 1271–1278. 10.1016/0895-4356(91)90160-B
51
PaceR.PluyeP.BartlettG.MacaulayA. C.SalsbergJ.JagoshJ.et al. (2012). Testing the reliability and efficiency of the pilot mixed methods appraisal tool (MMAT) for systematic mixed studies review. Int. J. Nurs. Stud.49, 47–53. 10.1016/j.ijnurstu.2011.07.002
52
PantelicM.ShenderovichY.CluverL.BoyesM. (2015). Predictors of internalised HIV-related stigma: a systematic review of studies in sub-Saharan Africa. Health Psychol. Rev.9, 469–490. 10.1080/17437199.2014.996243
53
PlotnikoffR. C.CostiganS. A.KarunamuniN.LubansD. R. (2013). Social cognitive theories used to explain physical activity behavior in adolescents: a systematic review and meta-analysis. Prevent. Med.56, 245–253. 10.1016/j.ypmed.2013.01.013
54
PontoJ. (2015). Understanding and evaluating survey research. J. Adv. Pract. Oncol.6, 168–171.
55
ProtogerouC.JohnsonB. T.HaggerM. S. (2018). An integrated model of condom use in sub-Saharan African youth: a meta-analysis. Health Psychol.37, 586–602. 10.1037/hea0000604
56
QuonE. C.McgrathJ. J. (2014). Subjective socioeconomic status and adolescent health: a meta-analysis. Health Psychol.33, 433–447. 10.1037/a0033716
57
RhodesR. E.FialaB.ConnerM. (2009). A review and meta-analysis of affective judgments and physical activity in adult populations. Ann. Behav. Med.38, 180–204. 10.1007/s12160-009-9147-y
58
SantosI.SniehottaF. F.MarquesM. M.CarraçaE. V.TeixeiraP. J. (2017). Prevalence of personal weight control attempts in adults: a systematic review and meta-analysis. Obesity Rev.18, 32–50. 10.1111/obr.12466
59
SimmonsJ. P.NelsonL. D.SimonsohnU. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci.22, 1359–1366. 10.1177/0956797611417632
60
Von ElmE.AltmanD. G.EggerM.PocockS. J.GøtzscheP. C.VandenbrouckeJ. P. (2007). The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Prevent. Med.45, 247–251. 10.1016/j.ypmed.2007.08.012
61
WaggonerJ.CarlineJ. D.DurningS. J. (2016). Is there a consensus on consensus methodology? Descriptions and recommendations for future consensus research. Acad. Med.91, 663–668. 10.1097/acm.0000000000001092
62
WongW. C.CheungC. S.HartG. J. (2008). Development of a quality assessment tool for systematic reviews of observational studies (QATSO) of HIV prevalence in men having sex with men and associated risk behaviours. Emerg. Themes Epidemiol.5, 1–23. 10.1186/1742-7622-5-23
63
YoungM. D.PlotnikoffR. C.CollinsC. E.CallisterR.MorganP. J. (2014). Social cognitive theory and physical activity: a systematic review and meta-analysis. Obesity Rev.15, 983–995. 10.1111/obr.12225
Summary
Keywords
study quality appraisal, psychology, correlational studies, survey studies, evidence syntheses, transparency
Citation
Protogerou C and Hagger MS (2019) A Case For a Study Quality Appraisal in Survey Studies in Psychology. Front. Psychol. 9:2788. doi: 10.3389/fpsyg.2018.02788
Received
26 October 2018
Accepted
31 December 2018
Published
24 January 2019
Volume
9 - 2018
Edited by
Edgar Erdfelder, Universität Mannheim, Germany
Reviewed by
Timo Gnambs, Leibniz Institute for Educational Trajectories (LG), Germany
Updates
Copyright
© 2019 Protogerou and Hagger.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Cleo Protogerou cleo.protogerou@uct.ac.za
This article was submitted to Quantitative Psychology and Measurement, a section of the journal Frontiers in Psychology
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.