The Use of Research Methods in Psychological Research: A Systematised Review

Abstract

Research methods play an imperative role in research quality as well as educating young researchers, however, the application thereof is unclear which can be detrimental to the field of psychology. Therefore, this systematised review aimed to determine what research methods are being used, how these methods are being used and for what topics in the field. Our review of 999 articles from five journals over a period of 5 years indicated that psychology research is conducted in 10 topics via predominantly quantitative research methods. Of these 10 topics, social psychology was the most popular. The remainder of the conducted methodology is described. It was also found that articles lacked rigour and transparency in the used methodology which has implications for replicability. In conclusion this article, provides an overview of all reported methodologies used in a sample of psychology journals. It highlights the popularity and application of methods and designs throughout the article sample as well as an unexpected lack of rigour with regard to most aspects of methodology. Possible sample bias should be considered when interpreting the results of this study. It is recommended that future research should utilise the results of this study to determine the possible impact on the field of psychology as a science and to further investigation into the use of research methods. Results should prompt the following future research into: a lack or rigour and its implication on replication, the use of certain methods above others, publication bias and choice of sampling method.

Introduction

Psychology is an ever-growing and popular field (Gough and Lyons, 2016; Clay, 2017). Due to this growth and the need for science-based research to base health decisions on (Perestelo-Pérez, 2013), the use of research methods in the broad field of psychology is an essential point of investigation (Stangor, 2011; Aanstoos, 2014). Research methods are therefore viewed as important tools used by researchers to collect data (Nieuwenhuis, 2016) and include the following: quantitative, qualitative, mixed method and multi method (Maree, 2016). Additionally, researchers also employ various types of literature reviews to address research questions (Grant and Booth, 2009). According to literature, what research method is used and why a certain research method is used is complex as it depends on various factors that may include paradigm (O'Neil and Koekemoer, 2016), research question (Grix, 2002), or the skill and exposure of the researcher (Nind et al., 2015). How these research methods are employed is also difficult to discern as research methods are often depicted as having fixed boundaries that are continuously crossed in research (Johnson et al., 2001; Sandelowski, 2011). Examples of this crossing include adding quantitative aspects to qualitative studies (Sandelowski et al., 2009), or stating that a study used a mixed-method design without the study having any characteristics of this design (Truscott et al., 2010).

The inappropriate use of research methods affects how students and researchers improve and utilise their research skills (Scott Jones and Goldring, 2015), how theories are developed (Ngulube, 2013), and the credibility of research results (Levitt et al., 2017). This, in turn, can be detrimental to the field (Nind et al., 2015), journal publication (Ketchen et al., 2008; Ezeh et al., 2010), and attempts to address public social issues through psychological research (Dweck, 2017). This is especially important given the now well-known replication crisis the field is facing (Earp and Trafimow, 2015; Hengartner, 2018).

Due to this lack of clarity on method use and the potential impact of inept use of research methods, the aim of this study was to explore the use of research methods in the field of psychology through a review of journal publications. Chaichanasakul et al. (2011) identify reviewing articles as the opportunity to examine the development, growth and progress of a research area and overall quality of a journal. Studies such as Lee et al. (1999) as well as Bluhm et al. (2011) review of qualitative methods has attempted to synthesis the use of research methods and indicated the growth of qualitative research in American and European journals. Research has also focused on the use of research methods in specific sub-disciplines of psychology, for example, in the field of Industrial and Organisational psychology Coetzee and Van Zyl (2014) found that South African publications tend to consist of cross-sectional quantitative research methods with underrepresented longitudinal studies. Qualitative studies were found to make up 21% of the articles published from 1995 to 2015 in a similar study by O'Neil and Koekemoer (2016). Other methods in health psychology, such as Mixed methods research have also been reportedly growing in popularity (O'Cathain, 2009).

A broad overview of the use of research methods in the field of psychology as a whole is however, not available in the literature. Therefore, our research focused on answering what research methods are being used, how these methods are being used and for what topics in practice (i.e., journal publications) in order to provide a general perspective of method used in psychology publication. We synthesised the collected data into the following format: research topic [areas of scientific discourse in a field or the current needs of a population (Bittermann and Fischer, 2018)], method [data-gathering tools (Nieuwenhuis, 2016)], sampling [elements chosen from a population to partake in research (Ritchie et al., 2009)], data collection [techniques and research strategy (Maree, 2016)], and data analysis [discovering information by examining bodies of data (Ktepi, 2016)]. A systematised review of recent articles (2013 to 2017) collected from five different journals in the field of psychological research was conducted.

Methods

Design

Grant and Booth (2009) describe systematised reviews as the review of choice for post-graduate studies, which is employed using some elements of a systematic review and seldom more than one or two databases to catalogue studies after a comprehensive literature search. The aspects used in this systematised review that are similar to that of a systematic review were a full search within the chosen database and data produced in tabular form (Grant and Booth, 2009).

Sampling

Sample sizes and timelines vary in systematised reviews (see Lowe and Moore, 2014; Pericall and Taylor, 2014; Barr-Walker, 2017). With no clear parameters identified in the literature (see Grant and Booth, 2009), the sample size of this study was determined by the purpose of the sample (Strydom, 2011), and time and cost constraints (Maree and Pietersen, 2016). Thus, a non-probability purposive sample (Ritchie et al., 2009) of the top five psychology journals from 2013 to 2017 was included in this research study. Per Lee (2015) American Psychological Association (APA) recommends the use of the most up-to-date sources for data collection with consideration of the context of the research study. As this research study focused on the most recent trends in research methods used in the broad field of psychology, the identified time frame was deemed appropriate.

Psychology journals were only included if they formed part of the top five English journals in the miscellaneous psychology domain of the Scimago Journal and Country Rank (Scimago Journal & Country Rank, 2017). The Scimago Journal and Country Rank provides a yearly updated list of publicly accessible journal and country-specific indicators derived from the Scopus® database (Scopus, 2017b) by means of the Scimago Journal Rank (SJR) indicator developed by Scimago from the algorithm Google PageRank™ (Scimago Journal & Country Rank, 2017). Scopus is the largest global database of abstracts and citations from peer-reviewed journals (Scopus, 2017a). Reasons for the development of the Scimago Journal and Country Rank list was to allow researchers to assess scientific domains, compare country rankings, and compare and analyse journals (Scimago Journal & Country Rank, 2017), which supported the aim of this research study. Additionally, the goals of the journals had to focus on topics in psychology in general with no preference to specific research methods and have full-text access to articles.

The following list of top five journals in 2018 fell within the abovementioned inclusion criteria (1) Australian Journal of Psychology, (2) British Journal of Psychology, (3) Europe's Journal of Psychology, (4) International Journal of Psychology and lastly the (5) Journal of Psychology Applied and Interdisciplinary.

Journals were excluded from this systematised review if no full-text versions of their articles were available, if journals explicitly stated a publication preference for certain research methods, or if the journal only published articles in a specific discipline of psychological research (for example, industrial psychology, clinical psychology etc.).

Procedure

The researchers followed a procedure (see Figure 1) adapted from that of Ferreira et al. (2016) for systematised reviews. Data collection and categorisation commenced on 4 December 2017 and continued until 30 June 2019. All the data was systematically collected and coded manually (Grant and Booth, 2009) with an independent person acting as co-coder. Codes of interest included the research topic, method used, the design used, sampling method, and methodology (the method used for data collection and data analysis). These codes were derived from the wording in each article. Themes were created based on the derived codes and checked by the co-coder. Lastly, these themes were catalogued into a table as per the systematised review design.

Figure 1

Rigour

According to Johnston et al. (2019), “literature screening, selection, and data extraction/analyses” (p. 7) are specifically tailored to the aim of a review. Therefore, the steps followed in a systematic review must be reported in a comprehensive and transparent manner. The chosen systematised design adhered to the rigour expected from systematic reviews with regard to full search and data produced in tabular form (Grant and Booth, 2009). The rigorous application of the systematic review is, therefore discussed in relation to these two elements.

Firstly, to ensure a comprehensive search, this research study promoted review transparency by following a clear protocol outlined according to each review stage before collecting data (Johnston et al., 2019). This protocol was similar to that of Ferreira et al. (2016) and approved by three research committees/stakeholders and the researchers (Johnston et al., 2019). The eligibility criteria for article inclusion was based on the research question and clearly stated, and the process of inclusion was recorded on an electronic spreadsheet to create an evidence trail (Bandara et al., 2015; Johnston et al., 2019). Microsoft Excel spreadsheets are a popular tool for review studies and can increase the rigour of the review process (Bandara et al., 2015). Screening for appropriate articles for inclusion forms an integral part of a systematic review process (Johnston et al., 2019). This step was applied to two aspects of this research study: the choice of eligible journals and articles to be included. Suitable journals were selected by the first author and reviewed by the second and third authors. Initially, all articles from the chosen journals were included. Then, by process of elimination, those irrelevant to the research aim, i.e., interview articles or discussions etc., were excluded.

To ensure rigourous data extraction, data was first extracted by one reviewer, and an independent person verified the results for completeness and accuracy (Johnston et al., 2019). The research question served as a guide for efficient, organised data extraction (Johnston et al., 2019). Data was categorised according to the codes of interest, along with article identifiers for audit trails such as authors, title and aims of articles. The categorised data was based on the aim of the review (Johnston et al., 2019) and synthesised in tabular form under methods used, how these methods were used, and for what topics in the field of psychology.

Results

The initial search produced a total of 1,145 articles from the 5 journals identified. Inclusion and exclusion criteria resulted in a final sample of 999 articles (Figure 2). Articles were co-coded into 84 codes, from which 10 themes were derived (Table 1).

Figure 2

These 10 themes represent the topic section of our research question (Figure 3). All these topics except, for the final one, psychological practice, were found to concur with the research areas in psychology as identified by Weiten (2010). These research areas were chosen to represent the derived codes as they provided broad definitions that allowed for clear, concise categorisation of the vast amount of data. Article codes were categorised under particular themes/topics if they adhered to the research area definitions created by Weiten (2010). It is important to note that these areas of research do not refer to specific disciplines in psychology, such as industrial psychology; but to broader fields that may encompass sub-interests of these disciplines.

Figure 3

In the case of developmental psychology, researchers conduct research into human development from childhood to old age. Social psychology includes research on behaviour governed by social drivers. Researchers in the field of educational psychology study how people learn and the best way to teach them. Health psychology aims to determine the effect of psychological factors on physiological health. Physiological psychology, on the other hand, looks at the influence of physiological aspects on behaviour. Experimental psychology is not the only theme that uses experimental research and focuses on the traditional core topics of psychology (for example, sensation). Cognitive psychology studies the higher mental processes. Psychometrics is concerned with measuring capacity or behaviour. Personality research aims to assess and describe consistency in human behaviour (Weiten, 2010). The final theme of psychological practice refers to the experiences, techniques, and interventions employed by practitioners, researchers, and academia in the field of psychology.

Articles under these themes were further subdivided into methodologies: method, sampling, design, data collection, and data analysis. The categorisation was based on information stated in the articles and not inferred by the researchers. Data were compiled into two sets of results presented in this article. The first set addresses the aim of this study from the perspective of the topics identified. The second set of results represents a broad overview of the results from the perspective of the methodology employed. The second set of results are discussed in this article, while the first set is presented in table format. The discussion thus provides a broad overview of methods use in psychology (across all themes), while the table format provides readers with in-depth insight into methods used in the individual themes identified. We believe that presenting the data from both perspectives allow readers a broad understanding of the results. Due a large amount of information that made up our results, we followed Cichocka and Jost (2014) in simplifying our results. Please note that the numbers indicated in the table in terms of methodology differ from the total number of articles. Some articles employed more than one method/sampling technique/design/data collection method/data analysis in their studies.

What follows is the results for what methods are used, how these methods are used, and which topics in psychology they are applied to. Percentages are reported to the second decimal in order to highlight small differences in the occurrence of methodology.

Firstly, with regard to the research methods used, our results show that researchers are more likely to use quantitative research methods (90.22%) compared to all other research methods. Qualitative research was the second most common research method but only made up about 4.79% of the general method usage. Reviews occurred almost as much as qualitative studies (3.91%), as the third most popular method. Mixed-methods research studies (0.98%) occurred across most themes, whereas multi-method research was indicated in only one study and amounted to 0.10% of the methods identified. The specific use of each method in the topics identified is shown in Table 2 and Figure 4.

Figure 4

Secondly, in the case of how these research methods are employed, our study indicated the following.

Sampling−78.34% of the studies in the collected articles did not specify a sampling method. From the remainder of the studies, 13 types of sampling methods were identified. These sampling methods included broad categorisation of a sample as, for example, a probability or non-probability sample. General samples of convenience were the methods most likely to be applied (10.34%), followed by random sampling (3.51%), snowball sampling (2.73%), and purposive (1.37%) and cluster sampling (1.27%). The remainder of the sampling methods occurred to a more limited extent (0–1.0%). See Table 3 and Figure 5 for sampling methods employed in each topic.

Figure 5

Designs were categorised based on the articles' statement thereof. Therefore, it is important to note that, in the case of quantitative studies, non-experimental designs (25.55%) were often indicated due to a lack of experiments and any other indication of design, which, according to Laher (2016), is a reasonable categorisation. Non-experimental designs should thus be compared with experimental designs only in the description of data, as it could include the use of correlational/cross-sectional designs, which were not overtly stated by the authors. For the remainder of the research methods, “not stated” (7.12%) was assigned to articles without design types indicated.

From the 36 identified designs the most popular designs were cross-sectional (23.17%) and experimental (25.64%), which concurred with the high number of quantitative studies. Longitudinal studies (3.80%), the third most popular design, was used in both quantitative and qualitative studies. Qualitative designs consisted of ethnography (0.38%), interpretative phenomenological designs/phenomenology (0.28%), as well as narrative designs (0.28%). Studies that employed the review method were mostly categorised as “not stated,” with the most often stated review designs being systematic reviews (0.57%). The few mixed method studies employed exploratory, explanatory (0.09%), and concurrent designs (0.19%), with some studies referring to separate designs for the qualitative and quantitative methods. The one study that identified itself as a multi-method study used a longitudinal design. Please see how these designs were employed in each specific topic in Table 4, Figure 6.

Figure 6

Data collection and analysis—data collection included 30 methods, with the data collection method most often employed being questionnaires (57.84%). The experimental task (16.56%) was the second most preferred collection method, which included established or unique tasks designed by the researchers. Cognitive ability tests (6.84%) were also regularly used along with various forms of interviewing (7.66%). Table 5 and Figure 7 represent data collection use in the various topics. Data analysis consisted of 3,857 occurrences of data analysis categorised into ±188 various data analysis techniques shown in Table 6 and Figures 1–7. Descriptive statistics were the most commonly used (23.49%) along with correlational analysis (17.19%). When using a qualitative method, researchers generally employed thematic analysis (0.52%) or different forms of analysis that led to coding and the creation of themes. Review studies presented few data analysis methods, with most studies categorising their results. Mixed method and multi-method studies followed the analysis methods identified for the qualitative and quantitative studies included.

Figure 7

Results of the topics researched in psychology can be seen in the tables, as previously stated in this article. It is noteworthy that, of the 10 topics, social psychology accounted for 43.54% of the studies, with cognitive psychology the second most popular research topic at 16.92%. The remainder of the topics only occurred in 4.0–7.0% of the articles considered. A list of the included 999 articles is available under the section “View Articles” on the following website: https://methodgarden.xtrapolate.io/. This website was created by Scholtz et al. (2019) to visually present a research framework based on this Article's results.

Discussion

This systematised review categorised full-length articles from five international journals across the span of 5 years to provide insight into the use of research methods in the field of psychology. Results indicated what methods are used how these methods are being used and for what topics (why) in the included sample of articles. The results should be seen as providing insight into method use and by no means a comprehensive representation of the aforementioned aim due to the limited sample. To our knowledge, this is the first research study to address this topic in this manner. Our discussion attempts to promote a productive way forward in terms of the key results for method use in psychology, especially in the field of academia (Holloway, 2008).

With regard to the methods used, our data stayed true to literature, finding only common research methods (Grant and Booth, 2009; Maree, 2016) that varied in the degree to which they were employed. Quantitative research was found to be the most popular method, as indicated by literature (Breen and Darlaston-Jones, 2010; Counsell and Harlow, 2017) and previous studies in specific areas of psychology (see Coetzee and Van Zyl, 2014). Its long history as the first research method (Leech et al., 2007) in the field of psychology as well as researchers' current application of mathematical approaches in their studies (Toomela, 2010) might contribute to its popularity today. Whatever the case may be, our results show that, despite the growth in qualitative research (Demuth, 2015; Smith and McGannon, 2018), quantitative research remains the first choice for article publication in these journals. Despite the included journals indicating openness to articles that apply any research methods. This finding may be due to qualitative research still being seen as a new method (Burman and Whelan, 2011) or reviewers' standards being higher for qualitative studies (Bluhm et al., 2011). Future research is encouraged into the possible biasness in publication of research methods, additionally further investigation with a different sample into the proclaimed growth of qualitative research may also provide different results.

Review studies were found to surpass that of multi-method and mixed method studies. To this effect Grant and Booth (2009), state that the increased awareness, journal contribution calls as well as its efficiency in procuring research funds all promote the popularity of reviews. The low frequency of mixed method studies contradicts the view in literature that it's the third most utilised research method (Tashakkori and Teddlie's, 2003). Its' low occurrence in this sample could be due to opposing views on mixing methods (Gunasekare, 2015) or that authors prefer publishing in mixed method journals, when using this method, or its relative novelty (Ivankova et al., 2016). Despite its low occurrence, the application of the mixed methods design in articles was methodologically clear in all cases which were not the case for the remainder of research methods.

Additionally, a substantial number of studies used a combination of methodologies that are not mixed or multi-method studies. Perceived fixed boundaries are according to literature often set aside, as confirmed by this result, in order to investigate the aim of a study, which could create a new and helpful way of understanding the world (Gunasekare, 2015). According to Toomela (2010), this is not unheard of and could be considered a form of “structural systemic science,” as in the case of qualitative methodology (observation) applied in quantitative studies (experimental design) for example. Based on this result, further research into this phenomenon as well as its implications for research methods such as multi and mixed methods is recommended.

Discerning how these research methods were applied, presented some difficulty. In the case of sampling, most studies—regardless of method—did mention some form of inclusion and exclusion criteria, but no definite sampling method. This result, along with the fact that samples often consisted of students from the researchers' own academic institutions, can contribute to literature and debates among academics (Peterson and Merunka, 2014; Laher, 2016). Samples of convenience and students as participants especially raise questions about the generalisability and applicability of results (Peterson and Merunka, 2014). This is because attention to sampling is important as inappropriate sampling can debilitate the legitimacy of interpretations (Onwuegbuzie and Collins, 2017). Future investigation into the possible implications of this reported popular use of convenience samples for the field of psychology as well as the reason for this use could provide interesting insight, and is encouraged by this study.

Additionally, and this is indicated in Table 6, articles seldom report the research designs used, which highlights the pressing aspect of the lack of rigour in the included sample. Rigour with regards to the applied empirical method is imperative in promoting psychology as a science (American Psychological Association, 2020). Omitting parts of the research process in publication when it could have been used to inform others' research skills should be questioned, and the influence on the process of replicating results should be considered. Publications are often rejected due to a lack of rigour in the applied method and designs (Fonseca, 2013; Laher, 2016), calling for increased clarity and knowledge of method application. Replication is a critical part of any field of scientific research and requires the “complete articulation” of the study methods used (Drotar, 2010, p. 804). The lack of thorough description could be explained by the requirements of certain journals to only report on certain aspects of a research process, especially with regard to the applied design (Laher, 20). However, naming aspects such as sampling and designs, is a requirement according to the APA's Journal Article Reporting Standards (JARS-Quant) (Appelbaum et al., 2018). With very little information on how a study was conducted, authors lose a valuable opportunity to enhance research validity, enrich the knowledge of others, and contribute to the growth of psychology and methodology as a whole. In the case of this research study, it also restricted our results to only reported samples and designs, which indicated a preference for certain designs, such as cross-sectional designs for quantitative studies.

Data collection and analysis were for the most part clearly stated. A key result was the versatile use of questionnaires. Researchers would apply a questionnaire in various ways, for example in questionnaire interviews, online surveys, and written questionnaires across most research methods. This may highlight a trend for future research.

With regard to the topics these methods were employed for, our research study found a new field named “psychological practice.” This result may show the growing consciousness of researchers as part of the research process (Denzin and Lincoln, 2003), psychological practice, and knowledge generation. The most popular of these topics was social psychology, which is generously covered in journals and by learning societies, as testaments of the institutional support and richness social psychology has in the field of psychology (Chryssochoou, 2015). The APA's perspective on 2018 trends in psychology also identifies an increased amount of psychology focus on how social determinants are influencing people's health (Deangelis, 2017).

This study was not without limitations and the following should be taken into account. Firstly, this study used a sample of five specific journals to address the aim of the research study, despite general journal aims (as stated on journal websites), this inclusion signified a bias towards the research methods published in these specific journals only and limited generalisability. A broader sample of journals over a different period of time, or a single journal over a longer period of time might provide different results. A second limitation is the use of Excel spreadsheets and an electronic system to log articles, which was a manual process and therefore left room for error (Bandara et al., 2015). To address this potential issue, co-coding was performed to reduce error. Lastly, this article categorised data based on the information presented in the article sample; there was no interpretation of what methodology could have been applied or whether the methods stated adhered to the criteria for the methods used. Thus, a large number of articles that did not clearly indicate a research method or design could influence the results of this review. However, this in itself was also a noteworthy result. Future research could review research methods of a broader sample of journals with an interpretive review tool that increases rigour. Additionally, the authors also encourage the future use of systematised review designs as a way to promote a concise procedure in applying this design.

Conclusion

Our research study presented the use of research methods for published articles in the field of psychology as well as recommendations for future research based on these results. Insight into the complex questions identified in literature, regarding what methods are used how these methods are being used and for what topics (why) was gained. This sample preferred quantitative methods, used convenience sampling and presented a lack of rigorous accounts for the remaining methodologies. All methodologies that were clearly indicated in the sample were tabulated to allow researchers insight into the general use of methods and not only the most frequently used methods. The lack of rigorous account of research methods in articles was represented in-depth for each step in the research process and can be of vital importance to address the current replication crisis within the field of psychology. Recommendations for future research aimed to motivate research into the practical implications of the results for psychology, for example, publication bias and the use of convenience samples.

References

• 1

  AanstoosC. M. (2014). Psychology. Available online at: http://eds.a.ebscohost.com.nwulib.nwu.ac.za/eds/detail/detail?sid=18de6c5c-2b03-4eac-94890145eb01bc70%40sessionmgr4006&vid$=$1&hid$=$4113&bdata$=$JnNpdGU9ZWRzL~WxpdmU%3d#AN$=$93871882&db$=$ers

  • Google Scholar
  • View reference in article

• 2

  American Psychological Association (2020). Science of Psychology. Available online at: https://www.apa.org/action/science/

  • Google Scholar
  • View reference in article

• 3

  AppelbaumM.CooperH.KlineR. B.Mayo-WilsonE.NezuA. M.RaoS. M. (2018). Journal article reporting standards for quantitative research in psychology: the APA Publications and Communications Board task force report. Am. Psychol.73:3. 10.1037/amp0000191

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 4

  BandaraW.FurtmuellerE.GorbachevaE.MiskonS.BeekhuyzenJ. (2015). Achieving rigor in literature reviews: insights from qualitative data analysis and tool-support. Commun. Ass. Inform. Syst.37, 154–204. 10.17705/1CAIS.03708

  • CrossRef
  • Google Scholar
  • View reference in article

• 5

  Barr-WalkerJ. (2017). Evidence-based information needs of public health workers: a systematized review. J. Med. Libr. Assoc.105, 69–79. 10.5195/JMLA.2017.109

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 6

  BittermannA.FischerA. (2018). How to identify hot topics in psychology using topic modeling. Z. Psychol.226, 3–13. 10.1027/2151-2604/a000318

  • CrossRef
  • Google Scholar
  • View reference in article

• 7

  BluhmD. J.HarmanW.LeeT. W.MitchellT. R. (2011). Qualitative research in management: a decade of progress. J. Manage. Stud.48, 1866–1891. 10.1111/j.1467-6486.2010.00972.x

  • CrossRef
  • Google Scholar
  • View reference in article

• 8

  BreenL. J.Darlaston-JonesD. (2010). Moving beyond the enduring dominance of positivism in psychological research: implications for psychology in Australia. Aust. Psychol.45, 67–76. 10.1080/00050060903127481

  • CrossRef
  • Google Scholar

• 9

  BurmanE.WhelanP. (2011). Problems in / of Qualitative Research. Maidenhead: Open University Press/McGraw Hill.

  • Google Scholar
  • View reference in article

• 10

  ChaichanasakulA.HeY.ChenH.AllenG. E. K.KhairallahT. S.RamosK. (2011). Journal of Career Development: a 36-year content analysis (1972–2007). J. Career. Dev.38, 440–455. 10.1177/0894845310380223

  • CrossRef
  • Google Scholar
  • View reference in article

• 11

  ChryssochoouX. (2015). Social Psychology. Inter. Encycl. Soc. Behav. Sci.22, 532–537. 10.1016/B978-0-08-097086-8.24095-6

  • CrossRef
  • Google Scholar
  • View reference in article

• 12

  CichockaA.JostJ. T. (2014). Stripped of illusions? Exploring system justification processes in capitalist and post-Communist societies. Inter. J. Psychol.49, 6–29. 10.1002/ijop.12011

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 13

  ClayR. A. (2017). Psychology is More Popular Than Ever. Monitor on Psychology: Trends Report. Available online at: https://www.apa.org/monitor/2017/11/trends-popular

  • Google Scholar
  • View reference in article

• 14

  CoetzeeM.Van ZylL. E. (2014). A review of a decade's scholarly publications (2004–2013) in the South African Journal of Industrial Psychology. SA. J. Psychol. 40, 1–16. 10.4102/sajip.v40i1.1227

  • CrossRef
  • Google Scholar
  • View reference in article

• 15

  CounsellA.HarlowL. (2017). Reporting practices and use of quantitative methods in Canadian journal articles in psychology. Can. Psychol.58, 140–147. 10.1037/cap0000074

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 16

  DeangelisT. (2017). Targeting Social Factors That Undermine Health. Monitor on Psychology: Trends Report. Available online at: https://www.apa.org/monitor/2017/11/trend-social-factors

  • Google Scholar
  • View reference in article

• 17

  DemuthC. (2015). New directions in qualitative research in psychology. Integr. Psychol. Behav. Sci.49, 125–133. 10.1007/s12124-015-9303-9

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 18

  DenzinN. K.LincolnY. (2003). The Landscape of Qualitative Research: Theories and Issues, 2nd Edn. London: Sage.

  • Google Scholar
  • View reference in article

• 19

  DrotarD. (2010). A call for replications of research in pediatric psychology and guidance for authors. J. Pediatr. Psychol.35, 801–805. 10.1093/jpepsy/jsq049

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 20

  DweckC. S. (2017). Is psychology headed in the right direction? Yes, no, and maybe. Perspect. Psychol. Sci.12, 656–659. 10.1177/1745691616687747

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 21

  EarpB. D.TrafimowD. (2015). Replication, falsification, and the crisis of confidence in social psychology. Front. Psychol.6:621. 10.3389/fpsyg.2015.00621

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 22

  EzehA. C.IzugbaraC. O.KabiruC. W.FonnS.KahnK.MandersonL.et al. (2010). Building capacity for public and population health research in Africa: the consortium for advanced research training in Africa (CARTA) model. Glob. Health Action3:5693. 10.3402/gha.v3i0.5693

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 23

  FerreiraA. L. L.BessaM. M. M.DrezettJ.De AbreuL. C. (2016). Quality of life of the woman carrier of endometriosis: systematized review. Reprod. Clim.31, 48–54. 10.1016/j.recli.2015.12.002

  • CrossRef
  • Google Scholar
  • View reference in article

• 24

  FonsecaM. (2013). Most Common Reasons for Journal Rejections. Available online at: http://www.editage.com/insights/most-common-reasons-for-journal-rejections

  • Google Scholar
  • View reference in article

• 25

  GoughB.LyonsA. (2016). The future of qualitative research in psychology: accentuating the positive. Integr. Psychol. Behav. Sci.50, 234–243. 10.1007/s12124-015-9320-8

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 26

  GrantM. J.BoothA. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info. Libr. J.26, 91–108. 10.1111/j.1471-1842.2009.00848.x

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 27

  GrixJ. (2002). Introducing students to the generic terminology of social research. Politics22, 175–186. 10.1111/1467-9256.00173

  • CrossRef
  • Google Scholar
  • View reference in article

• 28

  GunasekareU. L. T. P. (2015). Mixed research method as the third research paradigm: a literature review. Int. J. Sci. Res.4, 361–368. Available online at: https://ssrn.com/abstract=2735996

  • Google Scholar
  • View reference in article

• 29

  HengartnerM. P. (2018). Raising awareness for the replication crisis in clinical psychology by focusing on inconsistencies in psychotherapy Research: how much can we rely on published findings from efficacy trials?Front. Psychol.9:256. 10.3389/fpsyg.2018.00256

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 30

  HollowayW. (2008). Doing intellectual disagreement differently. Psychoanal. Cult. Soc.13, 385–396. 10.1057/pcs.2008.29

  • CrossRef
  • Google Scholar
  • View reference in article

• 31

  IvankovaN. V.CreswellJ. W.Plano ClarkV. L. (2016). Foundations and Approaches to mixed methods research, in First Steps in Research, 2nd Edn. K. Maree (Pretoria: Van Schaick Publishers), 306–335.

  • Google Scholar
  • View reference in article

• 32

  JohnsonM.LongT.WhiteA. (2001). Arguments for British pluralism in qualitative health research. J. Adv. Nurs.33, 243–249. 10.1046/j.1365-2648.2001.01659.x

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 33

  JohnstonA.KellyS. E.HsiehS. C.SkidmoreB.WellsG. A. (2019). Systematic reviews of clinical practice guidelines: a methodological guide. J. Clin. Epidemiol.108, 64–72. 10.1016/j.jclinepi.2018.11.030

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 34

  KetchenD. J.Jr.BoydB. K.BerghD. D. (2008). Research methodology in strategic management: past accomplishments and future challenges. Organ. Res. Methods11, 643–658. 10.1177/1094428108319843

  • CrossRef
  • Google Scholar
  • View reference in article

• 35

  KtepiB. (2016). Data Analytics (DA). Available online at: https://eds-b-ebscohost-com.nwulib.nwu.ac.za/eds/detail/detail?vid=2&sid=24c978f0-6685-4ed8-ad85-fa5bb04669b9%40sessionmgr101&bdata=JnNpdGU9ZWRzLWxpdmU%3d#AN=113931286&db=ers

  • Google Scholar
  • View reference in article

• 36

  LaherS. (2016). Ostinato rigore: establishing methodological rigour in quantitative research. S. Afr. J. Psychol.46, 316–327. 10.1177/0081246316649121

  • CrossRef
  • Google Scholar
  • View reference in article

• 37

  LeeC. (2015). The Myth of the Off-Limits Source. Available online at: http://blog.apastyle.org/apastyle/research/

  • Google Scholar
  • View reference in article

• 38

  LeeT. W.MitchellT. R.SablynskiC. J. (1999). Qualitative research in organizational and vocational psychology, 1979–1999. J. Vocat. Behav.55, 161–187. 10.1006/jvbe.1999.1707

  • CrossRef
  • Google Scholar

• 39

  LeechN. L.AnthonyJ.OnwuegbuzieA. J. (2007). A typology of mixed methods research designs. Sci. Bus. Media B. V Qual. Quant43, 265–275. 10.1007/s11135-007-9105-3

  • CrossRef
  • Google Scholar
  • View reference in article

• 40

  LevittH. M.MotulskyS. L.WertzF. J.MorrowS. L.PonterottoJ. G. (2017). Recommendations for designing and reviewing qualitative research in psychology: promoting methodological integrity. Qual. Psychol.4, 2–22. 10.1037/qup0000082

  • CrossRef
  • Google Scholar

• 41

  LoweS. M.MooreS. (2014). Social networks and female reproductive choices in the developing world: a systematized review. Rep. Health11:85. 10.1186/1742-4755-11-85

  • Pubmed Abstract
  • CrossRef
  • Google Scholar
  • View reference in article

• 42

  MareeK. (2016). Planning a research proposal, in First Steps in Research, 2nd Edn, ed MareeK. (Pretoria: Van Schaik Publishers), 49–70.

  • Google Scholar
  • View reference in article

• 43

  MareeK.PietersenJ. (2016). Sampling, in First Steps in Research, 2nd Edn, ed MareeK. (Pretoria: Van Schaik Publishers), 191–202.

  • Google Scholar
  • View reference in article

• 44

  NgulubeP. (2013). Blending qualitative and quantitative research methods in library and information science in sub-Saharan Africa. ESARBICA J.32, 10–23. Available online at: http://hdl.handle.net/10500/22397.

  • Google Scholar
  • View reference in article

• 45

  NieuwenhuisJ. (2016). Qualitative research designs and data-gathering techniques, in First Steps in Research, 2nd Edn, ed MareeK. (Pretoria: Van Schaik Publishers), 71–102.

  • Google Scholar
  • View reference in article

• 46

  NindM.KilburnD.WilesR. (2015). Using video and dialogue to generate pedagogic knowledge: teachers, learners and researchers reflecting together on the pedagogy of social research methods. Int. J. Soc. Res. Methodol.18, 561–576. 10.1080/13645579.2015.1062628

  • CrossRef
  • Google Scholar

• 47

  O'CathainA. (2009). Editorial: mixed methods research in the health sciences—a quiet revolution. J. Mix. Methods3, 1–6. 10.1177/1558689808326272

  • CrossRef
  • Google Scholar

• 48

  O'NeilS.KoekemoerE. (2016). Two decades of qualitative research in psychology, industrial and organisational psychology and human resource management within South Africa: a critical review. SA J. Indust. Psychol.42, 1–16. 10.4102/sajip.v42i1.1350

  • CrossRef
  • Google Scholar

• 49

  OnwuegbuzieA. J.CollinsK. M. (2017). The role of sampling in mixed methods research enhancing inference quality. Köln Z Soziol.2, 133–156. 10.1007/s11577-017-0455-0

  • CrossRef
  • Google Scholar

• 50

  Perestelo-PérezL. (2013). Standards on how to develop and report systematic reviews in psychology and health. Int. J. Clin. Health Psychol.13, 49–57. 10.1016/S1697-2600(13)70007-3

  • CrossRef
  • Google Scholar

• 51

  PericallL. M. T.TaylorE. (2014). Family function and its relationship to injury severity and psychiatric outcome in children with acquired brain injury: a systematized review. Dev. Med. Child Neurol.56, 19–30. 10.1111/dmcn.12237

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 52

  PetersonR. A.MerunkaD. R. (2014). Convenience samples of college students and research reproducibility. J. Bus. Res.67, 1035–1041. 10.1016/j.jbusres.2013.08.010

  • CrossRef
  • Google Scholar

• 53

  RitchieJ.LewisJ.ElamG. (2009). Designing and selecting samples, in Qualitative Research Practice: A Guide for Social Science Students and Researchers, 2nd Edn, ed RitchieJ.LewisJ. (London: Sage), 1–23.

  • Google Scholar

• 54

  SandelowskiM. (2011). When a cigar is not just a cigar: alternative perspectives on data and data analysis. Res. Nurs. Health34, 342–352. 10.1002/nur.20437

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 55

  SandelowskiM.VoilsC. I.KnaflG. (2009). On quantitizing. J. Mix. Methods Res.3, 208–222. 10.1177/1558689809334210

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 56

  ScholtzS. E.De KlerkW.De BeerL. T. (2019). A data generated research framework for conducting research methods in psychological research.

  • Google Scholar

• 57

  Scimago Journal & Country Rank (2017). Available online at: http://www.scimagojr.com/journalrank.php?category=3201&year=2015

  • Google Scholar

• 58

  Scopus (2017a). About Scopus. Available online at: https://www.scopus.com/home.uri (accessed February 01, 2017).

  • Google Scholar

• 59

  Scopus (2017b). Document Search. Available online at: https://www.scopus.com/home.uri (accessed February 01, 2017).

  • Google Scholar

• 60

  Scott JonesJ.GoldringJ. E. (2015). ‘I' m not a quants person'; key strategies in building competence and confidence in staff who teach quantitative research methods. Int. J. Soc. Res. Methodol.18, 479–494. 10.1080/13645579.2015.1062623

  • CrossRef
  • Google Scholar

• 61

  SmithB.McGannonK. R. (2018). Developing rigor in quantitative research: problems and opportunities within sport and exercise psychology. Int. Rev. Sport Exerc. Psychol.11, 101–121. 10.1080/1750984X.2017.1317357

  • CrossRef
  • Google Scholar

• 62

  StangorC. (2011). Introduction to Psychology. Available online at: http://www.saylor.org/books/

  • Google Scholar

• 63

  StrydomH. (2011). Sampling in the quantitative paradigm, in Research at Grass Roots; For the Social Sciences and Human Service Professions, 4th Edn, eds de VosA. S.StrydomH.FouchéC. B.DelportC. S. L. (Pretoria: Van Schaik Publishers), 221–234.

  • Google Scholar

• 64

  TashakkoriA.TeddlieC. (2003). Handbook of Mixed Methods in Social & Behavioural Research. Thousand Oaks, CA: SAGE publications.

  • Google Scholar

• 65

  ToomelaA. (2010). Quantitative methods in psychology: inevitable and useless. Front. Psychol.1:29. 10.3389/fpsyg.2010.00029

  • Pubmed Abstract
  • CrossRef
  • Google Scholar

• 66

  TruscottD. M.SwarsS.SmithS.Thornton-ReidF.ZhaoY.DooleyC.et al. (2010). A cross-disciplinary examination of the prevalence of mixed methods in educational research: 1995–2005. Int. J. Soc. Res. Methodol.13, 317–328. 10.1080/13645570903097950

  • CrossRef
  • Google Scholar

• 67

  WeitenW. (2010). Psychology Themes and Variations. Belmont, CA: Wadsworth.

  • Google Scholar