To What Extent Do Successive Cohorts Adopt Different Publication Patterns? Peer Review, Language Use, and Publication Types in the Social Sciences and Humanities

Introduction

Quantitative studies of scholarly communication have traditionally been mostly applied to STEM fields, with only scant attention being directed toward the social sciences and humanities (SSH). The reasons for this are manifold, but include the heterogeneity of the SSH and the fact that the SSH are generally less well-covered in Web of Science and Scopus, the traditional data sources for most bibliometric studies (Hicks, 1999; Archambault et al., 2006; Hicks and Wang, 2011). This lack of attention has partially changed in the last decade, however, with the introduction of studies that are based on more comprehensive databases, typically at the level of a country or region. In Europe, several national databases or repositories for research output have been set up (Sile et al., 2017, 2018). Examples of such databases include CROSBI in Croatia, RINC in Russia, and VIRTA in Finland. While most of these databases do not include citation information, they can be used for various analyses, such as studies of productivity (Rørstad and Aksnes, 2015), collaboration (Ossenblok et al., 2014), concentration in publication channels (Sivertsen and Larsen, 2012), or internationalization of book publishing (Verleysen and Engels, 2014).

Overview of the Literature: Publication Pattern Changes and Cohort Analysis

The lack of attention for the SSH is especially unfortunate, since research in the preceding decades has shown that the social sciences and especially the humanities may have different characteristics from STEM fields (Hornbostel, 2008). Hence, insights derived from and practices intended for STEM do not necessarily transfer well to the SSH. For instance, Nederhof et al. (1989) point out that monographs and popularizing articles are more frequent, although typically various publication types are used in conjunction (Hicks, 2004; Sivertsen, 2016; Verleysen and Weeren, 2016). In addition, some disciplines write a substantial share of publications in a local language, in addition to publications in English or other international languages. Such findings have implications for our understanding of how SSH scholars communicate about their research findings, as well as for applications of science studies, such as research evaluation and research policy-making. Hence, it is useful to develop further insight into SSH publication patterns, the share of publications with certain characteristics, e.g., by language, indexation, publication type, or some other property.

Publication patterns can be studied statically or dynamically, depending on whether or not the dimension of time is taken into account. In a study based on the Flemish VABB-SHW database (see further), Engels et al. (2012) were among the first to show that publication patterns—publication type, indexation in the Web of Science (WoS), and language—in the SSH in Flanders, Belgium are changing over the period 2000–2009 and that large differences exist between disciplines. We point out four important conclusions from their study:

1) Scholars from the social sciences differ from scholars working in the humanities in terms of which types of publications are used. Peer-reviewed book publications account for almost a quarter of the work produced by scholars from the humanities, whereas for researchers in the social sciences this is only 7%.

2) Evolutions in the use of publication types are diverging between the humanities and the social sciences. Only the latter exhibits an increase in preference for journal articles, while the share of book publications in the humanities remains stable.

3) There is an increase in the share of English-language publications, coupled with a decline in the share of Dutch (the local language) and other languages. This is observable for almost every SSH discipline.

4) There is an increase in the share of WoS-indexed publications, but this is less the case for humanities disciplines.

While the study of publication patterns originally was limited to specific countries/databases, this has broadened in more recent years with the introduction of bilateral and multilateral comparisons across countries and databases. Kulczycki et al. (2018) found that the West and North European countries that were studied exhibit similar publication patterns, while in Central and East European countries publishing in English is rarer and the trends sometimes much more abrupt. Engels et al. (2018) show that the share of monographs and book chapters in Flanders, Finland, Norway, Poland and Slovenia is gradually converging, decreasing in some cases and increasing in others.

Changes in publication patterns may occur following different mechanisms. On the one hand, a change in favor of some characteristic may be due to changes at the individual level, i.e., researchers changing their behavior in response to some external stimulus. On the other hand, changes may also occur due to the replacement of individuals: if older individuals disappear and younger ones with different preferences appear, this may lead to changes in the overall system. This can be studied by dividing researchers into cohorts and carrying out a cohort analysis.

Most bibliometric studies using cohorts hardly consider the SSH (exceptions include Horner et al., 1986; Savelsberg and Flood, 2004; O'Brien, 2012; Sugimoto et al., 2016). We mention two factors that may have contributed to this. First, the (presumably) main factor is the fact that large international databases usually do not cover the SSH as well. As a consequence, some studies (Dietz et al., 2000; Sugimoto et al., 2016) rely on the analysis of curricula vitae. Analyzing CVs is, however, labor-intensive and hard to automate. It may also lead to sampling bias, with authors whose CV is not readily available being excluded from the analysis. Second, researchers in the SSH tend to use multiple genres and publication types. This may complicate the analysis of, e.g., productivity in terms of academic output, since it is not straightforward how different publication types and genres should be weighted.

Sabharwal (2013) studied research productivity of doctoral recipients in the US from psychology and social sciences (besides the natural sciences) across three career stages: early, mid, and late career. Several conclusions can be drawn. First, compared to scholars from non-SSH disciplines, researchers working in the social sciences produce less journal articles. In relation to career stage, the author concludes that scholars in social sciences and psychology are most productive in their mid-career stages. When taking scholarly books into account, it is shown that late career stage faculty members produce more books.

The analysis of PhD cohorts of elite scholars from various social sciences at high prestige US institutions by Sugimoto et al. (2016) partly confirms and refines these conclusions. When dealing with productivity in terms of overall output, they find that “changes in communication patterns are more due to career aging (rank) than to cohort changes” (Sugimoto et al., 2016, p. 1014), but when it comes to publication types in the three social science disciplines under review (sociology, economics, and political sciences), the authors conclude that the output of recent PhD cohorts contains a growing proportion of journal articles.

In a case study based on data from the Norwegian CRISTIN database, Sivertsen (2016) compares the cohort of researchers under 45 with the cohort of those over 55, for both the social sciences and the humanities. While the younger cohorts tend to publish more in journals and in English than the older ones, the differences remain limited. These studies (Sabharwal, 2013; Sivertsen, 2016; Sugimoto et al., 2016) belong to the few to date that study cohort effects on publication types.

Research Rationale

The present study expands the work of Engels et al. (2012) both in breadth, by expanding it to more recent years, and in depth, by refining the analysis. Specifically, we investigate how four publication patterns evolve over a 15 years time period (2000–2014) in the VABB-SHW database. Since overall trends can hide a great deal of variety, we carry out a cohort analysis at the author level. Through the cohort analysis a distinction can be made between effects that reflect changes in the characteristics of how researchers from the same cohort publish (intracohort change) and effects that are due to the disappearance of researchers and/or introduction of new researchers (cohort succession or cohort replacement).

Our study is different from earlier bibliometric cohort analyses, since it is not based on CVs of scholars working in the US but on comprehensive data from a West-European region. Besides the short analysis of Sivertsen (2016), there are currently no studies that analyze the publication patterns between cohorts of authors from non-English speaking countries. In contrast to the sampling methods used in most of the aforementioned studies, we do not rely on a selective sample of (elite) scholars, nor do we rely on scholars that have their CVs readily available. Instead, we study the complete population of publishing scholars affiliated to an SSH research unit at a Flemish university. The publication patterns investigated are:

- peer review status: the proportion of publications that are regarded as peer-reviewed in the Flemish performance-based research funding system (henceforth PRFS);

- publication language: use of the local language (Dutch), use of English, and use of other languages;

- publication type: the proportion of journal articles, book publications, and proceedings;

- coverage in Web of Science: the proportion of journal articles and proceedings papers that are indexed in one or more of the following WoS databases: Science Citation Index Expanded, Social Sciences Citation Index, Arts & Humanities Citation Index, Conference Proceedings Citation Index- Science, and Conference Proceedings Citation Index- Social Sciences & Humanities.

The rationale for studying these patterns is as follows:

- Peer review status: In PRFSs, peer review status is typically an important aspect in the sense that peer-reviewed publications determine most if not all of the funding distribution (Hicks, 2013; Debackere et al., 2018; Zacharewicz et al., 2018). In particular for the SSH, however, the distinction between peer-reviewed and non-peer-reviewed publications is not always evident (Pölönen et al., 2017; Dahler-Larsen, 2018). Such ambiguity may translate into changes in publication patterns if, e.g., authors start to avoid publication channels like books, for which the peer review status is not made explicit. Indeed, Williams et al. (2018) report that younger authors in particular mention the lack of clarity regarding peer review of books as a reason to focus more on journal articles. Moreover, ambiguity may encourage authors, decision makers and/or publishers to take initiative to turn formerly non-peer-reviewed publication channels or types into peer-reviewed ones. For the Italian VQR, for example, Bonaccorsi (2018) reports changes that led to the inclusion of specialized university textbooks and edited books in the list of admissible publication types. In the Flemish context, changes such as the introduction of the Guaranteed Peer Reviewed Content label (GPRC; Verleysen and Engels, 2013) allowed for the inclusion in the PRFS of books by local Flemish publishers. In sum, in a context of PRFSs the share of peer-reviewed publications may well evolve—most likely, increase.

- Publication language: In non-English speaking countries and for the SSH in particular, a balanced multilingualism of the publications is an ideal (Sivertsen, 2018). Such a balance is crucial for the enlightenment mission or third mission of the SSH (Hicks, 2013; Verleysen and Engels, 2014; Basili and Lanzillo, 2018) and may serve important roles in the regional epistemic community. Yet the possible distortion of the multilingual ideal due to global evolutions as well as formal research evaluation procedures, including PRFSs, is a returning issue of concern (e.g., Loobuyck, 2007; Pölönen et al., 2018). Moreover, a recent comparison by Kulczycki et al. (2018) shows very considerable differences between eight European countries in terms of the share of publications in the local language, the part that is most likely to reach a local readership. The share of publications in English has continually increased in Flanders (Engels et al., 2012) and appears the highest among the countries studied by Kulczycki et al. (2018). Given the important role of publications in the local language, the need for a balanced multilingualism, and the remarkable patterns previously observed for Flanders, we include a cohort analysis of publication language in Flemish SSH in this paper.

- Publication type: Like balance in language use, balance in publication types is a returning concern in the study of SSH publication patterns. In particular, book publishing has often been reported to be under pressure (Thompson, 2002; Williams et al., 2009) and at risk of gradually being replaced by journal publishing (De Wever, 2007). Books, however, are very important sources in the SSH (Hicks, 2004; Nederhof, 2006; Bonaccorsi, 2018), with even particular types such as catalogs and encyclopedia entries fulfilling important epistemic and social roles in SSH academic communities (Basili and Lanzillo, 2018). For Flanders, the share of book publications in the humanities has been reported as stable (Engels et al., 2012), yet appears to be rather small compared to Finland, Norway, Poland and Slovenia (Engels et al., 2018). Hence this article analyses the evolution of the use of publication types across cohorts, in view of the importance of SSH book publications and their relative position in Flanders.

- Coverage in Web of Science: The WoS coverage of publications is studied in this article because of the history and particularities of the Flemish PRFS, which started off as a PRFS exclusively based on WoS (Debackere and Glänzel, 2004). Only later on, with the introduction of the VABB-SHW database, measures to ensure a more appropriate inclusion of SSH publications in the PRFS where taken (Verleysen et al., 2014; Engels and Guns, 2018). For the period before the VABB-SHW, however, comparative research showed that although WoS-coverage could not be considered a valid indicator of internationalization, Flemish SSH scholars did increase their share of articles indexed in WoS through two routes, viz. targeting WoS-indexed journals more often and making efforts to get locally relevant journals indexed in the WoS (Ossenblok et al., 2012). After the introduction of the VABB-SHW, real or perceived pressures motivating such behaviors may have evaporated, yet the gradual expansion of the WoS has continued (Guns and Engels, 2016). We therefore analyze the evolution of WoS coverage, in addition to the analysis of peer review, language use and publication types.

Article Structure

The article is structured as follows. Section Data and methods first introduces the database used for our analysis, the VABB-SHW, and its content. Hereafter we discuss how cohorts are operationalized for our purposes (with particular attention to differences with previous work) and the methods used for analysis. Section Results presents the results in four different parts, corresponding to the four publication patterns investigated in the article. In the final sections Discussion and Conclusion of this article, we discuss some limitations and directions for future work and conclude with the implications of our study.

Data and Methods

The VABB-SHW

Our data source is the VABB-SHW, a full-coverage database of peer-reviewed publications that have been authored by researchers affiliated to an SSH unit of a Flemish university. Since virtually all SSH research in Flanders (Belgium) is carried out at universities, the database can be considered comprehensive for SSH research in Flanders. The VABB-SHW contains metadata on journal articles (both journals indexed in WoS and journals not indexed in WoS), monographs, edited books, book chapters, and proceedings papers. The inclusion of non-WoS publication channels—journals, publishers, and book series—is decided upon by the Authoritative panel (“Gezaghebbende Panel” or GP), a panel of senior Flemish SSH researchers. For a full description of the VABB-SHW, we refer to Engels et al. (2012) and Verleysen et al. (2014).

While Flemish universities can submit all publications classified as one of the five mentioned publication types to the VABB-SHW, only publications that comply with all requirements are included in the public database and contribute to the allocation of funding in the regional PRFS (Debackere and Glänzel, 2004; Verleysen et al., 2014). The most important requirement is that a publication must have undergone peer review prior to being published. Table 1 provides an overview of the number of submitted publications per year, distinguishing between three categories:

- Peer-reviewed publications: comply with all requirements, including peer review, and hence are counted in the context of the PRFS;

- Non-peer-reviewed publications: comply with all requirements except for the requirement of peer review, and hence are not counted for the PRFS;

- Publications not counted in PRFS for reasons other than lack of peer review: do not comply with one or more criteria (be publicly accessible, be uniquely identifiable by ISBN or ISSN, contribute to the development of new insights or to applications thereof, have a minimum of four pages) or are outside the scope of the VABB-SHW (publication outside SSH or WoS publication not counted for the BOF-key, cf. Debackere and Glänzel, 2004), and hence are not counted for the PRFS. This heterogeneous category includes, for example, publications without a valid ISBN or ISSN, of less than four pages, without page information, or publications not authored by a researcher belonging to an SSH unit.

TABLE 1

Table 1. Number of peer-reviewed and non-peer-reviewed publications in VABB-SHW per year.

Clearly, the yearly number and percentage of peer-reviewed publications is steadily increasing while the number of non-peer-reviewed publications reaches a peak in 2008 and decreases afterwards.

Each publication is classified as one of five publication types that are curated in the VABB-SHW: journal articles, monographs (books as author), edited books, book chapters, or proceedings. In addition to regular metadata such as title, authors, and journal, the database also contains information per publication on language, WoS indexation, and peer review status as determined in the context of the PRFS. Each publication is assigned to one or more disciplines according to an organizational classification, i.e., a classification based on the departments or units to which the authors belong (Guns et al., 2018). The social sciences consist of 8 disciplines: Criminology, Economics & business, Educational sciences, Political sciences, Psychology, Social health sciences, Sociology, and Social sciences general. The humanities consist of 10 disciplines: Archaeology, Communication studies, History, History of arts, Law, Linguistics, Literature, Philosophy, Theology, and Humanities general.

For the remainder of this paper, the publications not counted in the PRFS for reasons other than lack of peer review are excluded from the analysis, although it is worth mentioning that this set contains many non-English publications and many book publications. The analysis of peer-reviewed vs. non-peer-reviewed publications in section Peer review status is based on the remaining set of peer-reviewed and non-peer-reviewed publications (N = 122,647). The analyses of publication language and publication type in sections Publication language and Publication type are based on the set of peer-reviewed publications (N = 66,591) to ensure comparability with earlier studies (e.g., Engels et al., 2012). Finally, the analysis of indexation in WoS is based on the set of all peer-reviewed journal articles and proceedings papers only (N = 53,040; book publications are excluded from this set, since they cannot be indexed in the WoS indexes that are used in the Flemish PRFS, cf. Engels and Guns, 2018).

Cohorts

The analysis focuses on the dynamic aspects of these publication patterns by studying their yearly evolution both at the level of publications and at the level of authors. At the author level, we carry out a cohort analysis. Authors are divided into different cohorts based on the year of their first publication in the database. We divide the total time period of 15 years into five cohorts: authors with a first publication in 2000–02 (cohort A), in 2003–05 (cohort B), in 2006–08 (cohort C), in 2009–11 (cohort D), and in 2012–14 (cohort E). Table 2 shows the number of authors in each cohort as well as each cohort's yearly number of publications. The rows “Authors (humanities)” and “Authors (social sciences)” contain the numbers of authors who have contributed to publications classified as, respectively, humanities and social sciences. Because of overlap between the two, the sum of both is greater than the number reported in the row “Authors (all SSH fields).”

TABLE 2

Table 2. Number of authors and publications per cohort.

Note that this operationalization of author cohorts differs from the one used in other studies. Most studies of aging and cohort effects select groups of authors or cohorts based on biological age, career age (e.g., studying the bibliography of authors starting from the date they obtained their PhD degree), or cohorts of authors that have passed a career milestone at a specific time (e.g., a cohort of authors having obtained professorship at a specific moment in time). The difference between these methods and the one used here has two important implications. First, there is a clear starting point in time (2000, the earliest publication year in the database) that does not specifically relate to the profile of the authors being studied. This means that cohort A is likely more heterogeneous than the other ones, since it contains both authors whose first publication appeared in 2000–02 and authors whose first publication appeared before 2000, a subset of which consists of authors that were already established researchers in 2000. The second implication is that some authors are in fact more senior than their cohort suggests: the VABB-SHW contains publications authored by researchers affiliated to an SSH unit of a Flemish university, but does not track publications outside of that set. To give an example, if a foreign researcher is appointed at a Flemish university in 2007 and publishes in that same year, he/she will be considered to belong to cohort C, even if he/she has published before.

The two implications mentioned above are tied to limitations of the data that are currently available. They should be kept in mind when interpreting the results, because there is a difference between, e.g., researchers who have started their career in 2003–05 and researchers whose first publication in VABB-SHW dates back to 2003–05, even if the two overlap considerably. There are multiple possible reasons why an individual is in a given cohort; these include, of course, the start of their career as a researcher and author, but also international mobility, national mobility (especially between French-speaking and Dutch-speaking universities in Belgium), mobility between the professional and the academic world, and disciplinary mobility (e.g., moving from non-SSH to SSH).

There is an additional nuance to point out with respect to these cohorts. It is tempting to interpret changes in the mean value for a single cohort as intracohort change, i.e., as due to changes in the patterns of (a subset of) individuals in the cohort. That interpretation is, however, only valid if the composition of the cohort remains the same. As Table 3 shows, this assumption is not met in the case of our data set. The percentages for cohort A are much higher than for the succeeding cohorts. More than 70% of cohort A researchers are still active in the second time period and even in the fourth time period, this is still the case for more than 50%. The most likely explanation is that a sizable number of cohort A researchers were already active before 2000. An additional factor is that the number of PhD students and postdoctoral researchers was considerably smaller in the early 2000s, leading to less temporary contracts and less outflow after a limited number of years. For cohorts B, C and D, only about 57% of authors per cohort publish in the time period following the one in which their first publication in the database appears. In the time period after that, this percentage drops to about 45%. In order to distinguish intracohort change effects from succession effects—disappearance of researchers from a cohort–, we have also done the analysis for the subset of “survivors,” authors that are active in every time period since their first publication in the database (bottom row of Table 3). The results of survivors do reflect changes at the aggregate level in publication patterns of individuals. On the whole, the results for survivors only are very similar to the ones for all authors per cohort. For this reason, the paper only shows survivor-related results in the analysis of peer review status. For the analyses of language, publication type and WoS coverage, we provide these results as Supplementary Material.

TABLE 3

Table 3. Number and percentage of active researchers per cohort and per 3-years time period.

Analysis

Per author one can determine their share of publications in a given time period with a given characteristic (e.g., share of book publications in 2006–08). Since different authors have different shares, we characterize a cohort by the mean share per author. These mean shares can be visualized in a line plot to study the evolution of a single cohort as well as to compare different cohorts. We calculate shares from the perspective of each single author. This means that co-authored publications are counted without fractionalization. While the focus of the paper is on changes in relative preference for certain kinds of publications and we therefore mostly work with percentages, we do show one result with absolute numbers in the analysis of peer review. Figures displaying absolute numbers for the other publication patterns are included in the Supplementary Material to this article.

Since the arithmetic mean is not very robust and can be influenced by outliers, we also display a 95% stability interval for each line. Stability intervals are obtained by a bootstrapping procedure, in which 1,000 random samples with replacement are taken from the cohort's author set and for each sample the mean is determined. From the resulting set of 1,000 bootstrap means, the stability interval can be determined (Efron and Tibshirani, 1994). For this, we use the bootstrapping procedure as implemented in the seaborn package (Waskom et al., 2018). A wider stability interval indicates that the value obtained is less stable due to outliers.

In summary, we analyze the relation between publication patterns and cohorts by visualizing the evolution of the mean share of a publication pattern for each cohort. The analysis is carried out for the SSH as a whole and for the humanities and the social sciences separately. Each figure contains three rows for, respectively, the SSH as a whole, the humanities and the social sciences. All data and code for the analysis is openly available (Guns, 2018).

Results

Peer Review Status

In this section we analyze the shares of peer-reviewed vs. non-peer-reviewed publications. As explained earlier, the set of non-peer-reviewed publications does not include publications that are not counted in the PRFS for reasons not related to peer review. Figure 1 shows that, in each cohort, there is a substantial shift toward publishing more in venues that carry out peer review. With the exception of cohort A, all cohorts are roughly comparable: several lines cross, and the stability intervals overlap. This shift toward peer-reviewed publications might be related to the introduction of and changes in the Flemish PRFS (Guns and Engels, 2016), but it is worth pointing out that the changes predate the addition of the VABB-SHW to the PRFS (Engels and Guns, 2018). This evolution is apparent within both the humanities and the social sciences, although the shares of peer-reviewed publications are, on average, higher in the social sciences.

FIGURE 1

Figure 1. Share of peer-reviewed and non-peer-reviewed publications per cohort. From top to bottom: all SSH fields, humanities only, and social sciences only.

In order to better understand the dynamics involved in this shift toward more peer review, we look at the mean absolute number of peer-reviewed and non-peer-reviewed publications per cohort in Figure 2. This figure only takes survivors into account. We can see that the mean number of peer-reviewed publications per time unit is almost linearly increasing. A rather different pattern emerges for the non-peer-reviewed publications: there, an initial increase is followed by a decrease. This pattern can be observed for the first three cohorts, and in both the social sciences and the humanities. One possible way to explain this finding is that there are two basic mechanisms at play: on the one hand, a trend toward more publications per author (which may itself be due to factors such as increased productivity and increased co-authorship) and, on the other hand, a trend away from non-peer-reviewed publications. In the early stage of a researcher's career, the trend toward more publications plays a major role, leading to increased numbers of both peer-reviewed and non-peer-reviewed publications. In the later stages, the trend away from non-peer-reviewed publications outweighs the other trend, leading to a decrease of non-peer-reviewed publications. Interestingly, we observe that Figure 2 shows no major discrepancies in average productivity between the social sciences and the humanities: the latter tend to publish more non-peer-reviewed publications and the former more peer-reviewed publications, but all in all, the differences are rather limited. The following sections of the results are only based on the set of peer-reviewed publications.

FIGURE 2

Figure 2. Mean number of peer-reviewed and non-peer-reviewed publications per cohort, survivors only. From top to bottom: all SSH fields, humanities only, and social sciences only.

Publication Language

Two languages make up the bulk of the peer-reviewed publications in VABB-SHW, namely English as the major international forum language (49.7%) and Dutch as the local language (41.2%). Other languages are relatively marginal, together accounting for 6.8% of peer-reviewed publications. We find that use of English is strongly linked to indexation in Web of Science. Indeed, 95.1% of WoS-indexed publications in the VABB-SHW are written in English.

Earlier research has already established that the share of English-language publications is increasing in Flanders (Engels et al., 2012) as well as elsewhere (Kulczycki et al., 2018) to the detriment of the share of local and other languages. Figure 3 shows the evolution of mean share of publications in Dutch, English, and other languages per cohort. The overall trend seems to be similar for each cohort: the share of English is increasing and that of Dutch is decreasing. There is, however, a difference between the first cohort and the others. The share of Dutch as well as other language publications is clearly higher for cohort A than for the others, the lines of which roughly overlap. For all cohorts and years, the share of other language publications is <10% and quite stable. This stability may be due to the fact that publications in the object language play an important role in linguistics and literature studies, e.g., studies on Italian literature are frequently written in Italian. The picture for the social sciences and the humanities separately are quite similar to the SSH as a whole, although the share of English-language publications of cohort A in the humanities has increased much less than the general trend.

FIGURE 3

Figure 3. Share of Dutch, English, and other languages per cohort. From top to bottom: all SSH fields, humanities only, and social sciences only.

While Figure 3 seems to suggest that there is a marked decrease in the number of Dutch-language publications, this is actually not the case. Rather, the annual number of Dutch-language publications is roughly the same between 2004 and 2014 (4,804 on average), but the yearly number of English-language publications has almost doubled over the same time period, from 4,251 in 2004 to 8,389 in 2014.

Publication Type

One of the most distinguishing characteristics of the SSH is the use of multiple publication types. In addition to journal articles and proceedings papers, books play an important role in many SSH disciplines. We aggregate the three types of book publications in the VABB-SHW (monographs, edited books, and book chapters) into one type “book publication” and visualize the results per cohort in Figure 4 (peer-reviewed publications only).

FIGURE 4

Figure 4. Share of publication type per cohort. From top to bottom: all SSH fields, humanities only, and social sciences only.

In spite of the problems of the academic book nowadays (Thompson, 2002; Williams et al., 2009), the results do not display a clear shift toward publishing more in journals. While more recent cohorts tend to publish relatively more journal articles, all cohorts exhibit a decline in the share of journal articles and an increase in share of book publications. This effect is visible in both the humanities and the social sciences, but is especially pronounced in the former, where the changes within each cohort as well as the inter-cohort differences are fairly pronounced. The mean share of journal articles in the social sciences, on the other hand, is stable or slightly decreasing for all cohorts. Furthermore, the differences between cohorts are quite pronounced in the humanities, while all cohorts have very similar shares in the social sciences.

Engels et al. (2012) noticed a diverging evolution in the use of publication types between social sciences and humanities. Our results here seem to indicate that, overall, the direction of the evolution is the same for the social sciences and the humanities but the speed and magnitude are different. Compared to other countries the share of book publications in Flemish SSH is rather low, yet trends seem to converge across countries (Engels et al., 2018).

Coverage in Web of Science

Since the Book Citation Index is not used in the Flemish PRFS, we calculate shares of WoS-indexed publications by only taking peer-reviewed journal articles and proceedings papers into account. Figure 5 displays the share of WoS-indexed publications per cohort. The strong increase across all cohorts bears witness to an increased orientation toward WoS-indexed journals and proceedings, but is also partly due to the expansion of the WoS itself (Engels et al., 2012; Ossenblok et al., 2012). We observe that the different cohorts are quite similar, with the exception of cohort A.

FIGURE 5

Figure 5. Share of Web of Science covered journal articles and proceedings papers per cohort. From top to bottom: all SSH fields, humanities only, and social sciences only.

The overall picture in the humanities and the social sciences is similar, the main difference being much lower shares of WoS-indexed publications in the humanities: the highest percentage of WoS publications across all time units and cohorts in the humanities is roughly the same as the lowest share for the social sciences.

Discussion

Cross-national comparative research into the evolution of publication patterns in the SSH has shown that publication patterns evolve continuously (e.g., Engels et al., 2018; Kulczycki et al., 2018). While there has been much speculation on the reasons behind these evolutions, not much is known about the dynamics of such changes. In this article, we have taken one step further by introducing an additional socio-structural variable, namely different cohorts of authors. While this step in itself is not sufficient to unambiguously point out the reasons behind the observed changes, it does give us further insight in the dynamic nature of publication patterns.

Overall, we do not find strong cohort effects. Cohorts have very similar shares of peer-reviewed publications, publication languages, and WoS-indexed publications, as witnessed by the frequent crossing of lines and overlaps of stability intervals. The exception is cohort A, the set of researchers whose oldest publication in the VABB-SHW dates back to 2000–02. The discrepancy between cohort A and the other cohorts may be caused by the fact that a substantial part of this cohort was already active before 2000. If this is indeed the major cause, it implies that larger inter-cohort differences may develop over longer stretches of time. There is, however, at least one additional factor to cohort A's special status: later cohorts have increasingly higher shares of researchers on temporary contracts. Furthermore, it is remarkable that in all cases the overall trend—increasing or decreasing—appears to be the same across cohorts and for both social sciences and humanities. All in all, these findings suggest that cohort effects are in place but inter-cohort differences are relatively small.

Now we discuss each publication pattern separately. We witness an increase across cohorts in share of peer-reviewed publications. The absolute counts show that this is due to an increase of peer-reviewed publications, whereas the average number of non-peer-reviewed publications tends to decrease after an initial increase. The decrease in number of non-peer-reviewed publications does not seem to be a period effect of changes in the PRFS, such as the introduction of the VABB-SHW: even after 2008, there is an initial increase of non-peer-reviewed publications for all cohorts (with the exception of cohort E, for which only one data point is available). The increase in share of peer-reviewed publications is somewhat steeper from 2008 onwards for the more “senior” cohorts (A and B), which might be linked to the introduction of the VABB-SHW. This change in slope is, however, not visible for the social sciences. Several factors may contribute to the observed increase of peer-reviewed publications, including greater preference for peer-reviewed outlets and initiatives to turn non-peer-reviewed publication channels and types into peer-reviewed ones (Verleysen and Engels, 2013; Bonaccorsi, 2018).

With regard to preferred language use, inter-cohort differences are quite limited, in spite of the influx of international researchers. Between 2003 and 2014 the proportion of foreign research staff at Flemish universities (all disciplines) has increased from 5.2 to 9.3% (and from 10.6 to 30.4% for PhD students only; Gilliot and Titeca, 2017). This increased share of foreign researchers has had only limited influence on publication language. In this respect, it is worth pointing out that the majority of foreign researchers in Flanders come from the Netherlands, where Dutch is also spoken. In summary, the main driver of the shift toward more English-language publications seems to be intracohort change rather than cohort succession or period effects relating to the inflow of foreign research staff in Flanders. This intracohort change may be related to internationalization in terms of research topics, collaborative partners, and target audience. The stability in the last decade of the absolute number of Dutch-language publications indicates that a balanced multilingualism (Sivertsen, 2018) can be obtained.

As mentioned in the introduction, it is commonly argued that funding and evaluation systems may lead to a growth in journal publishing, to the detriment of books (Williams et al., 2009). Engels et al. (2012, 2018) did not find empirical support for this argument, reporting an overall stable share of book publications for Flanders in the period 2000–2009, and converging trends across Flanders, Finland, Norway, Poland, and Slovenia in the period 2004–2015. Our findings in this paper confirm and reinforce the general notion that book publications are not on the decline in Flanders. On the contrary, the share of book publications increases for each separate cohort, especially in the humanities. The share of book publications during the first time unit of each cohort is also very similar, around 10%. It is possible that the recognition of book publications in the VABB-SHW has contributed to these results; after all, book chapters and edited books both have the same weight (1) as journal articles, while monographs have a weight of 4. In addition, several additional mechanisms for including book publications in the VABB-SHW have been introduced, most notably selection at the level of book series and the introduction of the GPRC label (Verleysen and Engels, 2013).

Considerable changes in Flemish research policy were the reshaping of the funding allocation model to a WoS-based PRFS in 2003 and the introduction of a comprehensive database for the SSH in 2010. Since the original PRFS only relied on WoS publications, one might expect a growth of WoS-indexed publications. This is indeed the case (Guns and Engels, 2016), but the introduction of the VABB-SHW does not seem to have slowed down the growing share of WoS-indexed articles and proceedings. Hence, this growth might be more due to expansion of the WoS databases and the value attributed to WoS-indexed publications outside of the funding system.

Limitations

The analysis presented here focuses on the dynamic aspects of publication practices by studying their yearly evolution and through a cohort analysis of the authors. This way a distinction can be made between effects that reflect changes in the characteristics of how researchers of a certain cohort publish (intracohort change) and effects that are due to the disappearance of researchers and/or introduction of new researchers (cohort succession). As already touched upon in section Results, the appearance and disappearance of authors forms a “blind spot” in our study, in that we lack information on these researchers' full publication portfolios. If one wants to understand changes in publication patterns across entire careers, one should direct more attention to the issue of mobility.

The literature on scholarly communication suggests that period effects can confound when one interprets cohort effects. A study by Savelsberg and Flood (2004), for instance, addresses the difference between period and cohort effects with regard to criminological scholarship in the US between 1951 and 1993. Although the authors do not specifically focus on publication patterns, they draw a relevant conclusion: cohort membership has some effect, but periods—defined as dominant ideological currents of different eras, historic events, changing institutional practices in academia, etc.—have considerable impact (on topic, type of theory examined, and the data which are used by the researchers) over longer stretches of time. The fact that cohort membership effects often weaken over time, can thus be explained by countervailing period effects. When it comes to the present article, two important factors that might exert period effects are changes in research policy (e.g., the introduction and subsequent changes of the PRFS) and the internationalization of academia.

Some other limitations relate to the data used for our study. First, our results are based on data from Flanders and may not carry over to other countries or regions. Indeed, Kulczycki et al. (2018) find that “discipline-level publication patterns differ more across countries than [Sivertsen or van Leeuwen] initially suggested.” Second, the VABB-SHW is limited to a set of classic publication types among a much wider set of possible research output types, such as corpora, blogs, data sets, software, catalogs, or designs. Such forms of research output are not tracked systematically. It is hence unknown how prevalent they are and to what extent taking them into account would change our overall perception of (changes in) publication patterns in the SSH. A third issue is the classification of disciplines. This study presents a general view of the social sciences and humanities. More fine-grained studies of subfields within these two clusters might exhibit differing trends. Language use for publications in language studies is a typical example of this.

Future Research

There are several options for future research to extend the analysis presented in this article. A first possible line of work involves a refined operationalization of cohorts, by including more direct information on the career stage, age, and/or title of authors. While such data are not directly available, it would for instance be possible to identify the starting professors in each cohort and study the evolution of their publication patterns. Secondly, our data allow for investigations into other publication practices, such as collaboration patterns between and within different author cohorts. Specifically, we could study co-authorships between disciplines as well as between cohorts. Thirdly, it would be very interesting if a similar cohort analysis could be carried out in other countries, in order to enable an international comparison. Indeed, while some of the general trends have also been observed in other countries, it is currently unknown to what extent they are due to cohort effects.

Conclusion

Our study of author cohorts in Flemish SSH provides an extension to and a more nuanced understanding of the insights from past inquiries into publication practices and patterns. To enhance our understanding of how changes in publication patterns come about, we have carried out a cohort analysis of 15 years of SSH research.

Overall, we find that there is a trend across all cohorts and across fields toward peer-reviewed publications, toward use of English, and toward publishing in WoS-indexed journals. While we witness very clear intracohort changes, cohort succession effects appear to be much weaker. A first exception is the cohort of researchers already active in 2000-2003 (cohort A), who often appear to maintain a more “traditional” SSH profile, with lower shares of peer-reviewed publications, of publications in English, of journal articles, and of publications indexed in WoS. The second exception pertains to publication types, where we do witness a clear cohort effect: each successive cohort is less likely to publish book publications, although the proportion of book publications increases over time for all cohorts.

The increase in share of peer-reviewed publications is partially due to the fact that cohorts A, B and C all show a decreasing trend for the absolute number of non-peer-reviewed publications. Regarding publication languages, cohort A again has a rather different profile. Authors in this cohort tend to make more use, relatively speaking, of local and other languages. Overall, however, it is found that English language publications are of increasing importance for all authors, although percentages in the social sciences are much higher than in the humanities. In general, the annual number of publications in Dutch is stable over the last 10 years, while the number of English-language publications has nearly doubled over the same time period.

With regard to publication types, we do not observe a shift away from book publications. For all cohorts, we find a slightly declining share of journal articles over time, in favor of book publications. This trend exists in the humanities as well as the social sciences, but is clearest in the humanities. With respect to WoS indexation, our analysis shows that the share of WoS-indexed publications is much lower in the humanities than in the social sciences, but continues to increase for both across all cohorts.

In summary, we have provided a more fine-grained picture of how five cohorts of authors compare and evolve in terms of peer review, publication type, publication language, and coverage in WoS. By distinguishing intracohort changes from cohort succession effects, the present analysis has enabled a better understanding of the mechanisms behind changes in publication patterns, specifically in relation to how they do or do not relate to research policy. This illustrates that cohort analysis is a useful instrument to gain better insight into the evolution of publication patterns.

Author Contributions

RG, JE, and TE: designed the study; RG: performed the analysis; RG, JE, and TE: interpreted the results; RG and JE: wrote the first version of the manuscript; RG, JE, and TE: revised the manuscript.

Funding

This investigation has been made possible by the financial support of the Flemish government to the Centre for R&D Monitoring (ECOOM). The opinions in the paper are the authors' and not necessarily those of the government.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We thank Linda Sīle for input on the study design and thoughts on an earlier version of the manuscript, which helped to improve the paper.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frma.2018.00038/full#supplementary-material

References

Archambault, E., Vignola-Gagne, E., Côte, G., Larivière, V., and Gingras, Y. (2006). Benchmarking scientific output in the social sciences and humanities: the limits of existing databases. Scientometrics 68, 329–342. doi: 10.1007/s11192-006-0115-z

CrossRef Full Text | Google Scholar

Basili, C., and Lanzillo, L. (2018). “Research quality criteria in the evaluation of books,” in The Evaluation of Research in Social Sciences and Humanities: Lessons from the Italian Experience, ed A. Bonaccorsi (Cham: Springer International Publishing), 159–184.doi: 10.1007/978-3-319-68554-0_7

CrossRef Full Text | Google Scholar

Bonaccorsi, A. (2018). “Toward an epistemic approach to evaluation in SSH,” in The Evaluation of Research in Social Sciences and Humanities: Lessons from the Italian Experience, ed A. Bonaccorsi (Cham: Springer International Publishing), 1–29. doi: 10.1007/978-3-319-68554-0_1

CrossRef Full Text | Google Scholar

Dahler-Larsen, P. (2018). Theory-based evaluation meets ambiguity: the role of janus variables. Am. J. Eval. 39, 6–23. doi: 10.1177/1098214017716325

CrossRef Full Text | Google Scholar

De Wever, B. (2007). “Van A1, A2, A3. Concurrentievervalsing in academia,” in Welke universiteit willen wij (niet)?, eds P. Loobuyck, G. Vanheeswijck, W. Van Herck, E. Grieten, and K. Vercauteren (Gent: Academia Press), 199–203.

Google Scholar

Debackere, K., Arnold, E., Sivertsen, G., Spaapen, J., and Sturn, D. (2018). Performance-Based Funding of University Research: Summary Report. Luxembourg: Publications Office of the European Union.

Google Scholar

Debackere, K., and Glänzel, W. (2004). Using a bibliometric approach to support research policy making: the case of the Flemish BOF-key. Scientometrics 59, 253–276. doi: 10.1023/B:SCIE.0000018532.70146.02

CrossRef Full Text | Google Scholar

Dietz, J. S., Chompalov, I., Bozeman, B., Lane, E. O., and Park, J. (2000). Using the curriculum vita to study the career paths of scientists and engineers: an exploratory assessment. Scientometrics 49, 419–442. doi: 10.1023/A:1010537606969

CrossRef Full Text | Google Scholar

Efron, B., and Tibshirani, R. J. (1994). An Introduction to the Bootstrap. Boca Raton, FL: CRC press.

Google Scholar

Engels, T. C. E., and Guns, R. (2018). The Flemish performance-based research funding system: a unique variant of the Norwegian model. J. Data Inform. Sci. 3, 44–59. doi: 10.2478/jdis-2018-0020

CrossRef Full Text | Google Scholar

Engels, T. C. E., Istenič Starčič, A., Kulczycki, E., Pölönen, J., and Sivertsen, G. (2018). Are book publications disappearing from scholarly communication in the social sciences and humanities? Aslib J. Inform. Manage. 70, 592–607. doi: 10.1108/AJIM-05-2018-0127

CrossRef Full Text | Google Scholar

Engels, T. C. E., Ossenblok, T. L. B., and Spruyt, E. H. J. (2012). Changing publication patterns in the social sciences and humanities, 2000–2009. Scientometrics 93, 373–390. doi: 10.1007/s11192-012-0680-2

CrossRef Full Text | Google Scholar

Gilliot, D., and Titeca, E. (2017). Werken aan een Vlaamse Universiteit. Available online at: https://www.vlaamsindicatorenboek.be/3.3/werken-aan-een-vlaamse-universiteit (Accessed September 26, 2018).

Guns, R. (2018). Cohort Analysis of Publication Patterns in SSH: Data and Code (Version 1.0.0). Zenodo. doi: 10.5281/zenodo.1438348

CrossRef Full Text

Guns, R., and Engels, T. C. E. (2016). “Effects of performance-based research funding on publication patterns in the social sciences and humanities,” in Proceedings of the 21st International Conference on Science and Technology Indicators, ed I. Ràfols (València: Editorial Universitat Politècnica de València), 893–900.

Google Scholar

Guns, R., Sile, L., Eykens, J., Verleysen, F. T., and Engels, T. C. E. (2018). A comparison of cognitive and organizational classification of publications in the social sciences and humanities. Scientometrics 116, 1093–1111. doi: 10.1007/s11192-018-2775-x

CrossRef Full Text | Google Scholar

Hicks, D. (1999). The difficulty of achieving full coverage of international social science literature and the bibliometric consequences. Scientometrics 44, 193–215.

Google Scholar

Hicks, D. (2004). “The four literatures of social science” in Handbook of quantitative Science and Technology Research: The use of publication and patent statistics in studies of S&T systems, eds H. F. Moed, W. Glänzel, and U. Schmoch (Dordrecht: Kluwer Academic), 473–496.

Google Scholar

Hicks, D. (2013). One size doesn't fit all: on the co-evolution of national evaluation systems and social science publishing. Confero 1, 67–90. doi: 10.3384/confero13v1121207b

CrossRef Full Text | Google Scholar

Hicks, D., and Wang, J. (2011). Coverage and overlap of the new social sciences and humanities journal lists. J. Am. Soc. Inform. Sci. Technol. 62, 284–294. doi: 10.1002/asi.21458

CrossRef Full Text | Google Scholar

Hornbostel, S. (2008). “Gesucht: aussagekräftige indikatoren und belastbare datenkollektionen. Desiderate geisteswissenschaftlicher evaluierung in Deutschland,” in What the Hell is Quality? eds E. Lack and C. Markschies, (Frankfurt, NY: Campus Verlag), 55–73.

Google Scholar

Horner, K. L., Rushton, J. P., and Vernon, P. A. (1986). Relation between aging and research productivity of academic psychologists. Psychol. Aging 1:319.

PubMed Abstract | Google Scholar

Kulczycki, E., Engels, T. C. E., Pölönen, J., Bruun, K., Dušková, M., Guns, R., et al. (2018). Publication patterns in the social sciences and humanities: evidence from eight European countries. Scientometrics 116, 463–486. doi: 10.1007/s11192-018-2711-0

CrossRef Full Text | Google Scholar

Loobuyck, P. (2007). “Welke academici willen we? Pleidooi voor meer maten en gewichten,” in Welke universiteit willen wij (niet)?, eds P. Loobuyck, G. Vanheeswijck, W. Van Herck, E. Grieten and K. Vercauteren (Gent: Academia Press), 11–25.

Google Scholar

Nederhof, A. J. (2006). Bibliometric monitoring of research performance in the social sciences and the humanities: a review. Scientometrics 66, 81–100. doi: 10.1007/s11192-006-0007-2

CrossRef Full Text | Google Scholar

Nederhof, A. J., Zwaan, R. A., Debruin, R. E., and Dekker, P. J. (1989). Assessing the usefulness of bibliometric indicators for the humanities and the social and behavioral sciences–A comparative study. Scientometrics 15, 423–435. doi: 10.1007/BF02017063

CrossRef Full Text | Google Scholar

O'Brien, T. L. (2012). Change in academic coauthorship, 1953–2003. Sci. Technol. Hum. Values 37, 210–234. doi: 10.1177/0162243911406744

CrossRef Full Text | Google Scholar

Ossenblok, T. L. B., Engels, T. C. E., and Sivertsen, G. (2012). The representation of the social sciences and humanities in the Web of Science—a comparison of publication patterns and incentive structures in Flanders and Norway (2005–9). Res. Eval. 21, 280–290. doi: 10.1093/reseval/rvs019

CrossRef Full Text | Google Scholar

Ossenblok, T. L. B., Verleysen, F. T., and Engels, T. C. E. (2014). Coauthorship of journal articles and book chapters in the social sciences and humanities (2000–2010). J. Assoc. Inform. Sci. Technol. 65, 882–897. doi: 10.1002/asi.23015

CrossRef Full Text | Google Scholar

Pölönen, J., Auranen, O., Engels, T. C. E., and Kulczycki, E. (2018). “Taking national language publications into account: the case of the Finnish performance-based research funding system. Presented at the STI 2018,” in 23rd International Conference on Science and Technology Indicators (Leiden).

Google Scholar

Pölönen, J., Engels, T. C. E., Guns, R., and Verleysen, F. T. (2017). “Is my publication peer reviewed? A comparison of top-down and bottom-up identification of peer review in the framework of the Finnish and Flemish performance-based research funding systems,” in Presented at the STI 2017: Conference on Science, Technology and Innovation Indicators (Paris).

Google Scholar

Rørstad, K., and Aksnes, D. W. (2015). Publication rate expressed by age, gender and academic position – A large-scale analysis of Norwegian academic staff. J. Inform. 9, 317–333. doi: 10.1016/j.joi.2015.02.003

CrossRef Full Text | Google Scholar

Sabharwal, M. (2013). Comparing research productivity across disciplines and career stages. J. Comp. Policy Anal. Res. Pract. 15, 141–163. doi: 10.1080/13876988.2013.785149

CrossRef Full Text | Google Scholar

Savelsberg, J. J., and Flood, S. M. (2004). Criminological knowledge: period and cohort effects in scholarship. Criminology 42, 1009–1042. doi: 10.1111/j.1745-9125.2004.tb00543.x

CrossRef Full Text | Google Scholar

Sile, L., Guns, R., Sivertsen, G., and Engels, T. C. E. (2017). European Databases and Repositories for Social Sciences and Humanities Research Output. doi: 10.6084/m9.figshare.5172322.v2

CrossRef Full Text

Sile, L., Pölönen, J., Sivertsen, G., Guns, R., Engels, T. C. E., Arefiev, P., et al. (2018). Comprehensiveness of national bibliographic databases for social sciences and humanities: findings from a European survey. Res. Eval. 27, 310–322. doi: 10.1093/reseval/rvy016

CrossRef Full Text | Google Scholar

Sivertsen, G. (2016). Patterns of internationalization and criteria for research assessment in the social sciences and humanities. Scientometrics 107, 357–368. doi: 10.1007/s11192-016-1845-1

PubMed Abstract | CrossRef Full Text | Google Scholar

Sivertsen, G. (2018). Balanced multilingualism in science. BiD Textos Univer. Bibliotec. Document. 40. doi: 10.1344/BiD2018.40.25

CrossRef Full Text | Google Scholar

Sivertsen, G., and Larsen, B. (2012). Comprehensive bibliographic coverage of the social sciences and humanities in a citation index: an empirical analysis of the potential. Scientometrics 91, 567–575. doi: 10.1007/s11192-011-0615-3

CrossRef Full Text | Google Scholar

Sugimoto, C. R., Sugimoto, T. J., Tsou, A., Milojević, S., and Larivière, V. (2016). Age stratification and cohort effects in scholarly communication: a study of social sciences. Scientometrics 109, 997–1016. doi: 10.1007/s11192-016-2087-y

CrossRef Full Text | Google Scholar

Thompson, J. (2002). The death of the scholarly monograph in the humanities? Citation patterns in literary scholarship. Libri 52, 121–136. doi: 10.1515/LIBR.2002.121

CrossRef Full Text | Google Scholar

Verleysen, F. T., and Engels, T. C. E. (2013). A label for peer-reviewed books. J. Am. Soc. Inform. Sci. Technol. 64, 428–430. doi: 10.1002/asi.22836

CrossRef Full Text | Google Scholar

Verleysen, F. T., and Engels, T. C. E. (2014). Barycenter representation of book publishing internationalization in the social sciences and humanities. J. Inform. 8, 234–240. doi: 10.1016/j.joi.2013.11.008

CrossRef Full Text | Google Scholar

Verleysen, F. T., Ghesquière, P., and Engels, T. C. E. (2014). “The objectives, design and selection process of the Flemish Academic Bibliographic Database for the Social Sciences and Humanities (VABB-SHW),” in Bibliometrics Use and Abuse in the Review of Research Performance, eds W. Blockmans, L. Engwall, and D. Weaire (London: Portland Press), 117–127.Available online at: http://www.portlandpress.com/pp/books/online/wg87/087/0117/0870117.pdf

Google Scholar

Verleysen, F. T., and Weeren, A. (2016). Clustering by publication patterns of senior authors in the social sciences and humanities. J. Inform. 10, 254–272. doi: 10.1016/j.joi.2016.01.004

CrossRef Full Text | Google Scholar

Waskom, M., Botvinnik, O., O'Kane, D., Hobson, P., Ostblom, J., Lukauskas, S., et al. (2018). mwaskom/seaborn: v0.9.0 (July 2018). Zenodo. doi: 10.5281/zenodo.1313201

CrossRef Full Text

Williams, G., Basso, A., Galleron, I., and Lippiello, T. (2018). “More, less or better: the problem of evaluating books in SSH research,” in The Evaluation of Research in Social Sciences and Humanities: Lessons from the Italian Experience, ed A. Bonaccorsi (Cham: Springer International Publishing), 133–158. doi: 10.1007/978-3-319-68554-0_6

CrossRef Full Text | Google Scholar

Williams, P., Stevenson, I., Nicholas, D., Watkinson, A., and Rowlands, I. (2009). The role and future of the monograph in arts and humanities research. Aslib Proc. 61, 67–82. doi: 10.1108/00012530910932294

CrossRef Full Text | Google Scholar

Zacharewicz, T., Lepori, B., Reale, E., and Jonkers, K. (2018). Performance-based research funding in EU Member States—a comparative assessment. Sci. Public Pol. scy041. doi: 10.1093/scipol/scy041

CrossRef Full Text | Google Scholar