This article was submitted to Scholarly Communication, a section of the journal Frontiers in Research Metrics and Analytics
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
The field of law has retained its distinctiveness regarding peer review to this day, and reviews are often conducted without following standardized rules and principles. External and independent evaluation of submissions has recently become adopted by European law journals, and peer review procedures are still poorly defined, investigated, and attuned to the legal science publishing landscape. The aim of our study was to gain a better insight into current editorial policies on peer review in law journals by exploring editorial documents (instructions, guidelines, policies) issued by 119 Croatian, Italian, and Spanish law journals. We relied on automatic content analysis of 135 publicly available documents collected from the journal websites to analyze the basic features of the peer review processes, manuscript evaluation criteria, and related ethical issues using WordStat8. Differences in covered topics between the countries were compared using the chi-square test. Our findings reveal that most law journals have adopted a traditional approach, in which the editorial board manages mostly anonymized peer review (104, 77%) engaging independent/external reviewers (65, 48%). Submissions are evaluated according to their originality and relevance (113, 84%), quality of writing and presentation (94, 70%), comprehensiveness of literature references (93, 69%), and adequacy of methods (57, 42%). The main ethical issues related to peer review addressed by these journals are reviewer’s competing interests (42, 31%), plagiarism (35, 26%), and biases (30, 22%). We observed statistically significant differences between countries in mentioning key concepts such as “Peer review ethics”, “Reviewer”, “Transparency of identities”, “Publication type”, and “Research misconduct”. Spanish journals favor reviewers’ “Independence” and “Competence” and “Anonymized” peer review process. Also, some manuscript types popular in one country are rarely mentioned in other countries. Even though peer review is equally conventional in all three countries, high transparency in Croatian law journals, respect for research integrity in Spanish ones, and diversity and inclusion in Italian are promising indicators of future development.
Research of the peer review process, “the backbone of modern science” according to
The description of law as a science includes a series of interpretations and offers many hypotheses about the meaning and scope of legal concepts, rules, and principles that may be confirmed or rejected through scholarly research. In the academic context legal science could be observed through different manifestations of law, namely “law as a practical discipline”, “law as humanities”, and “law as social science” (
As a body of work produced by academics, legal scholarship is in a remarkable and singular position, because not only is it the way to communicate the science of law but is also an influential and authoritative source of law (
Another peculiarity is related to authorship, as legal researchers usually work alone (
In general, research evaluation consists of two parts: qualitative (peer review, prominence of authors, publication’s or publisher’s prestige) and quantitative (number of publications, number of citations). Quantitative “research evaluation” relies on many metric indicators, some of which have prevailed thanks to ease of use, especially in the STEM (science, technology, and medicine) areas. One such indicator–journal impact factor (JIF)—has become an indispensable measure the “quality” of research results published in journals. However, because there are many complex and specific issues to consider, none of the numerous discussions between legal academics has yielded an evaluation system that would be globally accepted within the legal community. Instead, two ways to evaluate research have prevailed, although unresolved problems are present in both. The first is the peer review process to which manuscripts are subjected prior to publication, and the second is the ranking of journals based on citations or peer review.
Still, there is no commonly recognized transnational assessment system that would make it possible to benchmark and rank law journals and other legal science outputs (
The field of law has retained its distinctiveness regarding peer review to this day, and reviews are often conducted without following standardized rules and principles. In addition, the number of journals has consistently increased, and more and more papers are published in journals with low level of critical assessment. In most European countries evaluation of legal scholarship relies on both editorial and independent/external peer review. This co-existence reflects the lack of a general agreement about quality indicators and the lack of a lingua franca that would expand the audience and scrutiny beyond national borders (
For specific branches of law and for topics limited to the national context (e.g., taxes, employment procedures) the legal scholarly community might be so small that anonymized peer review makes no sense because colleagues would still recognize each other’s work. For this reason, blind peer review is rarely used in some countries. While involving reviewers from abroad might provide a solution, foreign reviewers may find themselves at a disadvantage if they do not know the language or specific national topics (
Like in other disciplines, peer review in the law domain entails no institutional recognition or reward in terms of academic appointments and promotion. Various studies show that peer review is slow and inefficient, expensive, ineffective (in detecting errors, plagiarism, and fraudulent research), biased, opaque, easily abused, unreliable, and without incentive (
Yet, its deficiencies and weaknesses do not diminish its value and relevance. Instead, they call for changes and more innovative approaches (
Publishing in peer-reviewed journals not only benefits researchers for their professional advancement but also improves the image of academic institutions and disciplines, as scholars generally believe that peer-reviewed journals produce quality science (
Transparency is also in the focus of our study. We wanted to see to which extent peer-reviewed Croatian, Italian, and Spanish law journals are transparent in their editorial policies by asking ourselves four questions: where is the their peer review process information located, how are the elements of the peer review process addressed, what evaluation criteria are reviewers asked to follow, and what ethical concerns do law journals mention in their documents on peer review?
We began research on various aspects of legal science assessment in 2018 as part of the COST Action ENRESSH–European Network for Research Evaluation in the Social Sciences and Humanities, which ended in February 2020 (
Journals to analyze were selected in two steps. In the first step, we limited our research to law journals from Croatia, Italy, and Spain to cover small and large European countries in which English is not native language, yet publish much of their legal research in English. In the second step we selected journals with a sufficient level of transparency of editorial policy which declare themselves peer-reviewed in publicly available documents.
Croatian law journals were selected from the HRČAK Portal of the Croatian Scientific and Professional Journals
Then we ran a pilot study collecting only instructions to reviewers (ItR) and peer review policies (PRPol) but found that ItRs mostly did not contain information on the peer review process, and only a few journals had publicly available PRPol. This is why we had to expand research to all the most recent versions of every publicly accessible document issued by the editorial board mentioning peer review for each journal. The documents were taken from journal or publisher websites, national journal databases, and where no document was found, from the content of a recent issue. Altogether we collected 135 documents in different formats (PDF, DOC, and HTML) with information on the journal title, publisher’s country, ISSN, format, language, document URL, and document title (
The documents were then categorized by where the information on peer review is located, as follows: instructions to authors (ItA), instructions to reviewers (ItR), peer review policies (PRPol), review forms (RefForm), editorial policies (EdPol), publishing ethics (PubEth), about the journal webpage (AboutJ), other journal webpages (WP), and notes (Note). Were no English version of the document was available we collected information in the local language (
The 135 collected documents addressing peer review were analyzed using automatic content analysis, in which each document (case) was the analytical unit. For text-mining, content analysis dictionary, descriptive statistical analysis, analysis of co-occurrences, and qualitative insight we used the QDA Miner5 and WordStat8. Additional variables “country” and “language” were defined in the QDA Miner and used to compare categories between the three countries and English and non-English documents. We did not convert the documents to lowercase, since WordStat is case-insensitive. In addition, we used a list of stopwords in English, Italian, Spanish, and Croatian to exclude the most common words from the analysis (
In WordStat we developed a multilevel and multilanguage categorization dictionary in four languages (English, Croatian, Italian, and Spanish) (
A simplified version of the peer review categorization dictionary.
Category | Subcategory | Examples of keywords |
PR_process | Goverance | Editor-in-chief, editorial board, scientific/advisory board, steering committee |
ID_transp | Transparency of reviewer’s, author’s and editor’s identity: visible, anonymous (single, double, triple) | |
Reviewer | Number of reviewers (one, two, three, or more), competencies, objectivity, autonomy, independence | |
PR_criteria | Publ_type | Article (scholarly, professional), review article, case note, essay |
Eval_criter | Design, methods, relevance, appropriateness, theory, writing style, literature | |
PR_outcome | Acceptance (with or without revision), rejection | |
PR_ethics | Coi | Conflict of interest/competing interest |
Confident | Confidentiality | |
EDI | Equity, diversity, inclusion, bias (gender, race, religion, nationality) | |
Res_misconduct | Data fabrication and falsification, text recycling | |
Text_similar | Plagiarism, self-plagiarism, text similarity-check |
Within the categorization dictionary we defined several categories that contained general terms in four languages irrelevant for analysis, namely, “PR_gen” (peer review, evaluation), “eval_cri_gen” (criteria), “reviewer_gen” (reviewer, referee), “submiss_gen” (article, contribution, submission, manuscript) and “ethics_gen”(ethics, ethical). These general categories served to gain insight into the overall representation of the most common terms in the documents and create rules by which to record the presence of more specific categories and keywords. For example, for the “Title” we created the rule @TITLE (TITLE NEAR #SUBMISS_GEN to register “article title”, “title of the paper”, “submission title”, etc., Such general categories are excluded from the tables below, because they are very frequent and irrelevant, but are included in
The spreadsheet with a list of all journals and documents in the sample, a full version of peer review categorization dictionary, all documents in RTF format, and a spreadsheet with all results from WordStat8 are available in the
Descriptive statistics were used to summarize the data, and comparisons between the groups were analyzed with chi-square test. A
Information on peer review is scattered across various types of documents issued by the journal. Most information was found in the instructions for authors, editorial policies, and web pages describing the journal, usually under the link ‘About journal’ (
Documents included in content analysis by type.
ItA | ItR | PRPol | RevForm | EdPol | PubEth | AboutJ | WP | Note | Total | |
Croatia | 21 | 8 | 0 | 7 | 3 | 2 | 0 | 7 | 0 | 48 |
Italy | 7 | 6 | 10 | 0 | 4 | 8 | 3 | 3 | 2 | 43 |
Spain | 11 | 2 | 2 | 1 | 13 | 0 | 15 | 0 | 0 | 44 |
ToTAL | 39 | 16 | 12 | 9 | 20 | 9 | 18 | 10 | 2 | 135 |
ItA: instructions to authors; ItR: instructions to reviewers; PRPol: peer review policy; RevForm: review form; EdPol: editorial policy; PubEth: publishing ethics; AboutJ: about journal; WP: WebPage.
Among the 135 documents, 69 were in English (29 Croatian, 13 Italian, and 27 Spanish documents), 30 in Italian, 17 in Spanish, and 19 in Croatian.
The top three categories are represented as follows: “PR evaluation” with 3,174 mentions (130, 96%), “PR process” with 1,290 mentions (129, 96%), and “PR ethics” with 478 mentions (73, 54%).
In order to find out what organizational and quality elements of the “PR process” are addressed in the documents, we analyzed three aspects: involvement of editorial and other boards and committees in the process, the transparency of identities during the peer review, and preferred characteristics of the reviewers. Accordingly, the “PR process” category in the dictionary includes three subcategories represented differently in the corpus of the collected documents (
Peer review process categories (found in more than 10% of the documents).
Mentions | Documents | ||||
Category (1st level) | Category (2nd level) | Category (3rd level) | Frequency | Number | % |
PR_process | — | — | 1,290 | 129 | 96 |
— | Goverance | 621 | 108 | 80 | |
— | — | ED_board | 392 | 81 | 60 |
— | — | Editor-in-cheif | 136 | 51 | 38 |
— | — | SCI/advisor_board | 59 | 23 | 17 |
— | ID_transp | 332 | 104 | 77 | |
— | — | Anonym | 331 | 104 | 77 |
— | Reviewer | — | 337 | 98 | 73 |
— | — | Independ | 116 | 65 | 48 |
— | — | Competence | 98 | 51 | 38 |
— | — | Objectivity | 58 | 31 | 23 |
— | — | Number | 65 | 29 | 21 |
The journal’s governance could be defined by journal owner (university, learned society), publisher, or journal. It usually involves different boards, committees, and councils with various duties and responsibilities. When it comes to the bodies that manage the journal, there is no uniform terminology in a particular language, nor are there uniform roles and responsibilities related to the peer review process. The most considerable differences in terminology were recorded in Italian journals, less in Spanish, while Croatian journals were uniform regarding the used terminology in both languages.
The main pillars of the journal’s governance are editor-in-chief, editorial board, and advisory board, responsible for the journal’s mission and composed of renowned experts in the fields covered by the journal. Their full names, affiliations, and contact information are usually given on the journal website. Members of the editorial board are primarily responsible for selecting submitted manuscripts for publication. Some journals may require additional assessment by members of the advisory board, which supports the editorial board in academic duties. The journal may also have a board of reviewers and a steering committee.
The editorial board comprises “renowned experts in the fields covered by the journal” (#123, Spain) or “university professors and researchers, magistrates and well-known lawyers” (#73, Italy). Additionally, members of the editorial board should have “research skills, proved and acknowledged by their active and regular participation in the assessment processes of leading national and foreign publications, experience in managing and editorial tasks in other scholarly journals; and full availability in the performance of their duties” (#84, Spain). The editorial board is responsible for “keeping the editorial line of the journal and for adequately running the manuscript selection process” (case 84, Spain), including the peer review process. “Journal’s ‘management approves a list of evaluators external to the bodies of the journal, made up of Italian and foreign scholars particularly qualified in the field of study, whose inclusion in the list can be proposed by each member of the management” (# 62, Italy).
In most law journals, the editor-in-chief and editorial board check whether the submission fits the journal’s scope and its overall quality. The manuscripts whose quality is low or outside the scope may be rejected without external review. Those that pass this first step are sent to external reviewers. When reviewer reports disagree or additional opinion is needed, the editorial board can send the manuscript to another reviewer. Once the editorial board has all reviews, it makes one of the following decisions and communicates them to the author(s): accept without revision, accept with revisions, or reject. The distinction between minor and major revisions was noted in only a few Croatian and Spanish journals.
If appointed, the advisory board usually monitors the development of the journal, suggests new topics and special editions, evaluates journal’s yearly performance, and submits comments, proposals, and complaints to the editor-in-chief, editorial board, or journal owner. Some law journals ask members of the advisory board to make the initial assessment of submitted manuscripts in terms of quality and fitting the journal scope: “All manuscripts are evaluated, first, both by the editors-in-chief and one or more members of the advisory board.” (#74, Spain). Members of the advisory board can also assist in checking and verifying substantially revised manuscripts in terms of journal requirements and suggestions given by external reviewers. Instead of “advisory board” some journals (primarily Italian) use the term “scientific board” to describe similar roles and responsibilities.
Under the “Governance” category, we identified three governing instances involved in the peer review process: the “Editorial board” (81, 60%), “Editor-in-chief” (51, 38%), and the “Advisory board” (23, 17%). The steering committee, board of reviewers, and other boards, councils and committees are referred to in less than 2% of the documents.
The ‘ID transparency’ category, which includes information about peer review being single-blind, double-blind, or open, was referred to 332 times across 104 documents (77%), and all of them spoke about anonymized peer review. “Double-anonymized” peer review was mentioned in 50 documents (37%) and “Single-anonymized” in 21 documents (16%). The single mention of open peer review (visible identities), not represented in
In our study, reviewer’s “Independency” was the most represented subcategory in the “Reviewer” category (65, 48%), followed by “Competencies” (51, 38%). A deeper look into the documents reveals that reviewers are sometimes recruited from advisory, scientific, reviewer, or even editorial boards. Some journals recruit “external academics not belonging to the editorial board” (#81, Spain) or “experts in the field external to both to the Editorial Board and the Advisory Board” (#122, Spain). When declared, “independence” most often means independence from a journal and its bodies and less often from an institution or publisher. Less represented are keywords describing reviewer’s “Objectivity” (31, 23%). Two reviewers per manuscript are the prevalent model (24, 18%), and “Three and more” are less common [(8, 6%) and (15, 11%) respectively]. Even though a reviewer is expected to keep manuscript information and ideas confidential in closed and blinded peer reviews (
Typically, manuscript evaluation relies on specific journal criteria and refers to peer review outcomes, represented accordingly by the categories “Evaluation criteria” and “Peer review outcome” (
Law journals welcome a variety of “genres”, and review practices differ with manuscript types. In some law journals all submissions undergo peer review, and in others only manuscripts with scientific content are sent to external reviewers. Subject to peer review are usually scholarly articles, professional articles, preliminary communications, review articles, essays, case comments/notes, and legislative comments/notes. Peer review is not always mandatory with regional observations, expert reviews, forums, and conference papers.
We included twelve publication types in this category: article (scholarly and professional), book review, preliminary communication, note, essay, conference paper, review paper, debate, experience, conference review, critical discussion, and comment. The most popular category, “Article” (68, 50%), has two subcategories: “Scholarly article”, which is more common (63, 47%), and “Professional article” (35, 26%). Law journals use the attribute “scientific,” “scholarly,” and “academic” for journal articles interchangeably to denote “the paper which is characterized by originality of conclusions, or which presents previously unpublished original results of scientific research” (#5, Croatia). “Book review,” “Preliminary communication” and “Note” share a 20% representation, give or take, while “Essay” (19, 14%), “Conference paper” (18, 13%), “Review paper” (14, 10%), and “Debate” (13, 10%) are less common (
Peer review evaluation criteria categories (found in more than 10% of the documents).
Mentions | Documents | ||||
Category (1st level) | Category (2nd level) | Category (3rd level) | Frequency | Number | % |
PR_evaluation | — | — | 3,174 | 130 | 96 |
— | Publ_type | 520 | 103 | 76 | |
— | — | Article | 244 | 68 | 50 |
— | — | Book_rev | 36 | 27 | 20 |
— | — | Pre_comm | 42 | 26 | 19 |
— | — | Note | 53 | 26 | 19 |
— | — | Essay | 41 | 19 | 14 |
— | — | Conf_paper | 51 | 18 | 13 |
— | — | Rev_paper | 16 | 14 | 10 |
— | — | Debate | 18 | 13 | 10 |
— | Eval_criteria | 2024 | 122 | 90 | |
— | — | Relevance | 709 | 113 | 84 |
— | — | Writing and present | 521 | 94 | 70 |
— | — | Ref_liter | 518 | 93 | 69 |
— | — | Method and stat | 148 | 57 | 42 |
— | — | Theory | 82 | 34 | 25 |
— | — | Design and concept | 30 | 18 | 13 |
— | PR_outcome | — | 630 | 110 | 81 |
— | — | Acceptance | 517 | 108 | 80 |
— | — | Rejection | 113 | 53 | 39 |
Journals are dedicated to publishing high-quality articles on legal scholarship that can be read by a large community of practitioners and the general public, and peer review serves to reach these goals. For the review process to ensure quality evaluation, journals should provide clear and complete evaluation criteria to authors, reviewers, and editors, in line with the journal’s scope and mission. To define categories for ‘PR evaluation’, we relied on the Bornmann, Nast, and Daniel study (2008) that identified 572 criteria and grouped them into nine main categories: 1) “relevance of contribution”, 2) “writing/presentation”, 3) “design/conception”, 4) “method/statistics”, 5) “discussion of results”, 6) “references to the literature and documentation”, 7) “theory”, 8) “author’s reputation/institutional affiliation”, and 9) “ethics”. We modified the “relevance” category by adding the “Originality”, “Appropriateness”, “Contribution to practice”, and “Significance” subcategories. To ‘Writing&Presentaton’ we added the “Clarity”, “Accuracy”, “Publication Guidelines”, and “Structure&Layout” subcategories. The last was further divided into “Discussion”, “Title”, “Abstract”, “Introduction”, and “Conclusion”. We also added the “Replicability” subcategory to “Design/Concept”. Being a category
The “Evaluation criteria” category was identified in 90% of the documents, including subcategories (
Other well-represented categories were “Writing&Presentation” (95, 70%), “References and literature” (93, 69%), and “Methods and Statistics” (58, 43%). Other categories in the “Evaluation criteria” were found in less than 50% of the documents. For example, “Publishing guidelines” were mentioned by 14 documents (25%) only.
Peer review outcome in the applied taxonomy contains two categories, “Acceptance” (108, 80%) and “Rejection” (53, 39%) (
“The main criteria for acceptance are methodological rigour and consistency, text structure and framework; scientific originality and relevance; the validity of the reasoning supporting the central thesis; accuracy/completeness of sources and bibliography.” (#51, Italy).
We did not include the concepts of minor and major revisions in our corpus, even though some journals describe the process of manuscript revision exhaustively:
“If the manuscript has been accepted with modifications, the authors will have to resubmit a new version of the article, taking into account the demands and suggestions of the external evaluators. If they wish, the authors can also provide a letter to the Editorial Board to indicate the content of the modifications to the article. Articles with important corrections may be sent to the Advisory Board to verify the validity of the modifications made by the author. Depending on the degree of compliance with the requested modifications, the Advisory Board will decide whether or not to publish the article. This decision will be communicated to the author by the director of the journal.” (#124, Spain).
Legal scholarship has adopted peer review as a cornerstone of modern legal science. It relies on legal experts to deliver objective, constructive, and consistent evaluation intended to help editors make an informed decision and authors to improve their manuscript. Still, peer review can be slow, expensive, inconsistent, biased, unreliable, unable to detect errors, and abused (
Studies show that peer review in SSH can be the source of all sorts of biases arising from differences between reviewer’s and author’s affiliation, nationality, language, gender, race, religion, interpretation, ideology, and level of conservatism (
Our analysis included six ethical issues: conflict of interests, research misconduct, (non)adherence to ethical guidelines, equity/diversity/inclusion (EDI), plagiarism, and confidentiality. The World Association of Medical Editors (WAME) suggests that editors should address conflict of interest as follows:
“Conflict of interest exists when a participant in the publication process (author, peer reviewer, or editor) has a competing interest that could unduly influence (or be reasonably seen to do so) his or her responsibilities in the publication process (submission of manuscripts, peer review, editorial decisions and communication between authors, reviewers and editors).” (
Detecting research misconduct, like data falsification or fabrication, could be one of the most challenging tasks for reviewers. However, the burden of upholding the journal’s ethical standards rests on editors’ shoulders, and they often rely on guidelines such as those issued by the Committee of Publication Ethics (COPE) or International Committee of Medical Journal Editors (ICMJE).
Of the three top categories represented in law journals “Peer review ethics” ranked the lowest (74, 54%).
Peer review ethics categories (found in more than 10% of the documents).
Mentions | Documents | ||||
Category (1st level) | Category (2nd level) | Category (3rd level) | Frequency | Number | % |
PR ethics | — | — | 478 | 73 | 54 |
— | Coi | 132 | 43 | 32 | |
— | Text_similar | 85 | 35 | 26 | |
— | Edi | 111 | 30 | 22 | |
— | Eth_guid | 64 | 28 | 21 | |
— | Res_misconduct | 70 | 25 | 19 |
As for the top ranking category, some journals clearly state that it is not acceptable for a reviewer to participate in the same project, to mentor, or otherwise be involved with the manuscript under review.
“Reviewers are required not to accept for reading articles for which there is a conflict of interest due to previous collaborative or competitive relationships with the author and/or his/her institution.” (76, Italy).
By accepting to review, the reviewers implicitly declare no such conflict of interest.
To compare possible differences between the three countries, we used the Crosstab feature of WordStat8, which tabulates case occurrences across the “country” variable. The difference in case occurrences was assessed with the χ2 (chi-squared) test, and the threshold of statistical significance was set at
Distribution of categories identified in at least 50% of the documents by country (Croatia 48, Italy 43, Spain 44).
Category (1st level) | Category (2nd level) | Category (3rd level) | Category (4th level) | Croatia | Italy | Spain | Chi2 | P (2-tails) |
Number of documents | ||||||||
PR process | — | — | — | 44 | 42 | 43 | 2.653 | 0.265 |
— | Governance | — | 38 | 19 | 35 | 16.692 | <0.001∗∗∗ | |
— | — | Edboard | 38 | 9 | 34 | 40,166 | <0.001∗∗∗ | |
— | ID_transp | — | 28 | 35 | 41 | 16.437 | <0.001∗∗∗ | |
— | — | Anonym | — | 28 | 35 | 41 | 16.437 | <0.001∗∗∗ |
— | — | — | Double anonym | 14 | 10 | 26 | 13.954 | 0.001∗∗ |
— | Reviewer | — | 27 | 33 | 37 | 9.545 | 0.008∗∗ | |
— | — | Independ | 11 | 19 | 35 | 29.884 | <0.001∗∗∗ | |
— | — | Competence | 8 | 19 | 23 | 13.863 | 0.001∗∗ | |
PR evaluation | — | — | 45 | 41 | 44 | 2.673 | 0.263 | |
— | Publ_type | — | 40 | 24 | 27 | 8.904 | 0.012∗ | |
— | — | Article | — | 41 | 9 | 15 | 42.946 | <0.001∗∗∗ |
— | — | Preliminary comm. | 26 | 0 | 0 | 58,366 | <0.001∗∗∗ | |
— | Eval_criter | — | 42 | 39 | 44 | 5.56 | 0.062 | |
— | — | Writing and present | — | 36 | 24 | 34 | 5.751 | 0.056 |
— | — | Relevance | — | 39 | 32 | 42 | 7.384 | 0.025∗ |
— | — | — | Originality | 38 | 28 | 40 | 8.596 | 0.014∗ |
— | — | Ref_liter | — | 35 | 25 | 33 | 3.448 | 0.178 |
— | — | Method and stat | — | 25 | 14 | 18 | 3.591 | 0.166 |
— | PR outcome | — | 40 | 32 | 38 | 2.226 | 0.329 | |
— | — | Acceptance | — | 37 | 19 | 30 | 11.182 | 0.004∗∗ |
— | — | — | Revision | 20 | 29 | 27 | 6.806 | 0.033∗ |
— | — | Rejection | — | 13 | 11 | 29 | 19.462 | <0.001∗∗∗ |
PR ethics | — | — | — | 21 | 21 | 30 | 6.018 | 0.049∗ |
Note: ∗∗∗<0.001, ∗∗<0.001, ∗<0.05.
The three countries show similarities in the distribution of “PR process” and “PR evaluation” first level categories. However, distribution starts to differ significantly with the second, third, and fourth level categories. Croatian and Spanish journals refer to the editorial board more often than Italian (
The “PR evaluation” category shows that publication types “Article” (
Ethics too seems to be closer to the heart of Spanish law journals, research misconduct in particular (
Between the documents in English (69) and those in national languages (non-English, 66), statistically significant differences were recorded in the “Article” (
Distribution of categories identified in at least 50% of the documents by document language (English 69, non-English 66).
Category (1st level) | Category (2nd level) | Category (3rd level) | English | Non-English | Chi2 | P (2-tails) |
PR process | Governance | Edboard | 53 | 53 | 0.244 | 0.621 |
PR process | ID_transp | Anonym | 49 | 55 | 2.894 | 0.089 |
PR evaluation | Publ_type r | Article | 44 | 24 | 10.134 | 0.001* |
PR evaluation | Eval_criter | Writing and present | 53 | 42 | 2.808 | 0.094 |
PR evaluation | Eval_criter | Relevance | 64 | 49 | 8.474 | 0.004* |
PR evaluation | Eval_criter | Ref_liter | 51 | 42 | 1.662 | 0.197 |
PR evaluation | PR outcom | Acceptance | 58 | 50 | 1.453 | 0.228 |
PR evaluation | PR outcome | Rejection | 36 | 17 | 9.871 | 0.002* |
Note: *<0.05.
Everyone agrees that journal editorial policies should be accessible, clear, and transparent to authors, reviewers, and readers alike (
“Journal content must be clearly marked as whether peer reviewed or not. Peer review is defined as obtaining advice on individual manuscripts from reviewers experts in the field who are not part of the journal’s editorial staff. This process, as well as any policies related to the journal’s peer review procedures, shall be clearly described on the journal website, including the method of peer review used.” (
For research journals it is essential to provide evidence of editorial policies that foster the quality of published content (
Finding information about peer review in law journals of the three countries was not that easy. It was scattered across different documents with different titles in different sections of journal websites. While most Croatian journals provide information on peer review in instructions to authors, Spanish journals provide it with journal description, and Italian journals provide it in separate documents specifically dedicated to peer review policy. Some journals have an additional online version, hosted by a repository or a publishing platform, which often organizes information about review transparency and editorial policy differently. Information given in English often reflects the desire to reach international audience and improve journal’s visibility and impact, even if the articles are published in the local language only. Journals that provide this information in local language alone, as many Italian journals do, show their national orientation.
Law journals in all three countries perceive anonymous (primarily double-blind) peer review as the necessary guarantee of the journal’s highest scientific quality. Despite increasing demands for open peer review as one of the core principles of open science, the editors of the law journals in our sample all prefer the blinded option. Similar to the
Some journals in our sample require authors to anonymize their manuscripts before submission and remove all identifying information, including references to the institution or project the manuscript is associated with, personal information in the file title or file properties, and bibliographic references which may identify the authors. Some journals even ask reviewers to contact the editorial board if they learn author identity by chance. We also noticed that some journals misinterpret double-anonymized peer review as the one in which two reviewers do not know each other’s identity. One journal in our sample misinterprets ‘open peer review’ as a single-anonymized peer review, because author identity is not hidden.
As a rule, law journals in our sample expect that reviewers are competent experts in the subject matter of the manuscript they are invited to evaluate, but not all invite independent/external reviewers. External review is more often requested by Spanish and Italian than Croatian journals. Also, only certain types of manuscripts are subject to peer review. Some law journals list types of manuscripts that are not sent to external review, and these are usually book reviews, doctoral dissertations, and conference papers. Instead, such manuscripts are accepted or rejected by the editor-in-chief and/or the editorial board. Such submissions have an additional time constraint of 2 years after the book publication date, conference date, or dissertation defense date, which seem unjustifiably long (who is interested in a conference paper after 2 years?). Surprisingly, some journals in our sample do not require peer review for papers authored by a professor emeritus or a recognized expert in a relevant position such as constitutional judge. Some law journals also seem to depart from their proclaimed peer review policies in practice.
Editors of smaller and less prestigious journals seem to rely more heavily on reviewers to categorize submissions by type, but the final decision remains with the editorial board. Such practice is common in Croatian journals. Once the paper is published, its category will be listed in the heading and table of contents. Decisions on manuscript type are not relevant only for journals but also for authors and their academic careers and could significantly impact journal metrics. Different types of papers are evaluated differently by academic institutions in terms of career advancement, which may be the main reason why editors ask reviewers to assist them in this matter.
Ethical issues are undoubtedly under-represented in documents describing the peer review process and editorial policies issued by Italian, Spanish, and Croatian law journals. Although it is essential to disclose possible competing interests that might prevent a reviewer from providing a fair and unbiased peer review, only one-third of the documents mention conflict of interests, and all other ethical concerns are even more neglected, although it is crucial for reviewers to remain unbiased towards author nationality, religious or political beliefs, gender, ethnicity, or geographical origin. Some journals pursue balanced gender representation in their editorial boards and reviewer pools. Low representation of research misconduct (e.g. fabrication or falsification of data) in policy documents may suggest that this is not an issue in legal scholarship as opposed to some other disciplines like biomedicine. This assumption, however, is yet to be verified or dismissed. Even so, editors of law journals should do more to check manuscripts for similarity and to preserve confidentiality.
There are several internationally prominent publishing and ethical guidelines created to improve the quality of scholarly publishing such as the EASE Guidelines for Authors and Translators of Scientific Articles to be published in English
Thanks to statistical analysis we have observed some significant differences in trends between the three countries. For example, Italian journals refer to the editorial board in their peer review documents far less often than the journals from the other two countries. One reason may be that peer review in these documents is addressed separately, while Croatian and Spanish journals tend to put both review and editorial roles together in the same document. We also noticed that, although journals in all three countries prefer traditional anonymized peer review, Croatian journals refer to author, reviewer, or editor anonymization significantly less often. Spanish journals also seem to hold reviewer independence and competence significantly more dear than Croatian journals.
Interesting differences were also observed in the “publication type” category. Croatian journals mention original scientific papers in their documents much more often than the journals from the other two countries. The difference is even more significant in preliminary communications, which Italian and Spanish journals do not mention at all in their documents. This may be owed to the journal subsidy evaluation criteria issued by the Croatian Ministry of Science and Education, which favor original scientific papers, review papers, and preliminary communications.
Croatian and Spanish journals insist on the originality of submitted manuscripts much more than Italian journals. Such manuscripts must not be previously published and must provide an original contribution to the field of law. The high prevalence of this term may arise from cultural distinctions, as what is termed an “original scientific work” in Croatia is simply termed a “research article” or simply an “article” in English-speaking countries.
Mentions of manuscript acceptance are significantly less frequent in Italian law journals, while mentions of manuscript rejection are significantly more frequent in Spanish journals. Spanish journals also refer to research misconduct, including fabrication, falsification, and manipulation of data and scientific fraud significantly more often than Croatian and Italian journals.
Although we found statistically significant differences for three categories (“Article”, “Rejection”, and “Relevance”) between documents in English and local languages, this does not mean that internationally oriented journals (those which publish their editorial policies in English), have adopted more advanced and globally accepted standards and peer review policies than local journals. Finally, the most important question is how to improve peer review in law journals. Some progressive ideas and experiments have already been described in the cited literature. Peer review can also be improved through contributions of editors to discussions about publishing and ethical standards, standardization of the peer review process in particular. A step further would be to make peer review open (unblinded and/or public), to incentivize and reward reviewers, to train editors and reviewers, and to provide a platform for efficient communication between authors, reviewers, and editors.
In legal science, peer review is considered the guardian of publication quality, or at least “the worst way to assess research, except for all the others” (
Our general finding is that law journals in our sample prefer double-blind peer review managed by the editorial board and require a high level of expertise from reviewers. Traditional approach is also characteristic for all other elements and there is little room for innovative approaches. Criticizing anonymous peer review in great many scholarly journals from all disciplines, the editor of the British Medical Journal once argued that “a court with an unidentified judge makes us think immediately of totalitarian states and the world of Franz Kafka” (
We were not surprised by the great variety of genres (article types), as legal scholars publish both peer-reviewed scholarly papers and several shorter and less formal types of articles specific to the field of law that are valued by their colleagues, but in view of this variety we were somewhat surprised to find that most law journals value manuscripts by relevance (originality), writing and presentation, and exhaustive and accurate referencing.
In terms of publication ethics, most law journals address conflict of interests, while other aspects such as transparency, plagiarism, best practices, bias, and research integrity are less represented and deserve more attention in future research. Law journals have yet to tackle more complex ethical issues specific to different areas of law. The most advanced in that respect are the Spanish journals.
Future research should also address other issues raised by our findings, such as the lack of diversity in editorial procedures across countries and reluctance to adopt innovative approaches to peer review. In fact, our findings could serve as an incentive to law journals to make their editorial policies more transparent and comprehensive. Future research could also compare actual review practices with the declared ones through qualitative evaluation of various stakeholders in the editorial process, including editorial boards, managing editors, publishers, authors, and reviewers.
With the multilingual peer review taxonomy developed for this research, it is also possible to include journals/documents from other countries and languages and increase the corpus to obtain more reliable and statistically more relevant analysis.
Our analysis is limited to publicly available documents and therefore does not provide a more comprehensive picture that would include some closed commercial law journals and their practices. Furthermore, it is possible that some journals included in this study have restricted access to more detailed information on peer review (such as instructions to reviewers) to reviewers who have accepted their invitation to review submissions. Some journals provide instructions to reviewers and peer review policy with evaluation criteria only through their editorial management systems, and these documents are not publicly available on journal websites.
Another limitation may be the design of our categorization dictionary. Is it intuitive and comprehensive enough? We could not find any validated instruments for our study besides the STM Standard Taxonomy on Peer Review, which is focused on four elements of the process (i.e. how transparent is the identity of those involved, with whom does the reviewer interact, what information about the review process is published, and whether post-publication commenting takes place) and on peer review evaluation criteria published by (
Another limitation could be that we used the number of mentions, documents, and shares in the total sample as a proxy for keyword relevance, which could be an imperfect solution for our corpus consisting of a small number of relatively short documents and could have affected our statistical analysis. Our future research will therefore expand to law journals from other countries.
The original contributions presented in the study are included in the article/
JS ES-C, TA, and GP designed the study, improved the methodology used in previous studies, collected the data, and created taxonomy. JS performed the content analysis. JS and GP drafted the manuscript. All authors contributed to manuscript revision, read, and approved the submitted version.
This work was supported by the national research project of R&D&i RTI The Role of Open Science in Spanish Universities: Institutional Transformation and Anticipatory Governance (ROSSUE, PID 2019-104052RB-C21) financed by the Spanish Ministry of Science and Innovation. The contribution of JS was partly supported by the H2020-INFRAEOSC-2019-1 project Transforming Research through Innovative Practices for Linked Interdisciplinary Exploration (TRIPLE, ID: 863420).
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
The authors would like to express their special gratitude to the European Network for Research Evaluation in the Social Sciences and the Humanities (ENRESSH) Cost Action CA15137 for funding preliminary studies through its Short Term Scientific Missions (STSM). We would also like to thank Dado Čakalo for help with language and the narrative. We are also grateful to the reviewers for their valuable feedback on our manuscript.
The Supplementary Material for this article can be found online at:
List of journals and collected documents with necessary metadata new version is uploaded.
Wordstat8 data at category and country level.
Categorization dictionary (Excel spreadsheet).
Categorization dictionary (CSV).
All collected documents on editorial policies related to the peer review process.
SCImago (n.d.). SJR—SCImago Journal and Country Rank (Portal). Retrieved August 13, 2021 from