Fair and open evaluation may call for temporarily hidden authorship, caution when counting the votes, and transparency of the full pre-publication procedure

The Problem of Imbalance

The traditional system of manuscript evaluation has created a certain imbalance bordering with unfairness: while the authors of submitted papers typically have had their identity disclosed already at the outset, the reviewers have remained mostly anonymous. With a new open evaluation system being currently envisaged, the main difference would be that evaluators become disclosed as well, which is a significant step toward balance and fairness. More openness and constructive interactivity in the reviewing process have become to be practiced increasingly more, including some noteworthy success (e.g., the Shepherding system at the European Conference on Pattern Languages of Programming and Computing and the Frontiers initiative). However, while both the authors and the reviewers have become disclosed, the past collective “instincts” and traditions of reviewers as evaluators could often remain unaffected by this change. Thus, if the new envisaged evaluation system fully discloses both the authors and evaluators, it nevertheless cannot tackle all potential sources of bias and unfairness. Although for the majority of researchers this system seems to be suitable, there are also researchers who feel that not all possible sources of unfairness would be eliminated. Therefore there should be also an additional optional format of review and publishing that goes even further in pursuit for minimizing the impact of subjectivity. Why so?

The Sources of Unfairness

First, de facto scientific policies have always featured certain elements of paradigmatic power-structure, impact of authority, regional interests, “pecking order.” This does not necessarily constitute a bias or animosity toward particular people, but rather a negative bias against “alien” theoretical approaches and positive attitudes in adhering to traditional views or views of the most authoritative scientists. Second, the history of personal relationships between authors (and/or their colleagues) on the one hand and evaluators (and/or their colleagues) on the other hand, may prejudice the whole process. (This includes an earnestly perceived but non-deliberately distorted understanding of the papers and views.) Third, as some field or tradition of research may be willy-nilly in a stage of stagnation, new ideas and approaches can be almost collectively resisted and negatively evaluated. Therefore, it is advisable to adopt two additional, even if not universally implemented, formats of evaluation of the written work. (1) Keeping the identity of author(s) undisclosed for up to a year post-publication (with later disclosure) if the author(s) wish so. This should diminish the author’s identity-based negative biases. (2) As science is inherently paradigmatic and because a large number of evaluators are inevitably accustomed to the currently prevailing paradigms, weighing votes or numbers of positively or negatively valenced evaluations can often be biased toward reactionary or conventional views. This weighing style of evaluation may also result in an opposite bias of (sometimes) undeserved praise and highly positive rating of dull or non-innovative works deriving from scientific–political influences and habits. Both of these biases should also be counterbalanced in the new envisaged evaluation system.

Possible Remedies and Design

In order to alleviate the above-mentioned problems, in case of each paper submission and the weighing procedure the following principles could be adopted. A set of concomitant open writings of evaluation are published in the finalized issue of the periodical together with the main article. Similarly to what has been practiced by Behavioral and Brain Sciences (BBS, Cambridge University Press) these evaluation papers may be accompanied by the authors’ reply and counter-criticism. Furthermore, the relatively informal pre-publication stage of preparing a paper and its comments should be transparent – all interested and involved parties can access all the submitted main-article manuscripts as well as all review/evaluation papers. In other words, the full portfolio of submissions by professional authors and full set of reviews should be transparent and made available for the scientific community. (The currently available electronic means help to overcome the endangering capacity problems.) Seeds of this format have been planted already by such outlets as BBS, Interdisciplines (supported by OpinioNet and LiquidPub), and some others. The open review could also adapt the format suggested by Lee (2011) in his Selected-Papers Network model: reviewers can endorse a paper for publication and also publish a concomitant review. After some critical time has elapsed, the unpublished submissions and reviews will eventually be removed from the public domain if authors wish so, but may also remain accessible under the label “unpublished.” The original timeframe with full disclosure has made it possible to copy the pre-publication versions of main papers and critical evaluative papers by all professionals interested anyway. It should be allowed, where necessary, to cite also the unpublished but temporarily accessible “pre-publication” works and data included there. How could this vision relate to the central design decisions when constructing a new system for open evaluation? The backbone of the procedure could look something like this:

pre-acceptance screening > open review > (non)acceptance > publication/closure > post-review.

Pre-acceptance screening. Some minimum screening for the obviously non-professional or mocking contributions or technically/formatively clearly non-conforming works is applied. This is a non-transparent step 1, based on editorial decision.

Reviewing. Step 2 marks the beginning of a review process, which in turn means a fully transparent display of both the complete submitted material as well as a full set of comments by reviewers and editors. As for the alternative metrics (e.g., paper downloads), I suggest not using this as a standard procedure in the reviewing stage. It would burden the already voluminous body of text in the evaluation treatment; furthermore, downloads are heavily biased by non-substantial factors such as journal rankings, visibility, and influence of authors, etc. Downloads data could be made accessible at request, not attached/displayed by default.

Therefore, it is important to guarantee that scientific objectivity prevails and political motivations are minimized. (i) Papers become published after minimal review, more thorough post-review and criticism follows publication. Criticism is highly professional and well-informed allowing for substantive commentary elements just as an old Estonian proverb suggests – the wolves are fed and the sheep alive. (ii) Special explicit sections or footnotes in the form of a short commentary regarding the views and theories that be in question, why so, and with what implications are advisable. (iii) The system should resort to transparent signed reviews and ratings. On the other hand, about 1/4 of future open-access journals could remain “traditional” in terms of anonymity of reviewers if they wish so. The authors can choose the type of journal they wish to be published in. (iv) Evaluation may continue in the post-publication review phase and for a considerable length of time (e.g., with promising or controversial papers, papers with possibly controversial or limited results), but need not. (v) Ratings should be used only if differentiated and specific enough – e.g., novelty of interpretations/theory, technical quality, methodological advances, discovery status, creativity of ideas, etc. Ratings should not be automatically revealed together with a paper, but only accessible at request by readers.

(Non)acceptance is step 3 followed by publication or closure (step 4). Published papers get their final unique article identification label with specification of volume/issue/pages/web-link/date added to the initial identifier attributed to the manuscript at submission. Unpublished papers keep their unique initial identifier supplemented by the label “closed.”

Post-publication affairs. This stage is optional, depending on evoked interest, potential reviewers’ incentives, new emerging circumstances, etc. In the post-publication evaluative open review (step 5) by the original or new reviewers, formatted as separate brief commentaries, the emphasis in informed comments would expectedly shift more toward refined debates, which remains an open discussion forum for quite long time (unless it dies out). The continuing evaluation should be useful because not everybody who may have something important to say may have seen the paper earlier and because some important evidence and related new results may appear just a bit later. On the other hand, evaluative prioritization and rankings based on downloads statistics etc., allowing readers to compare different papers should be only accessible at the readers’ request, but not publicly displayed. The time period covering months and a couple of years post-publication is too short for real evaluation that would stand the test of time, scientific-political factors and underdevelopment of the field of research may interfere too much with substance, and there are too many reasons for downloads other than that a paper is of really high quality, important, or truly innovative. It is questionable to evaluate the value of a scientific publication by numbers of downloads precisely for the above reasons. Let the citation databases live their separate lives without mixing publishing business with scientometrics. The upon-request post-publication ratings should be differentiated and concrete rather than based on overall general statistics. When considering whether to adopt comprehensive rules or varying formats for defining the evaluation formulae, we should leave some options open. Although publishers (i.e., collectives of scientists) may try to reach a consensus in unification, some other optional instrument should also be developed, e.g., authors may be allowed to evaluate their contribution in terms of ratings along various evaluation scales.

General strategies and specific formats. However, there should be a special publication format optional for use and even recommended to the authors, i.e., to remain anonymous for a year post-publication. The articles are cited for this period authored as temporarily anonymous (anon-temp). As soon as the year has passed authorship disclosure becomes compulsory. Each article, whether in the anonymity stage or post-disclosure stage, has a unique identifier which helps to be certain that the same article is referred to. (It is widely believed that despite attempts to remain anonymous, professional readers can in practice successfully guess the author’s identity. Preliminary information available from conferences, lab visits, previous publications, etc., could make it doubtful to guarantee anonymity. Anonymity also may discourage researchers from taking credit for their achievements and fostering one’s career. All this can be countered by special care in writing an article and optimizing the frequency of opting for one or another type of publication.) Most importantly, this new format of publication may not become a prevailing option, but an outlet especially useful for innovative research and cases where authors feel the need to remain anonymous for the time being and therefore take care in not including disclosing information in their papers.

Aspects of Implementation

How can we efficiently bring about a transition toward the future system? There will be inevitably some period of trial-and-error and perhaps the development should continue even further. However, there are some threats that the new system may not turn out as was originally expected or it could make a mockery out of what was initially envisaged as an aspiration toward fairness, speed, and openness. There tend to be two kinds of scholars – researchers immersed in high-quality top-level research vs scientific administrators and organizers, not so prominent as scientists, but influential in other ways. The former are not eager to devote time to implementing reforms and organizational matters whereas for the latter, reforms and “the so-called reforms” are their natural domain. Consequently, the future evaluation system may not attract many truly informed and complex-free academics as evaluators, but too many fresh post-docs and scientific administrators instead and thus the new system may fail to achieve its goals. An idealist hope is characterized by the following: new open-access periodicals will be managed and the tone set by teams of top-level scientists who are known for their objectivity, generosity, sharp analytical vision, love of creativity, and innovation, with preference for substance rather than nice “packaging,” and possession of wide contextuated knowledge combined with the ability to create new knowledge instead of the mere familiarity with the currently prevailing buzzwords. This group of scientists-by-heart will invite the best papers and the reform will be implemented through “magnetism” toward the highest impact, fast-track publication outlets. Furthermore, can the new journals, in minority among the prevailing earlier system survive the already established environment? It is hard to know and only time would tell whether the actual demand for this format of publishing will help its survival.

There is yet another threat. Under-developed countries with less financial and scientific-political power will have fewer chances to publish and wield influence as their institutions simply have a limited budget. The promise to take this into account is just an excuse and cannot be applied endlessly for financial reasons. Moreover, abandoning publication fees altogether would be embarrassing for the authors or their institution. So, a “promise of discrimination” is lurking behind the open-access, pay-per publication system. It would typically result in a situation where in order to overcome this obstacle, the less prosperous researchers will “sell” their ideas for joint authorship and although it might entail an essentially positive aspect of international integration and co-operation this also means that their scientific production will be controlled from outside of their own environment. As a remedy, I suggest the possibility of dispersing the leading open-access journals’ teams and facilities geographically in terms of choices/appointments of editors, editorial board members and reviewers, IT-facilities servicing a journal, etc. It is also important to avoid the excess of reviewer monopoly such as about three to eight authors having recently been published in a particular paradigm, review most of the submissions anonymously, including the review of their direct competitors. The excess reviewing by currently visibly publishing post-docs should be also moderated because many of them often tend to have too narrow a perspective, knowledge and expertise related strictly to their PhD topic without a broader contextuated knowledge and experience. (This is despite the fact that they tend to be more absorbed by the reviewing process and may be even better in spotting the errors. However, according my own extended experience with younger reviewers and fresh researchers, they tend to lack multifaceted, broad view and sufficient knowledge of the host of earlier published research.) The scope of reading even by several different reviewers may lack sufficient depth. Now, this is precisely the place where a fully transparent pre-publication evaluation system together with the continuing post-publication discussion may have its advantages over the traditional system.

In conclusion, the key proposals introduced above contain the following: pre-publication manuscripts selected for review and the reviewer’s work are both transparent, the reviewer’s identity is disclosed; the author of a paper may remain anonymous; discussion of a paper can continue post-publication; overly critical or overly flattering evaluation can be at least minimally counterbalanced; the author has an option to remain temporarily anonymous post-publication. Measures should be taken against bureaucratizing and politicizing the new review system, the choice of reviewers should not be restricted to junior scientists or “activists,” the new system of review and publishing should be introduced also in the less-developed regions accompanied by lower pay-per-publication costs. Last but not least, the traditional system of journal publishing should not be discarded instead it should be preserved as a viable option.

References

Lee, C. (2011). Open peer review by a Selected-Papers Network. Front. Comput. Neurosci. (in press).

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text