Innovating peer review, reconfiguring scholarly communication
Toward an evidence-informed understanding of innovations in peer review
This post is by Johanna Brumberg, Helen Buckley Woods, Wolfgang Kaltenbrunner, Stephen Pinfield and Ludo Waltman (of Research on Research Institute (RoRI))
Peer review is generally seen as one of the cornerstones of the scholarly publishing system. As such, it is also constantly subject to many simultaneous attempts to improve its perceived effectiveness, objectivity, transparency, and long-term sustainability.
Some of the more well-known instances of such innovation in recent decades are initiatives to deanonymize reviews and publish them alongside articles (so-called open or transparent peer review), preprint reviewing, and systems to formally acknowledge and thereby incentivize review labour. The actual range of ongoing innovation activities, however, is significantly broader, and there is sometimes considerable diversity in how innovating actors who pursue ostensibly similar projects design the exact means necessary to achieve certain types of change.
In an effort to provide a condensed overview of a sprawling field of activity, we – a group of scholars collaborating in the Research on Research Institute (RoRI) – have recently completed two studies that we hope will be useful to a broad range of readers from across the scholarly publishing ecosystem. One is a review of existing review articles of peer review innovations. The number of review articles have increased sufficiently to become an interesting meta-genre in which authors advance a broad range of problem diagnoses and implicit future visions of the scholarly publishing system. In our meta-summary of innovations, we identified three high-level categories: approaches to peer review, reviewer focussed initiatives and technology to support peer review.
Another is an empirical paper analyzing the results of a survey about ongoing innovation projects among publishers, academic journal editors, and other organizations in the scholarly communication ecosystem. The survey, announced in an earlier post on this blog, resulted in a dataset of 95 self-defined innovations. In our paper, we break down this material according to an inductively developed taxonomy consisting of five main dimensions. For example, what is the object of review in a given project? How are reviewers recruited, and how does a given project handle questions of transparency/openness?
The paper stands for itself in providing an analytical inventory of ongoing innovations in peer review – and as we hope, in a way that complements ASAPbio’s excellent ReimagineReview initiative, a registry of platforms and experiments innovating around peer review of scientific outputs. Yet for us as scholars of scientific publishing and peer review, it is only a first step in better understanding our research object. Perhaps the most important insight for us has been a sense of the tensions between simultaneous thrusts of innovation. For example, several initiatives aim to make peer review more efficient and less costly, while other initiatives aim to promote rigour of peer review, which is likely to increase the cost. Innovations based on a singular notion of “good scientific practice” in turn seem to be at odds with more pluralistic understandings of quality in scientific work. The idea of transparency in peer review – often seen as a remedy for gate-keeping and cronyism – is the antithesis to the notion that objectivity in peer review requires anonymization.
We are not surprised to find friction among the initiatives per se, since scholarly publishing practices are highly diverse across fields and thus can be expected to require a degree of tailoring on the part of peer review innovations. Nevertheless, the above charted fault lines suggest that the very success of individual innovations could directly undermine or go at the expense of other undertakings. Conversely, many initiatives would seem to offer opportunities for synergy if only the necessary connections among actors could be made.
It is exactly such challenges that we aim to pursue in the next phase of our activities as RoRI, starting in February 2022. Our initial insights about tensions and synergies between innovation activities will ideally become the basis for more targeted research projects co-organized with relevant stakeholders. We expect such projects to further a solid evidence-informed understanding of ongoing innovations in peer review. Our experiences in conducting the survey have left us impressed by the generosity and creativity of the many publishers, infrastructural organizations, and journal editors with whom we interacted in conducting the survey. We look forward to building on and deepening these interactions as we go along, and of course to extend the range of our collaborators. In this spirit, we very much welcome any feedback and/or ideas for collaboration – scholarly as well as in practical terms – from readers of this blog!
Feedback, suggestions and questions are welcomed – contact Wolfgang Kaltenbrunner, firstname.lastname@example.org