We recently posted a short announcement about the DOI Event Tracker project, linking to two more detailed posts at CrossRef. In this blog post, OASPA Board member Mark Patterson, who currently chairs the DET Working Group, provides a bit more background to the project and how he hopes DET will provide a core piece of infrastructure that will help to accelerate progress in open access publishing.
——————————————————————————————————————————————-
OASPA wants to support and create a vibrant and competitive market for open access publishing that also stimulates innovation. In recent years there’s been good progress towards these goals as reflected by the membership of OASPA. Not only have we seen strong growth in fully open content published by OASPA members, there are also experimenting with business models, different approaches to peer review, and new ways of presenting content.
This is all good, but it’s happening quite slowly, and so OASPA is also on the lookout for areas where members can work together to accelerate open access and innovation. One idea emerged over a year ago at an OASPA board meeting on the topic of article-level metrics and indicators – an idea which rapidly developed into the project called the DOI Event Tracker (DET).
One of our challenges is that OASPA is now quite a mixed bag of : old and new; hybrid and fully OA; mega and micro; radical and conservative; commercial and non-profit; journals and books; and all the scholarly disciplines. This is a sign of progress of course, but it does mean that there are topics where it’s hard for us to reach a consensus. One of the things many of us can agree on, however, is that current methods of research evaluation are deeply flawed.
We seem to be stuck in a system where researchers are judged by where they publish their work, rather than what they’ve actually accomplished and how it affects their field. As outlined in the San Francisco Declaration on Research Assessment, there are lots of reasons to change this. What is particularly relevant for OASPA is that most researchers feel they have to publish in established traditional journals, and that tends to stifle innovation and open access where most of the publication venues tend to be quite new. Publishers as well as scholars seem to be slaves to this antiquated system.
How can we break out of this, so that scholars can explore new publication venues without fearing that their work will be ‘downgraded’ simply because of where they chose to publish it? One approach is to move away from a single journal-based metric – the impact factor – towards indicators (qualitative and quantitative) about the article itself. In a digital environment, when an article is downloaded, bookmarked, written about, commented on, cited etc, all of this can be tracked. The same is true for other research outputs like data, code, blog posts and so on.
One of the pioneers in such article-level metrics was PLOS (disclaimer – I worked at PLOS when article-level metrics were first released), but there are now a host of projects and organisations with article metrics as a focus. It’s a promising area with a lot of potential to support more sophisticated and meaningful approaches to research evaluation. It’s also an area that needs to be approached with much caution – the importance of an individual research output cannot be reduced to a single number (an “article impact factor”). Used with care, however, article-based indicators can provide evidence of different kinds of outcomes, and they can also serve many other purposes such as revealing links between content, people, and other research outputs.
For many reasons then, the OASPA board got pretty excited about a group of members trying to work together on this. If could coordinate the way we gather metrics we would be able to compare and link articles and other outputs within and between journals, across and disciplines in ways that could provide new insights.
Before embarking on a collaborative project like this, we had to establish a couple of core principles. The first was that the data that we gather about articles should be free of access and reuse barriers to the greatest possible extent. The data can then be used to develop new tools and services (some of which might be commercial), but the data will be available for purposes of validation and reproduction in ways that are simply not possible with our current system.
The second principle is that the software itself should be open too. Here the collaboration got off to a great start because PLOS had already released its own article-level indicator software (now called Lagotto) as an open source tool. A recent blogpost by Geoffrey Bilder, Jennifer Lin and Cameron Neylon (all involved in DET) sets out some very useful guidance for the implementation of open scholarly infrastructure, which should be useful as the DET project develops.
Both of these principles are important if we are to avoid the mess that research evaluation has become when it is based on proprietary tools and data. Even though the project has grown quite a bit, and extends beyond OASPA members now, it’s fair to say that there is a strong consensus around the goal to make the data and tools as open as possible.
When we first started discussing the idea of a collaboration around article-level indicators, we were also made aware that CrossRef had been doing some exploratory work with the Lagotto software across journals indexed by CrossRef. We quickly established that the best way to take the project forward was to work with CrossRef and that was how the DOI Event Tracker project was properly initiated around a year ago.
Since then, thanks in particular to the work of CrossRef staff and Martin Fenner from PLOS as well, much progress has been made. The Lagotto software is now handling up to 12 million DOIs, which covers 2-3 years worth of articles across all . In the blog posts referred to in the introduction to this post, Geoffrey Bilder and Joe Wass provide much more detail on the technical status of the project. Suffice to say, that we have reached a point now when the project will shortly be formally raised with the CrossRef board.
Fig 1. DET collects data from a number of sources (left) relating to articles (right) or other research outputs. The data is then available to any interested party and can be used for services and research.
The interest in DET amongst CrossRef members (subscription as well as open access ) has been very encouraging as well, and a connected project using the same infrastructure has also been initiated to aggregate usage data from repositories and other venues.
Finally, we’ve focused our efforts so far on a relatively small subset of resources that can provide data (including Mendeley, WordPress, Wikipedia, EuropePMC, ScienceSeeker). There are some unresolved questions about the extent to which this data can be truly open, but there is one other data source that we also need to consider – citation data. Right now, give this data away to Thomson Reuters and Elsevier, who (with the exception of the journal impact factor) produce some very good services on top of that data. However, DET could itself provide a source of open citation data, upon which any interested party could build new services. Another hope is therefore that DET will encourage more (subscription as well as open access ) to make their reference lists freely available via CrossRef. It’s very easy to do that by simply contacting CrossRef.
If the DET project can evolve into a widely used source of open data about the reach and usage of research articles and other research objects, DET will be an important addition to the forces that are loosening the grip of journal names and impact factors on research assessment. The benefits for research communication, open access and scholarship could be profound.
Interesting. If research data is always assigned a DOI, would DET be able to track this too? Woud this work with the sources being tracked (Mendeley, Wikipedia)?
Yes, very much so. We do envisage that DET could be extended to other research objects, such as data, that have DOIs assigned. We’re starting with articles because that’s such an important use case.