Disruption eruption

Dispute about the validity of a metric for scientific innovation highlights the need for open research approaches in metascience

By Stephen Curry (RoRI) and Ludo Waltman (RoRI/CWTS)

Photo: Anat Zilkar via PikiWiki

Science has steadily lost its capacity to produce “disruptive” or ground-breaking ideas. This is the conclusion drawn by Michael Park, Erin Leahy and Russell Funk in a 2023 Nature paper reporting their analysis of trends in discovery and innovation over recent decades. But is it really true?

Park and colleagues used a citation-based metric referred to as the consolidation-disruption (CD). The metric itself seems to make intuitive sense. It is based on a reasonable assumption: a paper (or patent) that is cited by future papers (or patents) which don’t also cite its references is judged to be disruptive since it appears to have broken away from prior work; by contrast a paper (or patent) that is cited in future work alongside some of its own references is judged to be a more incremental consolidation of previous studies. The CD index is constructed from a relatively simple calculation reflecting these citation patterns that confines the indicator to a range between +1 (maximally disruptive) and -1 (maximally consolidating).

By tracking the decline in the average CD index of the corpus of research published between 1945 and 2010, Park et al. reported that papers and patents were becoming dramatically less disruptive, dropping by around 90% to values close to zero across various disciplines.

If this is correct, the reasons for the decline in the capacity of research to produce disruptive ideas need to be investigated, as they are likely to have major policy implications for governments and funding agencies. Even in the absence of those deeper investigations, the paper has already been influential, having been highly cited - including in the recently published policy paper “Science – A New Golden Age”, the Trump administration’s controversial attempt to restructure research funding in the US.

But is it correct? In the past couple of weeks, Nature has published a critique by Vincent Holst and colleagues suggesting that anomalies within the dataset used by Park et al. and their handling of papers and patents with zero references largely account for the observed decline in the CD index over time. Park et al have replied with criticisms of their own about the data and methods used by Holst et al., who have independently published a further reply on GitHub. These exchanges are extremely technical and rather fractious, both sides accusing the other of various sins of omission and misrepresentation.

Our main aim here is not to adjudicate on the relative merits of this dispute, important though it is. Readers with expertise in the relevant methods and databases can judge for themselves, though we would also note that there have been other criticisms both of the CD index and its claimed capacity to reveal a decline in scientific and technological disruption¹, and that there are many other broader discussions of the question of whether science is indeed losing its innovative mojo.² Rather, we wish to share two observations about this dispute, highlighting more general lessons that we feel the metascience community needs to draw from the controversy.

The importance of open data

First, the original Park et al. paper relies partly on metadata of research outputs obtained from the Web of Science database, which the authors weren’t allowed to share, thereby obliging Holst et al. to use a different database, SciSciNet, for their re-analysis. The two camps now seem to be criticizing each other on the basis of inconsistencies arising from the different databases and for not being more open about sharing information, data and methods.

For us, this confirms the importance of the open availability of the metadata of research outputs, and the responsibility of the metascience community to use openly available data and to support advocacy initiatives such as the Barcelona Declaration on Open Research Information. If all data used by Park et al. had been openly available, it would have been much easier for the two research groups to validate and challenge each other’s claims. Third parties could also have contributed to the discussion.

Leiden University’s Centre for Science and Technology Studies (CWTS), one of the metascience centers participating in RoRI, recently decided to discontinue the use of the Web of Science database, in part because of the lack of reproducibility of bibliometric studies based on this database. At RoRI, we are also committed to using open data (e.g., OpenAlex) as much as possible. Regardless of who is right or wrong, the dispute around the Park et al. paper shows how progress in metascience suffers from the reliance on proprietary data, and how this may have significant policy consequences.

The need for more open approaches to scholarly publishing

Second, the way the debate about the Park et al. paper is taking place shows the failure of Nature’s publishing model. It has taken far too long – three years – for Holst et al.’s critique and the authors’ reply to be published. Even now, the recent exchanges of fire in Nature have not settled the debate, which is continuing on social media and through other channels.

This illustrates a more general challenge: most research nowadays is extremely complex, for instance relying on large data sets, highly advanced analyses, and combinations of ideas and methods from multiple disciplines. Journal peer review processes are unable to guarantee the trustworthiness of such research. Inevitably, in many published studies, in Nature or any other journal, there will be mistakes, or at least issues that are open for debate.

To deal with the complexity of 21st century science, we need a publication system in which researchers can efficiently, rapidly, and openly discuss and critique each other’s work. Nature’s publishing model is the exact opposite of this! More innovative approaches to scholarly publishing, such as MetaROR, a publish-review-curate platform managed by RoRI and AIMOS, are getting closer to what we need, although even these still don’t offer the full level of flexibility we ideally should have. This is one of the reasons why RoRI’s Future of Peer Review report recommends a transition from peer review to peer engagement. Traditional peer review processes cannot deal with the complexity of most modern research.

In light of their expertise on how the science system operates, metascientists have a particular responsibility to reflect on their publication practices. Slow and opaque publishing models should be avoided. Timeliness and openness are crucial for healthy debate.

Embracing open science

All of that said, it remains of vital importance to researchers and policymakers that we can determine whether or not science is indeed losing its innovative or disruptive capacity. The question is very much a live one and bibliometric tools that can probe the rapidly growing corpus of new knowledge are likely to play an important role in answering it. The jury might still be out on the utility of the CD index for this task, but the work goes on.³

As it does, it is essential to keep in mind that bibliometrics generally serve as indicators rather than measurements of the phenomena they seek to track. They need to be carefully constructed, relatable to extant and shifting research practices and behaviours⁴, and – crucially – open to critique. Whether deployed in scientometrics studies, research assessment or policy formulation, they need to be used responsibly⁵ based on a full understanding of the meaning they draw from the underlying data. This can only happen if the indicators for interrogating science are themselves fully open to interrogation. Metascience needs to embrace open science!

1. See Macher et al. (2024). Is there a secular decline in disruptive patents? Correcting for measurement bias; Petersen et al. (2025). The disruption index suffers from citation inflation: re-analysis of temporal CD trend and relationship with team size reveal discrepancies; Bentley et al. (2023). Is disruption decreasing, or is it accelerating? For a broader discussion of disruption indicators see Bornmann & Tekles (2021). Convergent validity of several indicators measuring disruptiveness with milestone assignments to physics papers by experts.

2. Matt Clancy from Open Philanthropy has published a lengthy blog post (Science is getting harder) which discusses in more depth the various strands of evidence for an apparent decline in scientific discovery and innovation.

3. The latest entrant to this lively field is winner of UKRI’s Metascience Unit’s Novelty Indicators Challenge, which was announced earlier this summer. The winning indicator, dubbed LENS (LLM-Evaluated Novelty & Significance) goes beyond citation analysis by using AI analysis of the text and research context of individual papers to assess their novelty. To date, precise details of the methodology remain sparse.

4. See Wu et al (2021). Metrics and mechanisms: Measuring the unmeasurable in the science of science for a more in-depth discussion of this point.

5. See the Declaration on Research Assessment, the Leiden Manifesto and The Metric Tide report for more on responsible use of quantitative indicators.