Did you know this study was replicated?
Tracking and disseminating replications and reproductions
By Josefina Weinerova (Birkbeck University of London), Lukas Wallrich (Birkbeck University of London), and Lukas (Luke) Röseler (University of Münster)
Two weeks ago, I (LR) was watching a science TV show that featured the famous false-memory study by Loftus, which showed that people can be led to believe they have experienced events that never actually took place.
I first came across this study during my undergraduate degree and I had found it so fascinating. But knowing now that about half of studies didn’t replicate in psychology, I took out my phone, my pulse somewhat elevated. I opened the Replication Atlas and typed in “Loftus”. And was relieved: two successful replication studies, one from 1995 and one from 1978. Pulse steady again.
For decades, replications had been neglected across social sciences. Every article needed to be new and innovative. Some journals even have a section where authors have to explain how their study is unprecedented. While innovation is an important part of research, there is also value in revisiting past findings and checking their robustness and generalisability.
This is why we advocate for a shift in culture: we want to move toward a landscape where replications are everywhere. Not as a ‘police’ force for research, but as a constructive feedback loop that makes science more reliable and efficient. But to build that loop, we first have to face a difficult reality.
Not all discoveries stand the test of time. Ideally, all research would be conducted rigorously with appropriate statistical power, but reality falls some way short and errors creep in. Some widely cited findings are even based on errors in applying an analysis method, such as the now-discredited Dunning-Kruger effect. It is not uncommon for PhD students to spend time trying to replicate previous results as a prelude to further work, only to end up with “unpublishable null-results” and the sense that maybe they did something wrong. In 2015, the Reproducibility Project: Psychology found a replication success rate of one-third to one-half for 100 experiments from high-ranking psychology journals. This year, the SCORE sub-project on replicability corroborated these numbers, observing a success rate of around 50% for 164 papers from various social and behavioral sciences.
If roughly half of findings fail to replicate, then checking the track record of any study you want to rely on becomes essential - but that is precisely where things get difficult. Without a centralised platform that lists replications, finding out if a study has been replicated can be difficult. It means doing a manual Google Scholar search with fingers crossed that the results (which often go unpublished) have been indexed, or browsing the OSF or other preprint servers, or checking websites like PsychFiledrawer.org (now offline but accessible via the Internet Archive), or searching through the supplementary materials of over 20 large scale replication projects to find out what the target studies were.
“Somebody should make a list of replication studies”
Fortunately, things are changing. In 2021 at the Framework for Open and Reproducible Research Training (FORRT), Helena Hartmann led a community of people who were listing replications. Focusing primarily on seminal findings, they collated a list of replication attempts along with meta-analyses and relevant blog posts, comments, etc. Some of us (Luke and Lukas) approached her and Flavio Azevedo (FORRT’s director), suggesting we turn this listing into a database that could be used to make replications more visible in multiple ways.
Luke had started compiling the “Replication Database (ReD)” in spring 2022, which later became the foundation for the FORRT Replication Database (FReD). Things escalated quickly, and five years later we are a team of three researchers, four student research assistants, and more than two hundred volunteer contributors working on the successor to FReD: The FORRT Library of Reproduction and Replication Attempts, or as we call it, FLoRA. FLoRA currently contains over 2,100 pairs of original and replication studies with another 1,000 scheduled for validation. So far, we are mostly focusing on replications that self-identify as such. Since researchers do not choose studies to replicate at random, and keyword searches are difficult due to the many alternative uses of terms such as replication or reproduction, the database is neither comprehensive nor representative. But it’s a start. We went through some dark times, having to learn how to organise a large team, how to deal with numerous special cases of approximate replications, quasi-replications, and replications that were actually reproductions.
Thanks to a UKRI Metascience grant awarded to Lukas (LW) in 2025, the three of us are now coordinating “Making Replications Count”. That project includes analyses of whether and how replications affect citations of original studies, and is developing tools to help researchers find replications. We are creating and developing these tools in collaboration with experts from around the world using the UKRI funding to organise multiple hackathons to bring people together for days at a time.
Going from listing replications to pushing replications
Throughout, we have been amazed by the skill and dedication of everyone who has joined us for the hackathons. During the events so far, we developed:
an automated pipeline that searches the literature and spits out coded pairs of original and replication studies1,
an API that returns replication attempts based on original study’s DOIs (and vice-versa),
a gamified validation app where contributors can get high-scores for spotting errors in the database (i.e., crowdsourcing data validation),
the Replication Atlas as a look-up tool based on DOIs, search terms, or full reference lists,
a metacheck replication module that ensures relevant replications can be surfaced during self- and peer-review,
the Zotero Plugin that adds and links replication studies, thus bringing replications directly into researchers’ daily workflows,
and Guess the Replication, which is our take on Guess the Correlation, where you have to say based on an original abstract if the replication was successful or not.
We are also working on a few more things to support replications in different ways. FLoRA is about tracking replications. With tools such as the FReD-Explorer, you can browse the database and explore replications. We are presently writing a multidisciplinary handbook that will help researchers to conduct replications. And finally we have created a community-owned diamond open access journal for replications, Replication Research. The FORRT Replication Hub structures these and many more projects, so please subscribe to the FORRT Newsletter or join the FORRT Slack to stay up to date and join the team. For example, anybody who does 20 correct validations will be listed as a contributor and eventually, there will be the FLoRA Report where we invite everybody to co-author one big data paper.
If we succeed with our mission, it will be difficult to ignore replications. They will be everywhere, and we think they should be. Not in a personal way where people point fingers at others whose work could not be replicated, but in a constructive way, where we have accepted that replication is difficult and that results depend on numerous factors, but also that replications provide helpful information for future research. Then, we can build on each other’s work much more efficiently.
Currently, we are checking OpenAlex and Semantic Scholar for studies that self-identify as replication in the title or abstract. A pipeline combining rules-based extraction and LLM calls then checks if it is actually a replication, identifies the target study, the outcome, and specific quote reporting that outcome. The results get forwarded into our validation app, in which humans collect points for spotting errors and eventually gain contributorship. Every entry has to be checked by two human validators and disagreements require resolution by one of the project leads.
From RoRI
AFIRE Experimental Funders Group (17 June): Jessica Biddinger (American Heart Foundation), Svenja Vandertol (NWO), Tom Stafford/Stephen Pinfield (Volkswagen Foundation) and Sampsa Kaataja (NordForsk) will dive into Distributed Peer Review and share insights from its use at their organisations.
ECR Research on Research Seminar Series (17 June): Join Finn Luebber (Universität Lübeck) on 17 June to explore the history of project-based peer-reviewed funding, its current criticisms, and new developments like the lottery-first approach.