Can AI tell if your research is actually new?
What I learned from building an AI novelty detector
By Carlos Arleo (Independent researcher)
Last year I found myself asking a question that turned out to be surprisingly hard to answer: how novel is this paper, actually? Not “does it sound impressive,” not “did the author write confidently,” but something closer to a structured judgment. I’d been building AI governance systems for a while, and these systems seemed to offer an answer to my question.
So I built a tool called the Disruption & Novelty Indicator (DNI), and an earlier version of it got a special mention in the 2025-26 Metascience Novelty Indicators Challenge, the blind-judged competition run by the UK Metascience Unit, Nesta, RAND Europe, SPRU and Challenge Works. The DNI tool generates an indicator, not a measurement — it’s a proxy for novelty, not a direct read of it, and that distinction matters for how much weight the score it generates should carry.
Most AI tools are built as conversationalists. You ask for an opinion, hope for accuracy, and usually get a response calibrated to please you. I prefer to treat AI as an instrument. Instead of soliciting a view, you force the model to reason inside a fixed frame, a rubric, a defined set of evidence. What comes out isn’t a friendly conversation but an evaluation against a standard. That is the core framework behind the novelty indicator.
Why is the question “is this novel?” hard for a machine, and why do the obvious shortcuts fail?
The first shortcut is to paste a paper into a conversational chatbot and ask for an evaluation - which fails by design. Conversational LLMs are mathematically incentivised to satisfy the user. Ask a model if your paper is novel, and it will construct a plausible, flattering validation regardless of the facts. A recent benchmark, NovBench, tested whether AI systems can recognize genuine scientific contributions rather than merely describe them convincingly. Its core finding cuts to the heart of the problem: LLMs are better at expressing novelty than understanding it. Worse, when forced to justify their praise from ungrounded memory, they will readily fabricate citations to prove a point that was never true to begin with.
Broad, unconstrained AI audits try to do everything at once, which is usually why they fail. Evaluating novelty requires a simpler approach: don’t ask the machine to pass judgment on the entire paper, just ask it to evaluate one specific thing. That is how the DNI works. The system uses strict, hardcoded rules to compare a new paper or idea against existing research. It isn’t trying to peer-review the paper; it’s simply trying to assess how different the core idea is from what has already been published.
The second shortcut is the older academic approach: convert the paper’s text into a set of numbers, a “vector”, that places it on a mathematical map alongside every other paper in a reference database, then evaluate how far it sits from its neighbours. This is fast and cheap, but it discards the actual arguments. A paper might end up in an “unexplored” corner of the map simply because it uses weird jargon or an unusual combination of keywords, not because the underlying idea is actually new.
Instead of taking these shortcuts, or relying on journal prestige or citation counts as proxies for importance, DNI attempts to evaluate novelty directly from the text at the point of reading.
DNI takes an approach that is built on three core commitments:
Full-text analysis: It evaluates the entire manuscript, not just the abstract.
Real-time retrieval: Instead of relying on what a model half-remembers about “the literature,” DNI extracts the actual bibliography cited in the text. It cross-references those citations in real time against OpenAlex, Crossref, and Semantic Scholar to verify they exist, and pulls their true content before evaluating the paper.
Constrained multi-dimensional scoring: A panel of 5 to 10 independent AI judges scores the work across four specific dimensions:
Uniqueness: Originality of claims against retrieved literature.
Tension/Displacement: How much existing consensus the paper challenges.
Synthesis: How substantively it bridges separate disciplines.
Coherence: Used as a baseline gate for the other three scores.
Within DNI’s process, each judge is weighted differently so no single instance dominates the result; and high panel disagreement automatically triggers a second review panel rather than averaging out the variance.
An LLM should never rate novelty freely. The DNI model reasons inside a structured frame, but hardcoded code (not AI) checks and corrects the output. For example, a hardcoded Review Cap automatically caps literature reviews or syntheses at 0.45, regardless of the AI panel’s score. This ensures model variance remains minor noise within fixed boundaries rather than unconstrained opinion.
An early test revealed a key failure mode: when fed a deliberately fabricated paper, complete with invented units, physics equations misapplied to an unrelated field, and zero data, the system scored it as “Moderately Novel” (0.70). Confident nonsense easily tricks LLM judges because it mimics the surface features of groundbreaking work. Each judge was doing its job faithfully; the paper was just built to satisfy all of them at once
To counter this, an Integrity Screen was added to check for empirical grounding and falsifiability. When re-run, the fake paper was flagged and dropped to 0.40, with a note identifying the invented units.
In another test, the system was run on two related papers by the same author published five months apart: one a theoretical framework, the other an empirical follow-up. Despite covering the same foundational topic, the system automatically capped the theoretical paper as a synthesis (0.45) while rating the empirical continuation as highly novel (0.72), demonstrating that hardcoded rules can differentiate structural contributions without manual intervention.
While the system produces consistent results across repeated runs, it comes with a major caveat: it has not yet been calibrated against human expert judgment. Comparing these automated flags to what domain reviewers actually think is the decisive test still ahead.
DNI is not a finished product, a substitute for peer review, or a tool to dictate research value. Its primary role today is as an experimental testbed for bounded AI reasoning, offering authors and literature analysts a way to surface potential reviewer critiques, structural gaps, or ungrounded claims before submission. For funders, one further use case is worth naming directly: assessing whether schemes designed to support high-risk or unconventional research are actually reaching the work they’re meant to — an evaluation problem that has been discussed very recently on this Substack. DNI’s multi-dimensional scores could serve as one input into that kind of retrospective check.
Because this system has not yet been calibrated against human expert judgment, the most important work is still ahead. The goal of sharing this project is not to declare any kind of victory, but to find where automated reasoning fails.
We are therefore opening a small, open beta to test the system’s limits. If you submit a draft or DOI, you will receive a full DNI report detailing its scores, uncertainty bands, and any hardcoded caps or flags that fired. If the system flags a legitimate unconventional theory or misses a subtlety a human reviewer would spot instantly, that isn’t a failure; it is precisely the feedback needed to understand the boundaries of AI in research evaluation.
Technical companion: This article is written for a general research audience and deliberately omits many implementation details. The accompanying technical report documents the full methodology, architectural evolution, evidence tiers, evaluation results, reproducibility analyses, and known limitations of the Darwinian/Disruption Novelty Indicator (DNI). DOI: 10.5281/zenodo.21508045.