The Brief
How AI Works 4 min read

Fake Citations, Real Consequences: The arXiv Decision

NAVION

Share

A preprint server that hosts millions of scientific papers has drawn a clear line. arXiv is now banning submissions that contain references hallucinated by AI tools. The policy targets a specific and growing problem: AI systems that generate plausible-looking citations to sources that do not exist. The decision, reported by Nature, arrives at a moment when the scientific community has not yet agreed on whether such citations constitute research misconduct.

That unresolved debate is precisely what makes this worth examining carefully.

What a Hallucinated Citation Actually Is

To understand why this matters, it helps to be precise about what an AI hallucination looks like in a research context. Large language models generate text by predicting what comes next based on patterns in training data. When asked to produce a bibliography, they do not retrieve real sources from a database. They generate text that looks like a citation: author names, journal titles, volume numbers, page ranges, publication years. The result can be entirely convincing and entirely fictional.

This is not a bug in the traditional sense. It is a structural feature of how these systems work. They are optimized to produce fluent, coherent output, not to verify that the sources they name actually exist. A researcher who pastes AI-generated text into a manuscript without checking each reference may submit a paper containing citations to journals that never published those articles, or to articles that were never written.

arXiv’s response is to treat this as grounds for rejection. The policy does not require proof of intent. A hallucinated reference is a hallucinated reference, regardless of whether the author knew it was fabricated.

The Misconduct Question That Science Has Not Settled

Here is what most coverage of this story misses: the arXiv ban is an institutional response to a question that the broader scientific community has not answered. Nature has reported that researchers continue to debate whether AI-hallucinated citations constitute research misconduct.

That debate is not trivial. Research misconduct is a serious designation with professional consequences. It typically covers fabrication, falsification, and plagiarism. A hallucinated citation is, in one sense, a fabricated source. But the person who included it may have done so without realizing it was false, relying on an AI tool they trusted to assist with a routine task.

The distinction between negligence and fabrication matters enormously in academic culture. Negligence is a failure of process. Fabrication is an ethical violation. The two carry different weight in institutional review, in career consequences, and in how the community understands what went wrong.

arXiv’s policy sidesteps this distinction by focusing on the output rather than the intent. A paper with hallucinated references does not belong on the server, full stop. Whether the author acted in bad faith is a separate question, one that arXiv is not positioned to adjudicate. What the server can control is what it hosts.

Why This Extends Far Beyond One Preprint Server

The arXiv decision and the Nature debate around it point to something larger than a policy update. They signal that the integration of AI tools into scientific workflows has outpaced the norms designed to govern those workflows.

Peer review, citation standards, and research integrity frameworks were built around human authors who, in principle, had read the sources they cited. The assumption was that a reference in a bibliography represented a claim: this source exists, it says what I imply it says, and it supports the point I am making. AI tools break that assumption at the level of basic mechanics.

This creates a verification burden that did not previously exist at scale. Editors, reviewers, and readers now face the possibility that any given citation in any given paper may be a plausible fiction. Checking every reference in every submission is not a realistic expectation for volunteer peer reviewers. Yet the integrity of the scientific record depends on citations being real.

The broader implication is that research institutions, journals, and preprint servers will need to develop new infrastructure, whether technical tools for citation verification or updated submission requirements, to handle a problem that did not exist before AI writing assistance became widespread. arXiv’s ban is one early response. It will not be the last.

What this moment reveals is that AI tools, when used without adequate verification, do not just assist research. They can quietly corrupt the connective tissue of science: the references that link one finding to another, that allow readers to trace claims back to evidence, and that give the scientific literature its cumulative structure.

In Short

arXiv is banning papers with AI-hallucinated citations, meaning references to sources that do not exist, generated by AI tools. The scientific community has not yet agreed on whether this constitutes research misconduct. The debate matters because it sits at the intersection of how AI tools work, how research integrity is defined, and who bears responsibility when a machine produces a convincing falsehood. The deeper issue is that citation norms built for human authors are not equipped for a world where AI can generate bibliographies that look real but are not.

Based on reporting from Nature.

Written by

NAVION