November 2026 · 5 min read
A citation gate
4 of 12 drafts cited a 404 page

Key Definitions
Citation existence check Issuing a request to every cited address and reading the status code. It answers whether the target exists, not whether it is the right target.
Citation identity check Matching the master identifier a draft claims against a real record in the source system. An object can exist and still be the wrong object, and that is the case the existence check passes.
Enqueue gate A check placed before an artifact enters the ready queue, rejecting it on failure. Compared with a pre publish check it wastes minutes of generation instead of a finished batch.
On the morning of November 15, the sales queue held twelve finished drafts, every filename prefixed with the word ready. Four of them were platform versions of one topic. Each carried a title, a length inside the platform limit, UTM parameters, and a line in the front matter naming a source. One pointed at a white paper PDF hosted on servicenow.com. The others carried a master page address inferred from the filename. I opened that address in a browser: the Chinese site returned 404, and so did the English site.
The drafts were not badly written. Length, link syntax, platform tone all passed. One thing was missing, and it was the page they cited.
The queue ran for a full month before anything caught this, and what caught it was a script added that morning. It does one thing: take the master address each draft claims, issue a request, and require 200 on both locales plus a slug that resolves to a row in the article database. Twelve drafts went in. Four came back named. Eight returned 200.
Writing got cheap, checking did not
Writing four social posts used to take a day or two. The person writing them clicked a link along the way, saw the page load, and moved on with some confidence. A generator now produces twelve in an afternoon. It does not click links. It rewrites a summary into four platform voices. The only step in the whole chain that was never automated is the one where somebody opens the page.
This is not only our problem. A team led by Columbia University School of Nursing audited PMC open access papers published between January 2023 and February 2026: 2.5 million papers, 97.1 million references checked, 4,046 citations that point at nothing, spread across 2,810 papers. The rate was about 4 per 10,000 papers in 2023 and about 57 per 10,000 in early 2026, a twelve-fold rise in three years. The audit was published in The Lancet in May. At the end of it, more than 98% of the affected papers had no correction, no expression of concern, and no retraction.
The Royal College of Surgeons of England tested medical references produced by several models in April. The worst performer had 34% fabricated or unverifiable references, another had 25%, and some models produced none at all. The spread between vendors on this one measure is wider than most buyers assume.
Three numbers, one mechanism: fake citations grow at the rate of writing capacity, not at the rate of model capability. The review gate was adequate in 2023 because submission volume and reviewer seats were roughly matched. It stopped being adequate when the writing end multiplied twelve times while the checking end stayed the same people, the same form, the same hours. Our twelve drafts are that same mechanism shrunk to the size of one company.
Why this error costs more than a typo
Somebody fixes a typo. A reader treats it as typesetting, an editor corrects it the next day. Nobody dares to fix a fake citation, because it looks more complete than a real one: author, journal, year, volume, DOI, all present, and it maps onto a topic that genuinely exists. Its defect is not something missing. It is something extra that should not be there.
Our four drafts were the same. They were not blank. They carried a plausible address, correctly formatted and correctly sized. The format check passed them because the format was not wrong. The human eye passed them because nobody opens four browser tabs to ship one social post.
The cost is multiplicative. One topic becomes four platform versions, and one bad master voids all four, along with the hours already spent on them. If it ships first and the 404 is found later, the damage moves to another chain: every one of those posts carries a link back to the site, and a reader who lands on a 404 does not click the next one. Points of layering rest on that return path, and one broken link makes the upper layers pointless.
Existence and identity are two checks
The existence check is the cheap one. Send a request, read the status code, accept 200. Twelve requests took 3.6 seconds in total when measured. That layer can be added today, without approval and without touching architecture.
Existence alone misses half the problem. Our four drafts would all have passed a status check, because the ServiceNow white paper they cite is live and returns 200. The link was not broken. The reference was wrong: it pointed at a vendor PDF instead of our own master page, and derivative posts may only derive from our own master.
The second check is identity. The master slug a draft claims has to resolve to a real row in the article table, in both languages. Our script implements that as a database query plus slug matching, not as a test for the presence of a link. The gap between those two judgements is the gap between catching four drafts and missing four drafts. Judge a validator by what it refuses: one that only refuses broken links cannot refuse the wrong object.
One implementation detail matters here: what the check matches on. We match the slug, not the title. Titles change, and editing a title is normal work, whether the master is updated, the platform shifts, or somebody polishes the wording. A slug is fixed once published, and it is the identity document for that artifact. A gate that matches on titles expires along with the next edit, and it expires silently by passing.
The third layer is consistency, meaning the sentence a draft attributes to a source can be found in that source. This layer cannot be fully automated. The workable form is sampling plus a record: a sampling ratio written into the process, results archived. The first two layers catch the wrong object. The third catches the right object described wrongly, which is more expensive and needs a reader.
Why every format check passes
Format checks test shape. Whether the length is inside the limit, whether the structure has all its parts, whether the link is a well formed URL, whether the title carries a banned phrase. All four can be computed from the text itself, without reaching the outside world.
A fake citation is perfect on all four. It has an author and a year, the format is legal, the length fits, the tone reads plausibly. The longer the checklist, the safer the pass feels, and every item on it is unrelated to whether the cited object exists. This is why the error travels all the way to the end of the queue: it does not slip through by luck, it passes because no check ever asked.
Adding the missing item is harder than adding another row to a table. It requires an outbound request, which means contact with the outside world. Any check that touches the outside world needs somebody to own its failure: whose problem is a false alarm, does it stop the line, who gets called. Until that is decided, the check stays on the list of things to do later.
Why the check cannot be a model
The usual fix is a line in the prompt asking the model to verify every citation. That line stopped working in 2024. A model has no network state during generation. It guesses from training data that the address probably exists, and a guess is indistinguishable in text from an invention.
The deeper problem is the nature of the judgement. Model output is probabilistic; the same input twice can produce two answers. A compliance decision has to be reproducible: the same input, always the same conclusion, and a third party running the same script gets the same number. Handing verification to a model turns compliance into another generation, with the ledger kept by the party under audit.
Our split is generation by model, verification by script. The validator is a sequence of if statements and an exit code. It does not need to be clever. It needs to say the same thing every time and to keep a failing batch from leaving the building.
Where the gate sits decides what it destroys
The same check in three positions destroys different things. Before publish, it destroys an incident and voids four finished posts. Before enqueue, it destroys work, a few minutes of generation. Before topic selection it destroys nothing, because no citation exists yet.
The position also decides something quieter: whether the queue can be trusted. If ready is a human label, it decays, because the person labelling gets tired, skips, and defaults. If it is computed by a script, it is true again every day. After moving the gate to enqueue, ready means something again: eight of twelve can ship, four never enter the queue.
A pre-publish check also trains a habit: finish first, check later. By the time a batch is done the person is tired, everything looks fine, and the check becomes a ritual. An enqueue check trains the other habit, where each unit has to leave the gate once, which is the only way the speed of generation actually pays.
The cost comparison is not subtle. A human review of one social post takes minutes, and twelve of them take a morning. Twelve requests took 3.6 seconds. The real difference is not seconds against a morning. It is whether the check happens at all: a morning of review gets skipped when the week is busy, and a three second script runs itself, every day.
Where this does not work
External documents change. A link that returns 200 today may return a 301 to the homepage in three months, or a 200 page that says the content is gone. A status code catches the first case only. The workable pattern is to record the access date and an archived copy at citation time, and to test the body for a characteristic string rather than the code alone.
On internal networks and offline deployments the object under check is a mirror. Capture the content once, hash it, carry the hash in the citation, and verify that the mirrored copy has not changed. The difference matters because the first can be recomputed offline and the second needs a network.
False negatives need attention too. Some marketing pages answer machine requests with a 200 soft 404, others block unknown user agents. A gate that works sometimes teaches the team not to trust it, and once trust is gone people route around it. The failure mode a gate should fear is not a miss. It is ambiguity about whether it is right.
Do not build a system at small scale. For twelve items, a loop and a few requests are enough, printing one line of results. Wait until the queue runs several times a day and passes fifty items before treating it as a service. A validation platform for twelve drafts is another kind of waste, and the platform needs its own maintenance.
What we think
We do not believe in the path of asking models to cite more carefully. The 2026 numbers show where that path ends: writing capacity up twelve times, fake citations up twelve times, and 98% of them still uncorrected. The rate is a problem of checking seats, not model capability. Without more seats, a stronger model only writes a more convincing fake.
Our position: in an AI pipeline, generation is judgement and verification is the ledger, and the ledger cannot be kept by the generator. An AI deliverable that another party can accept is not one that reads well. It is one where every claim can be rerun by a third party with the same tools. That is how we write acceptance thresholds for clients: not content quality is good, but every citation resolves, recomputes, and traces.
A second position about the gate itself. The volume it catches is small, four items. What it changes is the status of every artifact behind it, from looks finished to proven finished. At twelve items that difference is invisible. At twelve hundred it is the whole difference, because nobody can check twelve hundred by eye.
Three things to do today
First, take the artifacts currently waiting to ship or to be delivered, pull every citation, and batch the requests, recording status codes. Where a soft 404 is possible, grep the body for a characteristic string. No architecture change is needed.
Second, add two fields to every citation: access date and archive address. With those fields, a review three months later tells you what you saw then rather than what is there now.
Third, put the check after generation and before enqueue, and reject on failure instead of marking it for human review. A needs review label hands the item to somebody who will not specialise in it, and the label itself becomes the reason it stays in the queue.
One question for the buyer: in your pipeline, is there a step that proves the cited object exists? If the answer is that somebody takes a look, then the capacity of that step is the real ceiling on your queue, measured in items per day.
References
Internal figures come from measurements of our own ready queue on the morning of November 15, 2026: twelve drafts, four failing the master check, twelve requests in 3.6 seconds, and the master address returning 404 on both locales.
FAQ
Why did four of twelve drafts cite a page that does not exist+
The generator inferred the master address from the filename and never requested it, and the existing format checks test length, structure and link syntax rather than whether the cited object exists.
Is a working link enough+
No. The ServiceNow white paper those drafts cite returns 200, but it is a vendor document rather than our own master. Existence has to be followed by identity, meaning the slug resolves to a row in the article table.
Why not use a model to verify citations+
A model has no network state at generation time; it guesses from memory whether an address exists, and a guess is textually indistinguishable from an invention. Verification also has to be reproducible, which a probabilistic generator cannot promise.
Where should the check sit+
After generation and before enqueue, rejecting on failure. A pre publish check voids a finished batch, and a pre topic check has nothing to validate.
What happens when an external document changes in three months+
Record the access date and an archived copy when citing, and test the body for a characteristic string rather than the status code alone, since soft 404 pages often return 200.
Do you need a platform for a queue of twelve+
No. A loop and a few requests are enough, printing one line of results. Treat it as a service once the queue runs several times a day and passes fifty items.
Run the same check on your own execution log
Send a redacted 30-day agent execution log and get an eight-row table in 48 hours: one row per technical item, each cell carrying its numerator, denominator and criterion, gaps sorted by whether a schema change is needed. First three engagements are ¥8,000.
Related Articles
Silent by config: 900 runs that failed unseen
931 runs went undelivered in thirty hours, seven carrying error lines, and four incidents sat detected with no alert sent.
AI pays inside functions and stalls at the seam
Surveys of 6,000 executives and 1,719 firms agree: AI lifts output inside functions while nine in ten report no productivity gain. The seam is unmeasured.
Your merged PRs are the coding-agent benchmark
Databricks ran coding agents on its own merged PRs: same model, another harness, over 2x cost per task. Public leaderboards cannot see your codebase.
AI agent simulation testing: 4.8x, not proof
Nubank screened agents with tool-boundary simulation: version cycles fell from 21.2 to 4.4 days. A gate proves eligibility, not production behaviour.