Can a laptop check an AI research report? 235 planted errors and one real report

What did the test find?

1. Checking quotes needs no AI model. The checker fetches every cited page and looks for each quoted passage. It gave the same grade as our earlier study to all 593 quotes, and when one word in each quote was replaced it accepted none of 588.
2. A 12B model on a laptop is a usable judge. Gemma 4 12B, running on a MacBook Air with 16 GB of memory, caught 222 of 235 planted errors and passed 94 of 105 correct claims, at about ten seconds a claim.
3. The smallest checker models are fast but not enough alone. MiniCheck caught 212 of 235 errors in under a second each, but rejected 21 of 105 correct claims. HHEM missed 23 of 35 negated claims.
4. Used together, few errors slip through. When the report shows the judge’s verdict and sends disagreements to a person, 6 of 235 planted errors passed without a flag. The price is that a reader checks about one claim in four by hand.
5. On a real report it narrows the search; it does not decide. In our research notes on how AI shopping agents choose products, it flagged 141 items. Checked against the sources one by one, 36 were wrong or not in any cited source, 7 were right but cited to the wrong page, and 91 were correct.
6. The errors were the kind a careful reader misses. Paraphrases inside quotation marks, a range quoted as +19.75 to +23.38 points that the paper gives as +13.0 to +37.9, and a finding reported as no effect that the paper calls statistically significant.

What does the checker do?

It reads a report from ChatGPT, Claude, Gemini, Perplexity or an open-source research agent, fetches every page the report cites, and keeps a copy of each. Then it runs three checks.

1. Quotes. Is each quoted passage on the page it cites, word for word, edited, or missing? This is plain text matching, no model.
2. Links. Does each link work? For a dead link, the Wayback Machine tells a page that moved from one that may never have existed.
3. Support. Does the page back the claim? The checker picks the page sentences closest to the claim, a small checker model scores them, and a local chat model reads them and answers supported, partly or not supported. It names the sentences it relied on by number; it never writes a quotation itself. Long claims with several figures are split into single facts first, and each fact is checked on its own.

Everything runs on the user’s own machine or servers, with any model behind an OpenAI-compatible endpoint (llama.cpp, Ollama, vLLM, LM Studio). The code is at github.com/jbastide/receipts under the Apache 2.0 licence.

How well does it catch errors?

We started from 105 claims in our earlier study of AI research runs that two graders had accepted and whose quotes were on the cited page. Each was checked as written, where it should pass, and in up to four damaged versions, where it should be flagged: one number changed, the first verb negated, the claim checked against a page from another website, or an invented fact added (“according to a 2019 survey of 4,000 law firms”). 17 negations of claims that were already negative made nonsense and are left out.

Splitting long claims into single facts made no difference here, because each planted error sits in one fact. It matters on real reports, where one sentence can carry five figures and the reader needs to know which one fails.

What did it find in a real report?

We ran it on the research notes behind our page How AI agents choose products: 112 claims citing 87 sources, most of them academic papers. The checker flagged 4 quotes that were not on their pages and 137 facts with figures it could not confirm. Each flagged item was then checked against its source, reading the paper or page itself.

So roughly one flag in four pointed at a real problem. That is the honest description of what a local checker does with dense research prose today: it cannot tell a missed sentence from a missing fact, but it turns a reading of 87 sources into a list of 141 places to look. Some of what it found:

• 7 passages in quotation marks were not the source’s words. One paper says user profiles act “more like categorical switches or thresholds”; our notes had them “act as switches that override other cues”.
• A defence that “left social-proof effects of +19.75 to +23.38 points intact” left them at +13.0 to +37.9 across the seven models in the paper’s table.
• Two negative-framing cues reported as having no effect were, in the paper, “both statistically significantly effective”.
• A crawl-to-referral ratio of 38,000:1 was right, but cited to a page with different, later numbers; it is Cloudflare’s own July 2025 figure (38,065.7 for Anthropic).

The published page had been checked before it went live and already used the sources’ wording. The research notes behind it had not. We corrected them: 30 edits from the flagged items and 7 more found while reviewing them. Afterwards every quote in the notes was on its cited page.

What are the limits?

• The planted errors are mechanical. Real mistakes are subtler, so the catch rates above are a ceiling, not a promise.
• On the real report we only know about the errors it flagged. Errors it passed as supported were not counted.
• The judge reads the sentences that best match the claim, not the whole page. A claim drawn from a table or another part of the page can be marked unsupported.
• One judge model, one machine, one run. Results with other models will differ; the checker is built so you can measure your own.
• The review of the real report was done with the help of AI agents and spot-checked: every quote they relied on was matched against the source page by script.

How do I run it on my own reports?

On a Mac or Linux machine with uv installed and a local model server running:

git clone https://github.com/jbastide/receipts
cd receipts
uv sync --extra support
uv run receipts check report.md --judge-url http://127.0.0.1:8080/v1

It writes a readable report, a JSON file and a CSV. The quote and link checks work without any model: uv run receipts check report.md.

Questions

Can a small AI model on a laptop check whether a research report’s sources support it?

Mostly, if it is used as a judge rather than as the last word. In a Copy for Originals test, Gemma 4 12B running on a MacBook Air with 16 GB of memory caught 222 of 235 claims we had deliberately damaged (94%) and passed 94 of 105 correct ones (90%), at about ten seconds a claim. Paired with a smaller checker model, 6 of the 235 damaged claims got through without a flag.

How do you check the quotes in an AI research report?

Fetch every cited page and look for each quoted passage in it, ignoring spacing, punctuation and case. That needs no AI model. Copy for Originals’ checker reproduced all 593 quote grades from an earlier study this way, and when one word in each quote was replaced, none of 588 damaged quotes was accepted. A quote that is on the page can still fail to support the claim made with it, so the claim needs a second check.

What errors does an automated check find in real research?

In our own published research notes on how AI shopping agents choose products, the checker flagged 141 items. A source-by-source review found 36 of them wrong or not in any cited source: paraphrases inside quotation marks, a range reported as +19.75 to +23.38 points that the paper gives as +13.0 to +37.9, a finding reported as no effect that the paper reports as statistically significant. 91 were correct. The checker narrowed where to look; it did not decide.

Data and method

• planted.csv: all 357 planted cases, the claim as checked, the page, and each method’s verdict and score.
• audit.csv: the 141 items flagged in our research notes, the review result, the source passage and whether the notes were corrected.
• method/prepare.py writes both files from the checker’s benchmark output; method/build.py writes this page from them.
• Models: MiniCheck-Flan-T5-Large, HHEM-2.1-Open, Gemma 4 12B instruction-tuned (QAT, 4-bit) on llama.cpp. The benchmark script is bench/planted.py in the code repository.