Haiku, Sonnet or Opus for online research? One brief, nine runs, 593 checked quotes

What did the study find?

1. No model made up a quote. A script fetched every cited page and looked for the quote. It found 593 of 593. The brief warned each model that its quotes would be checked by machine, and every run reported checking its own work first.
2. The cheapest model found most of the known facts, but different ones each time. Haiku runs found 19, 23 and 22 of 25. Only 16 facts turned up in all three Haiku runs, though the three together covered all 25. Sonnet runs found 24, 24 and 23. Opus runs found all 25 each time.
3. The extra money bought breadth. An Opus run returned about 97 claims from 23 to 24 pages. A Sonnet run returned about 57 from 15 to 20, and a Haiku run about 44 from 8 to 12.
4. A real quote is not always enough. Two blind graders both flagged 41 of 593 claims as saying more than their quote showed: 9.8% of Haiku’s, 9.9% of Sonnet’s and 3.8% of Opus’s.
5. A blocked website stopped every model. Reddit refused the search tool in all nine runs. One Opus run found a side door and spent 30 minutes at it. No run invented a thread.
6. An Opus run cost about 10 times a Haiku run. Estimated at list prices: $0.28 for Haiku, $1.07 for Sonnet, $2.79 for Opus. Search fees were about half of Haiku’s cost.
7. The setup mattered as much as the model. Our first attempt failed twice in ways no model caused: nine runs shared one cap of 200 searches, and three of them saved files on top of each other.

What was the assignment?

We picked a question we already knew the answer to. Earlier the same day we had published a guide to whether Google records calls to law firms from its ads, with every quotation checked against its source. That gave us 25 facts a good researcher ought to come back with.

The brief asked ten questions. Does Google record calls from Local Services Ads? From Google Ads? What does the caller hear? How long are recordings kept? Who can listen? Can a firm switch it off? What have state bars said? What do lawyers say about it in public threads?

It also set rules, and they are the heart of the test. One fact per claim. Every claim carries the address of the page and a passage copied from it character for character, 300 characters at most, with no ellipses and no stitching. Anything the model could not quote exactly goes in a separate list of leads. Anything it looked for and could not find goes in another list. And one sentence we suspected would matter: “Every quote will be machine-checked against the page at that URL.”

Each model got the brief three times, as nine separate agents with a search tool, a page-fetching tool and a command line. The three models were Claude Haiku 5.5, Sonnet 5.5 and Opus 5.5, the small, middle and large models in Anthropic’s Claude 5.5 family. The full brief is published with the data.

Did any model make up a quote?

No. The fear with AI research is the invented source: a confident sentence in quotation marks that nobody ever wrote. We went looking for one 593 times and did not find it.

The check was a script, not a second opinion from a model. It fetched each cited page, stripped out the markup, and searched for the whole quote, forgiving only spacing, punctuation and capital letters. All 132 Haiku quotes were there. So were all 171 from Sonnet and all 290 from Opus.

A checker that passes everything deserves suspicion, so we tested it. We swapped a single word in each of 588 quotes and ran it again. It accepted none.

The models did not get there by luck. Told that a machine would check them, they checked themselves. “Every claim quote was machine-checked as a verbatim substring of the page text I fetched,” one Haiku run reported. An Opus run wrote that every quote “was re-fetched live and matched exactly (96 of 96).” All nine runs said something similar.

That sentence in the brief appears to have done real work. We did not run the brief without it, so we cannot say how much.

What did the cheapest model miss?

Not the hard facts. The easy ones, at random.

Opus came back with all 25 known facts in each of its three runs. Sonnet came back with 24, 24 and 23. Haiku came back with 19, 23 and 22. A fact counts as found when a run quoted a passage that states it.

One Haiku run never cited the Google help page that says, in so many words, that all Local Services calls through the forwarding number are recorded. Every quote it did report was real. A writer working from that run alone would have been missing the first fact on the topic and would have had no way to know it.

The other two Haiku runs found that page. This is the pattern that matters more than the averages. Only 16 of the 25 facts appeared in all three Haiku runs, against 23 for Sonnet and 25 for Opus. But put side by side, the three Haiku runs covered all 25. The cheap model was not weak so much as uneven, and unevenness is something a second and third run can fix.

What did the extra money buy?

Reading. Opus cited roughly twice as many pages as Haiku and came back with roughly twice as many claims: about 97 a run, against 57 for Sonnet and 44 for Haiku.

The extra claims were not padding. Once the results were in, we listed 28 findings that went beyond our 25 known facts. Opus runs returned 19, 18 and 23 of them. Sonnet runs returned 14, 16 and 11, and Haiku runs 10, 6 and 3.

Some of those findings were news to us. Our guide says Google’s help pages give no date for the day call recording became the default in Google Ads. That is still true of the help pages. But five runs found the date anyway, July 1, 2026, in trade coverage of an email Google sent to advertisers. No Haiku run did. We have since read those sources ourselves and added them to the guide.

The Opus runs also noticed when Google disagreed with itself. A help page says that with recording on, “all calls will be recorded and evaluated by AI.” The terms an advertiser accepts say Google may record “a sample” of calls. All three Opus runs quoted both. Deciding which one governs is not a research task, and none of the models pretended it was.

Price did not settle everything. All three Haiku runs found a 2013 Connecticut bar opinion on an advertising service that records calls. Only one Sonnet run and one Opus run did. A single Haiku run was the only one of the nine to quote a 2026 New York City Bar opinion on recording calls with prospective clients.

Taken together the nine runs cited 55 different pages. 29 of those were cited by one run only. Just four were cited by all nine. Whatever model you use, a single run is one researcher’s afternoon, not the last word. The full list of 28 findings is in the tables below.

Is a real quote enough?

Not quite. A quote can be genuine and still not say what the sentence above it claims.

So we ran a second check. All 593 claims were shuffled, stripped of any sign of which model wrote them, and read against their quotes by two graders working from the same written rule. A claim was flagged when it stated a second fact of substance that its quote did not carry, or when the quote did not show the claim’s main point.

Both graders flagged 41 claims, about one in 14. For Haiku the share was 9.8%, for Sonnet 9.9%, for Opus 3.8%.

Most of the flags were the same small sin: two facts in one sentence and a quote for only one. A Sonnet run wrote that a business hears “Call from Google” when a call comes in, though call forwarding or an internet phone line can stop the message playing. Its quote covered the first half. The second half is true, and it is on the same page, and nothing in the quote shows it.

A few were quotes cut too short. A Haiku run claimed that recordings are not available for health care categories and offered, as proof, the words “available for health care verticals.” Those words are on Google’s page. So is the “aren’t” in front of them, which the quote left out.

Two cautions. The graders were AI models, and they agreed with each other on 580 of 593 claims (97.8%), which is good but is not a human read. And the gap between runs of one model was as wide as the gap between models: by run, the share flagged by both graders ran from 2% to 17%, and the two worst were one Haiku run and one Sonnet run.

Why couldn’t any of them find the threads?

Because the door was locked, and a better model does not pick locks.

The ninth question asked for up to eight public threads in which lawyers or their marketers talk about the recording first-hand. The obvious place is Reddit. Reddit refused the search tool in every one of the nine runs, 16 refusals in all, and it turns away plain page requests too. “Reddit was blocked in every route I tried,” a Sonnet run reported.

Only four of the nine runs returned a quote from any thread or post at all: from Reddit, from LinkedIn, or from a small forum for local search marketers. One run got into Reddit. An Opus run worked out that the site’s feed addresses still answered, read threads through them between rate-limit refusals, and took 30 minutes where its siblings took 15. It came back with 8 Reddit quotes and an honest verdict on them: “I found no public thread where a lawyer or law firm staffer discusses Google recording these calls.”

That honesty was the most reassuring thing in the study. No run invented a thread to fill the hole. Every run filed between 6 and 13 items under “not found,” and between 3 and 14 under leads it could not confirm, as the brief had asked.

For our own thread studies we do not ask a model to search Reddit. A script reads a public archive of it. When the source is closed to the tools, the fix is access, not a bigger model.

What went wrong the first time?

The numbers on this page come from our second attempt. The first one taught us more than we wanted.

We had started all nine runs at once. Eight minutes in, an Opus run asked for a search and got back a sentence we had never seen: “Web search was not performed: this turn’s web search budget is used up (limit: 200 WebSearch calls per turn, shared by every agent in it).” The nine runs had made 202 search calls between them. Calls 201 and 202 both belonged to Opus runs, the slowest of the nine, which were still searching after the cheaper models had stopped. Our test of whether the expensive model is worth it had quietly handicapped the expensive model.

The second problem came from one of the runs. A Sonnet run noted at the end of its report that “the scratchpad is shared with other agents, and someone else’s build.py replaced mine mid-run.” All three Sonnet runs had saved their working files in the same folder under the same short names. Their quotes still passed against the live pages. But three runs that can read each other’s files are not three independent runs.

Neither failure was a model’s doing, and both would have skewed the comparison. So we ran everything again. Each run got a private folder, and the runs were started in two groups so that neither group could reach the cap. The logs of the second set show no search refused by the cap and no run touching another’s folder.

The first set is published as a pilot. Its results were close to the second, which is some comfort: Haiku runs found 22, 21 and 20 of the 25 known facts, Sonnet runs 24, 24 and 24, Opus runs 24, 25 and 25. And all 574 of its quotes were on the page.

What did each run cost?

About $0.28 for Haiku, $1.07 for Sonnet and $2.79 for Opus, estimated at Anthropic’s list prices. Haiku and Sonnet runs took about 8 and 10 minutes. Opus runs took 14 to 30.

Two things in that table surprised us.

The searches cost as much as the cheap model. Anthropic lists web search at $10 per 1,000 searches, a cent each. Every run made 10 to 26 of them, whatever the model. That fee was 50% of a Haiku run’s cost and 8% of an Opus run’s.

Haiku’s low price has a ceiling. Haiku 5.5 costs five times as much on any request whose prompt is longer than 100,000 tokens. Search results and page text pushed every Haiku run past that line, to about 145,000. 26 of 88 Haiku requests paid the higher rate. Kept under the line, a run’s token cost would have been about $0.05, not $0.14.

Even so, three Haiku runs cost less than one Sonnet run. The three together came to an estimated $0.84 and covered all 25 known facts and 14 of the 28 further findings. One Sonnet run cost about $1.07 and covered 23 to 24 and 11 to 16. One Opus run cost about $2.79 and covered 25 and 18 to 23.

These are estimates. The logs record what each model read to the token. They do not record a final count of what it wrote, so that part is estimated from the visible text, and the model’s private reasoning is not counted. On a subscription the bill would look different again.

So which model should do your research?

On this evidence, the cheap one can gather, as long as a script checks its quotes and you run it more than once. The expensive one earns its price when the research runs once, when nothing can be missed, and when nobody will read the claims against the quotes afterward.

This was one topic on one day with three runs per model. Take the table as a place to start your own test, not as a ranking.

How should a marketing team set this up?

Split the job in two. Cheap models gather quoted facts. A script checks the quotes. The best model you have reads the checked facts and does the thinking and the writing. Each step below comes from something that happened in this study.

1. Write the questions down. Ten numbered questions let us score every run question by question. “Research this topic” cannot be scored.
2. Ask for one fact per claim. Most of the 41 flagged claims had packed two facts behind one quote.
3. Require the page address and a copied quote, and say a script will check it. All 593 quotes passed. Then actually run the script. Ours is under 200 lines.
4. Ask for a list of what was not found. Every run kept one. It is how we know Reddit was locked and not empty.
5. Give each run its own folder. In our first attempt three runs saved pages under the same names and overwrote each other.
6. Count the searches before you launch. Our tool allowed 200 across all runs started together. Nine runs at about 22 each used them up.
7. Run the cheap model more than once and merge. Three Haiku runs agreed on only 16 of 25 known facts and covered all 25 between them.
8. Read each claim against its quote before writing. A second model did this for all 593 claims in under two minutes and agreed with the first grader on 97.8% of them.
9. Reach locked sources another way. An archive, an export or an official data feed beats a stronger model that cannot open the page.
10. Keep the judgment with your best model. The gathering runs brought back Google pages that contradict each other. Deciding which one is current is the writer’s job.

For what answer engines do with the finished page, see how to write content that LLMs retrieve and cite. For a study built on public files alone, see do law firm websites block AI crawlers?

What are the full numbers?

How was the study done?

1. Models: Claude Haiku 5.5, Sonnet 5.5 and Opus 5.5, three runs each, on October 13, 2026. The logs record the model IDs claude-haiku-5-5, claude-sonnet-5-5 and claude-opus-5-5.
2. Setup: each run was a separate agent started from Claude Code 2.1.295 with default settings. Each had web search, web fetch and a shell, and no view of the other runs. All nine got the same brief, word for word, apart from the folder name.
3. Quote check: a script fetched every cited page once and looked for each quote, ignoring spacing, punctuation and letter case. Reddit threads were read from the Arctic Shift archive.
4. Known facts: 25 facts from our published guide to the topic, each tied to the passage that states it. The list was written before any run finished. After the runs we read every miss and added seven alternative passages and one website match, so that a fact quoted from a different page still counts.
5. Further findings: 28 findings outside the known facts, listed after reading the results and matched by passage or by website.
6. Claim against quote: all 593 claims were shuffled and stripped of their model labels. Grader 1 was the Claude Opus 5.5 session that ran the study. Grader 2 was a separate Claude Sonnet 5.5 agent given the same written rule. No person graded.
7. Cost: token counts come from each run’s log and prices from Anthropic’s pricing page, including $10 per 1,000 web searches and Haiku’s higher rate over 100,000 prompt tokens.
8. Audit: the logs were searched for any run opening our own site, the answer list or another run’s folder. None did.

The data files are data.csv (one row per claim), runs.csv, facts.csv and sources.csv. The brief, the grading rule, both graders’ verdicts and every script are listed in method/README.md.

Raw outputs as the models wrote them: Haiku 1, Haiku 2, Haiku 3, Sonnet 1, Sonnet 2, Sonnet 3, Opus 1, Opus 2, Opus 3. Pilot: pilot/runs.csv and pilot/data.csv.

Limitations

• One topic, one day, three runs per model. A gap of a few points between two models is inside the gap between two runs of the same model.
• The known facts came from our own guide, which a Claude session researched earlier the same day. A model could find true facts outside that list. Those are counted as further findings, on a list written after the results were in.
• The graders are AI models, and they are the same models as two of the three tested. They could not see which model wrote a claim. No person checked their verdicts.
• The quote check has limits. It shows that the words are on the page. It does not show that the page is right or up to date. Several runs flagged Google help pages that contradict each other.
• Cost is an estimate at list prices. Output tokens were estimated and hidden reasoning was not counted, so true costs are somewhat higher, most of all for Opus.
• The tools shaped the result. Another setup, with a Reddit archive or a different search tool, would give different thread results.
• The brief was a careful one. We did not test a loose brief, or a brief without the warning that quotes would be checked.
• Model versions change. These results describe three models as they ran on October 13, 2026.
• Not legal advice. Nothing here says whether a firm should accept recorded calls.

FAQ

Can a cheap AI model do online research without making up quotes?

In this test, yes. Copy for Originals gave the same research brief to Claude Haiku 5.5, Sonnet 5.5 and Opus 5.5 three times each on October 13, 2026. All 593 quotes the nine runs returned were found word for word on the pages they cited, including all 132 from Haiku, the cheapest model. The brief required a copied quote and a page address for every claim and said each quote would be checked by a script.

When is Claude Opus worth the cost as a research agent?

When the brief runs once and nothing may be missed. In the Copy for Originals test, every Opus 5.5 run found all 25 known facts, Sonnet 5.5 runs found 23 to 24, and Haiku 5.5 runs found 19 to 23. Opus also cited about twice as many pages as Haiku and had the lowest share of claims that said more than their quote showed (3.8%, against 9.9% for Sonnet and 9.8% for Haiku). An Opus run cost an estimated $2.79 at list prices, against $0.28 for Haiku.

How do you stop an AI research agent from inventing sources?

Require evidence that a script can check. In the Copy for Originals test, each claim had to carry the page address and a passage copied character for character, and the brief said every quote would be machine-checked. A separate script then fetched each page and looked for the quote. It found 593 of 593. A second check is still needed, because 41 of the 593 claims (6.9%) said more than their quote showed.

Sources

All read on October 13, 2026.

• Anthropic, “Pricing” (model prices, cache prices, web search fee, Haiku 5.5 prices over 100,000 prompt tokens). platform.claude.com/docs/en/about-claude/pricing
• Copy for Originals, “Does Google record calls to law firms from its ads?” (the source of the 25 known facts). tools.copyfororiginals.com
• Arctic Shift, public archive of Reddit posts and comments, used to check Reddit quotes. arctic-shift.photon-reddit.com
• The 55 pages the nine runs cited are listed with their addresses in sources.csv.

How to cite: Copy for Originals, “Haiku, Sonnet or Opus for online research? One brief, nine runs, 593 checked quotes,” October 13, 2026, https://tools.copyfororiginals.com/haiku-vs-sonnet-vs-opus-research-study/. Data licensed CC BY 4.0.