“Useful diagnostic, terrible KPI”: what 200,753 Reddit posts say about measuring AI visibility

What did the study find?

1. The conversation is growing fast. Posts that mention AI search rose from 1.5% of all posts in January 2025 to 8.7% in September 2026, peaking at 9.0% in June 2026. In the technical SEO forums it is now more than one post in ten.
2. Measurement is growing faster. Posts about tracking, reporting or attributing AI visibility rose about 21-fold as a share of all posts, from 0.14% to 2.97%.
3. The most common wall is Google’s own data. 60 of 308 threads struggled with how Search Console handles AI Overviews and AI Mode. The new generative AI report drew praise in 17 threads and the complaint that it has no query data in 23, no clicks in 18.
4. The tools are nearly as contested as the gap. 55 threads questioned the cost or accuracy of AI visibility trackers: synthetic prompts, API answers that differ from the app, tools that disagree, prices from $99 to $400 a month and up.
5. There is no data from the assistants themselves. 45 threads said so outright. 40 described answers that change between runs, and 40 debated which prompts to track, with one participant noting that the right choice of prompts can make any dashboard look good.
6. The number is wanted from above. 38 threads described clients, bosses or boards demanding an AI visibility report; 36 said nobody could tie the visibility to revenue.
7. Two models, one codebook, a stricter answer. A first coding pass by Claude Sonnet 5.5 judged 193 threads relevant; the published Opus 5.5 pass, told to count only substantive discussion, judged 155. The two agreed on 91% to 97% of threads per problem, and almost every disagreement was the first pass counting something the second did not.

How fast is the AI search conversation growing?

We pulled every post from seven marketing subreddits between January 1, 2025 and October 15, 2026 from the Arctic Shift archive, 200,753 posts in all, and matched each against a list of terms: AEO, GEO, generative or answer engine, AI Overviews, AI Mode, AI visibility, AI search, AI or LLM traffic, ChatGPT, Perplexity. 10,840 posts matched, 5.4% of the total. A stricter count that drops posts matched only by the name ChatGPT or Perplexity gives 7,239. Adding a tracking, reporting or attribution term to the strict count gives 2,649 posts about measurement.

The shares tell the story better than the counts, because the subreddits themselves shrank and grew over the period, and one of them changed its posting rules in July 2025. In January 2025, one post in 66 mentioned AI search. In September 2026, one in 11 did. The measurement share rose from 0.14% to 2.97%, which is the 21-fold figure in the summary.

The technical forums lead. In r/TechSEO and r/bigseo more than one post in ten mentions AI search; in r/marketing and r/PPC it is about one in fifty. Measurement follows the same gradient.

What can’t marketers measure?

Counting posts shows that people are talking. To learn what they are stuck on, we read 308 threads with 9,190 comments: the 60 measurement threads with the most comments (the smallest had 43), every thread since January 2026 that mentioned Search Console near a reference to AI (216), the 30 most-discussed threads about choosing prompts, and the 20 most-discussed threads that mentioned healthcare. Two sets of AI agents coded each thread against the same written list of twelve problems. 155 threads turned out to be about measuring AI visibility at all; the rest were news digests, removed posts or ordinary Search Console questions.

The table ranks the twelve problems by the number of threads in which someone discussed them in substance. A thread can raise several; the median relevant thread raised three.

What follows is the texture behind the counts: for each problem, how people describe it in their own words. The quotes were chosen to be representative and were checked against the archive; promotional comments were excluded.

Search Console and AI Overviews (60 threads)

The most common complaint is about the one tool everyone already has. Until June 2026, impressions from AI Overviews sat inside Search Console’s ordinary totals with no way to separate them, and every link in an AI Overview was given the same position. The new generative AI report, discussed below, fixed the first problem for impressions only.

Been feeling like impressions, CTR, and positions are all unreliable with the AI data lumped in. I really wish it was separated.

But the report still gives you no queries, clicks, or CTR. It also combines AI Overview and AI Mode impressions.

Tool cost and accuracy (55 threads)

The tools that sell a solution draw nearly as many threads as the gap they fill. The recurring objections: the prompts are synthetic, the answers come from an API that doesn’t behave like the app, two tools disagree, and the price is hard to justify to a client.

SEMRush is $99 a $99-a-month add-on, and the Ahrefs add-on is $199 per platform or $699 a month for all five. My opinion, not worth it.

We've looked at some pa͏id stuff but like good luck telling a client we need $400/mo for a tool that won't even explain why something is/isn't working.

There are tools that do this by pulling from the APIs, and I’m unsure if that’s a true result. So right now, manual tracking.

Content-format claims (50 threads)

Threads about measurement keep turning into threads about tactics, and the tactic most often proposed is a format: answer first, add an FAQ, use a table, add schema. Most of these claims are offered as experience, and some participants push back.

Answers wobble between runs, so read the trend across the whole list, not any single prompt.

Tbh everyone acts like schema clusters are some magic bullet for llm seo but it feels overhyped.

“It’s just SEO” (47 threads)

A large minority doubts that there is anything new to measure. The sharpest version of the argument concedes one point: whether a brand appears at all is worth tracking, and the rest is packaging.

GEO is basic SEO with new packaging. The only part worth tracking is whether your brand is being surfaced at all.

The amount of people that think SEO and GEO are the same thing is mad. SEO is driven by human searches. GEO by machine searching. Man and machine search differently.

No data from the chat assistants (45 threads)

For ChatGPT, Gemini, Claude and Perplexity there is nothing to open. No query report, no impression count, no equivalent of Search Console. Everything downstream, from the tools to the dashboards, is an estimate built on that absence.

The measurement problem is real though. There's no Google Search Console equivalent for AI citations yet.

Theres a whole bunch of AI mention tracking tools, semrush has one too. But honestly I don't think they're productive.

Answers change between runs (40 threads)

Ask the same question twice and the answer changes. Practitioners who have tried to track a position in an AI answer describe giving up, and those who kept going describe averaging many runs.

Run each prompt several times per check, because the answer set moves between sessions and a single run is noise dressed as a finding.

ai overviews feel like they reroll every other week with zero changelog. one tuesday youre in half your queries and by friday its a different set of urls and nobody at google says anything tbh

Which prompts to track (40 threads)

If there is no data on what people ask, someone has to decide which questions to track, and that decision shapes every number that follows. The threads describe shortlists of twenty to fifty buyer-intent prompts, built from sales calls or Search Console, and a shared unease about who picks them.

The prompt universe part is what makes this tricky. You can make the dashboard look amazing just by choosing the right prompts 😂.

The problem with these tools is that they’re creating synthetic prompts based on keywords and calling it prompt tracking..

The same prompt can return different brands depending on context, follow-up questions, and whether the model searches the web.

Client and boss pressure (38 threads)

The demand for a number comes from above. Clients ask why they are not in ChatGPT; boards want to be kept calm; the person in the middle has a report to deliver and no data to put in it.

My VP asked me for a "monthly AI visibility report" last quarter and I seriously considered just making the numbers up.

yeah we got asked about this a couple months ago by one of our clients. why aren't they in Cha͏tGPT for searches around theier product. and we had nothing to tell them. felt pretty dumb honestly.

They have about 6 new AI tools to monitor and report how we perform in llm visibility versus rivals, specially to keep leadership and the board calm.

Can’t tie it to revenue (36 threads)

Even where visibility can be counted, the path from an AI answer to a sale is dark. The threads treat AI search like an awareness channel: real, probably valuable, and resistant to the attribution models that run the rest of the budget.

GSC shows impressions but doesnt break out AI overview vs regular blue links separately which makes attribution messy

Ai search ROI is hard to nail down, kind of like broad awareness channels.

Analytics and dark traffic (33 threads)

Some AI visits do arrive, and analytics files many of them as direct traffic. The workarounds people trade are a referral filter for the AI domains, a UTM check, and the oldest instrument in marketing: asking the customer how they heard of you.

had a call last week where someone casually said they found us through ChatGPT, not Google. wasn’t even something we were tracking, so it caught me off guard.

GEO isn't a new discipline; it's just a fancy rebrand for technical SEO fundamentals we've been doing for years Schema, entity tagging, and proper citations, wrapped in a fresh layer of AI-induced anxiety.

Query fan-out (27 threads)

A smaller group has noticed that an AI assistant answers one question by running several searches of its own, and that those background searches, not the user’s prompt, decide which pages it reads. The point is raised both as a tactic and as another thing nobody can see.

Look at the core answer, but more importantly, figure out all the **hidden follow-up prompts** it used behind the scenes to build that response.

For the 1000th time: LLMs are not search engines - they do not have their own ranking engine,their crawler fetch output from a QFO mostly outsourced to Google

Diagnosing lost clicks (26 threads)

The pattern is familiar by now: rankings hold, impressions rise, clicks fall. Proving that an AI Overview did it is harder than noticing it, and some participants warn against blaming AI for every drop.

It’s frustrating. We’re #1 for our target search terms. Our data and content is high-quality enough that the AIO is constantly referring to us when giving answers. But we’re not getting the clicks, because why click when AIO is already giving you our content?

How do you know AI overviews are the problem? Or are you just suggesting that the general trend is more AI overviews, therefore they are contributing? But you don't know how much/how often?

What do practitioners say about Search Console’s generative AI report?

Google announced the “Search Generative AI performance reports” on June 3, 2026, first for a subset of sites and, from August 31, for every site. The report shows how often a site’s pages appeared in AI Overviews and AI Mode, by page, country, device and day. It shows nothing else: no clicks, no click-through rate, no queries, no split between the two features, and no API. Those are Google’s own terms, from its help pages, and they match what practitioners found when the report reached them.

40 of the threads we read discussed the report itself. The reaction divides neatly into relief and disappointment, often in the same comment.

The relief first. For two years the only view of AI visibility on Google was inference, and here was a number from Google.

And an "impression inside an AI Overview" isn't the same as being "cited as a source" - you can be summarised without being the named reference.

Super useful! It's like getting AI visibility for Google from the horse's mouth.

For now, though, the page-level data is already handy: if one URL suddenly starts picking up AI impressions, that's a pretty good signal to investigate what it's doing right.

Then the disappointment, which has two parts. The report shows that a page appeared, but not for what question or with what result.

No search queries either. It's utterly useless.

GSC finally reports AI impressions (June 2026): a dedicated Generative AI report, impressions only, no clicks/queries yet, and no API.

Only goes back to 18th May and shows only impressions (drilled down to pages / countries / devices / days).

And a number that only goes up as Google shows more AI features is a poor measure of anyone’s work. The comment that gave this study its title made the point in four words; the post it answered made it at length.

Google's language is that it will add additional metrics over time, which is not a commitment to a date or to clicks specifically.

useful diagnostic, terrible kpi

certainly interesting to look at, but I just don't really see how it is necessarily helpful to me. what do i do with the info?

8 threads pointed across the street. Bing Webmaster Tools has shown Copilot citations since February 2026, together with a sample of the “grounding queries,” the search phrases the AI actually ran before it answered. Google shows no equivalent.

Question for the group: what's your take on impression volume as a metric? Is AI impression count meaningful on its own without clicks?

Luckily, Bing WMT has grounded queries for us, and Google has some data (minus keywords) for AI performance.

One thing may change. On June 3, 2026 the UK Competition and Markets Authority required Google to give publishers AI impressions, clicks and click-through rate “fully disaggregated from other elements of general search,” through “a commonly accessible platform, such as Google Search Console,” within six months. That falls in early December 2026. Whether the same data reaches sites outside the UK is not stated.

Who is talking about healthcare?

Almost nobody. Health terms appear in about one in a hundred of the posts about AI search, and 8 of the 308 threads we read were about a healthcare business. That is a gap rather than a finding, and it matters for two reasons. Health is the industry where Google shows AI Overviews most often, at least in Conductor’s benchmark of ten industries, where they appeared on 48.7% of page-one health queries. And the sources those answers cite are rarely individual practices: in Google’s AI Overviews the most-cited health pages are news and health publishers, and in ChatGPT, Gemini and Perplexity, directories such as WebMD and Vitals account for 52.6% of healthcare citations, according to Yext. Both measurements come from vendors that sell visibility tools.

Where health does come up, the setting is local. A dental practice watches a competitor get named for “best dental clinic in my area” and asks how; the replies say ordinary local SEO. Another thread notices the thing that makes local measurement hard in the first place.

Do that consistently and you’ll start showing up in AI answers too. It’s just SEO with extra steps.

Same brand, same prompt, different city = different answer, built from different local sources.

How much did the coding depend on the coder?

Counts like these depend on judgment, and we wanted to know how much. So the same 308 threads were coded twice against the same codebook. The first pass used Claude Sonnet 5.5 agents. The second, which is the one published above, used Claude Opus 5.5 agents with one added instruction: count a problem only when a participant discusses it in substance, not when a keyword passes by.

The two passes agreed on 91% to 97% of threads per problem, with Cohen’s kappa between 0.65 and 0.91. The disagreements ran one way. Across the twelve problems, the first pass counted 190 thread-problem pairs that the second did not, and the second counted 12 that the first did not. The first pass also judged 193 threads relevant against 155. The ranking of problems is nearly the same in both passes; the counts in the first are 36% higher overall. Readers who want the looser count can rebuild it from first-pass/threads.csv.

Every quote both passes produced was checked by script against the archived post or comment: all 971 quotes from the published pass matched exactly, as did all 1,204 from the first.

How was the study done?

1. Source: the Arctic Shift archive of Reddit, read through its public API on October 15, 2026 from an ordinary Swedish home connection. Reddit itself blocks automated reading. The archive’s own monthly counts were compared with ours for every subreddit and month, and they match.
2. Scope: every post from January 1, 2025 to October 15, 2026 in r/SEO, r/bigseo, r/TechSEO, r/DigitalMarketing, r/marketing, r/PPC and r/agency: 200,753 posts. r/GoogleSearchConsole was also swept and had one post in the period.
3. Term matching: title and body, case-insensitive, against the term list above. “GEO” counts only in capitals, or in lower case when the post also mentions AI or LLMs and is not about geographic targeting, because “geo” is everyday advertising shorthand. In the forums where moderators remove most post bodies (74% in r/SEO, 61% in r/marketing), removed posts were matched on title alone.
4. Thread selection: four rules, described above, chose 308 threads. Comments were pulled for each: 9,190 in all.
5. Coding: a written codebook with twelve problems, a relevance flag, a healthcare flag and a list of specific points about Search Console’s generative AI report (method/CODEBOOK.txt). Each thread was coded by a Claude Sonnet 5.5 agent and, independently and without sight of the first pass, by a Claude Opus 5.5 agent told to count only substantive discussion. The Opus pass is published. No person coded.
6. Quote check: every quote the coders supplied was required to be an exact, contiguous substring of one archived post or comment. A script checked all of them. Quotes from comments the coders flagged as promotional are excluded from this page.
7. Themes: the coders’ points about the generative AI report were grouped into nine themes by keyword rules (method/gsc_themes.py), counting distinct threads.
8. Agreement: percent agreement and Cohen’s kappa between the two passes, per problem, over all 308 threads.

The data files are monthly_counts.csv, threads.csv (one row per coded thread), quotes.csv (one row per checked quote, with its permalink), gsc_report_themes.csv, agreement.csv and first-pass/threads.csv. The scripts that swept the archive, filtered, pulled comments, batched threads for coding, verified quotes and built this page are in method/. The raw archive pull (about 130 MB) is not published; the scripts rebuild it.

Limitations

• The threads were selected, not sampled. Long threads and threads mentioning Search Console near AI were favoured, so the counts describe those 308 threads, not Reddit as a whole. Long threads also collect more problems.
• The coding is by AI agents. Two passes with the same codebook differed by 36% in total count, almost all in one direction. The published counts are the stricter ones. No person coded a sample for comparison.
• The term list defines the trend. A post about “Google’s AI answers” without any listed term is not counted. The share figures are a floor.
• Most post bodies are gone. Moderators had removed the body of 44% to 74% of posts in six of the seven subreddits by the time of the sweep. Those posts were matched on title only, which also lowers the shares.
• Reddit is not the market. These forums skew toward SEO practitioners and agencies, and undisclosed promotion is common. Flagged promotional quotes were excluded; unflagged promotion surely remains.
• “Just SEO” and the format claims are the most interpretive codes, and the two passes agreed on them least often after the revenue code.
• The subreddits changed under us. r/marketing’s daily post count fell by about four fifths on July 25, 2025, and r/TechSEO’s roughly doubled from February 2026. Shares normalise for this; counts do not.

FAQ

How many marketers are asking about AI search on Reddit?

More every month. Across 200,753 posts in seven marketing subreddits, the share that named AI search (AEO, GEO, AI Overviews, ChatGPT and similar terms) rose from 1.5% in January 2025 to 8.7% in September 2026, and 9.7% in the first half of October. In r/TechSEO and r/bigseo it is now more than one post in ten. Posts specifically about measuring or reporting AI visibility rose about 21-fold, from 0.14% to 2.97% of all posts.

What is the hardest part of measuring AI visibility, according to practitioners?

Of 308 Reddit threads coded by Copy for Originals, the problem raised most often was Google Search Console’s handling of AI Overviews and AI Mode (60 threads), followed by the cost and accuracy of AI visibility tools (55), claims about which content formats get cited (50), the view that AEO is just SEO renamed (47), and the absence of any first-party data from ChatGPT, Gemini, Claude or Perplexity (45). Answers that change between runs (40 threads) and the choice of which prompts to track (40) follow close behind.

What do practitioners think of Search Console’s generative AI report?

Useful but thin. Of 40 threads that discussed the report, 17 welcomed it as the first separate view of AI visibility from Google, 23 complained that it has no query or prompt data, 18 that it shows impressions but no clicks, 15 that it reached some sites and not others, and 15 that it changes no decisions. The compressed verdict from one thread was “useful diagnostic, terrible kpi.” 8 threads said Bing Webmaster Tools, which shows the search phrases Copilot actually ran, does it better.

Sources

Reddit threads are cited by permalink under each quote and in quotes.csv. The archive and the Google, Bing and CMA documents below were read on October 15, 2026.

• Arctic Shift, public archive of Reddit posts and comments. arctic-shift.photon-reddit.com
• Google Search Central, “Introducing the Search Generative AI performance reports,” June 3, 2026, with the note that the reports reached all sites on August 31, 2026. developers.google.com
• Google Search Console Help, “Generative AI performance report (Search)” (impressions by page, country, device and date; no query dimension). support.google.com
• Google Search Console Help, “Performance report: AI features” (one position for every link in an AI Overview; an AI Mode follow-up counts as a new query). support.google.com
• Bing Webmaster Blog, “Introducing AI Performance in Bing Webmaster Tools (Public Preview),” February 10, 2026 (grounding queries as “a sample of overall citation activity”). blogs.bing.com
• Conductor, “Health Care Industry: 2026 AEO / GEO Benchmarks” (AI Overviews on 48.7% of page-one health queries, the highest of ten industries; news articles the most-cited page type), last updated April 14, 2026. Conductor sells AI visibility software. conductor.com
• Yext, “86% of AI Citations Come from Brand-Managed Sources” (6.8 million citations across ChatGPT, Gemini and Perplexity; healthcare listings 52.6%), October 9, 2025. Yext sells listings management. yext.com
• Competition and Markets Authority, Publisher conduct requirements: interpretative notes and notice, June 3, 2026. Interpretative notes (PDF), Notice (PDF)

How to cite: Copy for Originals, ““Useful diagnostic, terrible KPI”: what 200,753 Reddit posts say about measuring AI visibility,” October 16, 2026, https://tools.copyfororiginals.com/ai-visibility-measurement-reddit-study/. Data licensed CC BY 4.0.