Blog · Published · Updated · By Patrick Coombe

Do LLM Bots Actually Use Cloudflare's Markdown for Agents?

Live experiment

TL;DR - Cloudflare's Markdown for Agents works, it generates markdown when requested.. Ask for markdown with an Accept: text/markdown header and you get a clean markdown version of the page, roughly 95% smaller than the HTML. The open question is whether the big LLM crawlers actually request it. I am testing it on a high-traffic client site of mine that is cool with testing. First week of clean data (below): one small agent crawler I had never heard of took almost all the markdown, and every big-name LLM bot crawled the site and grabbed the HTML instead.

Scope: this ran on a client site - a busy car dealer on WordPress behind Cloudflare Pro (not this site) so the traffic numbers are real production numbers. Want to do this yourself? I wrote a step-by-step guide to enabling Markdown for Agents, gotchas and all.

Reset - Day 1 is today (July 8, 2026). I first logged this with a Cloudflare Worker piping hits into a Discord channel. It worked, but it had issues, it needed babysitting, and I let my logging get muddled enough that I stopped trusting the counts. So I wiped the slate and switched to Cloudflare Log Explorer - raw edge logs I query directly with SQL, which is a lot cleaner. I am restarting the clock from today, so every number below is Day 1 forward.

Day 1 to 8: who actually pulled the markdown

Here is the whole point of the experiment, straight off Log Explorer: every request from July 8 (Day 1) through July 15, real production traffic on the car dealer. "Markdown pulls" means the edge actually returned text/markdown, not just that the bot showed up.

Crawler activity on the car dealer, Log Explorer Day 1 to 8
CrawlerRequestsMarkdown pulls
ShapBot / Shap-User1,001185
Googlebot23,6570
Bingbot16,0820
ChatGPT-User3,1210
GPTBot1,8850
ClaudeBot1,5070
OAI-SearchBot5820
PerplexityBot2580

One crawler took almost all of it. It is ShapBot / Shap-User, running out of Google Cloud, and it was not on my radar at all. It uses the same split OpenAI does - a -Bot that crawls and a -User that fetches on demand - and across the week it made about a thousand requests and asked for markdown 185 times. The only other genuine markdown taker was a single hit from something calling itself AgentRadar-Research, out of the BCG Henderson Institute, doing agentic web research. That was the whole list of real markdown consumers.

Every crawler you would actually name - Googlebot, Bingbot, GPTBot, ChatGPT-User, ClaudeBot, OAI-SearchBot, PerplexityBot - crawled the site plenty and took the HTML every single time. Googlebot alone hit it more than 23,000 times and never once asked for the markdown version. So the feature is real and it is getting used, just not yet by the names that would move the needle.

One more thing worth flagging. I also logged 71 "markdown" pulls from a "Googlebot" user-agent that was not Googlebot - real Googlebot lives on one network (AS15169) and these came from 26 different random hosting networks. If you trust the user-agent string you will book those as real adoption and be wrong. I threw them out and counted by network, not by name. If you take one operational thing from this post, take that.

My goal for Cloudflare Markdown For Agents Monitoring

I enabled this a few weeks ago and started to log. I checked in on it this week (glad I did) and saw almost zero markdown hits except malware. I wanted to know whether any AI agents were actually requesting the markdown representation of the page, who they were, and how often - basically, is this worth keeping and optimizing, or is it a feature sitting there that nobody uses?

How I checked it even works

Instead of testing an existing source, I triangulated:

  • Edge tests (curl): confirmed Accept: text/markdown returns markdown.
  • Origin logs (SSH, nginx): measured AI-bot crawl volume by user-agent.
  • Origin direct test: proved the origin only ever serves HTML, so the markdown conversion is 100% Cloudflare's edge, not the server.
  • Cloudflare GraphQL Analytics: pulled the real edge content-type split and per-bot behavior.
  • AI Crawl Control dashboard: cross-checked the feature status and fulfillment.
  • I did not touch the server config - my network admin would kill me.

The feature works - and it is tiny

A GET with Accept: text/markdown returns content-type: text/markdown with a clean markdown body (YAML front-matter plus content), roughly 95% smaller than the HTML. AI Crawl Control reported "96% of markdown requests fulfilled."

What a markdown grab actually looks like

I wanted to see if what Cloudflare was outputting actually looked worthy of the page. I looked at a VDP (a vehicle page) because that is arguably the most important type of page on a car dealer. The highlights of what it grabbed:

  • Title and description with the VIN baked into the description
  • Basic info tables (Body: SUV, Mileage: 12,234, 21 hwy / 15 city MPG, Black/Black, 3.0L V6, ZF 8-speed)
  • Full DESCRIPTION, DETAILS (the entire feature list - nav, harman/kardon, etc.), TERMS & CONDITIONS
  • Price: $24,888.00, financing CTAs, VIN XYZ / STOCK MXYZ
  • "Other vehicles you may like," full dealer NAP and hours
  • It grabbed all the JSON-LD, which contains all the images of the car
  • All of Yoast's output

Size: 23 KB markdown vs 369 KB HTML - about 16x smaller.

Does this actually matter for SEO?

WHO KNOWS?! Maybe, and to be honest, probably. If LLMs can spend 96% less to get the exact same data, why wouldn't they? If they don't now, they will soon. Some now, some later. My opinion, not facts.

Things I fixed and learned the hard way

  • Vary: Accept was missing on HTML responses. Without it, caches can serve cached HTML to a markdown-negotiating client. Fixed with a Cloudflare Response Header Transform Rule (set static Vary = Accept-Encoding, Accept). This is what broke basically everything.
  • Edge caching and markdown coexistence. Added a Cache Rule (HTML GET pages, 4h Edge TTL) that excludes Accept: text/markdown and bots, so browsers get cached HTML (HIT) while agents still get freshly-converted markdown on the same URLs. I tested this several different ways.
  • The testing gotcha (this cost me a full hour): curl -I (HEAD) returns text/html even when the feature works, because HEAD has no body to convert. Always test with GET. A false "it broke" panic came entirely from using -I.

How I am logging it (and why I reset)

Version one was a homegrown Cloudflare Worker (a transparent passthrough) that logged every Accept: text/markdown GET to Workers Analytics Engine and fired a 30-minute Discord digest when there were hits. Here is what that looked like:

Discord digest reading: Markdown for Agents, last 30 min, 9 hits - other times 7, gptbot times 1, claudebot times 1, paths slash, slash all-vehicles, slash contact
The old approach: a 30-minute Discord digest from a homegrown Worker. It worked, but it was noisy and easy to muddle.

It did the job, but it was a moving part I had to babysit, and I let the logging get tangled enough that I stopped trusting the numbers. So I tore it out and switched to Cloudflare Log Explorer instead: raw, unsampled edge requests I query directly with SQL, including edgeresponsecontenttype, which flags a genuine markdown response so I can separate content-negotiated markdown from direct .md fetches in a single query. No Worker to maintain, no digest to babysit, and the data is straight off the edge rather than a sampled dashboard. It is a lot cleaner, which is why I reset the experiment to Day 1 today and started the count over on Log Explorer.

Conclusions

The first clean week backs up what I suspected. Markdown for Agents is technically working and worth keeping - it is a forward-looking bet with basically zero downside - but the crawlers you actually care about are taking the HTML and ignoring the markdown. The only real adoption came from one obscure agent (ShapBot / Shap-User) and a one-off research fetch. That is a reason to turn it on now and be ready, not a reason to expect fireworks in your analytics tomorrow.

I am leaving Log Explorer running and will keep updating this table as the weeks add up - the interesting question is whether the big names (GPTBot, ClaudeBot, PerplexityBot) ever flip from HTML to markdown, and how long that takes. If you want to set this up on your own site in the meantime, the how-to guide is here.