中文

MXDATA / Data and method

Data and method

Read the collection method, definitions and limits behind each count. MXDATA records published signals; it does not decide their legal meaning.

7,886 reachable sites · 5,767 valid robots.txt files · 2026-10-16

Signals we collect

SignalLocationRecorded fields
robots.txt AI rules (RFC 9309)/robots.txtFor each of 36 named AI crawlers: full block, partial block, allowed or not named. The wildcard * group is recorded separately.
Content SignalsContent-Signal: lines in robots.txtDeclared values for search, ai-input and ai-train
RSL (Really Simple Licensing)License: in robots.txt, or <link type="application/rsl+xml">License URL, permitted and prohibited uses, and up to 2 license files hosted on the same site
TDMRep/.well-known/tdmrep.json, tdm-reservation header or metaPresence of a reservation signal and the file when available
ai.txt/ai.txtPresence of the file and its contents
noai / noimageai<meta name="robots">, X-Robots-TagPresence of either directive

Named AI crawlers

2026-10-16 · 5,767 valid robots.txt files

CCBotCommon Crawltraining72398890
GPTBotOpenAItraining7062031,068
BytespiderByteDancetraining69161781
ClaudeBotAnthropictraining634149918
Google-ExtendedGoogletraining529149823
meta-externalagentMetatraining50776627
Applebot-ExtendedAppletraining49878624
cohere-aiCoheremixed48149554
anthropic-aiAnthropiclegacy48071603
AmazonbotAmazonmixed47783610
DiffbotDiffbotmixed47118502
omgiliWebz.iomixed46313479
PerplexityBotPerplexitysearch454172783
Claude-WebAnthropiclegacy38847480
ChatGPT-UserOpenAIuser triggered377197727
YouBotYou.comsearch37637435
FacebookBotMetalegacy37042443
TimpibotTimpimixed3278337
ImagesiftBotImageSiftmixed30810321
Webzio-ExtendedWebz.iotraining2718280
OAI-SearchBotOpenAIsearch262200618
AI2BotAi2training25720280
DuckAssistBotDuckDuckGouser triggered25447318
Meta-ExternalFetcherMetauser triggered25028294
Perplexity-UserPerplexityuser triggered24583381
Claude-UserAnthropicuser triggered23487384
cohere-training-data-crawlerCoheretraining22916248
Claude-SearchBotAnthropicsearch22698384
Google-CloudVertexBotGooglemixed21318240
MistralAI-UserMistral AIuser triggered19628242
PanguBotHuaweitraining1958203
Kangaroo BotKangaroo LLMtraining1695175
Ai2Bot-DolmaAi2training16311175
BrightbotBright Datamixed83386
Gemini-Deep-ResearchGoogleuser triggered8121109
TikTokSpiderByteDancemixed7710102

The site panel

We follow the top 10,000 domains in Tranco list Y8JGG, dated 2026-10-15. This panel stays fixed, so changes reflect sites rather than a moving sample. Its list file has SHA-256 1e49477006b7d63dced741c2ce51666463c31d9881248df5ec7f4fafbb1376cf. Expanded coverage will use a new panel while this one continues.

1e4947…76cf1e49477006b7d63dced741c2ce51666463c31d9881248df5ec7f4fafbb1376cf

Collection steps

  1. Each day, from 01:07 UTC, we request https://{domain}/robots.txt. If needed, we try the www. hostname.
  2. If robots.txt allows our crawler, we request /ai.txt, /.well-known/tdmrep.json, the homepage and up to two same-site RSL license files. From the homepage, we retain only the <meta> and <link> tags in <head>.
  3. Requests to a site run in sequence, at most one per second. Collecting the full panel of 10,000 sites takes about 21 minutes.
  4. User-Agent: MXDATA-signals/1.0 (+AI usage-rights signal archive; daily, low-rate; robots.txt respected). Collection details are on the crawler page.

Keeping the evidence

  • Response details: requested and final URLs, HTTP status, response headers with cookies removed, UTC timestamp, latency and body SHA-256.
  • Original files: robots.txt, ai.txt, tdmrep.json and RSL files are retained verbatim and deduplicated by SHA-256. For homepages, we retain <meta> and <link> tags and the hash of the full <head>.
  • Daily manifests: each snapshot has a manifest of file hashes, linked to the previous day's manifest hash. Editing an archived file breaks the verification chain. Latest manifest: 7e56985581858d76305bd18d7a9562b36950ef7a54c965acdbdc331615feb130.
  • Reproducible parsing: signal tables can be rebuilt from the stored files. Parser corrections update derived history; archived bytes remain unchanged.
7e5698…b1307e56985581858d76305bd18d7a9562b36950ef7a54c965acdbdc331615feb130

Detecting changes

We compare consecutive daily snapshots. Robots.txt signals are compared only when both days have a valid robots.txt; homepage signals only when both homepages return 200. For ai.txt and tdmrep.json, both days must return a definitive result: 200, 404 or 410. These checks reduce outage-related false alarms. Each detected change includes the before and after file hashes.

What the data misses

  • 2,114 of 10,000 sites were unreachable to our crawler on 2026-10-16, often because of bot protection. We count these as unreachable, not as "no signal".
  • We inspect only the homepage. Signals on individual articles or added by JavaScript are outside this collection.
  • Full and partial robots.txt blocks are our interpretation of RFC 9309 matching rules. The original file remains available for review.
  • The fixed panel favors popular sites and does not represent the web as a whole.
  • Our history starts on 16 October 2026. We have no records from earlier dates.
  • A signal records what a website states. Whether it is binding, and on whom, is a legal question we do not answer.

Download the aggregates

Download the aggregate data under CC BY 4.0. Site-level records are available through a sample report request.

latest.json ↓history.json ↓

Need records for your sources? Request a sample report.

Request a sample report