MXDATA / Crawler
Our crawler
Saw MXDATA-signals in your logs? Here is what it requests, how often it visits and how to opt out.
Crawler identity
MXDATA-signals/1.0 (+AI usage-rights signal archive; daily, low-rate; robots.txt respected)
robots.txt user-agent: MXDATA-signals
Files we request
| Resource | Collection rule |
|---|---|
/robots.txt | Once per day |
/ai.txt, /.well-known/tdmrep.json, / (homepage) | Once per day, if robots.txt allows MXDATA-signals |
RSL license files, same site only, up to 2 | Once per day, when referenced and allowed |
Request limits
Requests to a site run one at a time, at most one per second. We run the collection once per site per day.
Outside our scope
We do not log in, submit forms, run JavaScript, fetch article pages or images, or collect personal information.
Stop the crawler
User-agent: MXDATA-signals Disallow: /
We stop requesting other files when your robots.txt includes this rule. We continue to read robots.txt once a day to check your instructions and keep a record of that public statement. To remove a domain from the panel entirely, contact us by email.
Crawler questions
We aim to reply within 2 business days.
Crawler IPs
IP ranges will be listed here when fixed addresses are available.
Need records for your sources? Request a sample report.
Request a sample report