Free tool · by Zenrows
See what an agent is allowed to read.
Paste an llms.txt or a robots.txt. It parses the file and tells you, in plain terms, what an AI crawler may fetch, the sections and links you offer, or the rules you set for GPTBot, ClaudeBot and the rest.
Runs entirely in your browser. Recognises the common AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, and more). Nothing you paste leaves this page.
Agents read the rules before the page.
An llms.txt invites them to your best content; a robots.txt tells named crawlers what is off limits. Getting these right decides whether AI sees your site at all.
Invite
llms.txt points agents straight at the pages worth reading.
Gate
robots.txt names which AI crawlers may enter, and where.
Verify
One glance shows whether a bot is welcomed or shut out.
What it checks.
It reads the file the way a crawler would, then reports it back in plain language.
01
Detects the format
llms.txt or robots.txt, from the shape of the file.
02
Maps the access
Sections and links for llms.txt; allowed and blocked crawlers for robots.txt.
03
Flags the gaps
Missing title, no links, or an AI bot shut out at the root.
Read any page, rules or not.
When you need the content behind the gate, Fetch reaches it as a real browser and returns it clean.
# fetch any page, clean markdown back curl -G "https://api.zenrows.com/v1/" \ -d "apikey=ZR_••••••••" \ -d "url=https://example.com/page" \ -d "response_type=markdown"