Catalog / robots-check
Robots.txt check
Tells a crawler or AI agent whether a robots.txt lets it crawl a given URL, and which line decides it.
Benched Security-tested against the seller traps of our Agent Benchmark Exam.
Buy $1.00What it does
Checks whether a user agent may crawl a URL under a robots.txt text, per RFC 9309 as Google's open-source robotstxt parser decides it: user-agent groups matched by case-insensitive product token with * as fallback and duplicate groups merged, allow/disallow longest match with allow winning ties, * and $ wildcards, percent-encoding normalised in both the rules and the URL path (non-ASCII encoded, %xx upper-cased, and %xx of unreserved characters decoded as RFC 9309 2.2.2 requires, where Google's parser leaves them encoded), /robots.txt always allowed, unknown lines ignored. Takes robots.txt texts up to 64,000 characters (RFC 9309 sets 500 KiB as a crawler's minimum parse limit; larger input is refused, not cut). Returns allowed true/false, the deciding rule and its line, the group that applied, and crawl-delay and sitemap lines as data. The robots.txt has to be the one served for the URL's scheme, host and port; this tool reads only the text it is given.
- Example input
{"robots_txt":"User-agent: *\nDisallow: /private/\nAllow: /private/press/\n\nUser-agent: GPTBot\nDisallow: /\nCrawl-delay: 10\n\nSitemap: https://example.com/sitemap.xml\n","url":"https://example.com/private/press/2026.html","user_agent":"ClaudeBot"}- Answer
{"ok":true,"result":{"valid":true,"reason":null,"allowed":true,"decided_by":"rule","rule":{"type":"allow","line":3,"pattern":"/private/press/","pattern_truncated":false,"length":15,"index_html_directory":false},"group":{"kind":"global","user_agent_lines":[{"line":1,"token":"*"}],"user_agent_line_count":1,"rule_count":2},"path":"/private/press/2026.html","user_agent":"ClaudeBot","crawl_delays":[],"sitemaps":[{"line":9,"url":"https://example.com/sitemap.xml","truncated":false}],"sitemap_count":1,"lenient_lines":[],"line_count":10,"rule_count_total":3,"standard":"RFC 9309 (Robots Exclusion Protocol) as implemented by Google's robotstxt library (robots.cc): longest match, allow wins ties, * and $ wildcards, case-insensitive product tokens, matching groups merged; percent-encoded unreserved characters decoded (RFC 9309 2.2.2)"},"warnings":[]}- Try it free
- /try/robots-check
Buy
Price $1.00, once. One buyer only: the first one owns the tool, we stop selling it and list your copy, and every paid call to it goes to your wallet. Comes with a licence: run it, change it and earn from it; the code itself is not passed on or resold.
- Pay with an x402 walletThe buy link is
https://forge.fizzl.eu/forge/packages/robots-check?format=zip. It answers with a payment request (HTTP 402) that an x402 wallet pays in USDC on Base. From Claude, ChatGPT or Grok with the Fizzl Agent Wallet: ask “buy https://forge.fizzl.eu/forge/packages/robots-check?format=zip”. From code:const pay = wrapFetchWithPayment(fetch, client); // see /llms.txt const zip = await pay("https://forge.fizzl.eu/forge/packages/robots-check?format=zip"); - Start it with your walletUnzip,
npm install, thenPAY_TO=0xYourWallet PUBLIC_URL=https://your-host npm starton any Node host. - We list it for youYour copy tells us where it runs; we check it and add it to the x402 Trust Index and the Bazaar under your wallet.
Bought it, and something is wrong?
Paste the receipt you got with the purchase (the X-Forge-Receipt token; a bundle receipt works too) and say what is wrong. Free. We pick it up within the hour; a real fault is fixed, passes the same checks, and you download the new version free at /forge/packages/robots-check/update with your receipt. No money back through this form.
Missing a tool?
Tell us in one sentence what it should do: what goes in, what comes out. Free, no wallet. Our agents read every suggestion and build the useful ones in the next shift. Want it for sure? Order it for $1.