robots.txt Policy Lint — AI Crawler Index

What LocalMark checked

• Published Agent Card

  Inspect the source card ↗. The last successful fetch was 28 Oct 2026, 15:09 UTC.

• Latest card check: passed

  28 Oct 2026, 15:09 UTC · Agent Card validated

• Advertised endpoint: TLS connection passed

  28 Oct 2026, 15:09 UTC · Valid TLS connection to advertised endpoint host; no A2A request sent This does not test the A2A protocol or run a task.

• Read-only protocol probe: A2A task lookup response received

  28 Oct 2026, 15:09 UTC · A2A task-not-found response to a read-only task lookup LocalMark sent no message and did not request task creation.

• Unsigned card

  This card does not provide a digital signature. Checked 28 Oct 2026, 15:09 UTC.

• 30-day card check history

  1 of 1 recorded card checks passed in the last 30 days. These are periodic observations, not continuous uptime monitoring.

• Publisher claim

  No publisher claim has been completed for this listing.

Card availability and a valid signature do not prove provider identity, task performance, or safety. Opt-in synthetic fixture results, if any, are shown separately from these read-only observations.

Recent card checks

Periodic observations over the last 30 days; they are not continuous uptime monitoring.

View the 30-day check log as JSON →

Declared skills 9

• Add a ready-made stance to an existing file

  {"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"merge_policy\",\"robots_txt\":\"User-agent: GPTBot\\nDisallow: /\\n\\nUser-agent: *\\nAllow: /\\n\",\"stance\":\"block-ai-training\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. Merge one of eight maintained robots.txt stances (block-ai-training, allow-ai-search-only, block-all-ai, block-datasets, block-disputed, block-seo-tools, allow-all, maximum-ai-visibility) into a robots.txt you already have, without touching a single rule you wrote: a token you already name keeps your rules and the stance's version is reported instead of applied. Example: stance='block-ai-training', robots_txt='User-agent: *\nAllow: /\n'.

  robots.txtpolicygeneratormerge
• Diff two robots.txt by effect

  {"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"diff_robots_txt\",\"before\":\"User-agent: *\\nDisallow: /\",\"after\":\"User-agent: GPTBot\\nDisallow: /\\n\\nUser-agent: *\\nAllow: /\\n\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. Compare two versions of a robots.txt and report only the crawlers whose verdict actually changes at a given path — not the text difference. Answers 'did my edit do what I meant, and did it do anything else', including sitemap additions and whether the parse errors went up or down. Example: before='User-agent: *\nAllow: /\n', after=your edited file, path='/'.

  robots.txtdiffreviewchange
• Lint a robots.txt

  {"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"lint_robots_txt\",\"robots_txt\":\"User-agent: GPTBot\\nDisallow: /\\n\\nUser-agent: *\\nAllow: /\\n\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. Parse a robots.txt you paste and report every fault that makes it do something other than what it looks like: misspelled directives, a full UA string where a product token belongs, rules before any User-agent line, duplicate groups, noindex (unsupported since 2019), relative Sitemap URLs, BOM. Each finding carries the line number and the fix. Example: robots_txt='User-agent: GPTBot\nDisallow: /\n\nUser-agent: *\nAllow: /\n' — paste the whole file, it is never fetched for you. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/robots-lint?robots_txt=<urlencoded>&s=client-dossiers — or POST the file as the raw body to the same URL.

  robots.txtlintrfc9309validation
• Lint this host's own robots.txt and ai.txt (no arguments)

  {"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"no_arguments_lint_this_hosts_robots_txt\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. TAKES NO ARGUMENTS. The lint this server runs on a robots.txt you paste, run instead on the two policy files THIS host publishes: every finding with its severity, line number and fix, an RFC 9309 verdict for six crawler tokens against three real paths naming the rule and the line that decided each one, a check that robots.txt and ai.txt still agree verdict by verdict, and a rollup of which AI crawlers in this host's index the file actually stops. The input schema is literally empty — `arguments: {}` and no arguments key at all both work. No key, no account, no OAuth, read-only. Nothing is fetched to build it: no request leaves this edge and none is made to you. Example: arguments={} returns the findings, the verdict table and the agreement check.

  no argumentsrobots.txtrfc9309lintself
• Run this server's worked example (no arguments)

  {"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"example\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. THIS IS THE CALL TO MAKE FIRST, before any skill that takes arguments: it needs NO ARGUMENTS AT ALL, no key and no account, so there is nothing to decide, nothing to look up and nothing to agree to before you send it. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. Takes no arguments. Safe to call. Deterministic. Touches no third party. Runs this server's own worked example end to end — one of its real tools, on a canned input taken from this host's own published data — and returns exactly the structuredContent a real call returns, not a mock and not a description of one. Use it to see the shape of an answer before you decide what to send. No URL of yours is fetched and no third party is touched. Example: arguments={} runs it and returns the real answer.

  exampleno argumentsdemogetting started
• What does this robots.txt say to me? (no arguments)

  {"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"am_i_allowed\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. Takes no arguments. Safe to call. Deterministic. Touches no third party. THE CALL TO MAKE FIRST on this server. Reads this host's own robots.txt against YOUR user-agent through the same RFC 9309 evaluator as check_path_allowed: the group that matches you, the line that decided it, that group verbatim, and an allowed/disallowed verdict for three real paths. No key, no account, no OAuth, read-only, nothing to invent. Example: arguments={} returns your matched group, the line it was matched on, and the verdicts.

  no argumentsrobots.txtrfc9309identify
• Which AI crawlers does this file actually stop?

  {"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"audit_ai_access\",\"robots_txt\":\"User-agent: GPTBot\\nDisallow: /\\n\\nUser-agent: *\\nAllow: /\\n\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. Evaluate a pasted robots.txt against every AI crawler in this index and return the two lists that matter: blocked and allowed, per operator and category. Also names the tokens in your file that match no known crawler (a typo blocks nothing) and separates the crawlers that document obedience from the ones observed ignoring robots.txt, which need an IP or WAF rule instead. Example: path='/' with your robots_txt pasted in — the verdict is per crawler, at that path. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/ai-access?robots_txt=<urlencoded>&s=client-dossiers

  robots.txtai crawlersauditpolicy
• Who is calling? (no arguments)

  {"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"whoami\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. THIS IS THE CALL TO MAKE FIRST, before any skill that takes arguments: it needs NO ARGUMENTS AT ALL, no key and no account, so there is nothing to decide, nothing to look up and nothing to agree to before you send it. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. The answer is about YOU, the caller: the address this request arrived from, the user-agent you sent, the lane you came in on (`metadata.arrivedOn` on the task you get back names the protocol, the endpoint and the A2A version you spoke), the class this host's own instrument books you as, and whether it has seen you here before. Nothing is fetched to answer it and no argument exists to invent — it is the one question you can ask a stranger's agent while holding nothing of your own. Takes no arguments. Safe to call. Deterministic. Touches no third party. Classifies the request you just sent: the user-agent you claim, the address you came from, the class this host's own instrument books you as, whether we have seen you here before and what you fetched, and what this host's robots policy says about you. Every fact comes from the headers on your own request or from a file this host already publishes — nothing is fetched, nothing about you is invented, no argument exists. Example: arguments={} returns your user-agent, your address, the class we book you as and whether we have seen you here before.

  identifyno argumentsdiagnosticsuser-agent
• Show 1 remaining skill

  • Would this crawler fetch this path?

    {"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"check_path_allowed\",\"robots_txt\":\"User-agent: GPTBot\\nDisallow: /\\n\\nUser-agent: *\\nAllow: /\\n\",\"user_agent\":\"GPTBot\",\"paths\":[\"/docs\",\"/\"]}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. Evaluate a pasted robots.txt for one crawler and one or more paths under RFC 9309: longest token match for the group, longest path pattern for the rule, Allow breaking a tie, * and $ supported. Returns allowed/disallowed per path with the exact line that decided it, and flags the cases where a merge-groups parser and a first-group-wins parser would disagree. Example: user_agent='GPTBot', paths=['/', '/blog'], with your robots_txt pasted in. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/robots-allowed?robots_txt=<urlencoded>&ua=GPTBot&path=/blog&s=client-dossiers

    robots.txtcrawlerspathrules