Docs

Robots, Sitemap and llms.txt API

Paid x402 API for AI agents that checks robots.txt, sitemap availability, llms.txt discovery, AI crawler readiness, agent accessibility and AI scraping compatibility for a public URL.

Request

Method
POST
URL
https://api.x402dataapi.com/v1/agent-accessibility
Price
0.001 USDC

Tags

x402ai-agentsagent-accessibilityrobots.txtsitemap

Input Schema

{
  "type": "object",
  "properties": {
    "url": {
      "type": "string",
      "format": "uri",
      "description": "Public http or https URL to inspect"
    }
  },
  "required": [
    "url"
  ]
}

Input Example

{
  "url": "https://example.com"
}

Response Example

{
  "url": "https://example.com",
  "robots": {},
  "sitemap": {},
  "ai_ready": true
}

Errors

PAYMENT_REQUIREDNo valid x402 payment signature was provided.
INVALID_PAYMENTThe signature was malformed, expired, replayed or does not match the endpoint price/resource.
VALIDATION_ERRORThe JSON input does not match the documented schema.
SSRF_PRIVATE_HOSTThe target resolves to a private or blocked address.

curl without payment

curl -X POST https://api.x402dataapi.com/v1/agent-accessibility \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}'

curl with PAYMENT-SIGNATURE

curl -X POST https://api.x402dataapi.com/v1/agent-accessibility \
  -H "Content-Type: application/json" \
  -H "PAYMENT-SIGNATURE: <x402-payment-payload>" \
  -d '{"url":"https://example.com"}'

Intent

robots.txt sitemap llms.txt API, AI crawler readiness, agent accessibility checker API

robots.txtsitemapllms.txtai-crawleragent-accessibilityx402

x402 agent flow

  1. Read /llms.txt.
  2. Fetch /openapi.json or /.well-known/x402-catalog.json.
  3. Inspect schema, price and network.
  4. Pay with x402 and retry with PAYMENT-SIGNATURE.

Agent use case

Use this page when an AI agent needs to decide whether a public website can be crawled, indexed or inspected by automated tools.