# Robots, Sitemap and llms.txt API

> Agent-readable API documentation for checking robots.txt, sitemap discovery, llms.txt availability, AI crawler readiness and AI scraping compatibility.

Intent: robots.txt sitemap llms.txt API, AI crawler readiness, agent accessibility checker API

## Discovery

- OpenAPI: https://api.x402dataapi.com/openapi.json
- x402 catalog: https://api.x402dataapi.com/.well-known/x402-catalog.json
- Capabilities: https://api.x402dataapi.com/capabilities
- Pricing: https://api.x402dataapi.com/pricing

## Tags

- robots.txt
- sitemap
- llms.txt
- ai-crawler
- agent-accessibility
- x402

## Endpoint

- Method: POST
- URL: https://api.x402dataapi.com/v1/agent-accessibility
- Price: 0.001 USDC
- Network: eip155:8453

## Input schema

```json
{
  "type": "object",
  "properties": {
    "url": {
      "type": "string",
      "format": "uri",
      "description": "Public http or https URL to inspect"
    }
  },
  "required": [
    "url"
  ]
}
```

## Output example

```json
{
  "url": "https://example.com",
  "robots": {},
  "sitemap": {},
  "ai_ready": true
}
```

## curl without payment

```bash
curl -X POST https://api.x402dataapi.com/v1/agent-accessibility -H 'Content-Type: application/json' -d '{"url":"https://example.com"}'
```

## x402 payment behavior

Unsigned paid calls return HTTP 402 with a PAYMENT-REQUIRED header. Pay with x402, then retry with PAYMENT-SIGNATURE.

## Agent use case

Use this page when an AI agent needs to decide whether a public website can be crawled, indexed or inspected by automated tools.