Crawl and map sites

Use POST /map for URL discovery and POST /crawl for page content.

Map first

Mapping is the efficient choice when an agent needs to understand a site's shape before deciding what to read.

{
  "url": "https://example.com",
  "search": "documentation",
  "limit": 100
}

The result is a list of discovered URLs. Filter or rank that list in your application, then scrape only the relevant pages.

Start a crawl

{
  "url": "https://example.com/docs",
  "limit": 50,
  "maxDepth": 3,
  "scrapeOptions": {
    "formats": ["markdown"],
    "onlyMainContent": true
  }
}

POST /crawl returns a job identifier. Poll GET /crawl/{id} until the status is terminal. Use GET /crawl/{id}/errors to inspect page-level failures and DELETE /crawl/{id} to cancel work that is no longer needed.

Control scope

Set explicit page and depth limits. Start from the narrowest useful path, exclude account, cart, search, calendar, and faceted-navigation patterns, and avoid crawling the same normalized URL through multiple query strings.

Credits are charged for completed page work rather than the act of creating the job. A blocked or invalid page is recorded as an error without inventing content.