Start a website crawl
/crawlAuthorization
api-key Enter your API key with the bc_ prefix
In: header
Request Body
application/json
TypeScript Definitions
Use the request body type in TypeScript.
Whether to allow crawling external domains
falseWhether to allow crawling subdomains
falseWhether to crawl entire domain or just subtree
falseDeduplicate similar URLs using intelligent matching
falseDelay between page scrapes in milliseconds
0Regex patterns to exclude paths
Ignore query parameters when deduplicating URLs
falseRegex patterns to include paths
Maximum number of pages to crawl
10000Maximum number of concurrent scrapes for this crawl
Maximum depth for URL discovery
10Natural language instructions for crawl configuration
Scrape options to apply to each page. actions accepts at most 25 bounded browser actions. zeroDataRetention: true is currently rejected with 400 ZERO_DATA_RETENTION_NOT_SUPPORTED.
Sitemap handling strategy
"include"Value in
- "include"
- "skip"
- "only"
Starting URL to crawl
Webhook configuration
Reserved for a future zero-data-retention mode. true is currently rejected with 400 ZERO_DATA_RETENTION_NOT_SUPPORTED; omit or use false.
falseResponse Body
application/json
curl -X POST "https://example.com/crawl" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com" }'{ "id": "string", "invalidURLs": [ "string" ], "success": true, "url": "string"}