Start a new crawl job

POST/crawl

Authorization

api-key

AuthorizationBearer <token>

Enter your API key with the bc_ prefix

In: header

Request Body

application/json

TypeScript Definitions

Use the request body type in TypeScript.

url*string

Starting URL to crawl

limit?number

Maximum number of pages to crawl

Default10000

maxDiscoveryDepth?number

Maximum depth for URL discovery

Default10

includePaths?array<string>

Regex patterns to include paths

excludePaths?array<string>

Regex patterns to exclude paths

crawlEntireDomain?boolean

Whether to crawl entire domain or just subtree

Defaultfalse

allowExternalLinks?boolean

Whether to allow crawling external domains

Defaultfalse

allowSubdomains?boolean

Whether to allow crawling subdomains

Defaultfalse

delay?number

Delay between page scrapes in milliseconds

Default0

sitemap?string

Sitemap handling strategy

Default"include"

Value in

"include"
"skip"
"only"

prompt?string

Natural language instructions for crawl configuration

scrapeOptions?

Scrape options to apply to each page

webhook?

Webhook configuration

ignoreQueryParameters?boolean

Ignore query parameters when deduplicating URLs

Defaultfalse

deduplicateSimilarURLs?boolean

Deduplicate similar URLs using intelligent matching

Defaultfalse

maxConcurrency?number

Maximum number of concurrent scrapes for this crawl

zeroDataRetention?boolean

Enable zero data retention mode (requires activation)

Defaultfalse

Response Body

`application/json`

curl -X POST "https://example.com/crawl" \  -H "Content-Type: application/json" \  -d '{    "url": "https://example.com"  }'

{  "success": true,  "data": {    "id": "crawl_uuid",    "status": "PENDING",    "url": "/crawl/crawl_uuid"  }}

Empty

Authorization

Request Body

Response Body

201application/json

401

`application/json`