Async jobs
Start crawls and extractions, follow their status, and read stored output safely.
POST /crawl and POST /extract are asynchronous. The create request returns before page work finishes, so save the returned id and use it for all later reads.
Create a job
curl -X POST https://api.berrycrawl.com/api/v1/crawl \
-H "Authorization: Bearer $BERRYCRAWL_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/docs",
"limit": 25,
"maxDiscoveryDepth": 2,
"allowSubdomains": false,
"scrapeOptions": {"formats":["markdown"],"onlyMainContent":true}
}'The response contains id and a relative url:
{"success":true,"id":"crawl_...","url":"/jobs/crawl_..."}Poll status and output
curl https://api.berrycrawl.com/api/v1/jobs/crawl_... \
-H "Authorization: Bearer $BERRYCRAWL_API_KEY"The job response includes status, progress counters, creditsUsed, expiration information, and the results available so far. A terminal job has one of these statuses:
| Status | Meaning |
|---|---|
PENDING | Accepted and waiting for a worker |
RUNNING | Page work is in progress |
COMPLETED | All admitted work finished |
FAILED | The job stopped with an error |
CANCELLED | The caller stopped the job |
Poll every few seconds with capped backoff. A polling failure for a known job is safer to retry than creating the job again. For long-running production workflows, configure a webhook and use the job read as the source of truth.
Cancel a job
curl -X DELETE https://api.berrycrawl.com/api/v1/jobs/crawl_... \
-H "Authorization: Bearer $BERRYCRAWL_API_KEY"Cancellation is best effort for work already being processed. Stop polling after the API reports CANCELLED.
Dashboard view
The dashboard's Jobs screen lists crawl and extraction jobs, refreshes active jobs, and shows progress, failed pages, credits, input, final output, stored page results, and webhook deliveries. Playground responses link directly to this screen in a new tab whenever they contain a crawl or extraction job id.