Production operations
Errors and retries
Retry transient failures without duplicating work or hiding bad requests.
Berrycrawl uses standard HTTP status families and includes a concrete failure message in error responses.
| Status | Meaning | Action |
|---|---|---|
400 | Invalid URL, option, schema, or request body | Fix the request; do not retry unchanged |
401 | Missing, invalid, or revoked API key | Replace or rotate the key |
402 | Insufficient credits | Top up the workspace |
404 | Unknown job or route | Check the identifier and environment |
409 | Conflicting state | Read current state before another write |
429 | Rate or concurrency limit | Back off with jitter and reduce concurrency |
502 | Upstream site or provider failure | Retry a small number of times |
503 | Required browser, proxy, AI, or storage dependency unavailable | Retry later or use a cheaper compatible mode |
504 | Operation timed out | Retry, narrow the work, or use an async endpoint |
Retry policy
Use capped exponential backoff with jitter for 429, 502, 503, and 504. A practical sequence is 1, 2, 4, 8, then 16 seconds with random variation. Cap the total attempts and surface the final Berrycrawl reason to the caller.
Do not automatically resubmit a crawl or extract job if its creation response may have been delivered. First check your stored job identifier or idempotency boundary. Polling a known job is safer than creating a duplicate.