Agents and SDKs
Webhooks
Receive asynchronous job completion events safely and idempotently.
Crawl and extraction requests can notify your application when asynchronous work changes state. Configure a public HTTPS endpoint and subscribe only to events your worker handles.
Handler rules
- Read and verify the delivery before parsing trusted fields.
- Return a success response quickly; move expensive work to your own queue.
- Store the delivery or job identifier and ignore duplicates.
- Fetch the job from Berrycrawl when the event is only a notification.
- Keep the endpoint available during deploys and retries.
Deliveries can be retried after timeouts and transient failures. Event order is not guaranteed across independent jobs, so use each job's current API state as the source of truth.
Never put your Berrycrawl API key in a webhook URL. Use the webhook signing configuration for authenticity and keep any endpoint-specific secret in server-side configuration.