Agent quickstart

Read https://berrycrawl.com/llms.txt and https://docs.berrycrawl.com/docs/guides/agent-quickstart, then integrate Berrycrawl into this repository.

First inspect the repository's language, framework, server entry points, package manager, existing environment files, and test commands. Do not ask me to paste a secret into chat. Tell me to create a Berrycrawl API key in the dashboard and add this variable to the server-side environment file or secret store:

BERRYCRAWL_API_KEY=bc_...

Wait until that variable exists in the environment you will use. Then implement the smallest server-side integration for the feature I asked for. Use the endpoint and request shape from the Berrycrawl docs, keep the key out of browser bundles and source control, add a focused test or local verification path, and report the files changed and the command I can run to verify it.

If the feature starts a crawl or extraction job, persist the returned id and poll GET https://api.berrycrawl.com/api/v1/jobs/{id} until the status is COMPLETED, FAILED, or CANCELLED. Do not create a second job when polling a known id.

What the human needs to do

The human creates a key once in Dashboard → API keys, then adds it to the project's server environment:

BERRYCRAWL_API_KEY=bc_...

That is the complete agent setup. The key should live in the runtime that calls Berrycrawl: a server process, background worker, serverless function, or deployment secret store. It must not be a browser variable such as VITE_BERRYCRAWL_API_KEY or NEXT_PUBLIC_BERRYCRAWL_API_KEY.

API base and request format

The REST base is:

https://api.berrycrawl.com/api/v1

Every request uses the bearer header:

Authorization: Bearer $BERRYCRAWL_API_KEY
Content-Type: application/json

The smallest first test is a scrape:

curl -X POST https://api.berrycrawl.com/api/v1/scrape \
  -H "Authorization: Bearer $BERRYCRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://berrycrawl.com","formats":["markdown"]}'

Ordinary scrape requests for berrycrawl.com are free and cached, so an agent can verify wiring without spending workspace credits.

Choose the endpoint

Use the API reference for the full schema. Start with the smallest endpoint that answers the feature request.

Async jobs

POST /crawl and POST /extract return a job identifier immediately:

{
  "success": true,
  "id": "crawl_...",
  "url": "/jobs/crawl_..."
}

Poll the job with the same bearer key:

curl https://api.berrycrawl.com/api/v1/jobs/crawl_... \
  -H "Authorization: Bearer $BERRYCRAWL_API_KEY"

Persist the id before polling. Back off between requests, stop on COMPLETED, FAILED, or CANCELLED, and use DELETE /jobs/{id} when the caller explicitly cancels work. See Async jobs and Webhooks for delivery options.

Use a browser through MCP

Connect to https://api.berrycrawl.com/api/v1/mcp with the same bearer key. Open with berrycrawl_browser_open, choosing proxy: "none" for a Free account and setting max_credits and timeout_seconds. Drive it with fresh snapshot refs, then call berrycrawl_browser_close. Sessions retain state between tool calls; disconnecting MCP does not close them.

Use berrycrawl_browser_run_task for a hosted agent or berrycrawl_browser_run_recording for a saved path. Poll their job IDs with berrycrawl_get_job. Do not retry an uncertain browser action or task creation automatically. See the MCP browser example and browser guide.

Integration rules for agents

• Inspect and follow the repository's existing env naming and secret-loading conventions.
• Keep Berrycrawl calls on the server. Never put a bc_ key in frontend code, logs, screenshots, or test fixtures.
• Prefer the native HTTP client already used by the repository unless an installed Berrycrawl SDK is available.
• Pass explicit page, depth, URL, and schema limits to crawl and extraction jobs.
• Treat 400 as a request bug, 401 as a key problem, 402 as a credit problem, and 429, 502, 503, and 504 as bounded-retry candidates.
• In tests, mock the Berrycrawl call. Use the free berrycrawl.com endpoint only for an opt-in integration check.