Scrape a page

POST /scrape is the general-purpose endpoint for one URL. It renders the page, applies optional actions and filters, and returns the formats requested in the body.

{
  "url": "https://example.com/docs",
  "formats": ["markdown", "links"],
  "onlyMainContent": true,
  "proxy": "auto",
  "maxAge": 3600000
}

Formats

Use plain string formats for standard outputs. Some formats accept an object when they need options.

The base operation is 1 credit for a successful uncached scrape. Proxy escalation is included and never changes the customer price. JSON extraction adds 4 credits; summaries add 2. Failed, blocked, and cached ordinary page responses are free. YouTube transcripts use the separate flat price below.

YouTube transcripts

Pass a YouTube video URL to this endpoint to receive its timestamped transcript. Berrycrawl also provides the focused POST /youtube/transcript endpoint for the same pipeline. It recognizes standard watch links, youtu.be links, Shorts, live-video links, and embed URLs. The dedicated YouTube transcript guide covers both request paths, response metadata, pricing, and errors.

{
  "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
  "formats": ["markdown"]
}

The response uses the normal scrape shape. data.markdown contains the title, creator, and timestamped transcript. Metadata includes the resolved caption language, video ID, duration, creator, and thumbnail when available.

Every successful YouTube transcript scrape costs a flat 5 credits. Missing captions and failed requests are not charged. Browser actions and the dedicated screenshot endpoint load the YouTube page instead of using transcript mode.

Main-content filtering

Set onlyMainContent to remove navigation, cookie banners, footers, and repeated chrome. Use includeTags and excludeTags when a page needs more precise selection. If exact DOM fidelity matters, request rawHtml and leave main-content filtering off.

Cache control

maxAge is the maximum acceptable cache age in milliseconds. Set it to 0 to force a refresh. A larger value favors speed and lower infrastructure cost; a smaller value favors freshness.

Proxy policy

• auto: start direct and escalate only when the response is blocked or challenged.
• none or basic: do not use a proxy; return a clean failure if direct access cannot succeed.
• residential or stealth: start at that explicit paid tier.

The response metadata records the fetch and proxy tier used, which makes cost and reliability audits possible without exposing proxy credentials.

When enabled for a controlled production cohort, Berrycrawl can start one delayed secondary IPRoyal route on the residential rung and accept the first valid response. Browser actions, screenshots, PDFs, and Bright Data never use this hedge.

Browser actions

Actions let a scrape wait, click, type, press keys, scroll, or run supported page interactions before extraction. Keep action sequences short and deterministic. Prefer selectors tied to stable attributes over presentation classes.

Concurrency and queueing

When all active scrape slots are occupied, Berrycrawl places the request in a bounded, distributed FIFO queue. It does not immediately return 429. Queue time counts against the request's timeout, and successful responses include queue admission metadata.

Use GET /account to read the workspace's active count, queued count, active limit, and queue limit. For many unrelated URLs, send bounded parallel scrape calls and persist each response in your application.