Scrape a page

POST /scrape is the general-purpose endpoint for one URL. It renders the page, applies optional actions and filters, and returns the formats requested in the body.

{
  "url": "https://example.com/docs",
  "formats": ["markdown", "links"],
  "onlyMainContent": true,
  "proxy": "auto",
  "maxAge": 3600000
}

Formats

Use plain string formats for standard outputs. Some formats accept an object when they need options.

The base operation is 1 credit for a successful uncached scrape. Proxy escalation is included and never changes the customer price. Screenshots, PDFs, and JSON extraction add 4 credits each; summaries add 2. Failed, blocked, and cached responses are free.

Main-content filtering

Set onlyMainContent to remove navigation, cookie banners, footers, and repeated chrome. Use includeTags and excludeTags when a page needs more precise selection. If exact DOM fidelity matters, request rawHtml and leave main-content filtering off.

Cache control

maxAge is the maximum acceptable cache age in milliseconds. Set it to 0 to force a refresh. A larger value favors speed and lower infrastructure cost; a smaller value favors freshness.

Proxy policy

• auto: start direct and escalate only when the response is blocked or challenged.
• none or basic: do not use a proxy; return a clean failure if direct access cannot succeed.
• residential or stealth: start at that explicit paid tier.

The response metadata records the fetch and proxy tier used, which makes cost and reliability audits possible without exposing proxy credentials.

When enabled for a controlled production cohort, Berrycrawl can start one delayed secondary IPRoyal route on the residential rung and accept the first valid response. Browser actions, screenshots, PDFs, and Bright Data never use this hedge.

Browser actions

Actions let a scrape wait, click, type, press keys, scroll, or run supported page interactions before extraction. Keep action sequences short and deterministic. Prefer selectors tied to stable attributes over presentation classes.