Web Scrape

Scrape content from a URL and return it in various formats.

Scrapes content from a URL and returns it in various formats including markdown, HTML, or screenshots.

POST /v1/tools/web_scrape

Authentication

AuthorizationBearerrequired

API key as bearer token in the Authorization header. Create keys at Manage API Keys.

Headers

Content-Typestringrequired

Must be application/json.

Body

urlstringrequired

The URL to scrape.

formats(string | object)[]optional

Output formats to return. Default returns markdown. Each entry is either a format name string (e.g. "markdown") or an object { type: <name>, ...options } for formats that take options. See Format options below.

Allowed values:markdownsummaryhtmlrawHtmllinksimagesscreenshotjsonchangeTrackingbrandingaudiovideoquestionhighlights
only_main_contentbooleanoptionalDefault false

Extract only the main content, excluding headers, footers, etc.

only_clean_contentbooleanoptionalDefault false

Return cleaned content without ads and other noise.

include_tagsstring[]optional

HTML tags to include in extraction.

exclude_tagsstring[]optional

HTML tags to exclude from extraction.

max_ageintegeroptional

Maximum age of cached content in seconds. Minimum: 1.

min_ageintegeroptional

Minimum age of cached content in seconds. Minimum: 1.

headersobjectoptional

Custom headers to send with the request. Key-value pairs.

wait_forintegeroptional

Time to wait for page to load in milliseconds. Minimum: 1.

mobilebooleanoptionalDefault false

Emulate mobile device.

skip_tls_verificationbooleanoptionalDefault false

Skip TLS certificate verification.

timeoutintegeroptional

Request timeout in milliseconds. Minimum: 1.

parsersstring[]optional

Parsers to use for content extraction.

actionsobject[]optional

Browser actions to perform before scraping (click, scroll, etc.).

locationobjectoptional

Geographic location settings with country (string) and languages (string[]).

remove_base64_imagesbooleanoptionalDefault false

Remove base64 encoded images from output.

block_adsbooleanoptionalDefault false

Block ads during page load.

proxystringoptional

Proxy mode to use.

Allowed values:basicstealth
store_in_cachebooleanoptionalDefault true

Store result in cache.

lockdownbooleanoptionalDefault false

Enable lockdown mode.

zero_data_retentionbooleanoptionalDefault false

Do not retain any data after request.

Response

idstringoptional

Unique request ID for this call.

dataobjectoptional

Scraped content. Contains fields based on requested formats: markdown, summary, html, raw_html, links, images, screenshot, json, change_tracking, branding, audio, video, question, highlights. Every response key is snake_case (the rawHtml format returns raw_html, changeTracking returns change_tracking); the keys inside json are returned exactly as your extraction schema declares them.

data.metadataobjectoptional

Page metadata including title, description, language, and source_url.

Format options

Formats that accept options must be passed as objects. The type field selects the format; other fields configure it.

json — structured extraction

{
  "type": "json",
  "schema": {
    "type": "object",
    "properties": {
      "title": { "type": "string" },
      "points": { "type": "number" }
    }
  }
}

Result appears at data.json.

question — ask a question about the page

{
  "type": "question",
  "question": "When was this article published and by whom?"
}

Result appears at data.question.

changeTracking — diff against previous scrape

{
  "type": "changeTracking",
  "modes": ["git-diff"]
}

Result appears at data.change_tracking.

Use Cases

  • Content extraction: Pull article text from news sites
  • SEO analysis: Extract metadata and link structure
  • Screenshots: Capture visual snapshots of pages
  • Data mining: Scrape structured data from web pages