Web Map

Map all URLs within a website and return them as a list.

Maps all URLs within a website and returns them as a list. Useful for discovering all pages on a site before scraping.

POST /v1/tools/web_map

Authentication

AuthorizationBearerrequired

API key as bearer token in the Authorization header. Create keys at Manage API Keys.

Headers

Content-Typestringrequired

Must be application/json.

Body

urlstringrequired

The URL to map. Will discover all links within this domain.

searchstringoptional

Search query to filter URLs. Only returns URLs matching this pattern.

sitemapstringoptional

Sitemap handling mode:

  • include: Use sitemap in addition to crawling (default)
  • exclude: Ignore sitemap, only crawl
  • only: Only use sitemap, skip crawling
Allowed values:includeexcludeonly
include_subdomainsbooleanoptionalDefault false

Include subdomains in mapping.

ignore_query_parametersbooleanoptionalDefault false

Ignore query parameters when deduplicating URLs.

ignore_cachebooleanoptionalDefault false

Ignore cached results and fetch fresh data.

limitintegeroptionalDefault 100

Maximum number of URLs to return. Range: 1 to 5000.

locationobjectoptional

Geographic location settings with country (string) and languages (string[]).

timeoutintegeroptional

Request timeout in milliseconds. Minimum: 1.

Response

idstringoptional

Unique request ID for this call.

linksstring[]optional

Array of discovered URLs.

Use Cases

  • Site auditing: Discover all pages for SEO analysis
  • Content inventory: List all URLs before bulk scraping
  • Link checking: Find all pages to validate links
  • Sitemap generation: Build sitemaps programmatically

Combine with Web Scrape

Use web_map to discover URLs, then web_scrape to fetch content from each. This is more efficient than blind crawling.