Web Map

Map all URLs within a website and return them as a list.

Maps all URLs within a website and returns them as a list. Useful for discovering all pages on a site before scraping.

POST /v1/tools/web_map

Authentication

AuthorizationBearerrequired

API key as bearer token in the Authorization header. Create keys at Manage API Keys.

Headers

Content-Typestringrequired

Must be application/json.

Body

urlstringrequired

The URL to map. Will discover all links within this domain.

searchstringoptional

Search query to filter URLs. Only returns URLs matching this pattern.

sitemapstringoptional

Sitemap handling mode:

  • include: Use sitemap in addition to crawling (default)
  • exclude: Ignore sitemap, only crawl
  • only: Only use sitemap, skip crawling
Allowed values:includeexcludeonly
includeSubdomainsbooleanoptionalDefault false

Include subdomains in mapping.

ignoreQueryParametersbooleanoptionalDefault false

Ignore query parameters when deduplicating URLs.

ignoreCachebooleanoptionalDefault false

Ignore cached results and fetch fresh data.

limitintegeroptionalDefault 100

Maximum number of URLs to return. Range: 1 to 5000.

locationobjectoptional

Geographic location settings with country (string) and languages (string[]).

timeoutintegeroptional

Request timeout in milliseconds. Minimum: 1.

Response

idstringoptional

Unique request ID for this call.

linksstring[]optional

Array of discovered URLs.

Use Cases

  • Site auditing: Discover all pages for SEO analysis
  • Content inventory: List all URLs before bulk scraping
  • Link checking: Find all pages to validate links
  • Sitemap generation: Build sitemaps programmatically

Combine with Web Scrape

Use web_map to discover URLs, then web_scrape to fetch content from each. This is more efficient than blind crawling.