The Map API for
Website URL Discovery
Give DataBlue a domain to discover accessible URLs using sitemap.xml, robots.txt, and link discovery. Responses can contain links or a job ID for background work. Set discovery limits, inspect the result, and use the URL list to scope a crawl. Check current plans and signup terms on the pricing page.
A Site's Structure, Not Its Content.
A map API takes a domain and returns the URLs that live on it, without scraping what's on any of those pages as full content output. Most teams don't need every page's content right away. They need to know what pages exist first, then decide what's actually worth fetching.
DataBlue combines several discovery methods to build that list, covered in detail below. Link metadata such as title, description, and source is optional. Discovery depends on available sources, access, and request limits; use your own site records to check coverage.
DataBlue uses familiar map fields such as url, limit, and links. For a Firecrawl migration, review the documented authentication, request options, and job lifecycle before adapting your client.
From Domain to Discovered URLs.
A response may include links or a job_id. Poll GET /v1/map/{job_id}while status is pending or running, then inspect the completed or failed result.
# REST curl -X POST "https://api.datablue.dev/v1/map" \ -H "Authorization: Bearer wh_..." \ -H "Content-Type: application/json" \ -d '{ "url": "https://docs.python.org", "limit": 200, "include_subdomains": false }'
{
"success": true,
"total": 1,
"links": [
{ "url": "https://docs.python.org/3/",
"title": "Python 3 Documentation",
"source": "sitemap" }
]
}Available Sources,
One Combined List.
Discovery can use existing domain inventory, sitemaps, HTML links, and rendered-page fallbacks. Sources depend on settings and availability.
Sitemap parsing
Reads available sitemaps when use_sitemap is enabled. Sitemap links can include optional published metadata.
Robots.txt discovery
Uses sitemap references in robots.txt to locate additional sitemap files.
Link discovery
Can collect HTML links and use rendered-page discovery as a fallback. Accessible sources and request limits bound the results.
Search ranking
Pass search to rank candidates using the URL and available title or description.
Subdomain control
include_subdomains is on by default. Turn it off to keep a map scoped to a single subdomain.
Robots control
respect_robots_txt is on by default. Turn it off only when you are authorized to map paths excluded from browser discovery.
Result limit
limit defaults to 100 and accepts 1 through 5,000. A response may contain fewer discovered links.
Built for Production, Priced for Scale.
Map uses URL-based block pricing from the published pricing catalog. Review the current rate and billing policies before running a request; site size alone does not determine the number of returned URLs or the charge.
Cached discovery may be available for matching request settings. Background jobs have a status endpoint, so your client can track work and handle failures. Availability and response time depend on the target site and runtime conditions.
Inspect discovered links and optional metadata before a big Crawl job, and use the URL list to decide which sections are worth the credits. Link discovery may fetch pages to find URLs; Map returns links rather than full page-content output.
What Teams Build With It.
A discovered URL list can be useful beyond scoping a crawl.
Crawl scoping
Map a site to discover candidate paths, then aim Crawl's include_paths at the sections you need.
Site audits & migrations
Build a candidate URL inventory before a redesign or domain move, then validate coverage against your own records.
Sitemap validation
Compare what a sitemap claims against what link crawling actually finds, and catch pages that quietly fell out of it.
Competitive structure research
Inspect a site's discovered URL paths and sections without requesting full page-content output.
AI agent site orientation
Give an agent discovered URL candidates before deciding which pages to retrieve.
Firecrawl migration
Adapt an existing map integration using familiar URL and link fields, checking authentication and polling differences.
Start With a Domain.
No sales call and no contract to sign. Sign up, grab a key, and your first URL list is one request away.
Create a Free Account
Create an account and obtain an API key. Check the pricing page for current signup allowances and plan terms.
Map a Site
Call /v1/map with a Bearer token and a url. Set limit or include_subdomains to shape how wide the discovery goes.
Read the URLs
Inspect returned links, or poll GET /v1/map/{job_id} until the job reaches a terminal status. Handle failed and empty results explicitly.
URL discovery, catalog-based pricing.
Map is billed in URL blocks under the published pricing catalog. Review current rates, result policies, plans, and signup allowances before choosing your request limits.
Map API Questions.
What is a Map API?
A map API discovers website URLs to help you scope a crawl. DataBlue combines available discovery sources and returns links with optional titles, descriptions, and source metadata. Results depend on the site's accessibility, discovery sources, and request limits; they are not a guarantee of a complete site inventory.
How does DataBlue discover URLs on a site?
Discovery can use an existing domain inventory, published sitemaps, links in HTML, and rendered-page discovery when needed. Robots.txt can identify sitemap locations. Sources and fallbacks depend on request settings and what the site makes accessible; not every method runs on every request.
Is the Map API synchronous, or does it return a job?
The POST /v1/map response can contain discovered links or a job_id for background work. When a job_id is returned, poll GET /v1/map/{job_id} while status is pending or running, then inspect the completed or failed result. Cache availability does not guarantee a particular response time or completion in a single poll.
How much does the Map API cost?
Map uses URL-based block pricing from the published pricing catalog, rather than a fixed job price. Check the current Map rate and billing policies on the pricing page before running a request. Request limits bound discovery; the returned result and applicable pricing rule determine the charge.
Can I filter which URLs come back?
Use search to rank matching candidates by their URL and available title or description. Set limit to cap the returned links: it defaults to 100 and accepts values from 1 to 5,000. Search is not an exact path filter or a completeness guarantee.
Does it include subdomains?
include_subdomains defaults to true, allowing discovered links on subdomains within scope. Set it to false to restrict results to the requested host. Enabling it does not guarantee discovery of every subdomain or page.
Is this compatible with Firecrawl's map endpoint?
DataBlue supports familiar fields such as url, limit, and a links array. Review the request options, authentication, job polling, and response fields in the documentation before migrating; changing only the base URL is not a guaranteed drop-in migration.
How is Map different from Crawl?
Map returns discovered URLs and available link metadata. Crawl retrieves page content within its configured scope and limits. Use Map to identify candidate paths, then select the pages or sections needed for a crawl.
What does the source field mean on each URL?
The optional source field records discovery provenance when available, such as sitemap or HTML discovery. It does not verify that a URL is current, accessible, or complete. Validate the pages you plan to consume.
How many URLs can I get back in one call?
limit defaults to 100 and accepts 1 through 5,000. The response can contain fewer links when fewer candidates are discovered or match the requested scope. Increasing the limit does not ensure that all website URLs will be found.
Explore a Site's URL Structure.
Create an account, obtain a key, and discover candidate URLs for your next crawl. Review the documentation and current pricing before running a request.