The Map API for
Every URL on a Site
Give DataBlue a domain, and it returns every URL it finds, combining sitemap.xml, robots.txt, and link crawling into one call. Most requests come back instantly. A site nobody has mapped recently kicks off a short background job you poll once. Firecrawl-compatible fields mean an existing map integration usually just needs a new base URL. Sign up and get 1,000 one-time credits to map your first site.
A Site's Structure, Not Its Content.
A map APItakes a domain and returns the URLs that live on it, without scraping what's on any of those pages. That distinction matters, since discovery is cheap and scraping isn't. Most teams don't need every page's content right away. They need to know what pages exist first, then decide what's actually worth fetching.
DataBlue combines several discovery methods to build that list, covered in detail in the section below. Each URL comes back with a title and a source field naming which method found it. Popular or previously mapped sites return complete results immediately. A site nobody has mapped recently kicks off a short background job instead, tracked the same way a crawl job is.
Because the core request and response fields mirror Firecrawl's map endpoint, teams already using Firecrawl for site discovery usually just point an existing integration at DataBlue without touching how they parse the result.
From Domain to Every URL.
Most map requests return complete results in the same response. A rare cache miss returns a job_id instead, ready a few seconds later at GET /map/{job_id}.
# REST curl -X POST "https://api.datablue.dev/v1/map" \ -H "Authorization: Bearer wh_..." \ -H "Content-Type: application/json" \ -d '{ "url": "https://docs.python.org", "limit": 200, "include_subdomains": false }'
{
"success": true,
"total": 187,
"links": [
{ "url": "https://docs.python.org/3/",
"title": "Python 3 Documentation",
"source": "sitemap" }
]
}Three Sources,
One Combined List.
No single source of URLs is exhaustive on its own. DataBlue runs all three together and tags each URL with where it came from.
Sitemap parsing
Reads sitemap.xml directly, the fastest and most reliable source. Matching URLs carry source: "sitemap" in the response.
Robots.txt discovery
Follows the sitemap reference inside a site's robots.txt file, catching sitemaps that aren't in the obvious place. Marked source: "robots".
Link crawling
Follows links across the site to catch pages missing from both files. Marked source: "crawl", the fallback that fills in gaps.
Search filter
Pass search and only URLs matching that string come back, handy for pulling one section of a large site.
Subdomain control
include_subdomains is on by default. Turn it off to keep a map scoped to a single subdomain.
Robots control
respect_robots_txt is on by default. Turn it off only when you are authorized to map paths excluded from browser discovery.
Result limit
limit caps how many URLs return, set to 100 by default and adjustable up for a full site inventory.
Built for Production, Priced for Scale.
Most discovery tools charge the same way they charge for scraping, by the page. DataBlue doesn't. A map job costs one flat credit no matter how many URLs it returns, so mapping a ten thousand page site costs the same as mapping a ten page one. Monthly plan credits reset each billing period, and top-up credits never expire.
Requests that hit the cache come back without delay, since DataBlue remembers recently mapped sites across every user, not just you. A cache miss still resolves fast, running as a short background job instead of leaving you waiting on a slow synchronous call. The three discovery methods covered above all run automatically, so a single-source map tool's blind spots don't become yours.
Output stays close to what Firecrawl users already expect, since the core fields carry over directly. Point Map at your own site before a big Crawl job, and use the URL list to decide which sections are worth the credits. Run it against a competitor's domain just as easily, since discovery never touches page content.
What Teams Build With It.
A fast, cheap URL list turns out to be useful well beyond scoping a crawl.
Crawl scoping
Map a site first to see its real structure, then aim Crawl's include_paths at the sections actually worth scraping.
Site audits & migrations
Pull a complete URL inventory before a redesign or domain move, so nothing gets left behind.
Sitemap validation
Compare what a sitemap claims against what link crawling actually finds, and catch pages that quietly fell out of it.
Competitive structure research
See how a competitor organizes a site, sections and depth, without scraping a single page of their content.
AI agent site orientation
Give an agent a fast way to see what exists on a domain before deciding what's worth reading closely.
Firecrawl migration
Point an existing map integration at DataBlue. Get the same core fields back without rewriting a parser.
Your First Map in Seconds.
No sales call and no contract to sign. Sign up, grab a key, and your first URL list is one request away.
Create a Free Account
Sign up and grab an API key right off the bat. You get 1,000 one-time signup credits to test a real domain, no credit card required.
Map a Site
Call /v1/map with a Bearer token and a url. Set limit or include_subdomains to shape how wide the discovery goes.
Read the URLs
Most responses land with links already populated. Poll GET /map/{job_id} on the rare cache miss. Ship it.
One job, one credit, any size site.
Start free with 1,000 one-time signup credits. A map job costs one flat credit regardless of how many URLs it returns, so a small blog and a massive catalog site cost the same to map. There's no per-URL charge and no premium for large sites. Monthly plans reset on schedule, and top-up credits stay available until you spend them.
Map API Questions.
What is a Map API?
A map API takes a website's root URL and returns every URL it finds on that site, without scraping the content behind each one. DataBlue's version combines several discovery methods into one call, covered in the section below, so you get a single list of URLs with titles and descriptions attached. Point it at a domain and get the site's structure back in seconds. Use that list to scope a crawl before spending credits on the pages you don't actually need.
How does DataBlue discover URLs on a site?
Three methods run together on every request, and the entire breakdown lives in the section below. sitemap.xml parsing does most of the work, since that's what most sites publish for crawlers. Link crawling catches pages a sitemap misses, the ones that only exist because something else links to them. Each URL in the response carries a source field showing which method actually found it.
Is the Map API synchronous, or does it return a job?
Both, depending on the request. If DataBlue has mapped that exact URL recently, the response returns fully populated right away, links and all. If not, DataBlue starts a short background job and returns a job_id instead, with an empty links array until it finishes. Poll GET /map/{job_id} once, and the list is usually ready within seconds. Most repeat requests hit the fast path, since popular sites get mapped often.
How much does the Map API cost?
One flat credit per job, no matter how many URLs come back. Mapping a ten page site and a ten thousand page site cost the same. That's different from Scrape or Crawl, which charge per page instead. Run a map before a big crawl, and the reconnaissance itself barely registers against your credit balance.
Can I filter which URLs come back?
Yes, with the search parameter. Pass a string, and DataBlue only returns URLs that match it, useful for pulling just the /blog/ section or every page mentioning a product line. limit caps the total number of URLs returned, set to 100 by default. Combine both, and a map of a huge site narrows down to the slice you actually need.
Does it include subdomains?
Yes, by default. include_subdomains is true out of the box, so a map of example.com also picks up blog.example.com or docs.example.com if they exist. Set include_subdomains to false when a subdomain runs an unrelated product and you only want the main site covered. Turn it off before mapping a domain with a large subdomain you'd rather skip entirely.
Is this compatible with Firecrawl's map endpoint?
Yes, in the ways that matter for migration. The core request fields follow the same shape Firecrawl uses, right down to the url and limit parameters. The response returns the same core link fields too, so code built against Firecrawl's map endpoint keeps working. DataBlue adds a few extra fields on top, like a source tag per URL, without breaking what's already there. Point an existing map integration at DataBlue and watch the response land in a shape your parser already understands.
How is Map different from Crawl?
Map only discovers URLs. Crawl discovers URLs and scrapes every page behind them. Reach for Map when you need a site's structure fast and cheap, then decide which paths are worth a proper Crawl job afterward. Reach for Crawl directly when you already know you want the content itself, not just a list of where it lives. Run Map first on an unfamiliar site, and let the URL list guide how you scope the Crawl that follows.
What does the source field mean on each URL?
It tells you which discovery method found that particular link, whether that's a published sitemap or something turned up through link crawling. Each method is broken down in the section above, along with the source value it maps to. Sitemap entries are usually the most current, since they come straight from what the site published. Check the source field when you need to weigh how much to trust a given URL.
How many URLs can I get back in one call?
Up to whatever you set with limit, which defaults to 100 if you leave it out. Raise it for a whole-site inventory, or lower it when you only need a quick sample. There's no separate page count ceiling beyond what limit controls, unlike Crawl's max_pages and max_depth working together. Set limit high on the first call to a new domain, and the site's real scale shows up immediately.
Map a Site in One Call.
Grab a free key and map your first site. See every URL DataBlue finds in one call, no card required, with 1,000 one-time signup credits to get you started.