The Crawl API for
Entire Domains, One Job
Give DataBlue a starting URL, and it follows links outward, scraping every page it finds along the way. Set the boundaries, like depth or page count, and get markdown or schema-validated JSON back for each one. The Firecrawl-compatible surface means an existing crawl integration usually needs nothing more than a new base URL. Sign up and get 1,000 one-time credits to run your first crawl.
One URL In, a Whole Site Out.
A crawl API takes one URL and turns it into an entire site's worth of structured pages. Instead of writing a scraper that follows links and tracks which pages it has already visited, you hand DataBlue the seed URL and the boundaries. A background job does the rest. That difference matters once a site spans hundreds of pages across nested categories or a documentation tree several levels deep.
DataBlue supports several traversal strategies, covered in the section below, so the order pages get visited in matches how a site is actually built. Each page comes back in the same shape as a single scrape, markdown or JSON, with depth and status attached. Faceted URLs get deduplicated automatically, so a product category with dozens of sort and filter combinations doesn't turn into dozens of near-identical results.
Because the request shape and job lifecycle mirror Firecrawl's crawl endpoint, teams evaluating alternatives usually just redirect an existing integration at DataBlue without rewriting how they poll for results.
One Seed URL, Every Page Back.
Starting a crawl returns a job ID right away. Poll GET /crawl/{job_id} or set a webhook, and pages land as the job works through the site.
# REST curl -X POST "https://api.datablue.dev/v1/crawl" \ -H "Authorization: Bearer wh_..." \ -H "Content-Type: application/json" \ -d '{ "url": "https://docs.python.org/3/", "max_pages": 50, "crawl_strategy": "bfs" }'
{
"success": true,
"job_id": "550e8400-e29b-...",
"status": "started",
"message": "Crawl job started"
}
# then: GET /crawl/{job_id}
"completed_pages": 37, "total_pages": 50Shape the Crawl,
Not Just the Start URL.
A seed URL alone rarely tells the whole story. These controls decide how wide, how deep, and how gently a crawl moves through a site.
Breadth-first (default)
Spreads across a site level by level. The steady choice for most content crawls and documentation sites.
Depth-first
Follows one branch all the way down before backing out. Suits deeply nested trees where breadth wastes requests.
Best-first
Ranks pages by relevance to your goal and visits the most promising ones first, instead of strict link order.
Path filters
include_paths and exclude_paths accept glob patterns, so a crawl stays inside one section or skips another entirely.
Depth & page caps
max_depth and max_pages bound a crawl from two directions, keeping wide and narrow sites equally predictable.
Webhook notifications
Set webhook_url and a signed request lands the moment a job completes, verified with webhook_secret.
Built for Production, Priced for Scale.
Plenty of crawlers follow links well enough. What's harder is doing it politely and predictably at scale, across thousands of pages without tripping a site's defenses or blowing the budget. DataBlue prices each crawled page like a single scrape, so a 500 page crawl costs what 500 scrapes would and nothing more. Monthly plan credits reset each billing period, and top-up credits never expire.
Every page in a crawl gets the same rendering treatment as a standalone scrape request, HTTP first and a full browser render as the fallback, so JavaScript-heavy pages come back complete instead of half-loaded. Concurrency is configurable per job, from one worker up to ten, so speed trades directly against how gently a job treats the target site. robots.txt is respected by default, and a crawl that goes off the rails stops the moment you cancel the job.
Because DataBlue's crawl output is Firecrawl-compatible, every page arrives LLM-ready by default. Markdown from an entire site drops straight into a RAG pipeline without a separate cleanup pass. Nest an extract schema in scrape_options and the whole crawl returns typed JSON instead. Point a crawl at your own docs and see the whole corpus come back ready to use.
What Teams Build With It.
A crawl job replaces a surprising amount of custom scraper code, powering everything from internal search indexes to site-wide RAG pipelines.
Documentation & knowledge bases
Crawl an entire docs site into markdown for a support bot or internal search index, sections and all.
Ecommerce catalog sync
Walk a category tree and pull detailed product data from every listing, with faceted URLs collapsed on the fly.
Competitive content audits
Map a competitor's site structure and content in one job, instead of scraping page by page by hand.
RAG corpus building
Turn a whole domain into markdown chunks ready for embedding, with depth and path controls keeping the corpus focused.
Site migration QA
Crawl staging and production in parallel jobs, then diff the markdown output to catch content that didn't migrate cleanly.
Firecrawl migration
Point an existing crawl integration at DataBlue. Cut costs without touching the job or response handling.
Your First Crawl in Under a Minute.
No sales call and no contract to sign. Sign up, grab a key, and your first crawl job starts with one request.
Create a Free Account
Sign up and grab an API key instantly. You get 1,000 one-time signup credits to test a real crawl, no credit card required.
Start a Crawl Job
Call /v1/crawl with a Bearer token and a seed URL. Set max_pages, max_depth, or crawl_strategy to shape how far the job goes.
Read the Results
Poll GET /crawl/{job_id} or set a webhook, and pages arrive in the same shape as a single scrape. Ship it.
A crawl costs exactly what its pages would.
Start free with 1,000 one-time signup credits. Each crawled page costs the same flat credit weight as a single scrape, so a crawl's total cost is clear before the job even finishes. There are no separate crawl surcharges and no hidden fees for concurrency or strategy choice. Monthly plans reset on schedule, and top-up credits stay available until you spend them.
Crawl API Questions.
What is a Crawl API?
A crawl API starts at one URL and follows links outward, scraping every page it finds along the way. You set the boundaries, like how deep to go or how many pages to grab, and the job runs on its own from there. DataBlue's Crawl API returns each page in the same shape as a single scrape, markdown or JSON, with status and depth attached. Start a job and watch pages arrive as the crawl finds them.
How does DataBlue's Crawl API discover pages?
DataBlue crawls breadth-first by default, spreading across a site level by level, which suits most content crawls. Depth-first is available too, drilling down one branch before backing out, useful for deeply nested documentation trees. The complete set of strategies, including a relevance-ranked best-first mode, is broken down in the section below. Set crawl_strategy once and the job handles the traversal from there.
How do I limit a crawl to specific sections of a site?
Use include_paths and exclude_paths with glob patterns, like /blog/* to stay inside one section. max_depth caps how many links away from the seed URL the crawl will follow. max_pages sets a hard ceiling on the total pages fetched. Combine these controls and a crawl of a huge site stays precisely as wide as you intend.
Does the Crawl API respect robots.txt?
Yes, by default. respect_robots_txt is true out of the box, so the crawler skips paths a site has asked automated agents not to touch. Turn it off with respect_robots_txt: false if a job genuinely needs unrestricted access, though most teams leave the default alone. allow_external_links stays false by default too, so a crawl won't wander off the seed domain unless you say so.
How do I get notified when a crawl finishes?
Two ways work out of the box. Poll GET /crawl/{job_id} and read the status field until it flips to completed. Subscribe over SSE instead, and progress arrives as pages finish. Set webhook_url on top of either one, and DataBlue posts a signed notification the moment the job wraps up, verified with webhook_secret. Wire up a webhook once and skip polling for status altogether.
Can I control what each crawled page returns?
Yes, through scrape_options nested inside the crawl request. Set formats and only_main_content just like you would on a single scrape call, along with a handful of other per-page controls. Whatever you configure there applies uniformly across every page the crawl visits. Nest an extract schema in scrape_options and every page returns typed JSON instead of markdown.
How many pages can I crawl in one job?
Up to 1,000 pages per job, with max_pages defaulting to 100 if you don't set it. max_depth caps link distance from the seed URL separately, so a shallow, wide site and a deep, narrow one both stay within bounds. concurrency controls how many pages get scraped in parallel, from one worker up to ten. Raise concurrency for large jobs, and a 1,000 page crawl finishes in a fraction of the time a serial crawler would need.
How much does the Crawl API cost?
Each page crawled costs the same flat credit weight as a single scrape call, so a 200 page crawl costs roughly what 200 individual scrapes would. There's no separate crawl surcharge on top of that. Monthly plan credits reset each billing period, and top-up credits never expire. A page that fails to return data doesn't consume a credit, so a job with a few dead links won't quietly drain your balance.
Is this compatible with Firecrawl's crawl endpoint?
Yes. The request shape and the async job pattern follow the same conventions Firecrawl uses, down to the per-page response format. Teams migrating from Firecrawl usually just swap the base URL and API key. If you already have a crawl job wired up elsewhere, point it at DataBlue and rerun it as a first test.
What happens to duplicate or faceted URLs?
filter_faceted_urls is on by default, so DataBlue collapses navigation variants like sort and filter query parameters into a single canonical URL. That keeps a crawl of an ecommerce category page from ballooning into thousands of near-duplicate results. Turn it off if a site genuinely needs every parameter combination captured. Leave the default on for most crawls, and check the page count before running the whole thing.
Start Crawling in One Job.
Grab a free key and start your first crawl job. Watch pages arrive as markdown or schema-based JSON from an entire site, no card required, with 1,000 one-time signup credits to get you started.