Choose a Workflow

Choose by the result you need. Use the smallest workflow that can return it.

Start With the Result You Need

You have	You need	Use
One known URL	Readable content or page assets	Scrape
One domain	A list of its useful URLs	Map
One domain or section	Content from multiple linked pages	Crawl
A topic or question	Relevant web results and optional page content	Search
URLs, HTML, or text	Fields matching a JSON Schema	Extract
A supported commerce, search, local, app, ads, or media query	Purpose-built structured records	Data API

The Six Core Workflows

Scrape: read a known page

Use Scrape when you already know the page URL. It returns one processed page directly.

Important option	What it controls	Practical default
`url`	The public page to retrieve	Required
`formats`	Which outputs appear in `data`	`["markdown"]`
`only_main_content`	Whether navigation, footers, and sidebars are removed	Enable for reading and LLM context
`wait_for`	Additional milliseconds to wait after page load	Leave at `0` unless content appears late
`timeout`	Maximum processing time in milliseconds	`30000`
`mobile`	Whether to request a mobile-rendered page	`false`

curl --fail-with-body --silent --show-error \
  -X POST "https://api.datablue.dev/v1/scrape" \
  -H "Authorization: Bearer $DATABLUE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown", "links", "images"],
    "only_main_content": true
  }' | jq .

Response: Outputs are in data. Page details, status, quality, and timing use data.metadata, data.status, data.quality, and data.time_taken.

Map: discover before retrieving

Map returns a URL inventory without scraping every page. Use it to inspect and filter a site first.

Option	What it controls	Default
`url`	The domain or starting URL to inventory	Required
`search`	Filters discovered URLs by a text query	No filter
`limit`	Maximum URLs returned	`100`
`include_subdomains`	Includes URLs hosted on subdomains	`true`
`use_sitemap`	Uses sitemap data during discovery	`true`

curl --fail-with-body --silent --show-error \
  -X POST "https://api.datablue.dev/v1/map" \
  -H "Authorization: Bearer $DATABLUE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "search": "documentation",
    "limit": 100,
    "include_subdomains": true,
    "use_sitemap": true
  }' | jq .

Response: total counts the records. links contains discovered URLs and available page details; status identifies empty results.

Crawl: collect a bounded page set

Crawl collects linked pages as a job. Set page, depth, and path limits before starting.

Option	What it controls	Default
`url`	The first page and crawl origin	Required
`max_pages`	Maximum number of pages collected	`100`
`max_depth`	Maximum link distance from the seed	`3`
`concurrency`	Parallel page work inside the job	`3`
`crawl_strategy`	Breadth-first, depth-first, or best-first ordering	`bfs`
`include_paths` / `exclude_paths`	Glob rules that bound the crawl to useful sections	No path rules
`scrape_options`	Formats and extraction settings applied to every page	Page scrape defaults

START_RESPONSE="$(curl --fail-with-body --silent --show-error \
  -X POST "https://api.datablue.dev/v1/crawl" \
  -H "Authorization: Bearer $DATABLUE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/docs",
    "max_pages": 25,
    "max_depth": 3,
    "concurrency": 3,
    "include_paths": ["/docs/**"],
    "exclude_paths": ["/docs/archive/**"],
    "allow_external_links": false,
    "respect_robots_txt": true,
    "scrape_options": {
      "formats": ["markdown", "links"],
      "only_main_content": true
    }
  }')"

echo "$START_RESPONSE" | jq .
JOB_ID="$(echo "$START_RESPONSE" | jq -r '.job_id')"

curl --fail-with-body --silent --show-error \
  "https://api.datablue.dev/v1/crawl/$JOB_ID" \
  -H "Authorization: Bearer $DATABLUE_API_KEY" | jq .

Start response: Save job_id, then poll GET /crawl/{job_id} or use the event stream. Completed data contains one result per page.

Search: find sources, then optionally read them

Search finds sources when you know the query but not the URLs. It can also retrieve content from result pages.

Option	What it controls	Default
`query`	The web question or search expression	Required
`num_results`	How many result pages are considered	`5`
`engine`	The documented search provider choice	`google`
`formats`	Content formats requested from result pages	`["markdown"]`
`only_main_content`	Removes repeated page chrome from retrieved results	`false`
`extract`	Optional schema-guided extraction applied to each result	Not enabled

START_RESPONSE="$(curl --fail-with-body --silent --show-error \
  -X POST "https://api.datablue.dev/v1/search" \
  -H "Authorization: Bearer $DATABLUE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "site:example.com deployment guide",
    "num_results": 5,
    "engine": "google",
    "formats": ["markdown"]
  }')"

echo "$START_RESPONSE" | jq .
JOB_ID="$(echo "$START_RESPONSE" | jq -r '.job_id')"

curl --fail-with-body --silent --show-error \
  "https://api.datablue.dev/v1/search/$JOB_ID" \
  -H "Authorization: Bearer $DATABLUE_API_KEY" | jq .

Start response: Save job_id and poll GET /search/{job_id}. Completed data contains ranked results and requested page content.

Extract: enforce a data shape

Extract returns specific fields. Provide a source, a precise instruction, and an optional JSON Schema.

Option	What it controls	Notes
`url`	One page to retrieve before extraction	Use for a direct single-source result
`urls`	Multiple pages to retrieve and process	Runs as a longer job
`content` / `html`	Supplied source material that does not need retrieval	Use one source form
`prompt`	Plain-language extraction instruction	Be specific about units, missing values, and scope
`schema`	The required JSON result shape	Recommended for application logic
`only_main_content`	Removes surrounding page chrome before extraction	Useful for articles and documents

curl --fail-with-body --silent --show-error \
  -X POST "https://api.datablue.dev/v1/extract" \
  -H "Authorization: Bearer $DATABLUE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "prompt": "Extract every pricing plan and its displayed monthly price.",
    "schema": {
      "type": "object",
      "properties": {
        "plans": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "name": {"type": "string"},
              "monthly_price": {"type": "string"}
            },
            "required": ["name", "monthly_price"]
          }
        }
      },
      "required": ["plans"]
    }
  }' | jq .

Response: Direct results use data.extract. Multi-URL extraction returns a job that must be retrieved after completion.

Data APIs: query a specialized web surface

Use a Data API when DataBlue already supports the exact source and dataset you need.

This Google SERP request shows the authenticated GET pattern. Each Data API defines its own parameters.

curl --fail-with-body --silent --show-error --get \
  "https://api.datablue.dev/v1/data/google/serp" \
  -H "Authorization: Bearer $DATABLUE_API_KEY" \
  --data-urlencode "query=best crm software" \
  --data-urlencode "country=us" \
  --data-urlencode "results=10" \
  --data-urlencode "advanced=false" | jq .

Response: Data APIs return source-specific records. Check each endpoint's response fields before storing them.

Open Data API Status and then the endpoint-specific reference page to obtain its current parameters, examples, and credit behavior.

High-Value Workflow Combinations

Map → filter → ScrapeInventory a large site, select relevant URLs in your application, then scrape only those pages. Map → CrawlUnderstand the site structure first, then launch a bounded crawl with accurate include and exclude paths. Search → ScrapeDiscover authoritative pages for a question, then request richer formats from selected result URLs. Scrape → ExtractInspect source markdown first, then extract a stable schema after confirming the needed facts exist. Search → Scrape → Extract with MCPLet an agent discover sources, retrieve evidence, and structure facts through explicit DataBlue tools. Data API → ScrapeUse structured discovery records, then scrape selected source URLs for deeper page content.

Avoid These Common Mismatches

Mismatch	Why it hurts	Better choice
Crawling when one URL is known	Adds job management and retrieves unnecessary pages	Use Scrape
Scraping a homepage to discover an entire site	A homepage rarely exposes every useful URL	Use Map
Searching when a supported Data API exists	Requires parsing generic results into a domain model	Use the Data API
Extracting before inspecting source quality	A schema cannot recover facts absent from the source	Scrape and validate first
Starting an unrestricted crawl	Creates unpredictable scope, duration, and credit use	Set page, depth, and path boundaries

Next Step

After selecting a workflow, configure Authentication and complete the First Request walkthrough.