Understand Responses

Check the response envelope first, then read only the fields your operation requested.

Immediate Response Envelope

Example

{
  "success": true,
  "data": {
    "markdown": "# Example Domain...",
    "links": ["https://www.iana.org/domains/example"],
    "metadata": {
      "title": "Example Domain",
      "source_url": "https://example.com",
      "status_code": 200,
      "word_count": 28,
      "content_length": 191
    }
  },
  "job_id": "optional-job-id",
  "time_taken": 1.42
}

Field	Meaning	Handling rule
`success`	Whether the requested operation produced a successful public result	Check it before reading `data`
`data`	Returned content or structured records	Treat fields as conditional on operation and requested formats
`error`	Human-readable failure context	Log safely and show a user-appropriate message
`error_code`	Machine-readable failure category when available	Use it to select retry or recovery behavior
`job_id`	Identifier for persisted or asynchronous work	Store it with your internal operation record
`time_taken`	Public elapsed time in seconds when available	Use for observability, not as a fixed performance guarantee

Data Fields Are Format-Driven

Optional fields are omitted when they were not requested or produced.

Requested format or feature	Expected field	Notes
`markdown`	`data.markdown`	Readable normalized content
`html`	`data.html`	Normalized HTML
`raw_html`	`data.raw_html`	Potentially large source-oriented HTML
`links`	`data.links` and possibly `data.links_detail`	URL list plus richer link grouping when available
`images`	`data.images`	Image records can include source and descriptive metadata
`headings`	`data.headings`	Document outline information
`structured_data`	`data.structured_data`	Embedded page metadata such as JSON-LD or Open Graph
`screenshot`	`data.screenshot`	Large encoded visual output
LLM extraction	`data.extract`	Validate important values against source content

Metadata Describes the Source

Field	Use
`title`	Display name or document title
`description`	Page-provided summary when available
`language`	Language information when identified
`source_url`	The source associated with the returned content
`status_code`	Source response status represented in the result
`content_type`	Returned source media type when available
`word_count`	Approximate text volume
`reading_time_seconds`	Approximate reading duration
`content_length`	Returned content size measurement
`canonical_url`	Canonical source declared by the page when available
`og_image`	Primary social image when declared
`favicon`	Page or site icon when available

Inspect a Response With jq

Example

# Save the complete response
curl --fail-with-body --silent --show-error \
  -X POST "https://api.datablue.dev/v1/scrape" \
  -H "Authorization: Bearer $DATABLUE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown", "links", "images"]
  }' > response.json

# Summarize without printing the full content
jq '{
  success,
  error,
  error_code,
  time_taken,
  source: .data.metadata.source_url,
  status: .data.metadata.status_code,
  words: .data.metadata.word_count,
  markdown_chars: (.data.markdown | length),
  links: (.data.links | length),
  images: (.data.images | length)
}' response.json

Asynchronous Job Lifecycle

Crawl, Search, and multi-source Extract can return jobs. Store the job and follow its lifecycle.

Start: send the operation request and store the returned job_id.
Observe: poll the operation status endpoint or subscribe to the documented event flow.
Read partial progress: when supported, display completed counts and available result rows.
Stop on a terminal state: completed, failed, or cancelled.
Persist the outcome: store source URLs, result identifiers, and the final status needed by your product.

Start a crawl and capture its id

Example

export CRAWL_JOB_ID="$(
  curl --fail-with-body --silent --show-error \
    -X POST "https://api.datablue.dev/v1/crawl" \
    -H "Authorization: Bearer $DATABLUE_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "url": "https://example.com",
      "max_pages": 10,
      "max_depth": 2,
      "concurrency": 2,
      "scrape_options": {
        "formats": ["markdown", "links"]
      }
    }' | jq -r '.job_id'
)"

echo "$CRAWL_JOB_ID"

Poll status and page through results

Example

curl --fail-with-body --silent --show-error \
  "https://api.datablue.dev/v1/crawl/$CRAWL_JOB_ID?page=1&per_page=20" \
  -H "Authorization: Bearer $DATABLUE_API_KEY" | jq '{
    status,
    total_pages,
    completed_pages,
    total_results,
    page,
    per_page,
    returned_rows: (.data | length),
    error
  }'

Poll until a terminal state

Example

while true; do
  RESPONSE="$(
    curl --fail-with-body --silent --show-error \
      "https://api.datablue.dev/v1/crawl/$CRAWL_JOB_ID?page=1&per_page=20" \
      -H "Authorization: Bearer $DATABLUE_API_KEY"
  )"

  STATUS="$(printf '%s' "$RESPONSE" | jq -r '.status')"
  printf '%s
' "$RESPONSE" | jq '{
    status,
    completed_pages,
    total_pages,
    error
  }'

  case "$STATUS" in
    completed) break ;;
    failed|cancelled) exit 1 ;;
    *) sleep 2 ;;
  esac
done

Cancel work that is no longer needed

Example

curl --fail-with-body --silent --show-error \
  -X DELETE "https://api.datablue.dev/v1/crawl/$CRAWL_JOB_ID" \
  -H "Authorization: Bearer $DATABLUE_API_KEY" | jq .

Pages completed before cancellation can remain available. Treat cancellation as a terminal state with potentially useful partial results.

Failure Envelope

Example

{
  "success": false,
  "data": null,
  "error": "Human-readable failure context",
  "error_code": "TIMEOUT",
  "job_id": "optional-job-id",
  "time_taken": 30.0
}

Separate HTTP failure from operation failure

HTTP status: tells you whether authentication, validation, admission, or routing succeeded.
success: tells you whether DataBlue produced the requested public result.
status: tells you where an asynchronous job is in its lifecycle.
error_code: helps select a recovery path when the response provides one.
error: provides context for logs and user-facing translation.

Defensive Parsing Rules

Parse the HTTP response body even when the status is not 2xx.
Check success before dereferencing data.
Expect optional fields to be absent.
Validate arrays before assuming they contain rows.
Store job_id before beginning polling.
Keep original source URLs beside transformed or extracted data.
Do not treat elapsed time or word count as a completeness guarantee.
Validate high-impact extracted facts against source evidence.

Next Step

Use Production Readiness to turn this response model into retry, timeout, webhook, storage, and observability policies.