Production Readiness

Production integrations need boundaries, secure credentials, recovery, validation, and observability.

Production Architecture

1. Accept an internal request. Validate the source, operation, and allowed scope before calling DataBlue.
2. Create an internal operation record. Store your request id, user, selected DataBlue operation, and source.
3. Send a bounded DataBlue request. Set explicit formats, timeouts, page limits, depth, and path rules.
4. Record the external result. Store HTTP status, success, job_id, terminal state, and safe error context.
5. Validate and transform. Confirm required content exists before indexing, displaying, or acting on it.
6. Publish the outcome. Update your user-facing state independently from the request worker.

1. Protect Credentials

• Call DataBlue from trusted server-side code, workers, CLI processes, or locally configured MCP clients.
• Inject DATABLUE_API_KEY through a secrets manager or deployment environment.
• Use separate named keys for development, CI, staging, and production.
• Redact Authorization, cookies, user-supplied secrets, and webhook secrets from logs.
• Rotate without downtime: create a replacement, deploy it, verify traffic, then delete the old key.

2. Bound Every Operation

Never let an end user silently convert an unbounded domain into an unrestricted crawl. Translate product intent into explicit server-side limits.

3. Set Layered Timeouts

Use separate timeouts for the page operation, your HTTP client, and the overall business workflow.

4. Retry Selectively

Retries are safe only when they are bounded and based on a failure category. Use exponential backoff with jitter and a small maximum attempt count.

A bounded curl retry example

curl --fail-with-body --silent --show-error \
  --connect-timeout 10 \
  --max-time 75 \
  --retry 2 \
  --retry-delay 2 \
  --retry-max-time 160 \
  --retry-connrefused \
  -X POST "https://api.datablue.dev/v1/scrape" \
  -H "Authorization: Bearer $DATABLUE_API_KEY" \
  -H "Content-Type: application/json" \
  -H "X-Request-ID: your-internal-request-id" \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown", "links"],
    "timeout": 30000
  }' | tee response.json

Use both HTTP status and returned failure fields to decide whether to retry.

5. Persist Long-Running Jobs

Do not keep crawl or search progress only in process memory. Store enough information to resume observation after a restart.

Use a state machine

created -> submitted -> pending -> running -> completed
                                      |          |
                                      |          +-> processed
                                      +-> failed
                                      +-> cancelled

Make terminal transitions idempotent so duplicate status checks or webhook deliveries cannot process the same result twice.

6. Use Webhooks for Completion

Use signed webhooks for completion and keep polling as a recovery path.

curl --fail-with-body --silent --show-error \
  -X POST "https://api.datablue.dev/v1/crawl" \
  -H "Authorization: Bearer $DATABLUE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/docs",
    "max_pages": 25,
    "max_depth": 3,
    "scrape_options": {
      "formats": ["markdown"]
    },
    "webhook_url": "https://your-app.example/webhooks/datablue",
    "webhook_secret": "your-dedicated-webhook-secret"
  }'

• Use HTTPS.
• Verify the signature according to the Webhooks reference before parsing the payload.
• Acknowledge valid deliveries quickly and move heavy processing to a queue.
• Deduplicate by delivery identity and job identity.
• Do not trust a status change solely because a JSON body claims it.
• Reconcile missed events by polling the job endpoint.

7. Validate Content Before Use

For markdown and page content

• Require a source URL and successful source status.
• Set minimum and maximum content sizes appropriate to the workflow.
• Keep source metadata beside indexed or transformed content.
• Deduplicate repeated content before inserting it into a retrieval index.
• Sanitize rendered HTML in your own UI even when the source is trusted.

For structured extraction

• Validate against your JSON Schema again inside your application.
• Reject missing required fields instead of silently inserting defaults.
• Preserve the source URL and, where useful, source text supporting the field.
• Require human or deterministic verification for financial, legal, medical, or irreversible decisions.

8. Control Response Size and Storage

• Request only necessary formats.
• Store screenshots and raw HTML in object storage instead of frequently queried relational rows.
• Compress large text payloads at rest when appropriate.
• Set retention periods by data type and user expectation.
• Delete temporary response files and avoid leaving API output in shared build logs.
• Record content hashes when your pipeline uses them for deduplication.

9. Observe the Integration

Record metrics by DataBlue operation and your internal use case rather than one undifferentiated request counter.

Log request and job ids, operation, source host, status, and duration. Never log credentials.

10. Plan for Credits and Concurrency

• Read Plans & Credits before selecting default page and result limits.
• Read Concurrency before choosing worker parallelism.
• Queue work inside your product when user demand can exceed available concurrency.
• Apply per-user limits so one customer cannot consume the entire integration budget.
• Display progress for long jobs instead of encouraging repeated submissions.
• Alert before credits are exhausted and provide a clear product behavior when work cannot be admitted.

Failure Playbook

Launch Checklist

• One operation and one source class have been tested with fresh inputs.
• Credentials are server-side, named, redacted, and independently revocable.
• Every request has explicit scope and timeout boundaries.
• HTTP and operation failures are handled separately.
• Retries are selective, bounded, delayed, and observable.
• Long-running jobs survive process restarts.
• Webhook signatures are verified and deliveries are idempotent.
• Required response fields and content size are validated.
• Large formats have a storage and retention policy.
• Metrics, logs, and alerts expose failures without exposing secrets.
• Concurrency and credit behavior are reflected in product limits.
• There is a documented cancellation and incident-recovery path.

Continue to the Reference