Overview

scrapfetch is an API that extracts structured data from websites using LLM agents. You send a target URL and a JSON Schema describing the data you want. The agent crawls the site (up to 3 levels deep), reads the content, and returns structured JSON matching your schema.

The key insight: your schema's description fields serve as extraction instructions. They tell the LLM what to look for, what's relevant, and what to ignore.

Authentication

Every extraction and batch endpoint requires an API key. Create one in your dashboard — it is shown once, when it is created, and cannot be retrieved afterwards. Keys start with sf_.

Send it as a bearer token:

curl -X POST https://scrapfetch.app/extract \
  -H "Authorization: Bearer sf_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "schema": {...}}'

An X-API-Key: sf_your_key_here header works identically, for clients that reserve Authorization for something else.

GET /models and GET /health are public and need no key. Everything else answers 401 without one.

A key belongs to one account and spends that account's credit. Revoke a leaked key from the dashboard; revocation takes effect immediately.

Quickstart

Extract product data from a website in one request. Set SCRAPFETCH_API_KEY to a key from your dashboard first.

curl -X POST https://scrapfetch.app/extract \
  -H "Authorization: Bearer $SCRAPFETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example-shop.com",
    "model": "grok-4.5",
    "schema": {
      "type": "object",
      "description": "Extract product listings with prices.",
      "required": ["products"],
      "properties": {
        "products": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "name": {"type": "string"},
              "price": {"type": "number", "description": "Price in local currency"}
            }
          }
        }
      }
    }
  }'

Response:

{
  "data": {
    "products": [
      {"name": "Widget Pro", "price": 29.99},
      {"name": "Widget Basic", "price": 9.99}
    ]
  },
  "rejected": false,
  "usage": {
    "total_tokens": 12450,
    "prompt_tokens": 10200,
    "completion_tokens": 2250,
    "cached_tokens": 0,
    "iterations": 5
  },
  "meta": {
    "url": "https://example-shop.com",
    "model": "grok-4.5",
    "pages_fetched": 8,
    "errors": 0,
    "duration_ms": 34500
  }
}

Note the duration. Extractions routinely run for a minute or more, and most reverse proxies and CDNs give up before that — Cloudflare returns a 524 after 100 seconds without a response. This endpoint writes nothing until the run finishes, so anything slow needs POST /extract/stream, which starts emitting heartbeats immediately and keeps the connection alive.

Send a URL and a JSON Schema to extract structured data. The agent crawls the target site, reads all fetched pages, and returns data matching your schema.

Request body

Crawling costs more than it looks like it should. Every page fetched becomes a line in the site map the model is given, and that map is re-sent on every round of the extraction — so a page that links into a large site on the same host can spend the whole request budget before a single fact is read. If you know which page holds the data, "depth": 0 is both cheaper and more accurate.

Response body

Rejection

If your schema's root description specifies relevance criteria (e.g., "RELEVANT: sauna, wellness. NOT RELEVANT: e-shop, fitness"), the agent will return rejected: true for sites that don't match — without attempting full extraction. This saves tokens and time.

{
  "rejected": true,
  "rejection_reason": "This is a fitness center without sauna facilities.",
  "usage": {"total_tokens": 3200, ...},
  "meta": {"url": "https://example-gym.com", "pages_fetched": 4, ...}
}

Same as /extract, but returns a Server-Sent Events stream. Use it for anything that might run long: a heartbeat every 15 seconds keeps the connection alive through proxies that would otherwise time out an idle request — Cloudflare cuts one off after 100 seconds, and Nginx defaults to 60.

Request

Identical to POST /extract. Same request body format (including the optional model field).

SSE events

curl example

curl -N -X POST https://scrapfetch.app/extract/stream \
  -H "Authorization: Bearer $SCRAPFETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","schema":{...}}'

# Output:
# event: heartbeat
# data: {}
#
# event: heartbeat
# data: {}
#
# event: result
# data: {"data":{...},"rejected":false,"usage":{...},"meta":{...}}

JavaScript example

const response = await fetch("/extract/stream", {
  method: "POST",
  headers: {
    "Authorization": "Bearer " + apiKey,
    "Content-Type": "application/json"
  },
  body: JSON.stringify({url: "https://example.com", schema: {...}})
});

const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = "";

while (true) {
  const {done, value} = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, {stream: true});

  const lines = buffer.split("\n");
  buffer = lines.pop();

  for (const line of lines) {
    if (line.startsWith("event: ")) {
      const event = line.slice(7);
    } else if (line.startsWith("data: ")) {
      const data = JSON.parse(line.slice(6));
      if (event === "result") console.log("Extraction complete:", data);
    }
  }
}

List available LLM models. Use the model ID in extraction requests to select a specific model.

Response body

Array of model objects:

Example

curl https://scrapfetch.app/models

[
  {
    "id": "grok-4.5",
    "name": "Grok 4.5",
    "provider": "xAI",
    "is_default": true,
    "batch_compatible": true
  },
  {
    "id": "grok-4.3",
    "name": "Grok 4.3",
    "provider": "xAI",
    "is_default": false,
    "batch_compatible": true
  }
]

Health check endpoint. Returns 200 OK when the server is running.

{"status": "ok"}

Batch Processing

The Batch API provides asynchronous extraction at 50% lower token cost by using the xAI Batch API under the hood. Submit jobs via POST /batch/jobs and poll for results with GET /batch/jobs/{id}. Jobs are processed in the background by a separate worker process.

The same schema-driven extraction logic applies: the agent crawls the site, reads pages, uses tools, and returns structured data. The only difference is timing — results are available minutes to hours later instead of immediately.

When to use Batch vs Real-time

• Use POST /extract for interactive use cases where you need results immediately
• Use POST /batch/jobs for background pipelines, bulk updates, and cost-sensitive workloads

Requirements

Batch processing requires a CockroachDB database (DATABASE_URL env var) and a default model configured in the providers/models tables. The batch worker runs inside the API server process automatically.

Submit a new extraction job. Returns immediately with a job ID. The job is queued for background processing.

Request body

Response

// 201 Created
{
  "job_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "pending"
}

Example

curl -X POST https://scrapfetch.app/batch/jobs \
  -H "Authorization: Bearer $SCRAPFETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example-shop.com",
    "schema": {
      "type": "object",
      "description": "Extract product listings with prices.",
      "properties": {
        "products": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "name": {"type": "string"},
              "price": {"type": "number"}
            }
          }
        }
      }
    },
    "language": "en"
  }'

List batch jobs. Optionally filter by status.

Query parameters

Example

curl https://scrapfetch.app/batch/jobs?status=done \
  -H "Authorization: Bearer $SCRAPFETCH_API_KEY"

[
  {
    "id": "a1b2c3d4-...",
    "url": "https://example-shop.com",
    "status": "done",
    "iteration": 7,
    "rejected": false,
    "usage": {"total_tokens": 15200, "prompt_tokens": 12000, "completion_tokens": 3200, "cached_tokens": 8500},
    "pages_fetched": 12,
    "created_at": "2026-03-17T10:00:00Z",
    "completed_at": "2026-03-17T10:45:00Z"
  }
]

Get details and result for a specific batch job. Poll this endpoint to check if the job is complete.

Response body

Polling pattern

# Submit job
JOB_ID=$(curl -s -X POST https://scrapfetch.app/batch/jobs \
  -H "Authorization: Bearer $SCRAPFETCH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","schema":{...}}' | jq -r .job_id)

# Poll until done
while true; do
  STATUS=$(curl -s https://scrapfetch.app/batch/jobs/$JOB_ID \
    -H "Authorization: Bearer $SCRAPFETCH_API_KEY" | jq -r .status)
  echo "Status: $STATUS"
  [ "$STATUS" = "done" ] || [ "$STATUS" = "failed" ] || [ "$STATUS" = "rejected" ] && break
  sleep 60
done

# Get result
curl -s https://scrapfetch.app/batch/jobs/$JOB_ID \
  -H "Authorization: Bearer $SCRAPFETCH_API_KEY" | jq .data

Job Statuses

A batch job progresses through these statuses. The agent loop (llm_pending → polling → tools_pending → llm_pending) may repeat multiple times as the LLM reads files and searches for data.

Terminal statuses: done, rejected, failed. Once a job reaches a terminal status, it won't change.

Billing

Credit is prepaid and spent per extraction, priced on the tokens the run actually used. There is no subscription and no monthly minimum. Buy credit from your dashboard; the balance and recent activity are shown there too.

The per-request cap

Every request reserves a fixed ceiling before it starts, and the same number is the limit the agent loop enforces mid-run. A run that reaches it stops and returns budget_exhausted with HTTP 402. One page can therefore never cost more than the cap, whatever it contains — and the reservation is released as soon as the run settles.

If the available balance will not cover the cap, the request is refused up front with insufficient_credit and HTTP 402. That refusal always arrives before any data does, including on /extract/stream.

What is charged

Charged:

• A successful extraction.
• A run where the model examined the page and found no matching data.
• A run that reached the per-request cap.
• A run cancelled by the client after it started — the tokens were already spent.

Not charged:

• A model provider outage or rate limit.
• A request that hit the server timeout.
• Any internal error on our side.

Batch jobs are not metered yet and do not consume credit.

Error Codes

All errors return a JSON object with error, code, and optional details fields.

{"error": "Invalid JSON body", "code": "invalid_json", "details": "unexpected EOF"}

Schema Guide

The JSON Schema you send serves a dual purpose: it defines the output structure and provides extraction instructions via description fields. This is the most important concept in scrapfetch.

Root description

The top-level description field tells the agent what the schema is about, what's relevant, and what to reject. Use clear, direct language.

{
  "type": "object",
  "description": "Extract sauna/wellness data. RELEVANT: sauna, wellness center, spa. NOT RELEVANT: massage salon, fitness center, e-shop.",
  "properties": { ... }
}

Property descriptions

Each property's description guides the extraction for that specific field. Be specific about format, units, and what to include/exclude.

"price": {
  "type": "number",
  "description": "Price in CZK as a number without currency symbol. Must be an actual price, NOT a percentage."
}

"duration": {
  "type": "number",
  "description": "Duration in minutes. 1h=60, 2h=120. Unlimited or all-day access = 0."
}

Tips

• Use CAPS for emphasis: "IGNORE massages, pools, fitness"
• Define NOT_RELEVANT criteria to enable smart rejection and save tokens
• Specify exact formats: "HH:MM (24h)", "price as number without currency"
• Use the array description to explain expected structure: "Must contain exactly 7 items (index 0=Monday, ..., 6=Sunday)"
• The agent can process images — if data might be in images, mention it in the description