Schema-Based Extraction vs Prompted Scraping: Why Structure Wins

Pasting a page into a prompt and asking for fields works on one clean page and drifts on a thousand different ones. A schema based web extraction API removes the drift by fixing key names, types and required fields before the call runs, instead of hoping a good answer falls out of a well-worded request. NeuralVerge takes the schema as an optional JSON Schema string, over REST or as an MCP tool for agents, so a pipeline and an agent get the same pinned shape.

The short version

• —The claim: structure beats prompting once the output is consumed by code, because a declared shape makes the result checkable by machine instead of by eye. It does not make a value more accurate.
• —How it works here: AI extraction takes a URL, plain-language instructions and an optional JSON Schema in settings.extract_schema_json. With a schema the typed result is in machine; with none, machine is null and the JSON text is in human.
• —Who it is for: pipelines with no human reading each result, agents that reason over tool output, and teams handing extracted data to another team.
• —What it costs: 5 points per call, the same over REST and MCP, with or without a schema.
• —Who can skip it: anyone extracting from one page that a person reads before using it.

Where prompted scraping drifts

Prompted scraping is the pattern most teams reach for first: paste a page, or its raw HTML, into a prompt and ask for the fields in plain language. It is quick to set up and it works for a one-off task on a page you can check by eye. The model is not the problem. The problem is that nothing in the request fixes the shape of the answer, so the shape is free to change from one call to the next in ways a person reviewing one result at a time will not notice.

Three failures that repeat

Name drift. One response calls the field employees, the next employee_count, a third nests it under company.staff_size. Nothing forced a choice, and a slightly different page layout was enough to nudge the wording.

Type drift. founded arrives as "2019" on one page and 2019 on another. This stays invisible until something downstream does arithmetic on a string or inserts into a typed column.

Confident absence. A page does not list a founding year, and instead of an empty field the model writes a plausible one. This is the dangerous case, because a guess dressed as a fact passes casual review. A schema does not by itself stop a model from guessing, and we have not measured how often either approach does; the point is that a declared contract gives you something concrete to test for it, as the worked example below shows.

None of these means the model misread the page. They are structure failures: the request never specified a contract, so the response did not honour one consistently.

Why it only hurts at volume

A single extraction checked by a person tolerates a lot of looseness. At ten pages, drift is something a reviewer might still catch. At ten thousand, nobody reads every response; a pipeline consumes them, and a renamed field or a quiet guess does not raise an error. It produces a wrong downstream result that looks like a right one, which is a worse failure than a call that visibly breaks.

What a schema pins down

A schema is an explicit contract for the response: property names, types and which fields are required, declared once and applied to every call. In AI extraction it travels as a JSON Schema string in settings.extract_schema_json, alongside the page url and the plain-language instructions. The AI extraction guide covers the full mechanism (render, clean, map); that mechanism is identical with or without a schema. What changes is whether the final mapping step is asked to follow a contract or to choose a shape that felt natural for that page.

A request with a schema

This request asks for the same three fields as the Acme example used across the site, with the shape pinned. It is the product page's request plus a schema; the Acme company is fictional.

curl https://api.neuralverge.ai/functions/v1/run-extract \
  -H "Authorization: Bearer <API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/company/acme",
    "instructions": "Extract the company name, founding year and employee range.",
    "settings": {
      "extract_schema_json": "{\"type\":\"object\",\"properties\":{\"company_name\":{\"type\":\"string\"},\"founded\":{\"type\":\"string\"},\"employees\":{\"type\":\"string\"}},\"required\":[\"company_name\"]}"
    }
  }'

The schema is a string inside the JSON body, which is why the quotes are escaped. It declares three string properties and makes company_name required. The response shape below comes from the AI extraction product page. It is an illustration with the fictional company, not a recorded run:

{
  "session_id": "11111111-1111-1111-1111-111111111111",
  "kind": "extract",
  "url": "https://example.com/company/acme",
  "machine": {
    "company_name": "Acme",
    "founded": "2019",
    "employees": "50-100"
  }
}

The same call without a schema

Two details come from calls we ran rather than from a description. With a schema, machine is a typed object, as above. With no schema, machine is null and the result arrives as JSON text in the human field:

{
  "kind": "extract",
  "url": "https://example.com/company/acme",
  "machine": null,
  "human": "<the extracted fields, as JSON text>"
}

That is the shape of the response, with the content elided. The text in human parses as JSON, but its keys are the model's choice, so two calls on similar pages are not guaranteed to produce the same ones. Your code should read machine when you sent a schema and human when you did not, and handle a null in either case. This difference is the whole argument in miniature: one path hands your code an object with declared keys, the other hands it text to parse and hope about.

Required fields turn absence into a signal

A field being present most of the time is different from a field being guaranteed present, and that distinction is what a required list records. Listing company_name as required cannot make it appear on a page that has none. What it gives you is a rule your validator can apply the same way on every record: a required field that is empty or missing routes that record to review, instead of looking like any other empty field.

Be exact about who enforces what. The schema narrows what the model is asked to return. The product pages do not state how the service behaves when a required field is absent from the page, so test that on a page you know lacks one before you rely on it, and let your own validator be the enforcement. Describing fields in prose tells a model what to look for; a schema with required fields gives you a rule to check against.

What a schema does not do

A schema does not make extraction more accurate on a page that lacks the information; a missing field is empty either way, and nothing invents a founding year that was never published. It does not stop a wrong value of the right type, such as a group headcount returned where the entity headcount was wanted. That needs domain checks, covered in what changes in code when you describe data instead of parsing it, which also walks through validation and evals. And a schema does not replace looking at real pages first.

Inferred first, schema second

The two approaches are stages of the same pipeline, not rivals. A team meeting a new class of pages, such as unfamiliar registry pages or a competitor's product listings, reasonably starts with instructions only. It is faster to see what the pages contain before committing to a shape. Once the sample shows which fields matter and which pages reliably have them, you add a schema to the same call. There is no second tool and no rewrite; the request gains one setting.

Pipelines get this wrong two ways: staying on inferred structure forever and accumulating silent drift, or writing a rigid schema before anyone has looked at the pages and finding it wrong for a share of them in production. Treat inference as discovery and the schema as production.

A worked example: 500 profile pages

This case is illustrative, not a recorded run. You want company_name, founded and employees from 500 company profile pages with different layouts: some show the founding year prominently, some bury it in a footer, some never publish it.

With prompted scraping, each of the 500 responses looks reasonable alone. Loading them into one table is where the drift appears: some rows say founded, others founding_year or year_founded; some responses wrap the JSON in a sentence and do not parse; some pages with no listed year may carry a plausible one anyway. The cleanup is real work, and it grows with the number of pages.

With a schema, you declare the three keys and their types once. Every response is asked for the same keys, and machine arrives as an object your code reads without parsing prose. Two checks still apply, and you should run them. First, diff the key names across all 500 responses, which should now be identical. Second, take ten pages you know have no founding year and look at what founded holds; that tells you how your pages behave with a missing field, which is the one thing we do not claim to know in advance. Then add the validator: founded must look like a year and not be in the future, and company_name must be non-empty. The schema gave you the shape; these rules give you the guarantee.

REST and MCP

The same extraction is reachable two ways, with the same inputs and outputs. Over REST you POST to the run-extract endpoint shown above. Over MCP, the hosted server at https://mcp.neuralverge.ai/mcp exposes it as the run_extract tool. Tool names use underscores where REST paths use hyphens. You add that one URL to an MCP client such as Claude, ChatGPT, Cursor, VS Code or Gemini CLI, sign in through the browser and click Allow; there is no key to paste into a config file. Setup is covered in the MCP server for Claude guide and at docs.neuralverge.ai/connect-a-client.

An agent call carries the schema the same way, as a string in settings.extract_schema_json:

{
  "tool": "run_extract",
  "arguments": {
    "url": "https://example.com/company/acme",
    "instructions": "Extract the company name, founding year and employee range.",
    "settings": {
      "extract_schema_json": "{\"type\":\"object\",\"properties\":{\"company_name\":{\"type\":\"string\"},\"founded\":{\"type\":\"string\"},\"employees\":{\"type\":\"string\"}},\"required\":[\"company_name\"]}"
    }
  }
}

The price is the same as REST, and the machine and human behaviour is the same, so check the field that matches whether you sent a schema.

Pick REST when a pipeline already knows which pages to process and when, such as a weekly monitoring job or a batch of 500 profiles. Pick MCP when an agent decides mid-task that it needs a fact from a page it just found. The schema matters in both, and arguably more for the agent: it reasons over whatever comes back, and a stable shape stops it guessing what a renamed field means. With an agent the model sets the call count, so watch early runs to see how many 5-point calls a task makes.

Cost and limits

An AI Extract call costs 5 points, flat per call, with or without a schema. Passing a schema does not add a charge, and we have not seen evidence that it changes how long the render-and-clean steps take; measure latency on your own pages rather than trusting a general claim. Points come from your plan's monthly allowance, so a workflow costs calls multiplied by 5. Current plans and allowances are on the pricing page, and those numbers are the ones to trust over anything here.

The charge does not depend on the result. An extraction call on a sample-invoice PDF that returned null was still charged 5 points, so do not budget as if empty or failed calls are free. If your list includes many URLs that may not contain the fields, run a sample first.

The call reads one public page or document and returns fields. It does not crawl a site, log in or interact with a page. The product pages state no accuracy figure, no maximum page size and no throughput, so this article gives none. If any of those decides your choice, test it on your own pages. Check the documentation for current endpoint and parameter details before you build against them.

How to decide

Use a schema when most of these are true:

• —No human reads each result. Anything consumed programmatically turns drift into a silent downstream bug instead of something a reviewer catches.
• —The output feeds a typed destination. A database column, a typed API or a data frame breaks, or quietly coerces, when a field is a string on one page and a number on the next.
• —Several page templates are merged. Combining results from different layouts works only if every source returns the same names and types.
• —An agent reasons over the output. Predictable structure helps a model the same way it helps a developer.
• —A job runs repeatedly. A schema fixed at the start keeps each weekly or daily run comparable to the last, so a changed field name cannot break a comparison unnoticed.
• —Another team consumes the data. A documented schema is the interface contract and replaces a conversation you would otherwise have each time the shape shifts.

Stay on instructions only when you are exploring a new class of pages, or when a single page is checked by a person before use. Before trusting any pipeline's structure, run five or six genuinely different pages through it, not one clean example, and look at three things: whether a missing field comes back empty or filled, whether names and types stay identical across layouts, and whether every response parses as typed JSON. Diffing the field names across a real batch makes the gap concrete in a way a description cannot.

Frequently asked questions

Is asking an LLM for fields from a page the same as schema-based extraction?

On an easy page the output can look the same, which is why the difference is easy to miss. It shows up across many pages: with a bare prompt nothing fixes the key names, the types or what happens when a field is absent. With a schema, the shape is declared once and every call is asked to follow it. Your own validation still has to check the values.

Do I always need to define a schema?

No. Describing the fields in plain language works well while you are exploring what a class of pages contains. A schema earns its place when a downstream system, such as a database table or another team's code, expects the same keys and types on every call.

Where does the result come back, with and without a schema?

With a JSON Schema in settings.extract_schema_json, the typed result is in the machine field. With no schema, machine is null and the result arrives as JSON text in the human field. Read whichever field matches how you called it.

Does a schema guarantee the values are correct?

No. A schema fixes the shape: key names, types and which fields are required. It cannot make a value true. A page that lacks a field, or a page that shows two competing numbers, still needs a validator and a sample of hand-checked pages to catch it.

Can prompted scraping be fine for a one-off task?

Yes. For a single page that a person checks before using it, the gap this article describes barely matters. It appears at volume, on pages nobody has inspected, which describes most pipelines past the prototype.

Can I use schema-based extraction over MCP, or only through the API?

Both work. The hosted MCP server at mcp.neuralverge.ai/mcp exposes extraction as the run_extract tool, with the same inputs as the REST call, including a JSON Schema in settings.extract_schema_json. The price matches REST, at 5 points per call.

Should I start with a schema or with inferred structure?

Start with instructions only on a small sample of real pages, look at what comes back, then write the schema from what you saw. Committing to a rigid schema before looking at real pages tends to produce one that is wrong for a share of them.