From HTML to JSON: What Changes When You Add AI to Extraction
An html parsing alternative ai teams can adopt step by step: what changes in code, where determinism goes, and how to call it by API or MCP.
By the NeuralVerge team · 12 min read · Updated October 2026
▸On this page
An html parsing alternative ai teams can trust is one where you describe the data you want instead of the markup it sits in, and a model finds it on the rendered page. That changes the code you write, how it fails, what you test and what you pay. This article is for people who already maintain parsers and want to know what they would be trading. You can reach the extraction step over REST from code, or over MCP from an AI agent.
The short version
- —The claim: AI extraction swaps "where is the value" for "what is the value". Layout changes stop breaking your code, but you give up free determinism and have to rebuild it with schemas, validation and evals.
- —Who it is for: teams with many layouts they do not control, or a long tail of one-off sources, where selector upkeep costs more than the calls. Pages you own, feeds and huge volumes of near-identical pages still suit a classic parser.
- —What it costs: one AI extraction call is 5 points in NeuralVerge, the same over REST and MCP. Details are on the AI extraction product page and the pricing page.
- —The catch: a misread page can return a well-formed, plausible, wrong answer, so the checks around the extractor matter most.
What changes when you describe the data
A classic parser rule describes location: find a stable-looking anchor, reach from it to the value, return whatever sits there. CSS selectors, XPath, regular expressions and libraries like BeautifulSoup differ in syntax but share that contract. AI extraction inverts it: you say what you want, and the system works out where it is. In NeuralVerge's AI extraction product the page is rendered the way a browser shows it, boilerplate such as navigation and ads is stripped, and the remaining content is mapped to your fields and returned as typed JSON.
The trade in one paragraph
A parser rule is deterministic: the same HTML through the same rule gives the same output anywhere, so a unit test with a saved page means what it says. It is also brittle, because the rule knows nothing about what the value means. If a redesign moves the price, it returns nothing or the neighbouring element, and cannot notice that "Contact us" is not a price. For stable pages that trade is hard to beat; the pain comes from unstable pages and from the number of layouts you eventually cover. With extraction:
- —The unit of work is a field with a meaning, not a path to a node. "Founding year" survives a redesign that moves it, because nothing in your request mentioned the position.
- —The same request works across different layouts, and the output is typed JSON with no HTML in your code.
- —The behaviour is probabilistic at the edges: a model reads the page, so the determinism you got for free becomes something you build on purpose.
Before and after: the same task in code
Take a simple job: pull a company's name, founding year and headcount range from a profile page. First the parser version. The code below is illustrative, written for a made-up page layout, not taken from a real site.
# Illustrative parser: tied to one imaginary page's markup
from bs4 import BeautifulSoup
def parse_profile(html: str) -> dict:
soup = BeautifulSoup(html, "html.parser")
name = soup.select_one("div.company-header h1.title").get_text(strip=True)
facts = {}
for row in soup.select("table.facts tr"):
label = row.select_one("th").get_text(strip=True).lower()
value = row.select_one("td").get_text(strip=True)
facts[label] = value
return {
"company_name": name,
"founded_year": facts.get("founded"),
"employee_range": facts.get("employees"),
}
Every line of that function is a bet about markup: the heading class, the table class, the label text "founded". A second source that says "Established", or renders facts as a definition list, means a second function, and the whole thing assumes the HTML already contains the content, which fails once facts are rendered with JavaScript.
Now the schema-driven version, sent to the run-extract endpoint. The request carries the URL, a plain-language instruction and, optionally, a JSON Schema passed as a string in settings.extract_schema_json. The sketch below is illustrative too: the URL is fictional and the key is a placeholder. The full reference is at docs.neuralverge.ai.
# Illustrative request: describe the data, not the markup
import json, requests
SCHEMA = {
"type": "object",
"properties": {
"company_name": {"type": "string"},
"founded_year": {"type": "string"},
"employee_range": {"type": "string"},
},
"required": ["company_name"],
}
resp = requests.post(
"https://api.neuralverge.ai/functions/v1/run-extract",
headers={"Authorization": "Bearer <API_KEY>"},
json={
"url": "https://example.com/company/acme-oy",
"instructions": "Extract the company name, founding year and employee range.",
"settings": {"country_code": "fi", "extract_schema_json": json.dumps(SCHEMA)},
},
)
result = resp.json()["machine"]
There is no selector, no label mapping and no HTML here. A layout change does not touch this code, a second source uses the same call, and rendering happens on the extraction side, so client-side content needs no headless browser in your stack.
What you lose, and how to get it back
What you lose:
- —Bit-exact reproducibility of values. Two runs over the same page may word a free-text value differently. A year is far more likely to be stable, but check rather than assume.
- —A mapping you can read. With a parser you can point at line 14 and say why a value came out as it did. With extraction, debugging starts from the page and the request, not from a rule.
- —Failure that announces itself. A selector that matches nothing throws an error or returns
None. An extractor that misreads a page can return a well-formed, plausible, wrong answer. This is the change that most deserves your attention.
What you keep: the output shape (the schema decides keys and types), the ability to say "not found" as an empty field, and testability, if you build the test set. The certainty moves from the mechanism to the checks around it, in three layers.
Schemas pin the shape
A schema fixes property names, types and which fields are required. It removes a whole class of surprises: renamed keys, numbers that arrive as prose, fields that appear on some pages and vanish on others. It replaces the dictionary your parser returned, declared once and applied to every source; the schema-based extraction piece covers the mechanics.
A schema does not make the values right; it makes the shape checkable.
Validation checks the values
Treat every extraction result as untrusted input, like a form submission. Shape validation is the easy part; the more valuable checks encode what you know about the domain:
- —Format and range rules. A founding year should look like a year and not be in the future; a country code should belong to a known set; a headcount range should be ordered.
- —Required-field failures. A required field that comes back empty is a routing decision: review, retry, or mark incomplete.
These checks are ordinary, deterministic and cheap, and they do not care how the value was produced: the extractor supplies flexibility, the validator supplies the guarantee. Keep provenance (URL, time, schema version) with each record.
Evals catch drift over time
Validation checks one record; evals check the system. An eval is a fixed set of pages with hand-verified values, run on a schedule and whenever a schema or instruction changes. It answers the question your old unit tests answered: did anything change that should not have?
A workable eval set is smaller than people expect: a few dozen pages covering your templates, including missing and ambiguous fields and recent layout changes, with values you would accept. Track correctness per field and stability across repeated runs. If a field wanders between runs, tighten its description or treat it as advisory. We quote no accuracy percentages, because any number would be about somebody else's pages; the eval is how you find your own.
A worked example: one profile, two pipelines
Take the fictional Acme Oy (Finland). The directory that lists it ships a redesign and the facts table becomes cards. In the parser pipeline the selector returns None, the employee field is written as null, and someone reverse-engineers the new structure and re-runs the backlog. In the extraction pipeline the request is unchanged and the record passes validation. If the redesign also removes the headcount, both pipelines return an empty value, and your schema decides what happens: employee_range is not required here, so the record is accepted with a gap; if it were required, the validator would route it to review.
The subtler case is a quiet failure: the page shows a group headcount and a Finnish-entity headcount, and the extractor picks the wrong one. No selector broke and no schema was violated; only a cross-check or an eval built around that ambiguity catches it.
REST and MCP
The REST call above suits a scheduled job. If an AI agent should decide mid-task that it needs fields from a page, the same extraction is available as the run_extract tool on NeuralVerge's hosted MCP server at https://mcp.neuralverge.ai/mcp. Add that one URL to Claude, ChatGPT, Cursor, VS Code, Gemini CLI or any MCP client, sign in through the browser and click Allow. There is no server of your own to build or host. The MCP server for Claude walkthrough covers a full connection.
Here is the request from the AI extraction product page, over REST:
curl https://api.neuralverge.ai/functions/v1/run-extract \
-H "Authorization: Bearer <API_KEY>" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/company/acme",
"instructions": "Extract the company name, founding year and employee range.",
"settings": { "country_code": "us" }
}'
And the sample response shown there. It is an example payload for the fictional Acme, not a capture from a live site:
{
"session_id": "11111111-1111-1111-1111-111111111111",
"kind": "extract",
"url": "https://example.com/company/acme",
"machine": {
"company_name": "Acme",
"founded": "2019",
"employees": "50-100"
}
}
The machine object holds your fields; with a schema, its keys follow the schema. Over MCP the agent sees run_extract next to Search and the catalog's data-source tools, with arguments that mirror the REST body (url, instructions, settings). The response has the same shape and the price is the same, with no separate MCP rate. The tool is run_extract with an underscore; the REST path keeps the hyphen. The validation and eval layers apply unchanged: an agent's output is as untrusted as a script's. Pick REST for pipelines that process a known list on a schedule, and MCP when an agent chooses pages as it works. Endpoint and parameter details change over time, so confirm them in the documentation before you build.
Cost and limits
Per-page work goes up. A parser on your own machine does almost nothing per page once written. An extraction call renders the page, cleans it and runs a model, and in NeuralVerge costs 5 points. A call that finds nothing is still billed like any other, so do not treat misses as free. Compare total cost of ownership: count the engineering hours spent writing and repairing selectors across all your templates, not just the per-request bill. The pricing page has current point numbers.
Latency goes up. Rendering and model inference take longer than a selector over HTML you already hold. That is irrelevant for an overnight refresh and may matter where a user is waiting; fetch ahead, cache, or keep a fast path for pages you can parse cheaply.
Cost does not scale with your template count. A parser pipeline's cost grows with the number of distinct layouts, because each layout is code you own; an extraction pipeline's cost grows with the number of pages. Few templates and huge volume favour a parser. Many templates, moderate volume and layouts you do not control favour extraction. Measure your current maintenance time and set it against a sample extraction run on your own pages.
What it does not do. Extraction reads a public page or document at a URL. It is not a crawler, and it does not log in or click through a flow.
What still needs a classic parser
- —Structured feeds you already have. If a source publishes JSON, CSV, an API, a sitemap or RSS, use that directly; sending structured data through a model to get structured data back is a step backwards.
- —Pages you own. If your own application generates the markup, read from the source of truth behind it.
- —Fixed formats at very high volume, where a tested parser wins on price and speed, and bit-exact requirements such as audit reproducibility, where the model stays out of the path.
- —Pre-filtering and routing. Parsers are cheap at deciding which pages are worth an extraction call.
For a company profile, the usual mixed design is to pull whatever a registry or feed offers directly and use extraction for the unstructured remainder. NeuralVerge's ready-made data sources, such as corporate registries, come with documented output schemas, so for those you need neither a parser nor an extraction schema. AI research helps when you do not yet know which page holds the answer; extraction is the right tool once you do.
A migration path that does not bet the pipeline
- —Inventory your parsers by pain. Note how often each broke in the last year and how long repairs took. Frequent breakers covering many templates are the candidates.
- —Write the schema from the parser's output. Your function already defines the fields and types you rely on. Translate that into a schema, mark required fields deliberately, and add the validators your code did implicitly.
- —Run in shadow mode. Send the same pages down both paths, keep only the parser's output in production, and log every disagreement.
- —Turn disagreements into evals. The pages where the paths differed go into your eval set, labelled by hand once.
- —Cut over one page type at a time. Keep the parser as a fallback until the new path has survived a real change on the source side.
How to decide
- —Do you control the markup? If yes, read from the source of truth and skip both parsing and extraction.
- —How many layouts do you cover, and how often do they change? Many, changing often, favours extraction; a few stable ones favour a parser.
- —Can you require fields and get an explicit empty value? You need "not found", not a plausible fill.
- —Is the shape identical across sources? If you still need per-source cleanup, most of the gain is gone.
- —Can you run your own eval set before committing? Any provider that discourages this is telling you something.
- —Does an agent or a schedule pick the pages? Agents point to MCP; known lists on a schedule point to REST.
Frequently asked questions
Will AI extraction give me exactly the same output every time?
Not byte for byte, and it is better to plan for that than to assume it. What you can hold constant is the shape: a schema fixes field names, types and required fields, and a validation layer rejects anything that does not conform. Values on the same page should be stable when the page is stable, but you should verify that against your own pages with a small test set rather than take it on trust.
When should I keep using a classic parser?
Keep it for pages you own or that follow a fixed, documented format, for feeds and sitemaps that are already structured, for very high volumes of near-identical pages where per-page cost dominates, and for anything where you need bit-exact, fully reproducible output. A parser is also the right tool for cheap pre-filtering, such as deciding which pages are worth sending for extraction.
What happens when the page does not contain the field I asked for?
A well-behaved extractor returns the field as empty rather than inventing a plausible value. If the field is marked required in your schema, that empty result becomes an explicit signal that the record needs review, instead of a quiet gap that surfaces later in a report.
Is AI extraction more expensive than a parser?
Per page, yes: a parser you run yourself costs almost nothing once written, while an extraction call renders the page and runs a model, and in NeuralVerge each call costs 5 points. Per template, it can go the other way, because you stop paying engineers to write and repair selectors for every layout. Compare your maintenance hours against a sample run on your own pages.
Can I run AI extraction over MCP instead of calling an API from code?
Yes. Add NeuralVerge's hosted MCP server URL to a client such as Claude, ChatGPT or Cursor, sign in through the browser and click Allow, and extraction appears as the run_extract tool. It takes the same inputs and returns the same response shape as the REST endpoint, at the same price, so you can prototype with an agent and move the same call into a scheduled job later.
Related articles
An AI extraction API turns a page or PDF URL plus plain-language instructions into JSON, no selectors. How it works, 5 points per call, REST or MCP, and limits.
A structured data extraction llm call turns competitor pricing pages into JSON you can diff. Build the price-monitoring agent over API or MCP, step by step.
Extract company data from website API or MCP calls: instructions, JSON schemas, missing fields and PDFs, step by step. 5 points per call, REST or MCP.
About NeuralVerge
Give your agents structured, cited, real-world data from 300+ sources and the open web — through one API or MCP server.
AI Extraction on the NeuralVerge blog.