AI Extraction Explained: URL & Document to JSON Without Selectors

If you've ever maintained a scraper against a real website, you know the failure mode: it works fine until the site redesigns a page, and then every CSS selector your code depends on points at nothing. You're not fixing a bug at that point — you're rewriting the scraper. An AI extraction API is built to remove that failure mode entirely: instead of telling it where on the page to look, you tell it what you want, and it reads the page — or the document — the way a person would.

This guide covers what's actually happening when a URL or a document goes in and structured JSON comes out — rendering, cleaning, and field mapping — a worked example for each source type, and how to decide whether reaching for one makes sense for what you're building.

What is an AI extraction API?

An AI extraction API takes a URL or a document and returns its content as structured, typed JSON — the fields you asked for, cleaned of everything else on the page. It's easiest to place by contrasting it with what people usually reach for first.

AI extraction vs. selector-based scraping

A traditional scraper is a set of CSS or XPath selectors written against one specific page layout. It's fast and cheap to run once it's written, but it's coupled to that exact layout — a redesign, an A/B test, or even a class-name change can silently break it, and it only breaks loudly if you're lucky enough to have monitoring that catches empty fields rather than wrong ones.

AI extraction reads the page's actual content and maps it to the fields you asked for by meaning, not by position in the DOM. A layout change that a human reader would barely notice doesn't require you to touch the extraction call at all, because nothing about the request depended on where the company name happened to sit in the markup.

AI extraction vs. asking an LLM to "read this page"

Pasting raw HTML into an LLM prompt and asking for the interesting bits works as a one-off, but it's not the same as a production extraction pipeline: there's no guaranteed output shape, no handling for pages that need to be rendered before the content actually exists, and no consistent behavior for missing fields — the model might return a paragraph instead of JSON, truncate on a long page, or fill in a plausible-sounding guess where a field simply doesn't exist on the page.

AI extraction wraps that same underlying capability with the parts that make it usable in a pipeline: rendering, a defined or inferred schema, and a typed response every time, regardless of the source page's structure or how the request happened to be worded.

AI extraction vs. a PDF/OCR parser

Document parsers are built to pull raw text out of a file format — they're good at that and not much else. They don't know that the text three lines down is a "founding year" versus a "headcount"; they just return text and its position on the page. AI extraction takes that a step further by mapping the content to the semantic fields you actually asked for, whether the source was a rendered web page or a document, which is why the same call handles a company's "About" page and a company's PDF fact sheet without switching tools.

AI extraction vs. browser automation and RPA tools

Browser automation tools (the kind used for testing or task automation) are built to act on a page — click, type, navigate — and usually pull data out as a side effect using the same brittle selectors a scraper would. They're the right tool when the job is interacting with a page. AI extraction is narrower and more reliable at the one job it does: reading a page or document that's already in front of it and returning its content as structured data, without needing to simulate a user session to get there.

How AI extraction works, step by step

1. Render — loading the page the way a browser would

Static HTML fetching misses anything that loads client-side, which is most of the modern web — a price that only appears after an API call finishes, a description that streams in after the initial page load, a tab that has to be clicked to reveal its content. The page is rendered the same way a browser would show it, so content that only appears after JavaScript runs is actually present by the time extraction happens. For a document instead of a URL, this step is replaced by reading the document's actual content rather than rendering — a PDF or a similar file doesn't need a browser, just correct parsing of its layout into readable text.

2. Clean — stripping everything that isn't content

Every real page carries navigation, ads, cookie banners, and boilerplate around the content that actually matters. This gets stripped automatically before extraction runs, so the fields you get back reflect the page's substance, not its chrome. Documents get an equivalent treatment — headers, footers, and repeated boilerplate that appear on every page of a multi-page PDF don't leak into every extracted field just because they were physically present on the page.

3. Map — turning content into the fields you asked for

This is the step that replaces a selector. Instead of "grab the text inside .company-name," you get a mapping from meaning to field: whatever on the page reads as the company's name becomes the name field, regardless of what element it happens to sit in or how the underlying markup is structured. You can define the exact fields you want up front, or describe what you're after in plain language and let a sensible structure get inferred from what's actually present.

4. Return — structured JSON, not raw HTML

The output is typed JSON — never HTML, scripts, or leftover markup — so it drops directly into a database, a RAG index, or an agent's next tool call without an intermediate parsing step of your own. A missing field comes back empty rather than papered over, which is the detail that tells you whether the extraction step is reading the page or approximating it.

A worked example: extracting a company profile page

Take a concrete case: pulling a structured company profile from a company's page at https://example.com/company/acme — a fictional example.

1. —Render loads the page as rendered, including anything added by client-side scripts (a dynamically loaded "About" section, for instance).
2. —Clean strips the site's navigation bar, footer, cookie notice, and any promotional banners, leaving the actual profile content.
3. —Map takes the remaining content and fills in the requested fields — name: "Acme Inc.", founded: "2019", employees: "50-100" — pulling each from wherever it actually appears on the page, whether that's a hero section, a sidebar, or a table further down.
4. —Return hands back a single JSON object with those fields, with no HTML or boilerplate mixed in.

If the page doesn't list a founding year at all, a well-built extractor returns that field empty rather than inventing a plausible one — which is the detail worth checking for when you're evaluating one (see the checklist below).

A worked example: extracting from a document instead of a URL

The same request shape works when the source is a document rather than a page — say, a PDF one-pager for the same fictional company.

1. —The document's content is read in full, including multi-column layouts and any tables, rather than treated as one undifferentiated block of text.
2. —Repeated boilerplate — a header logo on every page, a footer disclaimer — is recognized as noise rather than as repeated "content."
3. —The same requested fields (name, founded, employees) get mapped from wherever they appear in the document's actual layout, whether that's a header block or a details table on page two.
4. —The response comes back in the same shape as the URL example above — your integration code doesn't need a separate code path depending on whether the source was a web page or a file.

That consistency is the practical payoff of "URL and document" being one API rather than two: a pipeline that ingests both company websites and company fact sheets doesn't need to know, at the call site, which kind of source it's looking at.

Schema-defined vs. inferred extraction

There are two ways to tell an extractor what you want, and they suit different situations. Defining an explicit schema up front is worth it when you need the exact same shape back every time — a fixed pipeline that expects name, founded, and employees on every call, run against thousands of pages, where a missing field needs to be a predictable null rather than an absent key. Describing the fields in plain language and letting a structure get inferred is faster to get started with and works well when you're exploring what a class of pages actually contains before you commit to a fixed schema, or when the pages you're pointing it at vary enough in what they contain that a single rigid schema would leave most fields empty most of the time. Both return typed JSON either way — the difference is only in whether you or the model decides the shape, and it's a decision worth revisiting once you've seen a sample of real pages rather than guessing up front.

What you get back

The response is built to be consumed by code, not read as a document:

• —Fields you explicitly defined, or a sensible structure the model inferred from the page's or document's content.
• —No HTML, scripts, or boilerplate leaking into the output — every field is clean text or a properly typed value (string, array, number).
• —A shape that's ready to drop into a database, a RAG index, or an agent's next tool call without an intermediate cleanup pass of your own.

That consistency matters more as volume grows — a one-off extraction can tolerate a slightly awkward response shape you clean up by hand, but a pipeline running the same call against thousands of sources needs the shape to hold reliably every time, including on the pages that don't quite match what you expected.

Calling it directly vs. exposing it as a tool call

Calling extraction directly from a pipeline makes sense when you already know which pages need processing — indexing a fixed set of company profile pages on a schedule, or pulling structured fields out of every document in an intake queue, for example. You control exactly when the call happens and what happens to the result.

Exposing it as a tool an agent can call mid-task makes sense when the agent encounters URLs or documents dynamically and needs to decide for itself, in the middle of a task, that it needs structured data from a source rather than raw content. The agent reasons about what it's looking at, recognizes it would benefit from a typed field instead of a page of prose, and calls the tool instead of trying to parse the content itself. The extraction step doesn't change between the two — only how it gets invoked.

Where teams use it

• —Competitor and market monitoring — pulling comparable fields from a set of competitor pages on a schedule, without a selector to maintain per competitor and without a rewrite every time one of them redesigns.
• —Lead and account enrichment — turning a company's own website into structured firmographic fields alongside other enrichment sources, rather than reading each site by hand.
• —RAG pipeline ingestion — feeding clean, structured content into a retrieval index instead of raw, boilerplate-heavy HTML that would otherwise pollute retrieval results with navigation text.
• —Document intake — turning incoming PDFs or similar files — invoices, one-pagers, filings — into structured fields a downstream system can act on directly.
• —Agent tool calls — giving an agent a way to turn any URL or document it encounters mid-task into structured data it can reason over directly, instead of a wall of raw text.

What to check before you commit to an AI extraction API

• —Does it render pages, or just fetch static HTML? If it can't run JavaScript, anything that loads client-side simply won't be in the output.
• —What happens to fields that aren't on the page? Returned empty is honest; a plausible-looking guess in every field regardless of the actual page content is a sign of inference standing in for reading.
• —Can you define an exact schema when you need one? If every response's shape is entirely up to the model with no way to pin it down, that's a problem for a pipeline expecting consistent fields at scale.
• —Does the same call work for URLs and documents? If extraction logic differs meaningfully between a web page and a PDF, you're maintaining two mental models and likely two code paths instead of one.
• —How does it behave on a page you already know well? Point it at a page whose content you can check by eye first — that's the fastest way to catch whether it's actually reading the page or approximating it before you commit to it at volume.

Running one real page and one real document through a candidate API is the fastest way to see whether it's actually reading the content or approximating it. For a deeper look at how a research-oriented pipeline handles the closely related problem of turning many sources into a verified answer rather than a single structured record, see what an AI deep research API does.

Frequently asked questions

Does AI extraction replace a traditional scraper entirely?

For most teams, yes, for the pages it targets — you stop maintaining selectors for those pages. It's still one HTTP-plus-render call per page, so very high-volume crawling of millions of pages a day is a different problem with different cost tradeoffs than extracting the specific pages a workflow actually needs.

What happens if a page doesn't have the field I asked for?

A well-behaved extractor returns the field as empty or null rather than guessing a plausible-sounding value. If a tool always fills in a complete-looking answer for every field regardless of what's actually on the page, that's a sign it's inferring rather than reading.

Can AI extraction handle JavaScript-heavy pages?

Yes — the page needs to be rendered the way a browser renders it, not fetched as raw HTML, or content that loads client-side simply won't be there to extract.

Do I need to define a schema before extracting anything?

No. You can describe the fields you want in plain language and let the model infer a reasonable structure, or define an explicit schema up front if you need the same exact shape every time. Both return typed JSON either way.

Does the same API handle both web pages and documents like PDFs?

Yes — a URL and a document go through the same field-mapping step once their content has been read, so you're not maintaining two separate mental models depending on the source format.