Extract Structured Data From News Articles at Scale
How to extract structured data from news articles over API or MCP: an event schema, dedup, paywalls and provenance, at a flat 5 points per page.
By the NeuralVerge team · 12 min read · Updated October 2026
▸On this page
To extract structured data from news articles, give an extraction call the article URL and a schema, and store what comes back as event records instead of text. NeuralVerge's AI Extract does the reading step over REST or as an MCP tool for agents. One article is easy; tens of thousands from outlets with different layouts, covering the same events repeatedly, is where schema, deduplication and provenance decisions start to matter. For general PDFs and filings, see the PDF guide.
The short version
- —The answer: one extraction call per article, with a JSON Schema for events, then separate steps for validation, deduplication and entity matching.
- —Who it is for: teams that monitor coverage of companies for sales, due diligence, competitive intelligence or research, and want rows they can filter instead of articles they have to read.
- —What it costs: AI Extract is a flat 5 points per call, whatever the page size. A URL that does not load still returned null and was billed 5 points in a live test.
- —What it will not do: read past a paywall, decide whether an article is true, or find the articles for you.
Why news needs its own schema
Every article is written for a human, not for a database. Four things make the step to a row harder than it looks.
The same event appears many times. A funding round or leadership change is usually reported by several outlets, then followed up. One row per article over-counts events.
Dates are slippery. An article has a publication date, and the event it describes has its own. Phrases like "last week" only make sense relative to the first.
Entities are ambiguous. A company appears as its legal name, a short name, a brand or "the Finnish group".
Not everything is the article's own claim. Stories quote spokespeople, cite anonymous sources and repeat rumours. A flat "acquirer" field cannot say whether the article states something as fact or reports that someone said it.
Versus the alternatives
A selector-based scraper finds fields by position in the markup and breaks on the next redesign; extraction that works from meaning applies one schema to every outlet. A full-text dump is a fine archive, but you cannot ask a pile of articles for every funding announcement last month and get a table back. A research pipeline starts from a question and decides which sources to read; extraction reads a page you already have and does not check whether it is right. AI research and extraction are complementary.
Designing the schema
The schema decides what you can query later, and changing it after you have stored many records means re-running extraction. A reasonable starting point, with Acme Oy (Finland) as the illustrative subject, sketched as a field outline (the request below shows how it becomes a JSON Schema):
{
"headline": "string",
"publication_date": "string, ISO 8601",
"outlet": "string",
"is_partial": "boolean, true if only a teaser or truncated text was visible",
"events": [
{
"event_type": "funding | acquisition | leadership_change | product_launch | layoff | partnership | legal | other",
"event_date": "string, ISO 8601, or null if not stated",
"event_date_precision": "day | month | quarter | year | unknown",
"summary": "string, one sentence",
"organizations": [{ "name": "string", "role": "subject | acquirer | target | investor | partner | other" }],
"amount": { "value": "number or null", "currency": "string or null" },
"attribution": "stated_as_fact | attributed_to_source | rumour",
"evidence_quote": "string, the sentence that supports this event"
}
]
}
Each choice prevents a specific problem.
- —Events as a list. One release can announce a funding round and a new finance chief. A single event field forces the extractor to drop one.
- —Two dates and a precision field. Separating publication date from event date resolves the ambiguity above. When an article says "in the third quarter", a fabricated exact date looks tidy and is wrong. A null with precision
unknownis honest and easy to filter. - —Roles, not a flat list of names. Knowing which company is the acquirer and which the target tells you what happened.
- —Attribution. This does not tell you whether a claim is true. It tells you how the article presented it, which the article really does contain.
- —An evidence quote per event. It is the seed of your provenance trail. If the quote does not contain the entities and event type in the record, the record is suspect.
- —Enums where you group. Fixed values make counting possible. Include
otherso unusual events are not forced into the wrong bucket.
How to extract structured data from news articles
The workflow has six steps. Only the second is the extraction call; the rest are what make the output usable at scale.
1. Find the URLs
Discovery is whatever mechanism you already use to find candidate articles: feeds, alerts, a search step or a watchlist. Extraction does not do it. Normalize each URL (strip tracking parameters and fragments, lower-case the host, resolve obvious redirects) and hash it, so the same page arriving through two feeds is one job.
2. Extract each article
Each article is an independent request, which is what lets you parallelize, retry and resume without coordination. Use a queue, not a loop, with each job holding a URL, a schema version, a status and an attempt count, so a page that times out fails one job, not the run. Key results by URL and schema version: re-running replaces instead of duplicating, and a schema change is a bump and a re-queue. The request is in the next section.
3. Validate before storing
Valid JSON is not a good record. Check required fields, enum values, parseable dates, numeric amounts and a non-empty quote. Failures go to a review queue, not the main table. Retry transient failures a few times, then park the job with its reason.
4. Deduplicate: one event, many articles
Separate two concepts: the article, a fetched document with its own URL and provenance, and the event, a real-world occurrence many articles may describe. Deduplication links the first to the second, in layers from cheap and exact to fuzzy.
- —Identical pages. The URL hash from step 1 catches these.
- —Near-identical text. Wire copy appears on many sites with only the headline changed. Hash the cleaned text to detect it. Do not delete the copies: link them to one cluster and mark them syndicated, because independent reporting versus republishing is meaningful.
- —Same event, different words. Two outlets write their own stories on one announcement. Here the extracted fields do the work: two records are candidates when they share event type, overlapping organizations in the same roles, and close event dates. Score the overlap, set a threshold and send borderline pairs to review.
Pick a canonical record, perhaps the earliest article or the most complete one, but keep links to the rest. Disagreements such as different amounts stay visible. Fuzzy matching will sometimes merge distinct events or split one, so bias towards splitting and log every merge so it can be undone.
5. Resolve entities
Extraction gives you names as written. Analysis needs identities. Treat resolution as its own step: store the extracted name exactly, then match it against a reference list using normalization (legal suffixes, punctuation, case) plus country and industry context from the same record. Link confident matches automatically and queue the rest. Registry records are better keys than name strings, and the catalog includes dedicated corporate registry sources for that lookup.
6. Keep provenance
For each stored record, keep the source URL and outlet, the publication date as read from the article, the fetch timestamp, the schema version, the evidence quote and attribution for each event, and status flags such as partial content or manual overrides. Never overwrite extracted values in place: store human corrections as a separate layer with the person and time.
Provenance also marks the limit of what extraction can claim. The quote proves the article said something, not that it was right. Where a claim matters, corroborate it: give it to a research step as a question and it checks other sources and returns a cited answer.
A worked example
Suppose your watchlist includes Acme Oy (Finland), and three articles appear on a Tuesday. After URL normalization, extraction runs per URL. The business daily yields a funding event with an amount and a quote. The regional outlet yields the same type with no amount and a month-level date. The paywalled piece yields a headline and date, is_partial true, and no events. Validation passes the first two and flags the third rather than rejecting it. Deduplication groups the first two by event type, subject and date, and entity resolution links "Acme" and "Acme Oy" to the watchlist entry. Only one article states the amount, so the cluster records it with a note that one source supports it, and a research question about recent funding for Acme Oy is raised for a reviewer to compare with the extracted record.
REST and MCP
With NeuralVerge's AI extraction, one article against a schema is a single run-extract request: the article URL, a plain-language instruction, and the schema passed as a JSON Schema string in settings.extract_schema_json. The example below is illustrative (the URL is a placeholder) and uses a trimmed version of the schema above:
import json, requests
SCHEMA = {
"type": "object",
"properties": {
"headline": {"type": "string"},
"publication_date": {"type": "string"},
"events": {
"type": "array",
"items": {
"type": "object",
"properties": {
"event_type": {"type": "string"},
"event_date": {"type": ["string", "null"]},
"summary": {"type": "string"},
"organizations": {"type": "array", "items": {"type": "object"}},
"evidence_quote": {"type": "string"},
},
},
},
},
"required": ["headline"],
}
resp = requests.post(
"https://api.neuralverge.ai/functions/v1/run-extract",
headers={"Authorization": "Bearer <API_KEY>"},
json={
"url": "https://news.example.com/2026/09/acme-oy-raises-round",
"instructions": "Extract the headline, publication date and every business event the article reports, with the organizations involved and a supporting quote.",
"settings": {"country_code": "fi", "extract_schema_json": json.dumps(SCHEMA)},
},
)
record = resp.json()["machine"]
The extracted fields come back under machine. Pages are rendered before extraction, and a press release published as a PDF goes through the same call.
The same step is an MCP tool. NeuralVerge hosts the MCP server at https://mcp.neuralverge.ai/mcp. You add that one URL to an MCP client such as Claude, ChatGPT, Cursor, VS Code or Gemini CLI, sign in through the browser and click Allow. There is no key to paste into a config file and nothing to host. The Claude walkthrough shows the steps, and the setup guides for other clients are at docs.neuralverge.ai. The agent sees extraction as the tool run_extract. REST paths keep hyphens (run-extract); tool names use underscores. The arguments are the fields of the REST request:
{
"tool": "run_extract",
"arguments": {
"url": "https://news.example.com/2026/09/acme-oy-raises-round",
"instructions": "Extract the headline, publication date and every business event the article reports, with the organizations involved and a supporting quote.",
"settings": {
"country_code": "fi",
"extract_schema_json": "{ \"type\": \"object\" }"
}
}
}
The response has the same shape as the REST endpoint and the price is the same, so choosing MCP is an integration decision, not a cost decision. A practical split: use REST for the queue-driven batch above, where your code already knows which URL to read and when. Use MCP when an agent is working through a question, finds an article that looks relevant and needs fields before it carries on. Research is the one piece that stays on REST, so the corroboration step in the worked example is a REST call.
Cost and limits
Extraction is a flat 5 points per call regardless of page size, so a backfill of 1,000 articles is 5,000 points. See pricing for the current action costs.
Failed fetches are billed. In a live test, AI Extract on a URL that did not load returned null and still cost 5 points. Do not assume dead links are free: check that addresses load before a large run, and count null results as billed.
Paywalls. A public request generally sees a teaser. Extraction can only work with what is visible, so it can often recover headline, date and outlet, but not the body. Set is_partial when the text is clearly truncated, and make null an explicit allowed answer in your schema, so an amount that is not on the visible page comes back empty rather than guessed. If you hold a licence for the full text, supply it yourself through your own authorized access. Circumventing a paywall is not something to build into a pipeline.
Script-heavy pages and blocks. Rendering handles pages that load body text with JavaScript, but consent walls, interaction-gated content and region-specific versions can still get in the way, and some publishers block automated requests. Record the reason against the job. If an outlet fails consistently, that is a coverage gap to report as unknown, not a quiet news day.
Concurrency. Let your store's write speed and the documented rate limits set it, and spread requests across outlets rather than draining one publisher at a time.
How to decide
Test with a sample of real articles before you commit to a pipeline.
- —Does one schema hold across outlets? Test at least ten publishers with different layouts.
- —Can it return null instead of guessing? Give it a teaser-only page and ask for the amount.
- —Can you get a supporting quote per event? Without it, review is slow.
- —Can you tell failures apart? Blocks, timeouts and paywalls should look different in the response.
- —Can you re-run cleanly after a schema change? Keyed by URL and version, it is a queue operation.
- —Is there a use for it? Teams run this for account monitoring (funding, leadership changes and launches routed to account owners), due diligence (a dated timeline of coverage with the quote behind each entry), competitive intelligence (launches clustered so one counts once) and analyst research (a stream of sector articles turned into a table to filter and chart).
Frequently asked questions
Should I extract one record per article or one record per event?
Both, in two layers. Extract per article first, because an article is the unit you fetch and the unit you can cite. Then group article-level event records into event clusters in a separate step. Extracting straight to events loses the link between a field and the article that supplied it, which is the link you need when someone asks where a fact came from.
How do I handle articles behind a paywall?
Extraction only works on what a public request can see. If a page returns a login wall or a truncated teaser, record that the article was partial and mark the affected fields as missing rather than let the model fill them in. If you have a licence for the full text, feed that text in yourself. Working around a paywall is not something to build into a pipeline.
How do I avoid storing the same story dozens of times?
Deduplicate in layers: hash normalized URLs for identical pages, compare extracted entities, event type and event date to catch the same story on different sites, and keep syndicated copies linked to one cluster instead of deleting them. The number of outlets carrying a story is a useful signal in its own right.
Can the extractor tell me when an article is wrong?
No. Extraction reports what the article says, not whether it is true. Store the publication date, the outlet and a quote for each field so a reviewer can judge the claim, and cross-check against other sources, for example with a research step, when a claim matters enough to verify.
What does it cost to run a batch of articles, and what if a URL is dead?
AI Extract is a flat 5 points per call, so 1,000 articles is 5,000 points. A URL that does not load returned null in a live test and was still billed 5 points, so check that addresses load before a large run. Check the documentation for current rate limits before you plan a big backfill.
Can I extract structured data from news articles over MCP, or only through the API?
Both. NeuralVerge hosts an MCP server at https://mcp.neuralverge.ai/mcp, and AI extraction is available there as the run_extract tool. It takes the same inputs and returns the same response shape as the REST endpoint, at the same price. Use REST for a scheduled batch pipeline and MCP when an agent decides mid-task which article to read.
Do I need to define a schema before I start?
You can let the extractor infer a structure while you explore, but for anything you store and query you should define the schema. Inferred structures vary between runs and articles, which makes comparison and deduplication harder. Start small, run it on a sample of real articles, and expand from what you see.
Related articles
A pdf to json api turns a public PDF into typed JSON. See a real IRS W-9 call over API or MCP, what it costs (5 points, empty results billed) and its limits.
A llamaparse alternative compared: file parsing to markdown versus one call that turns a document URL into schema-pinned JSON, over API or MCP, at 5 points.
An AI extraction API turns a page or PDF URL plus plain-language instructions into JSON, no selectors. How it works, 5 points per call, REST or MCP, and limits.
About NeuralVerge
Give your agents structured, cited, real-world data from 300+ sources and the open web — through one API or MCP server.
AI Extraction on the NeuralVerge blog.