How to Stop Rewriting Scrapers Every Time a Layout Changes

If you maintain scrapers, a web scraping API alternative is worth looking at when the same pages keep breaking your selectors after every redesign or A/B test. The fix is not a better selector. It is extraction that finds fields by what they mean instead of where they sit in the markup, so a moved element does not break the job. NeuralVerge's AI extraction works this way, over API or MCP.

The short version

• —The answer: a selector encodes position, so any layout change can break it. Extraction that maps content to fields by meaning does not depend on position, which removes the most common cause of routine breakage. It does not remove all maintenance.
• —Who it is for: teams whose pipelines read many external pages that change on someone else's schedule, and agents that meet URLs mid-task. For the build-or-buy question itself, see build vs buy web scraper.
• —What it costs: 5 points per AI Extract call, per the pricing page. A URL that did not load returned null and was still billed 5 points.
• —What you give up: control over exactly which DOM node a value comes from, and determinism. A model's reading of a page is something you check, not assume.

Why selectors break

A selector is a set of coordinates

A CSS or XPath selector says: the text inside the third div under this class, inside that section. It works because it encodes exactly where something sat in the markup when the selector was written. That is also its whole weakness. The selector has no idea what the content means, only where it happened to be.

A redesign does not need to remove the company name from the page to break a selector aimed at it. Moving the name from a hero block to a sidebar is enough. The selector was never tracking the company name. It was tracking a spot in the DOM that used to contain it.

Fallbacks and faster fixes do not change the cause

A more specific selector is still a set of coordinates, only a longer one. Fallback selectors reduce how often a change causes total failure, but each fallback is another path that can break on its own. The dependency on structure stays.

The same is true of process fixes: better monitoring, a faster on-call rotation, a runbook for the common failures. They shorten the time a scraper stays broken, and that is worth having. But the same class of incident keeps recurring. A team that has cut each fix to twenty minutes still pays those twenty minutes for the life of the scraper, and the yearly total rarely gets looked at the way a single incident does.

Silent failure is the expensive kind

A crash is loud. Someone sees the pipeline stop and fixes it. A selector that still runs but points at the wrong element, or at nothing, is quieter. It can return empty fields, or a plausible value from the wrong place, into a dashboard or database that nobody checks daily. That is the worst version of selector breakage, and it grows with the number of sites you depend on, because every site is an independent source of it.

The cost most teams do not count

A selector-based scraper looks free once written: no per-call fee, no vendor. The cost arrives later, scattered across small incidents instead of one line item. Someone notices empty fields, traces them to a layout change, rewrites the selectors, redeploys, and moves on. Repeated for as long as the target sites keep changing, that is real engineering time, and it never shows up in a budget the way an API bill does.

So the honest comparison is not a free scraper against an API bill. It is a free scraper plus recurring fix time against a per-call fee plus whatever checking you still do. How that lands depends on your numbers, and the build vs buy post gives a way to fill them in. For a scraper that rarely changes and is cheap to fix when it does, staying put is a fair answer.

A web scraping API alternative that reads by meaning

The alternative is not a smarter selector. It is not using position to find content at all. AI extraction renders the page, strips ads and navigation, and maps what remains to the fields you described. If the company name moves from a hero block to a sidebar, the call does not change, because nothing in it referred to where the name was, only to what it was.

This is a different failure mode, not a sturdier version of the same one. A selector fails when structure changes. Meaning-based mapping fails when the content is gone, renamed beyond recognition, or read wrongly. That is a narrower set of cases than "the markup shifted", which is most of what breaks scrapers day to day.

What the call needs from you

Moving away from selectors does not mean giving up control. You express it differently. Instead of saying where a field lives, you say what it is: a plain-language instructions string, optionally pinned with a JSON Schema in settings.extract_schema_json if every call must return the identical shape.

The call also renders JavaScript-heavy pages and can be pointed at public documents such as PDFs, which a scraper working from raw fetched HTML often handles badly. It is not a crawler, it does not log in or click through flows, and it reads pages you point it at. For many company and person records you can skip pages altogether, because the data sources catalog returns documented fields directly.

What you give up

A selector gives you, by construction, exact control over which DOM node a value comes from, and the same answer every time. Extraction by meaning gives that up. For most field-pulling jobs the precision was never the goal. The value was, and the DOM position was only a way to reach it. But there are jobs where position is the point, such as capturing a page's literal HTML structure, and for those a selector remains the right tool.

Because a model reads the page, the output needs checking the way any model output does. A schema fixes the keys that come back, not whether the values are right. Validate what arrives, and spot-check a sample against the live pages.

A worked example: the same redesign, two outcomes

Take an illustrative case: a scraper pulling plan_name, price and included_limits from ten competitor pricing pages on a weekly schedule.

With selectors, one competitor redesigns its pricing page in month three, a routine update not aimed at anyone's scraper. The next run returns empty fields for that competitor, quietly, because the selectors still execute without error. They just no longer point at anything. Someone eventually notices the gap, traces it to the redesign and rewrites the selectors. Until then the monitoring is broken for that competitor.

With meaning-based extraction, the same redesign usually does not interrupt the run. The call still looks for the plan name and the price, wherever they now sit. The data for that competitor stays continuous, and nobody has to notice or fix anything. The word "usually" matters: if the redesign removed the price from the public page, no approach will find it, and you would still want a check that flags an empty field.

REST and MCP

The same extraction is reachable two ways, with the same inputs, the same response shape and the same cost. Over REST you POST to the run-extract endpoint. Over MCP, the hosted server at https://mcp.neuralverge.ai/mcp exposes it as the run_extract tool. Tool names use underscores where REST paths use hyphens. A request carries a URL and plain-language instructions, with no selectors anywhere:

curl https://api.neuralverge.ai/functions/v1/run-extract \
  -H "Authorization: Bearer <API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/company/acme",
    "instructions": "Extract the company name, founding year and employee range.",
    "settings": { "country_code": "us" }
  }'

Over MCP the same call is the run_extract tool with url and instructions arguments:

{
  "tool": "run_extract",
  "arguments": {
    "url": "https://example.com/company/acme",
    "instructions": "Extract the company name, founding year and employee range."
  }
}

The response is JSON, not markup. This is the example response from the product page, for Acme, a fictional company, when a schema is passed:

{
  "session_id": "11111111-1111-1111-1111-111111111111",
  "kind": "extract",
  "url": "https://example.com/company/acme",
  "machine": {
    "company_name": "Acme",
    "founded": "2019",
    "employees": "50-100"
  }
}

One detail comes from calls we ran. When you pass a schema, the typed result is in machine. When you pass no schema, as in the requests above, the result comes back in the human field as JSON text and machine is null. Your code should read whichever field matches how you called it.

Pick REST when your own code already knows when a page should be read: a weekly pricing monitor, a nightly refresh. Pick MCP when an agent decides mid-task that a particular page should become fields, and which fields. An agent has a sharper version of the scraper problem, because it usually cannot tell that a selector has gone stale. When it states what it wants instead of where it sits, that problem does not arise. Both routes run the same extraction underneath, so the resilience described above applies either way. Setup for Claude, ChatGPT, Cursor and other clients is in MCP server for Claude, with client guides at docs.neuralverge.ai.

Cost and limits

An AI Extract call costs 5 points, flat per call, the same as a Search call. Points come from your plan's monthly allowance, and the pricing page has the current numbers. There is no separate MCP rate.

The charge does not depend on the result. A URL that did not load returned null in a call we ran and was still billed 5 points, so empty or failed calls are not free. If your list contains pages that may be gone or blocked, budget for the misses and test a sample first.

What we have not measured: accuracy across a broad set of page types, speed, and how it behaves on very large pages. Treat those as things to test with your own URLs. Run your real target pages through your current scraper and through meaning-based extraction, then see which one survives the next redesign without intervention. That is a more honest test than any description of either approach.

How to decide

Work through these questions for your own pipeline:

• —How many external sites does it depend on, and how often do they change? More sites and more frequent redesigns both push the maintenance math toward an approach that does not depend on structure. Ten sites is ten independent sources of breakage.
• —Is anyone watching for silent breakage? An unwatched pipeline feeding a dashboard or database is where selector fragility does the most damage before anyone notices.
• —Is a wrong-but-plausible value worse than a visible failure for you? If yes, pin the output with a schema and add validation, whichever approach you choose.
• —Do you monitor competitor or pricing pages? These change on someone else's schedule, with no reason to keep your scraper working.
• —Do you have spare capacity to fix scrapers? A broken scraper costs the same to fix on a team of five as on a team of one, but it is a much larger share of a small team's week.
• —Have you counted the time, not just the fee? Compare the full cost on both sides, including the maintenance hours that never reached a budget.
• —Does the position of the value matter, or only the value? If you need the literal markup, keep selectors. If you need the field, use extraction by meaning.
• —Can you migrate in stages? Replace the scrapers that break most often first and leave stable ones alone.
• —Does the current scraper struggle with client-rendered pages? Confirm any candidate renders pages the way a browser does before assuming it fixes that too.

Frequently asked questions

Do selectors really break that often in practice?

More often than teams expect once a scraper depends on more than a handful of pages. A redesign, an A/B test, or even a single class-name change can break a selector, and most of these happen without any announcement to whoever depends on the page's structure staying put.

Isn't a more defensive selector, with multiple fallbacks, a reasonable fix?

It buys some resilience, but it is still coupled to structure, just with more paths that can each break independently. It reduces how often a page change causes failure without removing the dependency that causes it.

Can I migrate an existing selector-based scraper incrementally?

Yes. Nothing requires a full rewrite on day one. Replacing the scrapers that need the most maintenance first, and leaving stable ones alone, captures most of the benefit without a large upfront project.

Will this eliminate scraper maintenance entirely?

No. No extraction approach is immune to a site becoming inaccessible or changing what content exists at all, and a model can misread a page. It removes one specific failure mode, markup changes breaking a scraper that depended on exact DOM position, which is the majority of routine breakage most teams see.

Can I use meaning-based extraction over MCP, or only through the API?

Yes, both. NeuralVerge hosts an MCP server at https://mcp.neuralverge.ai/mcp that exposes AI extraction (run_extract) as a tool. Add the URL to an MCP client such as Claude, ChatGPT or Cursor, sign in through the browser and click Allow. It takes the same inputs, returns the same response shape and costs the same as the REST call.

Is a silently wrong result actually worse than a scraper crashing outright?

Often, yes. A crash is visible and gets fixed because something obviously stopped working. A selector that shifts to the wrong element after a layout change can keep running and return plausible but incorrect values, which nothing flags until someone notices the numbers look off.

What does it cost, and am I charged when a URL does not load?

An AI Extract call costs 5 points. In a call we ran, a URL that did not load returned null and was still billed 5 points, so budget for misses and check a sample before running a long list.