Extract Company Data From Any Website: Step-by-Step API Tutorial

This tutorial shows how to extract company data from website API calls, instead of pulling structured fields — name, founding year, headcount, description — with hand-written CSS selectors that break every time the site redesigns. Point the AI extraction API at a page, tell it what you want back, and get typed JSON instead of a scraper to maintain. It covers inferred and defined schemas, pinning an exact response shape with a JSON schema, handling a missing field honestly, and extending the same call to a document instead of a URL.

Prerequisites

• —A NeuralVerge API key from the app dashboard.
• —A real company page you can check by eye — this tutorial uses a fictional example, https://example.com/company/acme, but running the same steps against a page you already know is the fastest way to judge whether the output actually matches what's on the page.
• —curl, Python, or any HTTP client — the examples below cover all three.

Step 1: Run an extraction without defining a schema

Start with the simplest possible call — a URL and nothing else. The extractor infers a sensible structure from the page's own content:

curl https://api.neuralverge.ai/v1/extract \
  -H "Authorization: Bearer <API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/company/acme"
  }'

import requests

response = requests.post(
    "https://api.neuralverge.ai/v1/extract",
    headers={"Authorization": "Bearer <API_KEY>"},
    json={"url": "https://example.com/company/acme"},
)
print(response.json())

const response = await fetch("https://api.neuralverge.ai/v1/extract", {
  method: "POST",
  headers: {
    Authorization: "Bearer <API_KEY>",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ url: "https://example.com/company/acme" }),
});

const data = await response.json();

A response like this comes back:

{
  "title": "Acme — Company Profile",
  "content": "Acme builds ...",
  "fields": { "founded": "2019", "employees": "50-100" }
}

Read this by eye against the real page before moving on. Confirm the inferred fields actually match what's visible on the page — that's the check that tells you whether the extraction is reading the content or approximating it.

Step 2: Describe the fields you want in plain language

Inferred structure is fine for exploring what a class of pages contains, but a pipeline that runs the same call across thousands of pages needs the same fields back every time, in the same shape. The fastest way to get there is describing what you want directly in plain language, via instructions:

curl https://api.neuralverge.ai/v1/extract \
  -H "Authorization: Bearer <API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/company/acme",
    "instructions": "Extract the company name, founding year, employee count band, headquarters location, and a one-sentence description."
  }'

Running this against ten different company pages — even ten with visibly different layouts — should return the same five field names every time, with a page that doesn't have one of them returning that field empty rather than omitting the key. That consistency is most of the way to what a pipeline actually needs. The part instructions alone doesn't guarantee is the exact shape of each field — a type, a required flag — which is what Step 3 adds.

Step 3: Pin the exact response shape with a JSON schema

instructions tells the model what to look for; it doesn't pin down the precise property names, types, or which fields must be present. For a pipeline that validates the response shape itself — a strict downstream consumer, a database column that can't accept the wrong type — pass an actual JSON Schema via settings.extract_schema_json instead:

curl https://api.neuralverge.ai/v1/extract \
  -H "Authorization: Bearer <API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/company/acme",
    "settings": {
      "extract_schema_json": "{\"type\":\"object\",\"properties\":{\"company_name\":{\"type\":\"string\"},\"founded_year\":{\"type\":\"string\"},\"employee_count\":{\"type\":\"string\"},\"headquarters\":{\"type\":\"string\"},\"description\":{\"type\":\"string\"}},\"required\":[\"company_name\"]}"
    }
  }'

import json
import requests

schema = {
    "type": "object",
    "properties": {
        "company_name": {"type": "string"},
        "founded_year": {"type": "string"},
        "employee_count": {"type": "string"},
        "headquarters": {"type": "string"},
        "description": {"type": "string"},
    },
    "required": ["company_name"],
}

response = requests.post(
    "https://api.neuralverge.ai/v1/extract",
    headers={"Authorization": "Bearer <API_KEY>"},
    json={
        "url": "https://example.com/company/acme",
        "settings": {"extract_schema_json": json.dumps(schema)},
    },
)
print(response.json())

extract_schema_json takes a JSON Schema as a string — type: "object" with a properties map, and an optional required array for the fields that must come back rather than being left empty. The response's field names then match your schema's property names exactly, every time, regardless of how the source page happens to label things internally. Nothing stops you from combining this with instructions in the same call — instructions can steer what on the page maps to a given field when it's ambiguous, while the schema pins the shape that comes back. For most pipelines, plain-language instructions (Step 2) is enough; reach for an explicit schema when a downstream system needs to validate types and required fields on its own, not just receive a reasonable-looking object.

Step 4: Handle rendering for JavaScript-heavy pages

Most company pages load some content — an "About" section, a specialties list — after the initial page load, via client-side JavaScript. Extraction renders the page the way a browser would rather than fetching raw HTML, so this content is present by the time extraction runs; there's no separate flag to set for it. If a field consistently comes back empty on a page where you can see it rendered in a browser, that's worth flagging directly rather than assuming the field genuinely isn't there — check the page in an incognito window first, since some content is genuinely gated behind a login or a cookie-consent interaction extraction won't click through.

Step 5: Check what comes back for a field that isn't on the page

Point the same call at a page you know is missing one of your requested fields — say, a company page with no listed founding year:

{
  "title": "Acme — Company Profile",
  "content": "Acme builds ...",
  "fields": { "founded": null, "employees": "50-100" }
}

A well-behaved extractor returns null or empty for a field that isn't present, rather than a plausible-sounding guess. This is the single most important thing to verify before trusting an extraction pipeline at volume — a tool that always fills in a complete-looking answer for every field, regardless of what's actually on the page, is inferring rather than reading, and that failure mode is much harder to catch downstream than an honest empty field. This is also where a schema's required array (Step 3) earns its keep: a field marked required that still comes back empty is a clean, explicit signal to flag the record, rather than something a pipeline has to notice on its own.

Step 6: Point the same call at a document instead of a URL

The same request shape works for a document — a PDF fact sheet, for instance — as it does for a URL. Swap the url parameter for a document reference per the API reference, and the same field-mapping logic applies — including an extract_schema_json from Step 3, if you're using one: multi-column layouts and tables are read in full, repeated boilerplate like a header logo on every page is treated as noise rather than content, and the response comes back in the identical shape as the URL example. A pipeline that ingests both a company's website and a company's PDF one-pager doesn't need a second code path — only the source changes, not the call or the schema.

Step 7: Run it on a real, messy batch

A single page proves the mechanism works. A real workflow needs to survive a batch of pages that don't all look alike — some missing fields, some redirecting, some genuinely not matching the company you expected. Before wiring this into a schedule or a larger pipeline, run it against twenty or thirty real, messy URLs from your actual target list and check three things directly: how often a field comes back empty versus populated, whether any responses look like a plausible guess rather than a grounded read, and how the call behaves on a URL that returns a 404 or redirects somewhere unexpected. This step catches the gap between "works on one clean example" and "works on the batch you're actually going to run."

A worked example: building a small monitoring pipeline

Take a concrete, illustrative case: tracking a fixed list of ten competitor pricing pages on a weekly schedule, watching for plan or price changes.

1. —Define a JSON schema once — plan_name, price, billing_period, included_limits, each typed and plan_name marked required (Step 3) — since the same fields need to come back from every page in the list, every week, in a shape a downstream table can trust without re-checking.
2. —Run the extraction call against each of the ten URLs on a schedule, storing the result alongside the date it was pulled.
3. —Compare each week's result against the previous week's for the same URL; a changed price or included_limits field is the actual signal worth alerting on.
4. —Because a missing field comes back empty rather than guessed (Step 5), a genuinely redesigned pricing page that temporarily breaks one field shows up as a real gap to investigate, not a silently wrong number that looks like a legitimate price change.

Where teams extract company data from website APIs like this

• —Competitor and market monitoring — tracking pricing, feature lists, or job postings across a known set of pages on a schedule, exactly as in the worked example above.
• —Lead and account enrichment — turning a company's own "About" or pricing page into structured firmographic fields alongside other enrichment sources.
• —Document intake — applying the same schema-driven approach from Step 6 to incoming PDFs like fact sheets or filings.
• —RAG pipeline ingestion — feeding clean, structured content into a retrieval index instead of raw, boilerplate-heavy HTML.

What to check before you run this at scale

• —Does the inferred structure from Step 1 actually match the page, or does it look plausible but wrong? Check by eye on a page you know before trusting it on pages you don't.
• —Do plain-language instructions (Step 2) or a JSON schema (Step 3) return the exact same field set on every page, including ones that don't have all the fields? Consistency of shape matters more than any single field's accuracy.
• —Does a missing field come back empty (Step 5), or does the tool always produce a complete-looking answer? This is the fastest way to catch a tool that's inferring rather than reading.
• —Have you run a real, messy batch (Step 7), not just one clean example? A pipeline's actual failure modes show up on the pages that don't match what you expected, not the one you tested first.
• —Does the same schema and call work for a document as well as a URL (Step 6)? If a workflow mixes both, confirm they share one code path before assuming they do.

Running your own real page and your own real document through this exact sequence is the fastest way to see whether it holds up before committing a pipeline to it. For the mechanism behind what happens between the request and the response — rendering, cleaning, field mapping — see how AI extraction works.

Frequently asked questions

Do I need to know the exact fields on the page before I start?

No — Step 1 in this tutorial runs an extraction without a defined schema, letting a sensible structure be inferred from the page. Describing fields in plain language (Step 2) or pinning them with a JSON schema (Step 3) are both worth doing once you know exactly what you want back on every call.

What's the difference between plain-language instructions and a JSON schema?

Instructions tell the model what to look for in words — fast to write, and enough for most pipelines. A JSON schema, passed as extract_schema_json, pins the exact property names, types, and which fields are required, which matters when a downstream system validates the response shape itself rather than trusting it loosely matches what was asked. You can pass both together: instructions for what to look for, schema for the exact shape it comes back in.

What happens if the company page doesn't have a field I asked for?

It comes back empty rather than filled in with a plausible-sounding guess — that's the behavior Step 5 checks for directly. A tool that always returns a complete-looking answer regardless of what's on the page is inferring rather than reading.

Can this tutorial's approach handle a PDF instead of a web page?

Yes — Step 6 covers pointing the same call at a document instead of a URL. The schema and field-mapping logic don't change; only the source does.

Should I call the extraction endpoint directly or expose it as an agent tool?

Call it directly, as this tutorial does, when you already know which pages need processing — a fixed list, a schedule. Expose it as a tool when an agent encounters URLs dynamically and needs to decide for itself that a page should become structured fields rather than raw content.

How do I avoid rewriting the schema every time a target site changes its layout?

A layout change doesn't require touching the schema at all, because extraction maps fields by meaning rather than by DOM position — the company name field still resolves correctly even if the site moves it from a hero section to a sidebar. That's the specific problem this approach is built to remove.