SEO audit scraping API. Crawl every URL. Status codes, redirects, render parity.
The data infrastructure under your own SEO audit tool. Burst concurrency on datacenter proxies, headless Chrome on demand for render-parity audits, redirect-chain capture via response headers — one credit covers fetch, render, and status check.
Burst concurrency · failed requests cost 0 · 100K URLs on the Enthusiast plan
# Status + headers for one URL — return_headers=true for redirect chains.
$ curl 'https://api.scrapingant.com/v2/general' \
--data-urlencode 'x-api-key=YOUR_KEY' \
--data-urlencode 'url=https://example.com/page' \
--data-urlencode 'return_text=false' \
--data-urlencode 'return_headers=true'# Crawl a URL list with concurrency.futures — burst hundreds in parallel.
from concurrent.futures import ThreadPoolExecutor
import requests
def check(url):
r = requests.get("https://api.scrapingant.com/v2/general", params={
"x-api-key": "YOUR_KEY",
"url": url,
"return_text": "false",
"return_headers": "true",
})
return url, r.status_code, r.headers.get("X-Status-Origin")
with ThreadPoolExecutor(max_workers=50) as ex:
for result in ex.map(check, sitemap_urls):
print(result)# browser=true for render-parity: page 200s on HTTP but shows nothing in Chrome?
import fetch from 'node-fetch';
const rendered = await fetch(
'https://api.scrapingant.com/v2/general?' +
new URLSearchParams({
'x-api-key': KEY,
url, browser: 'true',
})
);
const static_html = await fetch(
'https://api.scrapingant.com/v2/general?' +
new URLSearchParams({ 'x-api-key': KEY, url })
);
// diff the two — render-parity audit in two API calls.Why SEO audit tools build on us.
The infrastructure layer, not a finished crawler. Burst concurrency, render parity, geo-targeted exits — all on one credit pool.
Burst concurrency on datacenter pool
Hundreds of parallel workers, no per-key concurrency caps. Datacenter proxies for your own site, residential for competitive crawls.
How concurrency works →Render-parity diff in two calls
Plain HTTP fetch + browser=true on the same URL. Diff catches "200 OK but empty" pages.
browser=true → Geo-targeted audits
Verify hreflang variants from the right country. proxy_country=DE hits German egress IPs.
Status, headers, render. One call.
Most audit pipelines stitch three providers — HTTP-checker for status, Chrome cluster for render, headless crawler for link extraction. ScrapingAnt's /v2/general returns all three from one endpoint: return_headers=true exposes the redirect chain, browser=true flips to headless Chrome, the response body gives you the rendered DOM. Internal-link extraction layers in via /v2/extract with a schema like extract_properties=internal_links,canonical,meta_description.
- Full HTTP response: status, headers, body — not just a 200/404 boolean
- Redirect chain capture with
return_headers=true— diff against expected canonicals - Failed requests cost 0 credits — DNS errors, timeouts, 5xx never bill
Render-parity audits. Two calls. One diff.
JS-framework pages routinely 200 OK to a plain-HTTP crawler but render empty in a real browser — the kind of audit miss that ships broken pages to production. With ScrapingAnt the parity check is two API calls on the same URL: one with default options (HTTP fetch), one with browser=true (headless Chrome). Diff the two responses in your audit pipeline. Same key, same credit pool, failed calls cost zero.
- Catches "200 OK but empty <body>" pages — the classic JS-framework audit miss
- Hash the rendered body — diff against yesterday's hash to flag render regressions per URL
- One flag flip — no second provider, no second contract, same credit pool
Hreflang audits. Run from the right country.
Hreflang audits go wrong when you crawl from one geography and the site geo-redirects you to the wrong locale. proxy_country=DE hits German egress IPs; proxy_country=JP hits Japanese. Verify each declared variant actually serves the right language from the right region. Sticky sessions (proxy_country=DE&session=foo) keep multi-page crawls on the same exit IP — critical for sites tracking auth state or sequential page loads.
- 2M+ residential IPs across 100+ countries — no VPN tooling per region
- Country + city targeting via username parameters when you need state-level granularity
- Datacenter pool for your own site, residential for competitive crawls — same key
Six audit workloads teams build on top.
Same API, same credit pool — different ways of slicing the crawl data underneath.
Internal-link audits
Crawl every URL on a site, follow internal anchors, build the full link graph. Detect orphan pages, broken inbound links, and crawl-depth issues.
Redirect-chain validation
Follow 301/302 chains end-to-end with <code>return_headers=true</code>. Flag chains longer than two hops, mixed-protocol redirects, and loops.
Render-parity checks
Diff the raw HTML against the headless-Chrome render. Catch pages that 200 OK to crawlers but render nothing to real users — a common JS-framework gotcha.
Hreflang verification
Fetch each <code>hreflang</code> variant from the right country, confirm the rendered page matches the declared locale. Geo-targeted proxies do the routing.
Sitemap vs crawl gap
Pull the XML sitemap, crawl the actual site, diff the two sets. Find URLs the sitemap forgot and URLs that 404 despite being listed.
Pre-launch QA crawls
Burst-crawl a staging environment before pushing to production. Status codes, render checks, meta-tag inventory — all in one credit pool, no separate staging proxy.
Pricing
Industry leading pricing that scales with your business.
|
Plans
|
Enthusiast
100K credits / mo
$19/mo
|
★ Most Popular
Startup
500K credits / mo
$49/mo
|
Business
3M credits / mo
$249/mo
|
Business Pro
8M credits / mo
$599/mo
|
Custom
10M+ credits / mo
$699+/mo
|
|---|---|---|---|---|---|
| Monthly API credits | 100,000 | 500,000 | 3,000,000 | 8,000,000 | 10M+ |
| Support channel | Priority email | Priority email | Priority email | Priority + dedicated | |
| Integration help | Docs only | Custom code snippets | Debug sessions | Priority debug sessions | Full enterprise onboarding |
| Expert assistance | — |  |
|
|
|
| Custom proxy pools | — | — | |
|
|
| Custom anti-bot avoidances | — | — | |
|
|
| Dedicated account manager | — | — | |
|
|
| Start Free | Start Free → | Start Free | Start Free | Talk to Sales |
What teams are saying.
From solo developers shipping side projects to enterprise pipelines at Fortune 500s.
★★★★★ 5.0 on Capterra →★★★★★“Onboarding and API integration was smooth and clear. Everything works great. The support was excellent.”
★★★★★“Great communication with co-founders helped me to get the job done. Great proxy diversity and good price.”
★★★★★“This product helps me to scale and extend my business. The setup is easy and support is really good.”
What is an SEO audit API?
An SEO audit API is a managed endpoint that takes URLs and returns the data SEO audit tools build on: status codes, redirect chains, response headers, rendered HTML, internal-link inventories. ScrapingAnt's /v2/general covers static and JS-rendered fetches, return_headers=true exposes the redirect chain, and /v2/extract returns parser-free JSON for link graphs and meta-tag inventories. Datacenter-proxy concurrency keeps the per-URL cost predictable at scale.
How is this different from Screaming Frog or Sitebulb?
Screaming Frog and Sitebulb are finished crawler tools — they ship a GUI, a license model, and a single workstation's worth of concurrency. ScrapingAnt is the data infrastructure underneath an SEO audit product. You bring the URL list, the storage, the report layer; we hand back clean responses with predictable per-URL economics. Teams pick us when they want to embed audit data into their own dashboard, run scheduled audits across hundreds of client sites, or audit sites larger than a single Frog instance comfortably handles.
Does it respect robots.txt by default?
No — the API fetches exactly the URLs you pass. Robots-aware crawling is your audit tool's responsibility (parse robots.txt, filter the URL list before calling us). This is intentional: many audit workflows (broken-link checks on your own site, pre-launch QA crawls, render-parity audits) explicitly need to hit URLs that production robots.txt would block.
Can I get the JavaScript-rendered HTML?
Yes — pass browser=true and the response is the post-render DOM, exactly what a browser sees. Pair a non-browser fetch with a browser fetch on the same URL for a render-parity diff — the classic catch for "this page 200s for Googlebot but actually renders empty in Chrome".
How much does crawling 100,000 URLs cost?
A static-HTML fetch is about 1 credit; browser=true adds JS-render cost. 100K static URLs ≈ 100K credits — fits the Enthusiast plan ($19/month, 100K credits) for a one-shot audit. Recurring audits across many client sites land on the Startup plan ($49 / 500K credits) or Business ($249 / 3M credits). Failed fetches (DNS errors, timeouts, 5xx) cost zero credits.
Can I bring my own concurrency limits?
Yes — the API doesn't enforce a per-key concurrency cap inside paid plans, so your audit tool sets the worker-pool size. Datacenter proxies handle burst load (hundreds of parallel connections); for sites with strict rate limits, use sticky sessions via proxy_country + a stable proxy slot to keep each crawler-worker on one IP.
Datacenter vs residential proxies for audits?
For your own site, datacenter proxies are cheaper and faster — your site shouldn't be blocking datacenter ranges. For competitive audits against sites with aggressive bot detection (Cloudflare, Akamai), flip proxy_type=residential on the SKUs that 403. Same key, same credit pool.
Building an SEO audit tool?
Custom volume pricing, dedicated datacenter pools for concurrency-heavy workloads, audit-result caching strategies, or migration help from in-house Frog clusters — drop us a line and a real human gets back within a few hours.
“Our clients are pleasantly surprised by the response speed of our team.”
