C# HttpClient Web Scraping with ScrapingAnt and Proxies
Tested explicit C# request, field validation and failure handling offline. A real owned synthetic product also passed the full public client with a native JSON-LD contract and a 10-credit receipt; earlier marker captures verify raw/rendered behavior separately. Publication approved by Oleg Kulyk on October 12, 2026.
If HttpClient already returns the fields you need from a page you may retrieve, keep that direct request. A managed acquisition layer becomes useful when you want the same C# parser to consume HTML fetched through a chosen proxy pool or rendered with JavaScript.
This guide connects HttpClient to ScrapingAnt's /v2/general endpoint, then validates one product's SKU, title, decimal price and currency. It shows explicit raw and rendered choices, bounded failures, and a credit receipt check before you expand a workload. The general endpoint returns HTML; C# performs the field extraction in this example. Request and response documentation.
Run the offline example first
Install .NET SDK 10.0.401 and Python 3.12.11. The commands below require Bash in your .NET environment; the recorded run used macOS and the PR workflow is configured for Linux. Download the complete example, extract it, and enter its web-scraping-in-c-with-httpclient-and-proxies-a-2025 folder. The project pins HtmlAgilityPack 1.13.0 and restores it from a NuGet lock file.
dotnet --version
./run.sh
The recorded SDK version is:
10.0.401
run.sh builds the project, executes 59 default client cases and 62 separate native JSON-LD/CLI cases, checks literal expected fields, and replays the saved captures. It compares its outputs with committed results, uses no API key and makes no target or ScrapingAnt requests. Dependency restoration may download the pinned NuGet package.
The local static catalog contains:
<article data-sku="CSHARP-001" data-currency="USD"><h1>C# & HTTP</h1><span class="price">99.95</span></article>
A second fixture contains only an empty application shell. Its captured failure is:
PASS raw application shell has no product
That is a successful negative check: the parser refuses to call an empty shell a product. The authored catalog rendered.html fixture represents a possible final DOM; it was not produced by a browser or a live ScrapingAnt call. These fixtures verify the client contract. The separate owned-page captures below verify retrieval and rendering.
Construct the API request explicitly
Your C# application connects to the API endpoint. The proxy_type and proxy_country parameters select the target-side retrieval settings; they are separate from configuring a WebProxy on your application's HttpClientHandler.
The following method is copied from ManagedClient.cs in the downloadable project:
public static Uri BuildUri(Uri target, bool browser)
{
if (!target.IsAbsoluteUri || target.Scheme != Uri.UriSchemeHttps ||
target.UserInfo.Length != 0 || target.Fragment.Length != 0)
throw new ArgumentException("Use an absolute HTTPS target without user info or fragment");
var query = new Dictionary<string, string>
{
["url"] = target.AbsoluteUri,
["browser"] = browser ? "true" : "false",
["timeout"] = "30",
["proxy_type"] = "datacenter",
["proxy_country"] = "us",
["return_page_source"] = "false"
};
return new Uri("https://api.scrapingant.com/v2/general?" + string.Join("&",
query.Select(p => $"{Uri.EscapeDataString(p.Key)}={Uri.EscapeDataString(p.Value)}")));
}
Encode the complete target URL as one query value, including its own query string. The offline handler checks that it decodes back to the original URL and that no API key appears in the request URL. Authentication is added to each request through the supported x-api-key header:
using var request = new HttpRequestMessage(HttpMethod.Get, uri);
request.Headers.Add("x-api-key", apiKey);
using var response = await http.SendAsync(request,
HttpCompletionOption.ResponseHeadersRead, cancellationToken);
Use browser=false when the server's HTML contains the required data. With browser=true, ScrapingAnt renders the target page with JavaScript and returns its HTML when return_page_source is false. This example explicitly sets both values and a 30-second API timeout so it does not rely on defaults. Browser rendering documentation.
The caller's cancellation deadline is a separate limit. Program.cs disables automatic API redirects and supplies a 95-second overall deadline; cancellation never creates a replacement request.
Check retrieval on an owned page
On October 11, 2026, a separate C# marker probe retrieved the owned delayed fixture once in each mode, with no redirects or retries. It reused only ManagedClient.BuildUri from the pinned source, used the explicit settings above, and added wait_for_selector=#loaded for the rendered request.
The raw response contained #test with Web Scraping is hard and no actual #loaded element. The rendered response contained #test with I ❤️ ScrapingAnt (after 1500 ms) and an actual div#loaded with loaded. Both API and target statuses were 200; the actual credit receipts were 1 and 10. The saved HTML, sanitized receipts and offline replay are included in the download.
This confirms retrieval and rendering on that owned page. The probe used its own send and marker-validation path; it did not execute ManagedClient.FetchAsync, the product parser or the live CLI below.
Parse HTML, then validate the fields
Do not deserialize the general endpoint's body as a product JSON object. The response is HTML. HtmlAgilityPack loads that HTML and queries the fixture's DOM:
var document = new HtmlDocument();
document.LoadHtml(html);
if (contract == ProductContract.SingleProductOfferJsonLd) return ParseSingleProductOfferJsonLd(document);
if (contract != ProductContract.ArticleFields) throw new ContentFailure("Unsupported product contract");
var products = document.DocumentNode.SelectNodes("//article[@data-sku]");
if (products is null || products.Count != 1)
throw new ContentFailure("Expected exactly one product");
var node = products[0];
string Text(string xpath) => HtmlEntity.DeEntitize(node.SelectSingleNode(xpath)?.InnerText ?? "").Trim();
var sku = node.GetAttributeValue("data-sku", "").Trim();
var title = Text(".//h1");
var priceText = Text(".//*[@class='price']");
var currency = node.GetAttributeValue("data-currency", "");
The complete parser also requires a nonempty SKU and title, a nonnegative decimal price parsed with invariant culture, and a three-letter uppercase currency. It refuses a missing or duplicate product, missing fields and invalid values. Those rules describe this fixture's schema; adapt them for your page rather than accepting the first node that happens to match.
The client prefers the forwarded Ant-original-header-content-type, with the outer content type as fallback. It rejects a declared non-HTML or malformed type; when the type is absent, the body still has to pass the HTML document and required field checks. Target headers use the Ant-original-header- prefix. Response header documentation.
A successful Ant-page-status-code is also required. The outer HTTP status describes the API request; the target status is a different layer. An API 200 containing a target 403 is rejected by the offline control. Error reference.
The record oracle is independent of the parser: both the C# checks and a separate Python verifier compare the result with literal expected fields. Mutation controls show that a wrong price or title, missing record, false live label, or invented credit value fails the verifier.
Captured output excerpts:
SIMULATED raw: CSHARP-001 | C# & HTTP | 99.95 | USD; credits=1
SIMULATED rendered: CSHARP-001 | C# & HTTP | 99.95 | USD; credits=10
These are two simulated records. The receipt numbers are mock headers chosen to match the documented modes, not observed charges.
Select a native Product/Offer contract explicitly
The default ArticleFields contract expects the attributes shown above. A page that carries its product in JSON-LD needs a different, explicit contract. SingleProductOfferJsonLd selects exactly one native application/ld+json script containing a top-level Product, https://schema.org context, one Offer object, and string SKU/name/price/currency values. It rejects arrays, graphs, duplicate scripts or properties and invalid fields. It performs no automatic fallback or rewriting into the article-attribute layout.
On October 12, 2026, one rendered request for the existing owned synthetic product fixture passed the public ManagedClient.FetchAsync path: request construction/send, target and declared-type checks, native parsing, and credit receipt validation. The observed API and target statuses were 200, with one client attempt and an actual 10-credit receipt. Its checked record was:
{"sku":"SYNTH-000","title":"Synthetic product 000","price":12.34,"currency":"USD"}
The source, accepted body, sanitized receipt and independent replay are in the download. The received document has 601 bytes; the direct source has 588. Their full hashes differ, while the native JSON-LD script is unchanged. A separate observer invoked the public method and preserved the returned bytes; it did not transform the page into the default schema or execute the public live CLI.
The dated capture retains the client source pin used by that request. The current download adds only a currency-length guard to the default ArticleFields parser; its native JSON-LD path is unchanged and replays the saved response offline. No live request was repeated.
This is full-client acceptance on one owned synthetic product. The fields already appear in its static source, so the observation does not establish JavaScript-driven product updates, real-store coverage or a universal JSON-LD parser.
Stop corrective failures and cap transient retries
The full ManagedClient.cs implementation is included in the download. Its policy applies only to GET requests:
| Response or failure | Tested client action |
|---|---|
| Outer 400, 403 or 422 | Stop once; correct the request or access condition |
| Outer 409 | Stop once; queue work instead of creating more concurrent requests |
| Outer 404, 405, 423 or 429 | Stop once; inspect the cause or server schedule before another attempt |
| Outer 500, 502, 503 or 504; HTTP or body-stream transport exception | At most three attempts, with scheduled 1-second then 2-second delays |
| Cancellation during send, read or backoff | Stop; no automatic replacement |
| Missing target status, invalid fields, invalid or missing credit receipt | Reject the result; do not retry a validation failure automatically |
The tests inject the delays and inspect their schedule; they do not measure elapsed latency. The live command below deliberately uses just one attempt. Keep any later retry budget explicit because it can create additional successful retrievals.
For a 403, the code classifies the documented detail locally as authentication, quota or IP access. Those conditions need different fixes; changing a valid key does not restore credits. It prints fixed diagnostics instead of raw bodies, request URLs, key headers or transport exception messages. Read sensitive error details privately when the fixed category is insufficient. Troubleshooting by status and detail.
The capture ends with:
PASS 59/59 offline cases; live API calls=0
PASS independent literal oracle: 2 simulated records, 4 fields each
PASS 5/5 independent oracle mutation controls
PASS 62/62 native JSON-LD and CLI offline cases; live API calls=0
PASS 7/7 native literal-oracle mutation controls
PASS 8/8 saved native-capture corruption controls
The denominator is offline test cases. It says nothing about a production success rate.
One permitted URL, then a useful workload
The optional CLI reads SCRAPINGANT_API_KEY from your environment. Set PERMITTED_URL to a complete HTTPS page you may retrieve, after selecting or adapting an explicit contract for its markup. Keep the key out of command lines, URLs and source files.
These live commands are illustrative and were not executed for this article:
dotnet run --no-restore --no-build --configuration Release -- --live "$PERMITTED_URL" raw
dotnet run --no-restore --no-build --configuration Release -- --live "$PERMITTED_URL" rendered
For the owned synthetic Product/Offer page, the explicit selector is:
PERMITTED_URL='https://scrapingant.github.io/scrapingant-examples/fixtures/apify-continuation-v1/commerce/product-000.html'
dotnet run --no-restore --no-build --configuration Release -- --live "$PERMITTED_URL" rendered --contract single-product-offer-jsonld
This command is also illustrative and unexecuted. It would make one paid request when you provide a key. The recorded acceptance used an observer around FetchAsync; CLI option selection is tested offline.
Each command sends one GET with the explicit settings above and no automatic paid retries. The product parser is specific to the catalog fixture; it does not validate an unrelated dynamic-page marker. Start with the mode that should contain your fields, inspect the target status and checked record, and confirm the response's Ant-credits-cost header before scheduling repeated retrievals. If a receipt is missing or malformed, this client stops with unknown cost rather than inserting zero.
For a non-Google target, a request without a browser through a datacenter proxy costs 1 API credit. A request with JavaScript rendering through a datacenter proxy costs 10 API credits. These documented costs match the owned marker receipts and the native product's rendered receipt; check the actual receipt for each request. Credit schedule and receipt header.
Use the actual receipts and your required refresh frequency to budget the workload. Count validated records separately from HTTP successes, and leave room for deliberate retries or a change in retrieval settings. Avoid buying rendering for a page whose raw HTML already contains the fields. If direct HttpClient is sufficient, ScrapingAnt is not needed to parse that HTML.
When managed retrieval fits your task, create a key and follow the request documentation. As documented on October 11, 2026, the Free plan offers 10,000 API credits per month with no credit card required; the Enthusiast plan lists 100,000 credits per month at $19 per month. Check the current plans against your measured workload before upgrading. These plan allowances are not a capacity benchmark.
Limits and related workflows
Failure controls and CLI selection are tested offline. The full-client result covers one owned synthetic Product/Offer fixture; it does not establish real-store acquisition, arbitrary JSON-LD, production latency, throughput or anti-bot reliability. The optional public live CLI remains unexecuted. HtmlAgilityPack can tolerate malformed markup; required field checks do not certify every HTML document as valid. The client caps each body at 1,048,576 characters to bound this example's input. UTF-8 fixtures are the tested encoding; adapt decoding for targets using another encoding.
Use browser automation when the task requires page interactions or localStorage state. Managed rendering does not turn this HTTP client into a browser automation session. The broader C# dynamic scraping guide covers that tool-selection path. For a structured AI response instead of deterministic DOM parsing, the AI extractor is a separate endpoint with separate extraction behavior and cost.
Examples tested on 2026-10-12 with .NET SDK 10.0.401, .NET runtime 10.0.12, HtmlAgilityPack 1.13.0 and Python 3.12.11; the historical marker captures are dated October 11. Code and captured outputs: immutable evidence packet. The local download contains the same packet files.
This article was drafted with AI assistance and verified with offline controls, owned marker captures and an owned synthetic full-client capture. Publication approved by Oleg Kulyk on October 12, 2026.