Retail Shelf Intelligence - Validate SKU Price and Stock Data
The original article described broad CPG analytics and made unsupported claims about automatic extraction, layout adaptation and retailer coverage. This revision narrows the task to validated product observations and corrects the API parameter names to browser and proxy_country. Download the reproducibility packet.
A retail price observation needs a product identity, the selected variant and pack size, the offer being quoted, and evidence for its availability. Otherwise, a cheaper subscription offer or a smaller pack can look like a price drop. A failed page can look like an out-of-stock product if the collector treats every missing value as a stock signal.
This walkthrough builds a small source-specific Python collector for repeated SKU observations. It keeps unknown availability, rejected fields and acquisition failures distinct from explicit out-of-stock evidence. The runnable controls use a fictional tea family with two selected weight packs. A separate direct retail observation and optional managed retrieval are recorded separately. Completeness refers to those selected offers, not a retailer’s full inventory.
Select the source and the offer before collecting
Start with a small list of product URLs you are permitted to collect. Check access and reuse requirements, then inspect whether an official feed, product endpoint or direct HTML already supplies the required fields. If it does, use that source before adding a managed retrieval service. A robots directive alone is not a complete source-selection or redistribution review.
The source investigation observed Adagio's Earl Grey Lavender product page through direct raw HTTP on October 13, 2026 at 10:37:02 UTC. Its ProductGroup.hasVariant JSON-LD supplied these selected listed offers:
SKU, group 001036 | Weight | Listed price | Explicit source availability |
|---|---|---|---|
001036002 | 3oz | 9.00 USD | https://schema.org/OutOfStock |
001036004 | 16oz | 29.00 USD | https://schema.org/OutOfStock |
Both offers share the product URL. Their group, SKU and exact size bind the observations; no separate variant ID is invented. The page also supplies a US shipping-destination marker. That is page context, not a selected checkout destination: fulfilment remains unselected and one-time/subscription purchase basis unspecified. Separate auto-delivery discounts are not substituted for the listed variant prices.
Direct HTML already supplied the selected fields. No additional feed route was probed and no JavaScript requirement was established. The packet retains a minimal factual projection for offline inspection; full retailer HTML is withheld. Recheck the source's robots directives and its linked policies, including the privacy policy, before choosing a new collection scope. This observation is not blanket permission to reuse the site.
For this example, the useful identity is more specific than a product title: the selected product group, SKU, variant and pack quantity with its unit. Keep the offer context beside that identity. If the source does not provide market or fulfilment evidence, leave that context unknown; a proxy country cannot establish it. A subscription discount or an aggregate “from” price is a different observation from the selected base offer.
Do not infer the selected variant from a title substring or choose the lowest price across offers. Resolve it against the inspected source structure and reject an ambiguous result. This collector is deliberately tied to a small configured source contract. Adding another retailer requires its own identity rules, expected fields and fixtures.
Define an accepted observation
The collector emits the following fields for each accepted selected offer:
| Fields | Acceptance rule |
|---|---|
product_id, sku, variant_id, variant | Bind the configured product group to one selected SKU and exact size; variant_id is null because this source has no separate variant ID. |
pack_quantity, pack_unit | Normalize the selected size to a weight in ounces. A changed pack is not silently treated as the same offer. |
price, currency | Keep a decimal price string and an explicit currency. Reject missing or ambiguous values. |
availability | Preserve in_stock, explicit out_of_stock, or unknown; never derive stock from a missing record. |
market, market_basis | Retain a market only when page shipping metadata supplies it; this is page context, not a selected destination. |
fulfilment, offer_kind, purchase_basis, offer_context | Keep fulfilment unselected, purchase basis unspecified and the listed public offer context; no one-time/subscription price is inferred. |
observed_at_utc | Record when the observation was collected; this is not a retailer publication or update date. |
requested_source_url, observed_source_url | Retain the requested URL and the accepted source URL separately. |
source_evidence | Retain bounded field paths and literal source tokens supporting the selected identity, pack, price and stock signal. |
Store price values without converting them through a binary floating-point calculation. Compare only observations with the same identity, pack, currency and offer context. A collector cannot repair a missing currency by guessing it from a country label.
Reproduce the fixture failure and the bounded collector
Use CPython 3.12.11; the packet uses only the standard library. Download the archive, extract it and run:
unzip retail-sku-observations.zip
cd retail-sku-observations
PYTHON=python3 ./run.sh
In the captured run, python3 resolved to CPython 3.12.11. Set PYTHON to your 3.12.11 executable if its name differs; the runner checks the exact version. run.sh runs the scoped tests and captures JSON, CSV and change-report outputs from owned fixtures and a saved field projection. The test process rejects external connection attempts; the demo invokes local-file CLI replays. A configured API key does not make fixture replay dispatch a managed request.
The failing naive choice is the fictional group’s “from” price. It belongs to a sample, rather than the selected weight offer. expected_output/naive-price.json captures:
{
"currency": "USD",
"matches": false,
"naive_group_low_price": "2.00",
"scope": "owned fictional data; aggregate sample price cannot be assigned to the selected weight offer",
"selected_SKU": "000701002",
"selected_offer_price": "8.25",
"selected_pack": "3oz"
}
The captured runner summary is:
Runtime: cpython 3.12.11; standard library only
Default scope: owned fixtures and saved factual projection; no external/provider/model calls
Owned baseline: 2/2 selected offers accepted; complete=true
Owned changed: 2/2 selected offers accepted; comparable field changes=2
Owned partial: 1/2 selected offers accepted; unobserved prior records=1
Changed-pack control: 1/2 accepted; reason=pack_identity_changed; prior pack retained=1
Missing observations make no out-of-stock or disappearance claim; prior timestamps retained
Naive from/sample price=2.00; selected 3oz offer=8.25 USD
Adagio saved projection: 2/2 literal offers match independent source oracle; no fresh HTTP request
Recorded managed observation: 1 request; API/target=200/200; 2/2 accepted; receipt credits=1; credits per accepted record=0.50
Recorded managed data are read locally; standalone managed CLI excluded and no paid observation rerun
Fixture prices and timestamps are controlled inputs. They are not current retailer observations. The tests compare full records against literal expected values; JSON and CSV retain the leading zeroes in source IDs.
The controls should establish both that acquisition finished within the selected limits and that the returned data satisfied the source contract. HTTP 200 alone does not establish field acceptance. An error shell, missing price or ambiguous variant must fail acceptance even when its transport status is successful.
Here is the first record excerpt from expected_output/baseline.json:
{
"availability": "in_stock",
"currency": "USD",
"fulfilment": "unselected",
"market": "unknown",
"market_basis": "unspecified",
"observed_at_utc": "2026-10-13T11:00:00Z",
"observed_source_url": "https://scrapingant.github.io/scrapingant-examples/examples/retail-shelf-intelligence-scraping-digital-shelves-for-cpg/fixtures/baseline.html",
"offer_context": "owned fictional USD offer",
"offer_kind": "listed_variant_offer",
"pack_quantity": 3,
"pack_unit": "ounce",
"price": "8.25",
"product_id": "000701",
"purchase_basis": "unspecified",
"requested_source_url": "https://scrapingant.github.io/scrapingant-examples/examples/retail-shelf-intelligence-scraping-digital-shelves-for-cpg/fixtures/baseline.html",
"sku": "000701002",
"source_evidence": {
"availability_token": "https://schema.org/InStock",
"offer_path": "$.hasVariant[1].offers",
"pack_label": "3oz",
"price_token": "8.25",
"product_path": "$.hasVariant[1]"
},
"variant": "3oz",
"variant_id": null
}
The expected records are literal fixtures rather than values computed through the parser. The source investigator supplied a separate factual oracle for the direct retail observation. Controls alter material record fields and reject the mismatch. A test that computes its expected price using the same selection code could repeat the same mistake.
Keep acquisition and stock states separate
Acquisition completeness and the availability field answer different questions:
| Observed condition | Record or run state | Safe downstream interpretation |
|---|---|---|
| An explicit accepted stock token for the selected variant | in_stock or out_of_stock | Report that source observation in its retained context. |
| No recognized availability token | unknown | No stock assertion. |
| A required price, currency, pack or identity fails validation | Rejected record; incomplete run | Preserve the failure rather than manufacture a product observation. |
| A request fails transport/status checks | Acquisition failure; incomplete run | The selected SKU was not observed. |
| HTTP 200 returns an error shell that fails the source contract | Content rejected; no accepted record; incomplete run | The selected SKU was not observed; HTTP success does not imply stock. |
| A configured SKU is absent from a partial collection | Unobserved | Retain the prior observation without calling it current or out of stock. |
| Selected pack quantity changes | Rejected pack_identity_changed; incomplete run | Retain the prior pack observation; do not report a comparable price change. |
The controls include an HTTP 200 error shell, missing price/currency, duplicate or conflicting selected SKUs, unknown availability, explicit out-of-stock offers, a failed acquisition and a changed pack. A complete result with availability="unknown" still makes no stock assertion.
Compare repeated snapshots conservatively
Use the accepted records to compare like-for-like offers. Emit a price change only when the required identity and context match and both snapshots contain accepted decimal prices. Preserve availability-field changes as observed states. A transition to or from unknown does not establish a stock-in or out-of-stock event.
The complete changed fixture produces this expected_output/changes.json:
{
"changes": [
{
"fields": {
"availability": {
"after": "out_of_stock",
"before": "in_stock"
},
"price": {
"after": "8.75",
"before": "8.25"
}
},
"sku": "000701002",
"variant": "3oz"
},
{
"fields": {
"availability": {
"after": "unknown",
"before": "out_of_stock"
}
},
"sku": "000701004",
"variant": "16oz"
}
],
"current_complete": true,
"new_observations": [],
"scope": "two selected offers; no inventory/disappearance inference",
"unobserved_prior_records": []
}
The 3-ounce fixture price changes from 8.25 to 8.75 and its explicit token changes from in stock to out of stock. The 16-ounce offer becomes unknown; that is loss of an explicit stock signal. In the partial fixture, the 16-ounce offer is missing and its prior record remains in unobserved_prior_records with timestamp 2026-10-13T11:00:00Z. A changed 16-to-12-ounce pack fails the fixed identity check. Neither control manufactures a disappearance or stock event.
The result is a small observation and change dataset. It does not establish retailer-wide assortment completeness, cross-retailer matching, share of shelf, revenue effects or MAP compliance. Those tasks need additional inputs and definitions beyond this collector.
Add managed retrieval only when the source needs it
If permitted direct HTTP or an adequate product feed provides the fields, ScrapingAnt is not needed for this task. If your inspected source requires managed HTML retrieval, keep the same parser and field acceptance rules when changing the transport.
The general endpoint returns HTML. Set browser, proxy_type, proxy_country, return_page_source and timeout explicitly; the examples use timeout=30 seconds. browser=false requests raw retrieval. browser=true with return_page_source=false requests JavaScript-rendered HTML. The customer parser owns product-field extraction. These are documented interface choices; choosing rendered mode does not prove JavaScript was necessary for the selected source. See the request and response format.
Send the API key in the x-api-key request header and keep it out of logged request URLs. The optional CLI reads SCRAPINGANT_API_KEY from the environment; it prints normalized records and a safe receipt rather than the returned HTML. Its fixture tests check the explicit request settings, header authentication and encoded credential echoes.
From the extracted folder, the optional direct command makes one request to the fixed Adagio product page:
python3 collector.py direct
The optional managed command makes one paid request to that same fixed page. Supply SCRAPINGANT_API_KEY through your environment before using it:
python3 collector.py managed --browser false
These network commands are excluded from run.sh. They accept no arbitrary retailer URL, and add no retries or schedule. The standalone managed CLI has offline fake-response coverage; the recorded live check below exercised the native collector functions through a bounded controller.
The documented non-Google datacenter retrieval costs are:
| Explicit mode | Documented request cost |
|---|---|
browser=false, proxy_type=datacenter | 1 API credit |
browser=true, return_page_source=false, proxy_type=datacenter | 10 API credits |
These values come from the credit schedule. They describe retrieval cost, not accepted records. Record the Ant-credits-cost response header when present; a missing or malformed receipt leaves the actual request cost unknown. Do not infer zero credits from an absent receipt. Keep the API status, target status, acceptance result and actual receipt separate. The error documentation distinguishes the outer API HTTP status from the target status reported by Ant-page-status-code.
The default runner made no external or paid API calls. A separate bounded managed check on October 13, 2026 at 10:59:39 UTC made one raw datacenter request for the selected Adagio page. The collector accepted both selected offers: 001036002, 3oz, 9.00 USD, out_of_stock; and 001036004, 16oz, 29.00 USD, out_of_stock. These matched the separately inspected direct-source oracle.
| Managed observation | API / target status | Accepted offers | Actual receipt |
|---|---|---|---|
One raw request, browser=false | 200 / 200, text/html | 2 / 2; complete | 1 API credit |
For this one observation, the receipt divided by accepted offers is 0.50 credits per record, with no unknown charge. It is not a production yield estimate. The check exercised the collector’s acquire and collection functions through a bounded controller. The public managed CLI was tested with offline fakes; a live CLI invocation was not performed. Direct HTML already supplied the same selected fields, so this observation does not establish a need for managed retrieval or rendering. The receipt and accepted records preserve the measured result. Its provenance records the executed source snapshot; the packet strengthens validation of saved projections and normalizes malformed HTML parser failures after that capture.
For a first managed test, select one permitted URL and verify its source identity, selected variant and required fields. Repeat only a bounded workload after that acceptance check. If managed retrieval is needed, create an account and review the current pricing; the field validator remains your code.
Limits and review status
The parser recognizes the fixed Adagio product group and its two selected weight SKUs, plus the fictional fixture equivalent. It requires one matching canonical URL and one ProductGroup JSON-LD object. It does not implement arbitrary JSON-LD graphs, product discovery or another retailer’s selectors.
Acquisition uses one attempt, a 262,144-byte response cap, a 30-second socket timeout and elapsed-time checks before and after bounded reads. A blocking read can finish before the elapsed check rejects it; this is not a promise of an exact 30-second total runtime. Direct HTTP and API endpoint redirects are rejected; the provider's remote target redirect behavior was not independently validated. There are no retries or automatic rendering/proxy fallbacks. Transport fixtures cover target statuses, media types, byte limits, errors and unknown receipts; mocked headers are not paid measurements.
Owned fixture success does not demonstrate arbitrary retailer reliability. Source layouts and offer semantics need their own validation; source evidence should be sufficient to audit the fields without redistributing full retailer page bodies. There is no scheduler or automatic variant matching in this example. The public field projection cannot regenerate the hash of the withheld original retailer HTML, and replay does not create a fresh source observation.
Examples tested on 2026-10-13 with CPython 3.12.11 and the standard library. Code: Code and evidence.
This article was drafted with AI assistance from a tested evidence packet and independently reviewed by an AI assistant. Publication was authorized by the named author, who is responsible for the code, measurements and corrections.