Collect Employer Job Postings with Python and ScrapingAnt
This draft replaces broad labor-market and scraping-provider claims with a bounded job-posting collector, captured fixtures and explicit limits. The original URL, publication date and banner are preserved. Download the reproducibility packet.
A job-posting refresh needs more than a list of titles. It needs stable posting IDs, source URLs, validated fields, and a way to distinguish a complete collection from a failed second page. Otherwise, a transient failure can turn into a false disappearance alert.
This walkthrough collects one employer listing that you are authorized to access. The runnable pagination and refresh example uses a fictional, owned careers site. Separate evidence includes an official Greenhouse JSON alternative and one accepted raw managed retrieval of a selected published Greenhouse posting. ScrapingAnt is an optional HTML transport; parsing and validation remain explicit Python code.
Start with the source and required fields
Choose one authorized employer careers URL. Inspect its official API/feed and any existing structured data before writing selectors. If a permitted feed provides the fields you need, use it. If direct HTTP returns the complete permitted HTML, the direct collector is sufficient.
Greenhouse's official Job Board API documentation describes published-job GET endpoints as public and unauthenticated. Its listing endpoint is GET /v1/boards/{board_token}/jobs. On October 11, 2026 at 21:03:15 UTC, a direct public GET of the Greenhouse company's board returned HTTP 200 and 19 job posts, matching meta.total=19.
The packet retains a projection of that genuine response and replays it without fetching the live board. It keeps updated_at as source_updated_at, supplies the employer label explicitly, and leaves the unavailable posting date null. A separate application URL was not verified from the retained list, so the replay leaves applicationURL null and labels all 19 rows as candidates unsuitable for the validated refresh sink. This feed observation does not establish managed employer HTML retrieval or a universal ATS interface.
For the owned HTML listing example, the required record is:
jobID: Nonblank, matches this employer'sNS-000pattern.title,employer,location: Nonblank; employer matches the configured source.sourceURL: Approved origin and job path matching the ID.applicationURL: Approved application-page path matching the ID; only recorded, never submitted.observedUTC: UTC collection timestamp, distinct from source dates.sourcePostedDate: Valid source-provided date or null; never inferred from observation time.
Keep IDs scoped to their source. This example does not resolve the same vacancy across employers or job boards.
Run the owned HTML example
Use CPython 3.12.11. The packet uses the standard library and requires no package installation or API key. Download the archive, extract it, and run:
unzip employer-job-collection.zip
cd employer-job-collection
PYTHON_BIN=python3.12 ./run.sh
Set PYTHON_BIN to your Python 3.12.11 executable if its name differs. run.sh starts a temporary loopback server, runs the tests, captures outputs, and validates them against independently authored literal records. The default runner makes no live ATS, ScrapingAnt or model call, even if an API key is configured.
The captured summary is:
python=3.12.11
first_page_only: pages=1 unique_jobs=2 oracle_matches=false
bounded_collection: pages=2 cards=4 duplicates=1 unique_jobs=3 oracle_matches=true
complete_refresh: added=NS-004 changed=NS-001:title,location absent=NS-002
partial_refresh: failures=http_503 removed=0 retained_previous=NS-002,NS-003
managed_api_calls=0 paid_calls=0
greenhouse_replay: candidate_posts=19 validated_for_refresh=false missing_required=applicationURL
Stopping after the first page finds two fixture jobs. Following the validated two-page sequence sees four cards, including one repeated posting, and produces three unique records that match the full-record oracle. The test suite has 50 passing tests, including a control where a plausible but wrong title fails the independent oracle.
Here is a captured record from the owned fixture. The loopback port varies between runs:
{
"applicationURL": "http://127.0.0.1:55325/careers/apply/NS-002",
"employer": "Northstar Demonstration Labs",
"jobID": "NS-002",
"location": "Remote - UK",
"observedUTC": "2026-10-11T21:14:49Z",
"sourcePostedDate": null,
"sourceURL": "http://127.0.0.1:55325/careers/jobs/NS-002",
"title": "Data Analyst"
}
SourcePolicy and ListingParser recognize only this fixture's /careers/roles?page=1 contract, versioned listing envelope, page sequence, IDs and links. Before collecting another employer or ATS, replace those adapters after permission and schema validation, then create source-specific expected records. A transport change cannot supply missing selectors or establish source completeness.
Treat partial collection as a distinct result
The collector limits acquisition to max_pages=3 in the demo. It validates each page number and the declared page total, follows only the expected next URL on the configured origin/path/filter set, and rejects redirects in its direct HTTP transport. It stops on an unsafe next link, a loop, inconsistent pagination, an exhausted budget or a transport failure.
Identical duplicate IDs produce one record. Conflicting versions of an ID are excluded from the acquired records, and the run becomes partial. Missing required fields also make the run partial. An HTTP 200 error page has no accepted listing envelope; it is not an empty job inventory. A legitimate empty fixture carries an explicit empty state.
These are the actual loopback controls. unique_records counts accepted distinct records, not HTTP successes:
| case | requested_pages | unique_records | complete | failure |
|---|---|---|---|---|
| baseline | 2 | 3 | true | none |
| hostile_next | 1 | 2 | false | next_url_rejected |
| loop | 1 | 2 | false | pagination_loop |
| page_budget | 1 | 2 | false | page_budget |
| conflict | 2 | 2 | false | conflicting_id |
| missing_location | 2 | 2 | false | record_invalid |
| denied | 2 | 2 | false | listing_envelope |
| http_fail | 2 | 1 | false | http_503 |
| empty | 1 | 0 | true | none |
| empty_unconfirmed | 1 | 0 | false | unconfirmed_empty |
The CLI returns 0 for a complete validated acquisition and 2 for a partial one. Inspect complete, counters and failures before updating a downstream inventory. Outputs report requested/fetched/accepted pages, cards, valid/rejected cards, duplicates, conflicts and unique records; HTTP status alone is insufficient.
Refresh without false disappearance alerts
The complete refresh fixture changes NS-001's title and location, adds NS-004, and no longer lists NS-002. refresh(previous, current) emits field-level before/after alerts and marks NS-002 absent from that complete source listing.
The failure fixture updates NS-001 on page one, then returns HTTP 503 on page two. Its refresh keeps the previously observed NS-002 and NS-003, emits no removals, and sets removals_suppressed=true. Retained records keep their earlier observation timestamps. Valid newly observed records can still update the snapshot; the snapshot remains partial.
A complete run's removed_ids means absent from the validated source listing. It does not establish that a role was filled or that employment changed. A timestamp-only refresh produces no content-change alert. Review an incomplete run's failure before using its counts as a current inventory.
Optional managed HTML acquisition with ScrapingAnt
If an authorized source needs managed HTML retrieval, keep the same validation boundary around the acquired content. ScrapingAnt's /v2/general endpoint returns page HTML; it is not the structured job extractor in this example.
The packet's optional transport sets all of these explicitly:
| Parameter | Value |
|---|---|
browser | false for HTML without JavaScript rendering; explicitly true when rendered HTML is needed |
proxy_type | datacenter |
proxy_country | us |
timeout | 30 seconds |
return_page_source | false |
The packet's code reads the API key from SCRAPINGANT_API_KEY and sends it in a header, keeping it out of request URLs and error messages. The code checks the API response and target status, uses the original target content-type header when present, and rejects an explicit non-HTML target type. A missing target content-type leaves validation to the required listing schema. An outer response content-type alone does not establish the target's format. Its redirect guard controls API-endpoint redirects; it does not establish how a remote target redirect is handled.
One separate call through the unchanged managed_html transport fetched an existing owned raw fixture from 2026-10-11T21:38:39.695211 to 21:38:40.432850 UTC, using the explicit raw parameters above. API and target statuses were 200, the credit receipt was 1 API credit, and the captured HTML was 752 bytes. Its initial Web Scraping is hard marker and full body matched the pinned fixture source. The receipt, response body and source pins are included in the reproducibility packet.
This observation validates owned raw HTML transport and marker acceptance. It does not validate an employer/ATS HTML source, the job parser, pagination, managed CLI or JavaScript rendering. The default runner stays offline and does not repeat this call.
One selected published posting: raw HTML to a validated record
On October 12, 2026, from 09:42:19.062837 to 09:42:20.237010 UTC, a separate raw request acquired Greenhouse posting 8126988 through managed_html with the explicit parameters above. API and target statuses were 200, the observed receipt was 1 API credit, and the response was 104,223 bytes. A bounded source-specific adapter accepted exactly one literal record:
jobID:8126988;title:Commercial Counsel.employer:Greenhouse;location:Eastern Timezone.sourceURLandapplicationURL: the same HTTPS posting page linked above.observedUTC:2026-10-12T09:42:20Z;sourcePostedDate: null.
The observed form uses method get and an action resolving to that hosted page. Recording it as applicationURL establishes the hosted form page, without validating a distinct submission endpoint or submitting an application. The HTML had no posting-date marker, so the adapter does not borrow dates from the API. Its canonical URL used HTTP with the same host/path/query; the adapter records this discrepancy and keeps the source and application URLs HTTPS.
This ordinary public page was linked by its official public job API and inspected directly without authentication, a redirect or a challenge. Public JSON access is not blanket permission for arbitrary HTML. Both that API and direct raw HTML already supplied relevant facts, so the managed call demonstrates an optional acquisition path.
The separate greenhouse_single_post.py adapter checks one title, location, document title, matching Open Graph fields, canonical URL and a closed matching application form. Acceptance also requires an independently authored literal record, ignoring only observation-time differences. complete=true applies to one selected posting page, with max_pages=1 and board_inventory_complete=false. Empty, missing, challenge or truncated HTML fails this detail-page contract.
The packet adds 46 offline tests for this adapter and receipt controls, alongside the original 50 tests and 10 listing cases. They reject coherent field drift, unsafe application URLs, ambiguous credit receipts and encoded key echoes; harmless wrapper changes may pass. Its public CLI parses the short owned fixture or verifies a local file and makes no provider request. The recorded managed acquisition used a separately reviewed bounded helper; the public CLI was not used for that call.
Only safe receipt and field projections are distributed. The full employer HTML and API bodies are withheld. The measured HTML SHA-256 is b8385bbc2da7e5d34e469dad34117c059449bb68786df0e5063dc21ed09bb886; neither the projection nor the hand-authored structural fixture can reproduce that full-body hash. This one raw result does not establish employer inventory, pagination, refresh, JavaScript rendering, generic ATS support or production reliability.
After replacing and validating the source adapter, this documented opt-in command illustrates the managed path. It was not run for this packet and can consume credits. Set CAREERS_URL to the single source URL you have authorized and tested:
: "${SCRAPINGANT_API_KEY:?set it in your environment}"
: "${CAREERS_URL:?set one authorized careers URL}"
python3.12 collector.py --start-url "$CAREERS_URL" --max-pages 3 \
--transport scrapingant --browser false --allow-live-api
Use --browser true when the source requires JavaScript rendering, then validate that returned HTML against the source-specific schema. The bundled listing adapter still rejects URL shapes outside its known contract. Offline mocks verify request parameters and acceptance checks; the separate one-posting observation supplies the narrowly scoped live result above.
For ordinary non-Google targets, the documented credit schedule lists 1 API credit for a request without a browser through a datacenter proxy, and 10 API credits for JavaScript rendering through a datacenter proxy. The separate owned and selected-post raw receipts each recorded 1 credit. These two observations do not estimate yield or cost for other jobs or sources.
Try one authorized careers URL, inspect the required fields and pagination, and trial that bounded workload only if managed retrieval is needed. As checked on October 11, 2026, the signup offer lists 10,000 monthly free API credits with no credit card; the current plans list Enthusiast at $19/month for 100,000 credits. Recheck current plans before choosing capacity. An official feed or working direct HTTP collector may already complete your task.
Limits of the evidence
The HTML fixture proves the demonstrated parser, page bounds and refresh behavior on owned inputs. It does not prove live ATS rendering, arbitrary Indeed/LinkedIn support, production reliability, success rates or a ready-made jobs database. No applications, profiles, skills classification or wage inference are collected.
The Greenhouse list and selected-post evidence contain retained fields and capture metadata. Their recorded full-response hashes identify bodies that are not distributed. Saved projections have separate hashes and cannot reproduce the original full-response hashes. Replays validate the retained fields and owned structural inputs; they do not re-acquire the withheld source or establish a current live inventory.
Check the adapter again when a source changes its layout, page contract or required fields. Keep incomplete runs visible and avoid downstream removal alerts until the source passes the full validation gate.
Examples tested on October 11 and 12, 2026 with CPython 3.12.11, standard library only. Code and captured outputs: reproducibility packet.
This article was drafted with AI assistance from a tested evidence packet. Publication approved by Oleg Kulyk on October 12, 2026.