Python Requests Cookies: Sessions, Scope and Scraped Data
Replaced untested persistence and recovery recipes with a runnable Requests cookie experiment. The new examples test redirect cookies, path collisions, prepared requests and session restoration against extracted records. Corrected unsupported cookie keyword arguments and the claim that mounting an HTTPS adapter forces HTTPS. The original URL, publication date and video are preserved.
Use a requests.Session() when later requests need cookies received earlier. Keep its cookie jar intact: a cookie is identified by more than its name, and a plain dictionary cannot represent two region cookies with different paths. For a single call, the cookies= parameter can supply cookies without making them persistent session state.
We tested those distinctions against a small catalog. A dictionary backup retained the member session but changed the selected currency from EUR to USD. The request returned HTTP 200 and all four expected product IDs. Only checking the currency and prices caught the wrong dataset.
This guide gives you a complete save/restore example and the failures behind it. The catalog, identities and prices are synthetic; the measurements demonstrate mechanisms, not success rates on real websites.
Choose the cookie object that matches the task
Requests documents session persistence and the difference between session-level and per-call state. These distinctions matter before you write a persistence helper:
| Task | API | Important distinction |
|---|---|---|
| Reuse cookies across calls | Session() and session.cookies | The jar accumulates accepted response cookies |
| Supply cookies for one request | session.get(url, cookies=jar) | The supplied jar does not itself become persistent session state |
| Inspect one response | response.cookies | This is not necessarily the accumulated jar from a redirect chain |
| Look up a scoped cookie | requests_jar.get(name, domain=host, path=path) | Name-only lookup can be ambiguous |
| Delete one scoped cookie | jar.clear(domain=host, path=path, name=name) | Choose the stored domain/path tuple you intend to remove |
| Prepare a request with session state | session.prepare_request(request) | Bare request.prepare() does not apply the session's cookies |
The name-based lookup helper in this table belongs to RequestsCookieJar, the default jar in a new Session. The persistence example below restores a standard-library LWPCookieJar; that jar sends the restored cookies but does not provide Requests' get() helper. Iterate its Cookie objects when inspecting names, domains, paths and expiry. Avoid depending on the private _cookies structure, and avoid printing real cookie values. The cookie-jar API exposes the operations directly.
Run a complete session save-and-restore example
Download or check out the tested packet, then change into examples/manage-cookies-python-requests. The fixture, persistence helper and record oracle are included. With Python 3.12 installed:
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-lock.txt
python quickstart.py
./run.sh
The recorded runs used Python 3.12.10 and Requests 2.34.2 on macOS 26.6.2 arm64. The lockfile records the complete installed dependencies. The example starts an HTTP server on loopback with a temporary port; no real account or API key is involved. It uses HTTP only for that synthetic local fixture. Use HTTPS and normal certificate verification for real authentication traffic.
Here is the complete quickstart.py. CatalogFixture, the jar helpers and score() are supplied by the packet; Session is the ordinary Requests API.
"""Complete no-secret example: seed, save scoped cookies, restore and extract."""
from datetime import datetime, timezone
import json
from pathlib import Path
import tempfile
from requests import Session
from catalog_fixture import CatalogFixture
from cookie_state import load_jar, save_private_jar
from oracle import score
from requests_matrix import environment
def main():
with CatalogFixture() as fixture, tempfile.TemporaryDirectory(prefix="requests-cookie-demo-") as tmp:
snapshot = Path(tmp) / "cookies.lwp"
with Session() as first:
first.trust_env = False # The local demo ignores ambient proxies/.netrc.
final = first.get(fixture.base_url + "/login", timeout=5)
final.raise_for_status()
accumulation = {
"redirect_responses": len(final.history),
"final_response_cookies": len(final.cookies),
"accumulated_session_cookies": len(first.cookies),
}
# New-file only; opt in to saving otherwise-discarded session cookies.
save_private_jar(first.cookies, snapshot, include_session=True)
with Session() as restored:
restored.trust_env = False
# Opt in at load too. Expired cookies remain excluded at both steps.
restored.cookies = load_jar(snapshot, include_session=True)
response = restored.get(fixture.base_url + "/catalog/eu/products", timeout=5)
response.raise_for_status()
records = response.json()["records"]
result = {
"tested_at": datetime.now(timezone.utc).isoformat(),
"environment": environment(), "cookie_accumulation": accumulation,
"restored_cookie_count": len(restored.cookies),
"status": response.status_code, "records": records, **score(records),
}
assert result["exact_match"], "The records differ from the literal expected dataset"
print(json.dumps(result, indent=2)) # Records and counts; no cookie values.
if __name__ == "__main__":
main()
The fixture's /login route redirects once. That redirect sets the session cookie; the final /configure response sets two same-named region cookies on different paths. The snapshot therefore contains three cookies, even though the final response contains only two.
Selected fields from the captured output:
{
"cookie_accumulation": {
"redirect_responses": 1,
"final_response_cookies": 2,
"accumulated_session_cookies": 3
},
"restored_cookie_count": 3,
"status": 200,
"matching_records": 4,
"exact_match": true
}
The restored session retrieves four member/EUR records. The full capture contains their SKU, currency and integer minor-unit prices. The assertion compares the records with an independent literal dataset; it does not derive the expected answer from the server's price table.
The demo disables ambient proxy and .netrc lookup using trust_env=False so the localhost experiment is self-contained. That is a fixture choice, not a recommendation to discard your production environment configuration. Both sessions close, and the temporary snapshot is removed on exit.
What the extraction matrix found
We ran 12 extraction cases in three rounds: 36 extraction observations, plus 18 diagnostic observations, for 54 matrix checks. Every extraction returned HTTP 200 and four rows. Each table row below has three observations, with the same outcome each time.
| Cookie handling | Correct member/EUR records | Observed result |
|---|---|---|
| Separate short-lived sessions | 0/4 | Retail USD; state did not carry between calls |
| One persistent session | 4/4 | Member EUR |
Complete jar passed in cookies= | 4/4 | Correct for that call; receiving session stored no cookies |
| Follow-up without that per-call jar | 0/4 | Retail USD |
Copy only final response.cookies | 0/4 | Retail EUR; redirect's session cookie missing |
| Restore the complete LWP jar | 4/4 | Member EUR |
get_dict() then rebuild a jar | 0/4 | Member USD; path distinction lost |
Bare Request.prepare() | 0/4 | Retail USD despite cookies existing in the Session |
Session.prepare_request() | 4/4 | Member EUR |
| Session cookie given a past expiry | 0/4 | Retail EUR; expired session cookie not sent |
| Session cookie cleared | 0/4 | Retail EUR |
| Server-revoked session | 0/4 | Retail EUR although the session cookie was still received |
The independent-request control uses a separate short-lived Session for each call with ambient environment lookup disabled. It does not measure a separate implementation of top-level requests.get().
See the report and raw matrix. A passing matrix check means the observed outcome matched the case being tested, including an intentional failure. It does not mean every strategy returned correct data.
The record oracle matches each expected SKU/currency/price tuple once and requires the whole dataset to match. Repeating one correct product four times cannot substitute for four different products; an extra duplicate also fails exact matching. Repeated runs of this deterministic fixture do not estimate real-world failure frequencies.
Why the dictionary backup changed the currency
The fixture issues region=EUR for /catalog/eu and region=USD for /catalog/us. The intact jar sends the EUR preference to /catalog/eu/products and the USD preference to the other catalog. Those are two cookies, even though they share a name.
In the measured seed order, get_dict() reduced the three-entry jar to two name/value entries, retaining the USD region value. Rebuilding a jar from that dictionary lost the original path distinction. The EUR catalog URL then received the USD preference and returned member/USD prices. The URL's spelling did not force the server to select EUR.
This is a reason to preserve cookie objects, not a promise that every dictionary conversion produces USD. The Requests implementation explains the name-based mapping; the captured request establishes what happened in this fixture.
Our name-only lookup on the original RequestsCookieJar, jar.get("region"), also raised CookieConflictError. The explicit domain/path lookup selected EUR, and clearing that tuple left the USD cookie intact. A successful dictionary conversion can therefore hide a distinction that an ambiguous lookup would have exposed.
Persist a jar without silently renewing the session
The packet's persistence helper uses the standard library's LWPCookieJar. It preserves the scoped cookies needed by this example. It creates a new file with owner-only POSIX permissions, refuses to overwrite an existing path, and checks that a loaded snapshot is a regular file with restricted POSIX permissions.
Session cookies are deliberately opt-in: the quickstart uses include_session=True at both save and load. Expired entries remain excluded. That lets the example resume a valid synthetic session without pretending that serialization renews it. The separate revocation case proves the distinction: the server received the same session cookie and still returned retail data.
An LWP file is not encrypted. Treat a real snapshot as a credential, keep it outside version control and logs, and choose a private directory. This helper is a narrow example, not a cross-platform secret store; Windows file-permission behavior was not tested.
Prepare requests and diagnose failures before retrying
If you manually prepare a request, use Session.prepare_request() when you need the session's cookies. In our test, Session.send() did not repair a request that had already been prepared without them. For production environment-dependent settings, follow Requests' prepared-request guidance, including its separate environment-settings step.
Two other diagnostics correct the earlier examples. Passing httponly= or samesite= directly to session.cookies.set() raised TypeError in the tested version. Mounting an adapter on https:// left the http:// adapter available, and an HTTP fixture request still succeeded. Neither snippet established the security behavior previously claimed.
Requests cookie storage is not a browser environment. These tests do not establish browser SameSite enforcement, JavaScript execution, SSO, MFA or portability of real login sessions. A retry loop that merely checks for HTTP 200 would accept every wrong dataset in the table. Before retrying, determine whether the missing state is a preference, a request-preparation error or a session the server no longer accepts.
Receive and resend target cookies with ScrapingAnt
When moving page retrieval to ScrapingAnt, keep the same target-selection discipline. ScrapingAnt accepts cookies as name/value pairs through the cookies parameter, with multiple pairs separated by semicolons. ScrapingAnt's extended endpoint returns received cookies in its cookies field, so you can pass the relevant pairs into a later request.
We verified that receive/replay pattern in a separate September 27 cookie-echo experiment, published with the Selenium evidence packet. It made four calls without a browser and four with a browser, using datacenter settings: 44 credits total, with all eight receipts captured. Both modes received and explicitly replayed the synthetic pairs. A separate request without replay returned no demo cookies. These are reused transport results; the local Requests catalog was not fetched through ScrapingAnt.
The product rules for those settings are: A request without a browser through a datacenter proxy costs 1 API credit. A request with JavaScript rendering through a datacenter proxy costs 10 API credits.
Do not send an entire multi-site jar as get_dict(). Select the pairs appropriate to the intended target. The API's name/value string does not transfer the jar's domain, path or expiry metadata, and replaying it does not renew revoked authentication. ScrapingAnt is not needed for this local example or a direct Requests workflow that already retrieves the data. For service-based retrieval with explicit cookie chaining, use the cookie guide and extended response documentation.
Reproduce the result and apply the right checks
The README explains the matrix, offline checks and saved-results summary. ./run.sh uses local fixtures only and makes no paid API calls. Rerun files go into ignored run_output/; the committed captures stay unchanged.
For a production collector, check the data contract after restoring or changing state: account role, currency, required fields, unique product IDs and plausible values. The fixed prices here are test fixtures, not production constants. A valid session is one input to correct extraction, not the final acceptance check.
For browser-managed state, see Playwright cookies and Selenium cookies. If you are choosing the HTTP client itself, see Requests versus HTTPX.
Earlier video walkthrough
The original video is retained below. Use the tested examples and corrections above for this revision.
Examples tested on 2026-09-27 with Python 3.12.10 and Requests 2.34.2. Code and captures: pinned evidence packet.
This article was drafted with AI assistance from a tested evidence packet. The named author is responsible for the code, measurements and corrections.