How to Create a Proxy Server in Python Using Proxy.py

Updated 2026-09-22

Re-tested HTTP forwarding, HTTPS CONNECT, authentication and plugins with proxy.py 2.4.10. Replaced the broken rotation recipe with a tested upstream pool. HTTPS forwarding does not require TLS interception, and local proxy ports do not establish different public exit IPs. Commands, fixtures and captured logs.

To create a local Python proxy server, start proxy.py on an explicit loopback address and point your client at it. Then verify both the response and the proxy log: receiving a page is not enough to prove which route the client used.

This guide builds that path with local HTTP and HTTPS origins, cURL and Requests clients, authentication, a header plugin and an upstream pool. You can run the complete lab without an external target, a ScrapingAnt account or an API key.

Install proxy.py and start the local lab

The recorded environment was Python 3.12.11 on macOS 26.6.2, with proxy.py 2.4.10, Requests 2.34.2 and cURL 8.7.1. Use Git, Python 3.12, Bash, cURL and OpenSSL for this reproduction. Keep loopback ports 8000, 8443 and 8899–8905 free.

Get the pinned packet and install its dependencies:

git clone https://github.com/ScrapingAnt/scrapingant-examples.git
cd scrapingant-examples
git checkout 53aea6a71ce869c648f1fb31dcf8c384a73bb6fd
cd examples/python-proxy-server-proxy-py
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python lab.py serve

The requirements file pins proxy.py and Requests, including their installed dependencies. Wait for the READY line. The helper creates two JSON origins and the proxy listeners used below:

Address	Purpose
http://localhost:8000	HTTP origin
https://localhost:8443	HTTPS origin with a generated test certificate
http://127.0.0.1:8899	Plain forward proxy
http://127.0.0.1:8900	Authenticated proxy
http://127.0.0.1:8901 and :8902	Two local upstream proxies
http://127.0.0.1:8903	Pool proxy
http://127.0.0.1:8904	Header-plugin proxy

Port 8905 is reserved for reproducing the old recipe's failure. Keep the lab running in this terminal. In a second terminal, enter the same packet directory and activate .venv before running the clients.

For reference, this is the plain proxy's startup command, with the captured virtualenv executable path shortened to python:

python -m proxy --hostname 127.0.0.1 --port 8899 --num-workers 1 --num-acceptors 1 --log-level INFO

The helper already starts that process; do not start it again while the lab is running. An explicit 127.0.0.1 bind keeps this example on loopback. The initial log confirms the HTTP proxy plugin loaded; the access log appears after a client request.

Send an HTTP request through the proxy

Run this cURL command from the packet directory:

curl --noproxy '' --silent --show-error --fail --verbose --max-time 5 --proxy http://127.0.0.1:8899 http://localhost:8000/curl-False

Captured response body:

{"marker": null, "origin": "proxy-py-local-fixture", "path": "/curl-False", "tls": false}

The origin identifies our fixture, and path identifies this request. The matching line in expected_output/plain.log was:

2026-09-22 20:24:19,075 - pid:9686 [I] server.access_log:393 - 127.0.0.1:60456 - GET localhost:8000/curl-False - 200 OK - 235 bytes - 1.51ms

Your timestamp, process ID, client port and duration will differ. The destination and request path are what connect the client output to the proxy log. The recorded duration is a log field, not a performance benchmark.

Use --noproxy '' even for this local target so environment bypass settings cannot silently exclude it. The runner also clears inherited proxy environment variables. For more client-side options, see using cURL with a proxy.

Use Python Requests

The packet includes this complete client.py:

"""Usage: python client.py [http|https]; start `python lab.py serve` first."""
import sys
import requests

scheme = sys.argv[1] if len(sys.argv) > 1 else 'http'
if scheme not in {'http', 'https'}:
    raise SystemExit('Use http or https')
url = f'{scheme}://localhost:{8443 if scheme == "https" else 8000}/reader-client'
proxies = {'http': 'http://127.0.0.1:8899', 'https': 'http://127.0.0.1:8899'}
with requests.Session() as session:
    session.trust_env = False
    response = session.get(url, proxies=proxies, timeout=5, verify='generated/ca.pem')
    response.raise_for_status()
    print(response.status_code)
    print(response.text, end='')

Run it for HTTP:

python client.py http

Captured output:

200
{"marker": null, "origin": "proxy-py-local-fixture", "path": "/reader-client", "tls": false}

trust_env=False makes the client use the explicit settings in this example, and the timeout bounds its connection/read waits. The https dictionary key also points to an http:// proxy URL: it describes the proxy connection, not the destination's scheme. The Requests proxy guide covers the client configuration separately.

Forward HTTPS with CONNECT, without interception

The lab generates a test CA and a certificate for localhost. It passes that CA file to the clients explicitly; it does not install a CA into your system trust store. The proxy is started without interception certificate flags.

curl --noproxy '' --silent --show-error --fail --verbose --max-time 5 --proxy http://127.0.0.1:8899 --cacert generated/ca.pem https://localhost:8443/curl-True

These lines are excerpts from the captured verbose output:

> CONNECT localhost:8443 HTTP/1.1
< HTTP/1.1 200 Connection established
* SSL certificate verify ok.

The final origin response was:

{"marker": null, "origin": "proxy-py-local-fixture", "path": "/curl-True", "tls": true}

CONNECT establishes the tunnel; the client then performs TLS with the origin and verifies its certificate. This is enough for the demonstrated HTTPS request. Decrypting traffic at the proxy is a separate task and is outside this lab.

The equivalent Requests run is:

python client.py https

Captured output:

200
{"marker": null, "origin": "proxy-py-local-fixture", "path": "/reader-client", "tls": true}

The generated ca.pem belongs to this local fixture. Keep certificate verification enabled; do not use this lab CA as a general solution for unrelated HTTPS sites.

Require proxy authentication

The lab's port 8900 uses --basic-auth lab:secret. These are disposable fixture credentials. Its complete startup configuration, again using the activated interpreter, is:

python -m proxy --hostname 127.0.0.1 --port 8900 --num-workers 1 --num-acceptors 1 --log-level INFO --basic-auth lab:secret

It is already running under the helper. Test it with:

curl --noproxy '' --silent --show-error --fail --verbose --max-time 5 --proxy http://127.0.0.1:8900 --proxy-user lab:secret http://localhost:8000/auth-curl-False

Captured body:

{"marker": null, "origin": "proxy-py-local-fixture", "path": "/auth-curl-False", "tls": false}

To observe rejection, omit the credentials:

curl --noproxy '' --silent --show-error --fail --verbose --max-time 5 --proxy http://127.0.0.1:8900 http://localhost:8000/rejected-False-None

Captured error excerpt:

< HTTP/1.1 407 Proxy Authentication Required
curl: (22) The requested URL returned error: 407

The runner also tests wrong credentials and both failure cases over HTTPS, with both clients. In this recorded cURL build, all four rejected cURL requests exited 22. Requests returned an HTTP response with status 407 for HTTP and raised ProxyError containing Tunnel connection failed: 407 Proxy Authentication Required for HTTPS.

The rejected requests produced zero origin hits across eight attempts. Authentication success was checked separately for all four client/protocol combinations. These counts cover the declared test matrix; they are not a security audit.

Add a small HTTP header plugin

A plugin should produce something you can observe at the destination. Here is the working class from lab_plugins.py:

class MarkerPlugin(HttpProxyBasePlugin):
    marker = b"local-lab"

    def before_upstream_connection(self, request):
        if not request.is_https_tunnel:
            request.add_header(b"X-Lab-Proxy", self.marker)
        return request

The file imports HttpProxyBasePlugin from proxy.http.proxy. The helper makes the packet directory importable and loads the class on port 8904 with --plugins lab_plugins.MarkerPlugin.

curl --noproxy '' --silent --show-error --fail --verbose --max-time 5 --proxy http://127.0.0.1:8904 http://localhost:8000/marker

Captured body:

{"marker": "local-lab", "origin": "proxy-py-local-fixture", "path": "/marker", "tls": false}

The origin received the new header. For the HTTPS test through the same plugin, the origin returned "marker": null: the plugin skips CONNECT and does not rewrite headers inside the encrypted tunnel.

Route through an upstream pool

The earlier version of this article selected a proxy by assigning its address to request.host and request.port. We retained that algorithm in the packet solely as a negative test, substituting the two fixture upstream ports. It returned HTTP 400 and never reached the intended origin.

Use the release's ProxyPoolPlugin for the demonstrated chain. Its release-specific implementation opens a separate upstream connection and builds the request for that proxy.

The helper starts upstreams on 8901 and 8902 first. Each adds a distinct marker to plain HTTP requests. It then starts the pool with this configuration:

python -m proxy --hostname 127.0.0.1 --port 8903 --num-workers 1 --num-acceptors 1 --log-level INFO --plugins proxy.plugin.ProxyPoolPlugin --proxy-pool 127.0.0.1:8901 --proxy-pool 127.0.0.1:8902

For this pool test, keep the destination hostname localhost. In proxy.py 2.4.10, a literal private IP destination bypasses the pool. The packet tests that difference: http://127.0.0.1:8000/private-ip-bypass reaches the origin without either upstream marker.

To reproduce the batch, stop serve with Ctrl-C first, then run:

./run.sh

This starts its own listeners, checks each upstream individually and sends 20 pool requests using fresh Requests sessions. Every response must have the expected origin, request path and upstream marker. The selected upstream's access log must also contain that path.

Captured pool summary:

POOL: {"upstream-a": 10, "upstream-b": 10}; correct destinations=20/20

That split is an observation from this run. The plugin selects randomly when a client connection is created; it does not promise round-robin selection or a different endpoint on each request over a reused connection. Your split can differ. Both upstreams here run on one machine, so this proves routing through two ports, not rotation across different public IP addresses. No failover or throughput claim follows from these results.

Read failures and stop cleanly

The packet asserts failures as well as successes. A traceback in the captured negative case is expected; the overall runner still exits zero only when its assertions pass.

Symptom	Observed case or next check
cURL exit 7, connection refused	The controlled unavailable-listener case. Check that your proxy started and that the client's host/port match it.
Address already in use, proxy process exit 1	The packet deliberately starts a second proxy on occupied port 8899. Stop your earlier lab process or free the required port.
HTTP 407 or HTTPS ProxyError mentioning 407	Missing or incorrect proxy credentials in this lab. Check the proxy URL and authentication setting.
A response arrives but no expected access log appears	Check the explicit proxy address and bypass settings. Match the request path against plain.log and origin.jsonl.
The HTTPS fixture fails after being left running	Its generated certificate is valid for two days. Stop and restart the lab to generate a fresh certificate; keep the matching CA file.
The pool response has no upstream marker	Check whether the destination is a literal private IP; that branch bypasses this release's pool.

Press Ctrl-C in the serve terminal when finished. The helper stops the subprocesses it started and closes the origins. The automated run.sh does that cleanup itself. Do not run serve and run.sh together: they use the same ports and output files.

What was verified, and when a remote proxy is relevant

The recorded test matrix was:

Check	Result
cURL and Requests × HTTP and HTTPS	4 successful origin responses / 4 attempts
Same matrix with correct proxy credentials	4 successful origin responses / 4 attempts
Missing/wrong credentials × both clients × both protocols	8 rejections / 8 attempts; 0 origin hits
Pool requests over fresh client connections	20 correct destinations / 20 attempts

There are additional reader-script, plugin, individual-upstream, private-IP and tunnel cases in the packet. This table is not a total count of every request.

ScrapingAnt is not needed to run this local lab. If your task instead requires a separately operated upstream, the next step is the ScrapingAnt datacenter proxy setup documentation. No paid service was called in these tests. For the broader choice of proxy types, see proxies for web scraping.

The evidence covers the pinned release and local fixtures. It does not demonstrate SOCKS support, production capacity, anonymity, CAPTCHA handling or avoidance of target-site blocks. Review the packet again when changing proxy.py versions or plugin behavior.

Examples tested on 2026-09-22 with Python 3.12.11, proxy.py 2.4.10, Requests 2.34.2 and cURL 8.7.1. Code: reproducible proxy.py packet and raw logs.

This article was drafted with AI assistance from a tested evidence packet and reviewed by the named author, who is responsible for the code, measurements and corrections.