IP Blocked While Scraping? Rate Limits, IP Bans and How to Avoid Them
Rewritten around a tested rate-limit lab: every Python example now has its output, including what a burst of requests runs into, how Retry-After and pacing avoid it, and how a second exit IP splits the limit. Code: evidence packet.
Most IP blocks a scraper hits are not mysterious. The site counts requests per IP address, answers 429 Too Many Requests when you pass its limit, and blocks the address (403, a CAPTCHA, or an empty page) when you keep going. Slowing down to the site's rate and honouring Retry-After prevents most of it; spreading requests across more exit IPs is how you go faster without tripping the per-IP limit.
What an IP restriction looks like
Sites limit requests per IP to keep their servers responsive and to make bulk copying expensive. The usual signals:
| Response | What it means |
|---|---|
429 Too Many Requests, often with Retry-After | You passed a rate limit; the header says how many seconds to wait |
403 Forbidden after earlier 200s | Your IP (or session) is blocked, sometimes for minutes, sometimes longer |
A CAPTCHA or "unusual traffic" page with status 200 | Same block, served as a page; check the content, not just the status |
| Connection reset or timeouts only from your machine | Blocked at the network edge |
To show the pattern without hammering a real site, the examples below use a local lab server with a strict rule: 5 requests per 10 seconds per client, 429 with Retry-After beyond that, and a ban (403) for a client that keeps going after 3 refusals. It runs on your machine, so it can't see real exit IPs; it treats each of the lab's two proxies as a different client IP. Real sites set their own limits and rarely publish them. To run the snippets yourself, clone the evidence packet, install its requirements, start the lab with python lab.py serve in one terminal, and run each snippet in another. The lab remembers bans, so reset it before each snippet with curl http://127.0.0.1:8000/reset.
What happens when you send a burst
A plain loop with no pauses:
import requests
for i in range(1, 11):
response = requests.get(f"http://127.0.0.1:8000/item/{i}", timeout=10)
print(i, response.status_code, response.headers.get("Retry-After"), response.json().get("error"))
1 200 None None
2 200 None None
3 200 None None
4 200 None None
5 200 None None
6 429 10 rate limited
7 429 10 rate limited
8 429 10 rate limited
9 403 None banned
10 403 None banned
Five requests succeed, the next three are told to wait 10 seconds, and ignoring that gets the client banned. On a real site the ban usually lasts longer than the rate-limit window, so the cheapest fix is to never reach it.
Honour Retry-After
When you get a 429, wait as long as Retry-After says (or back off exponentially if it is missing) and retry the same URL:
import time
import requests
def get(url, max_tries=5):
for attempt in range(max_tries):
response = requests.get(url, timeout=10)
if response.status_code != 429:
return response
wait = int(response.headers.get("Retry-After", 2 ** attempt))
print(f" 429 on {url}, waiting {wait}s")
time.sleep(wait)
return response
for i in range(1, 9):
response = get(f"http://127.0.0.1:8000/item/{i}")
print(i, response.status_code)
1 200
2 200
3 200
4 200
5 200
429 on http://127.0.0.1:8000/item/6, waiting 10s
6 200
7 200
8 200
All eight items arrive and the client is never banned. Retry-After can also be an HTTP date instead of seconds; handle that if your target uses it.
Pace requests to the site's rate
Better still, stay under the limit. Spacing requests so that you never send more than the site allows avoids the 429s entirely:
import time
import requests
MIN_INTERVAL = 2.0 # seconds between requests: 5 per 10 s, the site's limit
last = 0.0
for i in range(1, 9):
time.sleep(max(0.0, last + MIN_INTERVAL - time.monotonic()))
last = time.monotonic()
response = requests.get(f"http://127.0.0.1:8000/item/{i}", timeout=10)
print(i, response.status_code)
1 200
2 200
3 200
4 200
5 200
6 200
7 200
8 200
On a real site you don't know the limit, so start slow, watch for the first 429 or block page, and keep a margin below the rate where it appeared. Adaptive approaches are covered in smart throttling algorithms and avoiding IP rate limiting.
Spread requests across exit IPs
A per-IP limit applies to each address separately, so two exit IPs get twice the budget. Here ten requests alternate between two proxies with no pause:
import itertools
import requests
PROXIES = itertools.cycle(["http://127.0.0.1:8901", "http://127.0.0.1:8902"])
for i in range(1, 11):
proxy = next(PROXIES)
response = requests.get(f"http://127.0.0.1:8000/item/{i}", proxies={"http": proxy, "https": proxy}, timeout=10)
print(i, response.status_code, "seen as", response.json()["client"])
1 200 seen as proxy-a-8901
2 200 seen as proxy-b-8902
3 200 seen as proxy-a-8901
4 200 seen as proxy-b-8902
5 200 seen as proxy-a-8901
6 200 seen as proxy-b-8902
7 200 seen as proxy-a-8901
8 200 seen as proxy-b-8902
9 200 seen as proxy-a-8901
10 200 seen as proxy-b-8902
The same ten requests that got the single client banned all succeed, because each exit stays at five. Rotation does not replace pacing: each IP still has to stay under the limit, and sites that also track cookies, browser fingerprints or behaviour will connect the requests anyway. For setting up proxies in Requests, including authentication and failover, see the Python Requests proxy guide; for which IP type to choose, see residential vs datacenter proxies.
ScrapingAnt API
Residential exit IPs in one API call
With the ScrapingAnt API you don't manage exit IPs at all: with proxy_type=residential and a proxy_country, each request goes out through a residential IP in that country, and in the run below 5 requests came from 5 different IPs. Any country from the documented list costs nothing extra, requests the API answers with a non-200 status are not charged, and there is a free plan to start.
import os
import requests
seen = set()
for i in range(1, 6):
response = requests.get(
"https://api.scrapingant.com/v2/general",
params={
"url": "https://ipinfo.io/json",
"x-api-key": os.environ["SCRAPINGANT_API_KEY"],
"browser": "false",
"proxy_type": "residential",
"proxy_country": "US",
},
timeout=60,
)
exit_ip = response.json()
seen.add(exit_ip["ip"])
print(i, response.headers["Ant-Page-Status-Code"], exit_ip["country"], exit_ip["region"])
print(len(seen), "different exit IPs in 5 requests")
1 200 US Pennsylvania
2 200 US Texas
3 200 US North Carolina
4 200 US New York
5 200 US Texas
5 different exit IPs in 5 requests
The second value is Ant-Page-Status-Code, the target's own status; the script prints the region, not the IP. Run on 2026-10-26 with requests 2.34.2; set SCRAPINGANT_API_KEY to your API key first. Leave out browser=false to load the page in headless Chrome.
Other things that get an IP blocked
- Bot-like headers. The default
python-requests/2.xUser-Agent is easy to filter. Send the headers a browser would; see changing the User-Agent in Python Requests. - Ignoring JavaScript challenges. Some sites only serve data after a browser check. A plain HTTP client sees the challenge page, not a
429. - Datacenter IP ranges. Some sites treat traffic from cloud and hosting networks with more suspicion than home connections; residential IPs are the usual answer.
- One session for everything. Cookies tie requests together even across IPs.
The broader checklist is in web scraping without getting blocked, and ethical IP ban bypass techniques covers where the line is. Respect the site's terms and robots.txt, and don't scrape faster than the site can serve its own users.
Examples tested on 2026-10-26 with Python 3.12.11, requests 2.34.2 and proxy.py 2.4.10. Code: scrapingant-examples/examples/ip-restriction-protection.
This article was drafted with AI assistance from a tested evidence packet and reviewed by the named author, who is responsible for the code, measurements and corrections.