Python Requests
The Python HTTP baseline — and our negative control.
Requests is the default answer to "fetch a page in Python," so we use it as the control condition: whatever it passes needs no special tooling, and whatever blocks it is doing at least basic bot filtering. We ran requests 2.34.2 (requests.Session.get, redirects on, cookies kept) against our 21-site public corpus with no proxy.
It passed 2 of 21 targets: Target search results and Rightmove listings. Everything else refused plain HTTP at the front door — Amazon answered 503, eBay/Tripadvisor/Zillow/Indeed/Yelp/Trustpilot/Craigslist sent hard 403s, Expedia/Realtor/Yahoo rate-limited with 429, and Walmart, Airbnb, CoinMarketCap, Reddit, and YouTube served robot-check or captcha interstitials. Even Wikipedia returned 403 to the default configuration, which is its documented policy response to generic client User-Agents.
The redeeming quality is how it fails: refusals arrived fast (mostly under 400 ms) with clean status codes and no ambiguity. As a plumbing check, an API client, or the baseline in a comparison, that honesty is exactly what you want.
Test conditions
21 real public sites (public-tough-v1), 2026-05-24, no proxy, single residential IP, Linux. Deterministic validators: expected status code plus required and forbidden content markers. Pass rates measure the tool and the network position together. Without a proxy, results reflect how each site's protections treat one plain residential connection as much as the tool itself.
What worked
- +Target search and Rightmove listings passed with zero configuration.
- +Sessions, cookies, and redirects behaved exactly as documented.
- +Fastest, cleanest failure signals of any tool tested — most blocks arrived in under 400 ms with an unambiguous status code.
What failed
- −19 of 21 targets blocked: hard 403s, 429 rate limits, a 503 from Amazon, and captcha interstitials.
- −Wikipedia 403s the default User-Agent (their API/UA policy) — set a descriptive UA.
- −No TLS fingerprinting and no JavaScript rendering, so there is nothing to tune when a site says no.
When to use it
APIs, feeds, and unprotected pages — and as the control condition in any scraping comparison. For anything with bot protection, pair it with proxies and proper headers, or reach for a TLS-impersonating client instead.
Does exactly what it claims, nothing more. Usable — as a baseline, not as a bypass.
Run evidence
Every attempt below is backed by saved artifacts: request/response metadata, body content, and (for browsers) screenshots. Raw JSON and artifacts live in the runner data directory.
Run · requests-common-sites-v1-20260524
adapter: requests.Session.get
| Target | Result | Detail | Latency |
|---|---|---|---|
| amazon.comamazon_search_laptop | fail | expected 200, got 503 | 270 ms |
| walmart.comwalmart_search_laptop | fail | block marker: “Robot or human” | 384 ms |
| ebay.comebay_search_laptop | fail | expected 200, got 403 | 104 ms |
| aliexpress.comaliexpress_search_laptop | fail | expected content missing | 123 ms |
| target.comtarget_search_laptop | pass | status 200 | 1.0 s |
| booking.combooking_search_paris | fail | expected 200, got 202 | 142 ms |
| airbnb.comairbnb_search_paris | fail | block marker: “captcha” | 1.6 s |
| tripadvisor.comtripadvisor_restaurants_paris | fail | expected 200, got 403 | 213 ms |
| expedia.comexpedia_hotels_paris | fail | expected 200, got 429 | 259 ms |
| zillow.comzillow_homes_for_sale | fail | expected 200, got 403 | 265 ms |
| realtor.comrealtor_new_york | fail | expected 200, got 429 | 228 ms |
| rightmove.co.ukrightmove_london | pass | status 200 | 715 ms |
| indeed.comindeed_python_remote | fail | expected 200, got 403 | 52 ms |
| yelp.comyelp_sf_restaurants | fail | expected 200, got 403 | 269 ms |
| trustpilot.comtrustpilot_amazon | fail | expected 200, got 403 | 109 ms |
| newyork.craigslist.orgcraigslist_nyc_bike | fail | expected 200, got 403 | 650 ms |
| finance.yahoo.comyahoo_finance_aapl | fail | expected 200, got 429 | 193 ms |
| coinmarketcap.comcoinmarketcap_home | fail | block marker: “captcha” | 104 ms |
| reddit.comreddit_webscraping | fail | block marker: “Please wait for verification” | 163 ms |
| youtube.comyoutube_search_scraping | fail | block marker: “captcha” | 766 ms |
| en.wikipedia.orgwikipedia_web_scraping | fail | expected 200, got 403 | 181 ms |