HTTPX
Modern sync/async HTTP for Python, same wall as Requests.
HTTPX is the modern Python HTTP client — sync and async APIs, HTTP/2 support, sane defaults. We ran httpx 0.28.1 (httpx.Client.get, redirects on, cookies kept) against the 21-site public corpus with no proxy.
It passed 3 of 21: Amazon search, Target search, and Rightmove. Amazon is the interesting one — plain Requests got a 503 on the same target minutes earlier, so small differences in default headers and protocol negotiation do move the needle at the margin. Everything else failed the same way it failed for Requests: 403s from eBay, Tripadvisor, Zillow, Indeed, Yelp, Trustpilot, Craigslist, and Wikipedia, 429s from Expedia and Realtor and Yahoo, robot checks from Walmart, AliExpress, Airbnb, CoinMarketCap, Reddit, and YouTube.
Like Requests, it fails fast and honestly — most refusals in well under half a second, with clean status codes.
Test conditions
21 real public sites (public-tough-v1), 2026-05-24, no proxy, single residential IP, Linux. Deterministic validators: expected status code plus required and forbidden content markers. Pass rates measure the tool and the network position together. Without a proxy, results reflect how each site's protections treat one plain residential connection as much as the tool itself.
What worked
- +Amazon search passed where plain Requests drew a 503 — default header/protocol differences are measurable.
- +Target and Rightmove passed with zero configuration.
- +Clean, fast failure signals; the async API makes it the natural high-throughput baseline.
What failed
- −18 of 21 targets blocked with the same 403/429/captcha wall that stops every plain HTTP client.
- −Wikipedia 403s the default User-Agent, same as Requests.
- −No TLS impersonation — protected sites identify it immediately.
When to use it
The default pick for new Python scraping code that talks to APIs and unprotected pages, especially when you want async. Against bot protection it needs the same help Requests does.
A better Requests for new code; identical odds against real protection.
Run evidence
Every attempt below is backed by saved artifacts: request/response metadata, body content, and (for browsers) screenshots. Raw JSON and artifacts live in the runner data directory.
Run · httpx-common-sites-v1-20260524
adapter: httpx.Client.get
| Target | Result | Detail | Latency |
|---|---|---|---|
| amazon.comamazon_search_laptop | pass | status 200 | 226 ms |
| walmart.comwalmart_search_laptop | fail | block marker: “Robot or human” | 317 ms |
| ebay.comebay_search_laptop | fail | expected 200, got 403 | 45 ms |
| aliexpress.comaliexpress_search_laptop | fail | block marker: “Robot” | 1.8 s |
| target.comtarget_search_laptop | pass | status 200 | 861 ms |
| booking.combooking_search_paris | fail | expected 200, got 202 | 103 ms |
| airbnb.comairbnb_search_paris | fail | block marker: “captcha” | 1.8 s |
| tripadvisor.comtripadvisor_restaurants_paris | fail | expected 200, got 403 | 184 ms |
| expedia.comexpedia_hotels_paris | fail | expected 200, got 429 | 219 ms |
| zillow.comzillow_homes_for_sale | fail | expected 200, got 403 | 133 ms |
| realtor.comrealtor_new_york | fail | expected 200, got 429 | 273 ms |
| rightmove.co.ukrightmove_london | pass | status 200 | 494 ms |
| indeed.comindeed_python_remote | fail | expected 200, got 403 | 39 ms |
| yelp.comyelp_sf_restaurants | fail | expected 200, got 403 | 119 ms |
| trustpilot.comtrustpilot_amazon | fail | expected 200, got 403 | 20 ms |
| newyork.craigslist.orgcraigslist_nyc_bike | fail | expected 200, got 403 | 513 ms |
| finance.yahoo.comyahoo_finance_aapl | fail | expected 200, got 429 | 156 ms |
| coinmarketcap.comcoinmarketcap_home | fail | block marker: “captcha” | 33 ms |
| reddit.comreddit_webscraping | fail | block marker: “Please wait for verification” | 116 ms |
| youtube.comyoutube_search_scraping | fail | block marker: “captcha” | 711 ms |
| en.wikipedia.orgwikipedia_web_scraping | fail | expected 200, got 403 | 149 ms |