[jehrr]

Problem

A run through a paid residential proxy could start a browser per URL on a proxy that refuses every request (a gateway 407 or 402). The canary's own hint said such a 407 "reaches Chromium as a navigation timeout". Once a browser was running behind the proxy, it also downloaded every image, video and web font through the metered proxy, although every value is read from the JSON the page embeds.

What changes

Proxy pre-check (all three browser engines). Before launching a local browser behind --proxy / --proxy-file, playwright_scraper.py, puppeteer_scraper.py and selenium_scraper.py send ONE HEAD https://www.instagram.com/robots.txt through the exit the first URL would use. This is ProxyPool.peek(), which does not move the rotation cursor. The shared logic is in page_flow.proxy_precheck, and the helpers are in proxy_pool.py.

• 407: exit 5, with the message the gateway refused the credential (or the traffic balance is exhausted).
• 402: exit 5, with the message proxy traffic/balance exhausted.
• In both cases the message names INSTAGRAM_PROXY and never prints its value. The new output_writer.write_failed_attempt writes only <out>.latest_attempt.meta.json (status: failed, stop_reason: proxy_auth_failed, data_updated: false), and <out> and <out>.meta.json are left as they were.
• Any other answer, or a transport error, is not a verdict on the proxy, and the run continues as it did before.
• The pre-check is one request and is never retried or rotated.
• It is not sent with --cdp-endpoint, with --scraper-api (no local proxy on that path) or with no proxy.
• Proxy.url() rebuilds the URL with percent-encoded credentials, for the HTTP client only.

Media blocking. With a local browser behind a proxy, images, media and fonts are aborted:

• Playwright uses context.route by resource_type, on the per-URL context. The async API needs an awaited handler (_async_media_route), because the shared sync playwright_media_route would create the abort/continue coroutines and drop them unawaited.
• Puppeteer already uses CDP Fetch to answer the proxy's password prompt (Fetch.enable with handleAuthRequests). Its existing requestPaused handler now calls Fetch.failRequest (BlockedByClient) for Image/Media/Font, so no second interception is opened. Without a password, proxy_pool.pyppeteer_block_media pauses only those three resource types.
• Selenium uses Network.setBlockedURLs by file extension, right after the driver is built, so an image served without an extension still loads. If the CDP command fails, the engine warns and loads the media.
• --load-media opts out. A hidden --force-block-media turns blocking on without a proxy, for measurement.
• Nothing is blocked over CDP, with --scraper-api or with no proxy. <out>.meta.json records media_blocked.
• No env variable was added, because env_config.ENV_KEYS maps only credentials and URLs.

The canary's ::error:: hint for exit 5 now points at the pre-check.

Evidence

• Offline suite, python3 smoke_test.py: 79/79 in a core-only venv, and 79/79 in each of the playwright, pyppeteer and selenium venvs. The engine-import assertion from tests.yml passes in each venv. unittest discover, ci_checks.py (credential scan), ci_checks.py --sample-check and every engine's --help are green.

• New checks:

  • Each engine's real run() against a local stub proxy that answers every request, CONNECT included, with 407 or 402. Each exits 5 after exactly one CONNECT www.instagram.com:443, and the patched browser launch is never reached. The message names the variable and does not print the credential. The attempt sidecar reads failed/proxy_auth_failed with the right engine. Earlier o.json/o.json.meta.json keep their bytes, and nothing else is written.
  • With a site 403, the browser does start. A proxy that cannot be reached is not treated as a failure.
  • No request is sent with --cdp-endpoint, with --scraper-api or with no proxy. The probe is the first URL's robots.txt through the first exit, and peek() does not consume the exit.
  • The media-policy matrix, and --load-media shown in --help while --force-block-media is hidden.
  • The Playwright handler on fake routes, and the route installed on the context only behind a proxy (not with --load-media, not over CDP).
  • Puppeteer: one Fetch.enable, images/fonts/media failed and documents/scripts continued, nothing failed with --load-media, the resource-type-only path without a password, and no Fetch opened when neither applies.
  • The Selenium blocked-URL list, applied behind a proxy only.
  • media_blocked in the sidecar, and a source check per engine.

• The real CLIs against the stub proxy (--proxy http://user:…@127.0.0.1:<stub>): Playwright, pyppeteer and Selenium each gave rc=5 for both 407 and 402. The stub saw one CONNECT, the secret did not appear in the output, and only o.json.latest_attempt.meta.json was written.

• The real engine classes were driven through a local forwarding proxy, with Basic auth where the engine supports it, to a page containing png/jpg/woff2/mp4/js:

  Engine	Default (behind the proxy)	With --load-media
  Playwright, with auth	only / and /app.js reached the proxy, and the JS ran	all assets
  Puppeteer, with auth through its Fetch handler	/, /app.js and the browser's own /favicon.ico	all assets
  Puppeteer, no auth	/, /app.js and /favicon.ico
  Selenium, no auth	/ and /app.js	all assets

• Live comparison on 2026-10-20, Playwright, no proxy, --url natgeo, with and without --force-block-media: not possible. Both runs got Instagram's login page (exit 3), the datacentre behaviour already noted in the README. Instead, the parser was read for anything that depends on loaded media. Every field, image and video URLs included, comes from the embedded JSON blocks, and nothing reads image bytes or load state.

• Planted-fault controls, each restored afterwards and green again:

  • Short-circuiting page_flow.proxy_precheck turns the three pre-check checks red.
  • Removing the engine call in selenium_scraper.run turns three checks red, among them selenium_scraper 407: a browser was started.
  • Making _async_media_route continue everything turns the Playwright media check red. So does not installing the route.
  • Removing the failRequest branch turns the Puppeteer check red.
  • Removing selenium_block_media(driver) turns the Selenium check and the source check red.

• No paid calls, no proxy and no credentials were used.

Not changed

• The Scraper API path (--scraper-api, scraper_api_engine.py) uses no local proxy, so it has no pre-check and no media blocking.
• No version bump and no release. The canary was not dispatched.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKsd7q25N19WzUZjrAdzx8