[jehrr]

Problem

A run through a residential proxy is billed per byte, and nothing measured or limited what a run sent through it. A run that loops, or a page that grew, could spend a lot before anyone looked.

Resulting behaviour

Only for a local browser behind --proxy / --proxy-file / INSTAGRAM_PROXY. Nothing changes over --cdp-endpoint, with --scraper-api, or with no proxy.

• Meter (proxy_meter.py), an estimate of the tunnelled payload. TLS/CONNECT overhead, WebSocket frames and Chrome's own background requests are not included.
  • Playwright: request.sizes() (request headers + body, response headers + body) on requestfinished / requestfailed.
  • Puppeteer: a CDP session on each page counts Network.loadingFinished.encodedDataLength for responses that came over the network, plus the request line, headers and post body from Network.requestWillBeSent. Chrome's own error page (about 250 KB, reported as a finished load of the failed URL) and cache hits are not counted.
  • Selenium: the same events from Chrome's performance log, which is now turned on only behind a local proxy.
  • The proxy pre-check's HEAD is counted too.
• Budget --proxy-budget-mb (default 10, 0 = unlimited; a negative or non-number value is exit 2). It is a flag only, because ENV_KEYS maps only credentials and URLs. Once the total passes the budget:
  • no further URL (or --posts post) is fetched;
  • Playwright aborts every further request through a route (registered after the media route, then fallback()). Puppeteer and Selenium set Network.setBlockedURLs ["*"];
  • the rows already read are kept, and the run ends with stop_reason: proxy_budget_exceeded: exit 6 with rows, 5 without. finish_run puts this above blocked or failed-page outcomes.
• Sidecar: proxy_bytes, proxy_budget_bytes, proxy_budget_exceeded. These are null / null / false without a local proxy.
  • A proxied run that writes no rows now records its outcome and bytes in <out>.latest_attempt.meta.json. The data and .meta.json are left as they were.
  • One line is printed at the end: Proxy traffic: X.X MB of Y MB budget.
• Canary: the run passes --proxy-budget-mb 10. An always-run step, .github/proxy_traffic_summary.py, appends **canary**: proxy traffic X.X MB of 10 MB to $GITHUB_STEP_SUMMARY. When the budget was hit, it prints ::error::stopped: proxy budget 10 MB exceeded and fails. A missing sidecar, an unreadable sidecar or a run without a proxy is reported and does not fail.

Evidence (offline; no proxy, no paid call, no canary dispatched)

• tests/test_proxy_meter.py (the same file as facebook-ads-scraper Proxy meter: stop counting chrome:// and other local loads in Selenium and Puppeteer #8) runs a stub forward proxy that serves a page of known size (about 152 KB) and counts the bytes crossing its sockets. Meter against the proxy's count:
  • requests: within 2%;
  • real Chromium via Playwright, pyppeteer and Selenium: within 10%.
• CI engine jobs run this file with PROXY_METER_REQUIRE=<engine>, so a missing browser fails instead of skipping.
• The real CLIs (--url natgeo) through the stub proxy (CONNECT answered 403, so instagram.com was never reached): rc 5 for playwright, selenium and pyppeteer, Proxy traffic: 0.0 MB of 10 MB budget, and an attempt sidecar with proxy_bytes 489 / 846 / 282. Selenium's figure shows that the performance log it now enables is read.
• smoke_test.py: 86/86 in core, playwright, pyppeteer and selenium venvs. unittest discover: OK in each, with that engine's real-browser test run, not skipped.
• New checks:
  • budget hit on the 2nd of 3 URLs: exit 6, profile + post kept, the 3rd URL not_attempted, sidecar (1200, 1000, True);
  • budget hit before any rows: exit 5, attempt sidecar, data untouched;
  • a failed navigation is not retried after the budget;
  • no proxy: nothing metered; budget 0 is unlimited;
  • finish_run precedence;
  • the flag on every engine;
  • per-engine wiring, including Selenium's logging prefs, only with a local proxy;
  • the canary step and its script.
• The two existing media-wiring checks now reset the run's meter first. The engine runs earlier in the suite left a metered run behind, which would have added the budget route to their exact route list.
• Planted-fault controls. Each went red on the named check and came back green once restored:
  • meter disabled: 4 smoke checks and 8 test_proxy_meter tests red;
  • run-loop budget check removed: 1 red;
  • navigation-retry check removed: 1 red;
  • finish_run override removed: 1 red;
  • summary step replaced: the canary check red;
  • Selenium performance-log line removed: the wiring check red.

Not changed

No version bump or release, and no canary dispatch. The proxy is never used without the user's flag or env. Chrome's background traffic is not suppressed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKsd7q25N19WzUZjrAdzx8