From dc8cd6cf7b433cb54bba21a10854badc1f165d45 Mon Sep 17 00:00:00 2001 From: jehrr <79864894+jehrr@users.noreply.github.com> Date: Wed, 21 Oct 2026 11:23:54 +0000 Subject: [PATCH] Count proxy traffic per run and stop at --proxy-budget-mb (default 10 MB) A local browser run through --proxy / --proxy-file now counts the bytes it sends through the proxy (Playwright: request.sizes(); Puppeteer and Selenium: CDP Network.loadingFinished.encodedDataLength plus request sizes; Selenium turns Chrome's performance log on for this behind a proxy). Once the total passes --proxy-budget-mb no further URL or post is fetched, the browser aborts what it still tries, the rows read so far are kept, and the run exits 6 (5 with no rows) with stop_reason proxy_budget_exceeded. The sidecar records proxy_bytes, proxy_budget_bytes and proxy_budget_exceeded, and the run ends with one "Proxy traffic: X.X MB of Y MB budget" line. Nothing is metered over --cdp-endpoint, --scraper-api or with no proxy. The canary passes --proxy-budget-mb 10, writes its traffic to the job summary and fails when the budget was hit. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01CKsd7q25N19WzUZjrAdzx8 --- .github/proxy_traffic_summary.py | 71 +++++ .github/workflows/canary.yml | 8 +- .github/workflows/tests.yml | 4 + CHANGELOG.md | 20 ++ Dockerfile | 2 +- README.md | 41 ++- output_writer.py | 47 +++ page_flow.py | 19 +- playwright_scraper.py | 6 + proxy_meter.py | 442 ++++++++++++++++++++++++++++ proxy_pool.py | 4 +- puppeteer_scraper.py | 4 + pyproject.toml | 1 + selenium_scraper.py | 9 + smoke_test.py | 306 +++++++++++++++++++ tests/test_proxy_meter.py | 485 +++++++++++++++++++++++++++++++ 16 files changed, 1458 insertions(+), 11 deletions(-) create mode 100644 .github/proxy_traffic_summary.py create mode 100644 proxy_meter.py create mode 100644 tests/test_proxy_meter.py diff --git a/.github/proxy_traffic_summary.py b/.github/proxy_traffic_summary.py new file mode 100644 index 0000000..20add55 --- /dev/null +++ b/.github/proxy_traffic_summary.py @@ -0,0 +1,71 @@ +#!/usr/bin/env python3 +"""Canary step: one line about a run's proxy traffic for the job summary. + + python3 .github/proxy_traffic_summary.py