Stop on a refused proxy before starting a browser, and skip media behind a proxy

.github/workflows/canary.yml

@@ -68,7 +68,7 @@ jobs:
                         'If it was refused (exit 3), set the INSTAGRAM_PROXY repo secret (a residential '
                         'http://login:password@host:port proxy) so the canary can reach the site.' % code)
               else:
-                  print('::error::canary exit %d through the INSTAGRAM_PROXY secret (1 crash, 3 blocked, 4 zero rows, 5 content never obtained, 6 partial). A proxy that rejects its credential (HTTP 407) reaches Chromium as a navigation timeout, so on exit 5 check the INSTAGRAM_PROXY secret first.' % code)
+                  print('::error::canary exit %d through the INSTAGRAM_PROXY secret (1 crash, 3 blocked, 4 zero rows, 5 content never obtained, 6 partial). A proxy that rejects its credential (HTTP 407) or has no traffic left (HTTP 402) stops the run at its proxy pre-check with exit 5 and says so in the log above, so on exit 5 check the INSTAGRAM_PROXY secret first.' % code)
               raise SystemExit(f'canary requires EXIT_OK=0, got {code}')
           meta = json.load(open('canary.json.meta.json'))
           print(json.dumps(meta, indent=2))

CHANGELOG.md

@@ -8,13 +8,36 @@ that changes a default is called out at the top of its entry.
 
 ## [Unreleased]
 
+> **Behaviour changes.**
+> * Behind `--proxy` / `--proxy-file`, all three browser engines send one
+>   `HEAD https://www.instagram.com/robots.txt` through the first exit before
+>   launching a browser. A gateway `407` (credential refused, or traffic
+>   exhausted) or `402` (traffic/balance exhausted) now ends the run with exit
+>   5 and `stop_reason: proxy_auth_failed` in the new
+>   `<out>.latest_attempt.meta.json`. No browser is started, and `<out>` and
+>   `<out>.meta.json` are left as they were. Any other answer, or a transport
+>   error, leaves the run as it was. Not sent with `--cdp-endpoint`,
+>   `--scraper-api` or without a proxy.
+> * Behind a local proxy, images, video/audio and web fonts are no longer
+>   downloaded. Playwright aborts them by resource type. Puppeteer fails them
+>   in the `Fetch` handler that answers the proxy's password prompt. Selenium
+>   blocks them by URL extension. `--load-media` restores the old behaviour.
+>   Nothing changes with `--cdp-endpoint`, `--scraper-api` or with no proxy.
+>   `<out>.meta.json` records `media_blocked`.
+
+### Added
+
+- `--load-media` in all three engines.
+
 ### Changed
 
 - CI only: a failed canary check now always ends with an `::error::` line. A run
   through the `INSTAGRAM_PROXY` secret that exited nonzero used to end red with no
   annotation, and a missing or wrong output file showed only a traceback.
   The message notes that a proxy rejecting its credential (HTTP 407) shows
   up in Chromium as a navigation timeout (exit 5), not as a 407.
+- CI only: that message now points at the proxy pre-check, which reports a
+  `407` or `402` by name before any browser is started.
 - CI only: the canary now runs weekly (Mondays 06:41 UTC) and on a manual dispatch, with its optional secret `INSTAGRAM_PROXY` passed on every run; the daily schedule is removed.
 
 ## [0.3.0] - 2026-10-15

README.md

@@ -208,6 +208,55 @@ getting blocked is dropped; once every exit is dropped the run stops, and
 never falls back to your own address), keep `--delay-between-pages` at 2s
 or more, or spread a big list over time.
 
+## Traffic through a paid proxy
+
+A residential proxy bills per byte, so two things happen only while a
+LOCAL browser goes through `--proxy` / `--proxy-file` (or
+`INSTAGRAM_PROXY`). Neither applies with `--cdp-endpoint` (that browser
+brings its own exit and is billed on its own side) or with `--scraper-api`
+(no local proxy is used), or with no proxy, where nothing is metered.
+
+**Pre-check before the browser starts.** All three engines send ONE
+`HEAD https://www.instagram.com/robots.txt` through the exit the first URL
+would use, before launching a browser:
+
+- `407`: the gateway refused the credential, or the traffic balance is
+  exhausted. Exit `5`.
+- `402`: the proxy's traffic or balance is exhausted. Exit `5`.
+
+The message names `INSTAGRAM_PROXY` and never prints its value. The run
+writes `<out>.latest_attempt.meta.json` with `status: failed` and
+`stop_reason: proxy_auth_failed`, and leaves `<out>` and `<out>.meta.json`
+from an earlier run exactly as they were. Any other answer (a `200`, a
+`403` from the site, a timeout, a DNS error) is not a verdict on the proxy,
+and the run carries on exactly as before. It is one request, never retried
+and never rotated to another exit.
+
+**Media is not downloaded.** Every value comes from the JSON the page
+embeds, image and video URLs included, so images, video/audio and web fonts
+are aborted:
+
+| Engine | How | What still loads |
+|---|---|---|
+| Playwright | `context.route` by resource type | nothing of those three types |
+| Puppeteer | the CDP `Fetch` handler that already answers the proxy's password prompt fails them by resource type; with no password, a `Fetch` interception paused on those three types only | nothing of those three types |
+| Selenium | CDP `Network.setBlockedURLs` by file extension | an image or font served from a URL **without** an extension |
+
+`--load-media` keeps them. A `--dump-html` capture is unaffected, because
+it is the page's HTML. Playwright switches its HTTP cache off while a
+route is installed, so a script or stylesheet repeated across URLs is
+fetched again. Each browser context is new per URL anyway, so this costs
+little here. `<out>.meta.json` records `media_blocked`.
+
+Live comparison, 2026-10-20: `playwright_scraper.py --url natgeo` with no
+proxy, once as it is and once with blocking forced on, to compare the rows.
+It was not possible from this datacentre address: both runs got Instagram's
+login page (exit 3), the same as the datacentre measurement under
+[Known limitations](#known-limitations). The verification is therefore
+offline: the fixtures (the parser reads embedded JSON only), and each real
+browser driven through a local proxy to a test page. With blocking, only the
+HTML and the script went through the proxy, and the script still ran.
+
 ## Scraper API mode
 
 With `--scraper-api` no browser is driven: each page is one 2Captcha
@@ -228,15 +277,17 @@ solves spent, and a hash of the output file). With the options below it
 also lists the posts left out (`excluded_urls`), every post seen so far
 (`seen_post_skus`) and whether the run was incremental. A run that
 collects nothing writes neither, so it never replaces your previous good
-file.
+file. The one exception is a proxy refused at the pre-check (see
+[Traffic through a paid proxy](#traffic-through-a-paid-proxy)): that run
+records itself in `<out>.latest_attempt.meta.json`, a separate file.
 
 | Exit | Meaning |
 |---|---|
 | `0` | complete |
 | `6` | partial: rows were written, but the run did not finish cleanly — `stop_reason` says why (`blocked`, `rate_limited`, `remote_api_error`, `failed_pages`, `parse_error`, `proxy_pool_exhausted`) |
 | `3` | blocked or throttled, no rows |
 | `4` | no rows: every input was not found, or every post was older than `--since` or already collected by `--incremental-from` |
-| `5` | nothing could be read due to fetch, parse, or remote-service failure; no rows |
+| `5` | nothing could be read due to fetch, parse, or remote-service failure; no rows. Also a proxy that answered `407`/`402` at the pre-check, before any browser was started |
 | `2` | bad usage, including input where every line was skipped |
 | `1` | crash (a bug; please report it) |
 
@@ -358,6 +409,7 @@ the command line.
 | `--format` / `--out` | json / `instagram_results.<format>` | output format and path |
 | `--proxy` / `--proxy-file` / `--proxy-shuffle` | `INSTAGRAM_PROXY` | one proxy or a rotating pool, for a local browser |
 | `--proxy-block-retries` | 3 | blocked answers before that proxy is dropped from the pool |
+| `--load-media` | off | load images, video and fonts behind a proxy too (they are aborted by default there; see [Traffic through a paid proxy](#traffic-through-a-paid-proxy)) |
 | `--scraper-api` | off | no browser: fetch each page through 2Captcha's Scraper API (needs `TWOCAPTCHA_KEY`; routed through `INSTAGRAM_CDP_ENDPOINT` when set) — posts only, see [Scraper API mode](#scraper-api-mode) |
 | `--cdp-endpoint` | `INSTAGRAM_CDP_ENDPOINT` | connect to a Scraping Browser API profile instead of launching a browser |
 | `--solve-captcha` | when-blocked | `off` disables the Browser API's own captcha auto-solve; there is no local solver |

output_writer.py

@@ -329,6 +329,41 @@ def write_meta(
         json.dump(meta, f, ensure_ascii=False, indent=2)
 
 
+# A run that stopped before it fetched anything — the proxy refused its
+# credential at the pre-check — has no dataset to describe, and
+# `<out>.meta.json` must keep describing the output still beside it. What
+# happened to THIS attempt goes into its own file instead.
+ATTEMPT_META_SUFFIX = ".latest_attempt.meta.json"
+
+
+def write_failed_attempt(
+    out_path: str, *, stop_reason: str, engine: str, url: str, pages_requested: int,
+    started_at: float, transport_error: str,
+) -> str:
+    """Record an attempt that ended before `finish_run` could run. Only
+    `<out>.latest_attempt.meta.json` is written; the output file and
+    `<out>.meta.json` are left exactly as they were."""
+    meta = {
+        "status": "failed",
+        "stop_reason": stop_reason,
+        "engine": engine,
+        "url": url,
+        "pages_requested": pages_requested,
+        "pages_completed": 0,
+        "failed_pages": [],
+        "product_count": 0,
+        "started_at": started_at,
+        "finished_at": time.time(),
+        "blocked": False,
+        "transport_error": transport_error,
+        "data_updated": False,
+    }
+    path = f"{out_path}{ATTEMPT_META_SUFFIX}"
+    with _atomic(path) as f:
+        json.dump(meta, f, ensure_ascii=False, indent=2)
+    return path
+
+
 # --------------------------------------------------------------------------- #
 # finish_run — the single place every engine calls to decide exit code,
 # whether to write output at all, and whether to write a sidecar. Keeping

page_flow.py

@@ -425,20 +425,22 @@ async def run(engine: Engine, args, *, urls: List[str], proxy_pool, client, star
                   engine_name=engine.name, urls=urls, fetched=queue[:args.max_results], started_at=started_at,
                   pages_completed=completed, failed_pages=failed_pages, failures=failures,
                   rate_limited=rate_limited, not_found=not_found,
-                  capped=len(queue) > args.max_results or posts_per_profile > 0, exclusions=exclusions)
+                  capped=len(queue) > args.max_results or posts_per_profile > 0, exclusions=exclusions,
+                  media_blocked=bool(getattr(engine, "media_blocked", False)))
 
 
 def finish(args, *, products: List[Product], blocked: bool, remote_api_error: bool, engine_name: str,
            urls: List[str], started_at: float, pages_completed: int, failed_pages: List[int],
            failures=None, rate_limited=False, fetched: Optional[List[str]] = None, not_found=None,
-           capped: Optional[bool] = None, exclusions=None) -> int:
+           capped: Optional[bool] = None, exclusions=None, media_blocked: bool = False) -> int:
     budget = getattr(args, "_solve_budget", None)
     fetched = list(urls if fetched is None else fetched)
     selection = run_state.selection(args, urls)
     extra = {"solves_spent": budget.spent if budget is not None else 0, "selection": selection,
              "failed_urls": failures or [], "not_found_urls": not_found or [],
              "excluded_urls": exclusions or [], "incremental": bool(getattr(args, "incremental_from", None)),
              "post_window": "embedded_pinned_first" if getattr(args, "posts", 0) else None,
+             "media_blocked": bool(media_blocked),
              # A post's date never changes, so one before --since stays excluded:
              # remembering it spares the next delta a fetch that only drops it.
              "seen_post_skus": sorted(set(getattr(args, "_known_posts", set())) |
@@ -479,3 +481,63 @@ def validate_common(args, *, urls, skipped, print_err) -> Optional[int]:
         print_err(f"Error: {_redact(str(exc))}")
         return EXIT_BAD_USAGE
     return None
+
+
+# --------------------------------------------------------------------------- #
+# Traffic savers behind a metered proxy (the helpers are in proxy_pool.py)
+# --------------------------------------------------------------------------- #
+# One HEAD for www.instagram.com/robots.txt, through the proxy, before a local
+# browser is launched. Short on purpose: a slow answer is not a verdict, and
+# the run then goes ahead exactly as it did before.
+PROXY_PRECHECK_TIMEOUT_S = 20.0
+
+
+def proxy_precheck(args, proxy_pool, *, urls: List[str], engine_name: str, started_at: float,
+                   print_err=None) -> Optional[int]:
+    """Pre-check the exit the first URL will use; return the exit code to
+    stop with, or None to go on.
+
+    A gateway 407 or 402 prints why (naming INSTAGRAM_PROXY, never its
+    value), writes `<out>.latest_attempt.meta.json` (status failed,
+    stop_reason proxy_auth_failed), leaves the output and `<out>.meta.json`
+    untouched, and returns exit 5. Never sent over --cdp-endpoint or
+    --scraper-api (no local proxy there) or without a proxy. Anything else —
+    a 200, a 403 from the site, a timeout — returns None: the run carries on
+    unchanged. One request, never retried and never rotated to another exit.
+    """
+    import sys
+    import output_writer
+    import proxy_pool as _savers
+    print_err = print_err or (lambda m: print(m, file=sys.stderr))
+    if proxy_pool is None or getattr(args, "cdp_endpoint", None) or getattr(args, "scraper_api", False):
+        return None
+    proxy = proxy_pool.peek()
+    if proxy is None or not urls:
+        return None
+    try:
+        _savers.precheck_proxy(proxy.url(), _savers.robots_url(urls[0]), timeout=PROXY_PRECHECK_TIMEOUT_S)
+    except _savers.ProxyPrecheckError as exc:
+        print_err(_savers.precheck_message(exc, "INSTAGRAM_PROXY", "www.instagram.com"))
+        try:
+            output_writer.write_failed_attempt(
+                args.out, stop_reason=_savers.PROXY_PRECHECK_STOP_REASON, engine=engine_name,
+                url=urls[0], pages_requested=len(urls), started_at=started_at,
+                transport_error="proxy pre-check: %s" % exc)
+        except OSError as write_exc:
+            # The verdict stands without its record: still exit 5, not a crash.
+            print_err("[!] could not write the attempt metadata: %s" % write_exc)
+        return output_writer.EXIT_REMOTE_API_ERROR
+    return None
+
+
+def add_media_arguments(parser) -> None:
+    """`--load-media` (documented) and the hidden `--force-block-media`."""
+    import argparse
+    parser.add_argument(
+        "--load-media", action="store_true",
+        help="Load images, video and web fonts. By default they are aborted while a local browser goes "
+             "through --proxy / --proxy-file (the parser reads the page's embedded JSON, and a proxy bills "
+             "per byte); nothing is blocked with --cdp-endpoint, --scraper-api or no proxy.")
+    # Turns blocking on without a proxy, so its effect on parsing can be
+    # measured for free. Not a user feature.
+    parser.add_argument("--force-block-media", action="store_true", help=argparse.SUPPRESS)

playwright_scraper.py

@@ -46,7 +46,8 @@
 import page_parser as pp
 from fingerprint_client import fetch_fingerprint, refuse_if_cdp, user_agent_from
 from output_writer import EXIT_BAD_USAGE, EXIT_CRASH
-from proxy_pool import Proxy, ProxyPool, ProxyParseError, load_proxies, redact_credentials
+from proxy_pool import (BLOCKED_RESOURCE_TYPES, Proxy, ProxyPool, ProxyParseError, load_proxies, redact_credentials,
+                        should_block_media)
 from scraper_api_client import TwoCaptchaClient
 
 ENGINE_NAME = "playwright"
@@ -99,6 +100,7 @@ def build_arg_parser() -> argparse.ArgumentParser:
     p.add_argument("--dump-html", action="store_true", help="Save each fetched page's HTML next to --out, on success too")
     p.add_argument("--headless", dest="headless", action="store_true", default=True)
     p.add_argument("--headful", dest="headless", action="store_false")
+    page_flow.add_media_arguments(p)
     page_flow.add_run_arguments(p)
     return p
 
@@ -112,6 +114,7 @@ def _default_out(fmt: str) -> str:
 
 async def _new_context(
     browser: Browser, proxy: Optional[Proxy], user_agent: Optional[str], *, reuse_default: bool = False,
+    block_media: bool = False,
 ) -> BrowserContext:
     """Over --cdp-endpoint, reuse the profile's own default context: it
     holds the cookies that make the profile worth reusing. A local browser
@@ -124,7 +127,22 @@ async def _new_context(
         kwargs["proxy"] = proxy.playwright_proxy_dict()
     if user_agent:
         kwargs["user_agent"] = user_agent
-    return await browser.new_context(**kwargs)
+    context = await browser.new_context(**kwargs)
+    if block_media:
+        # Images, video and fonts through a metered proxy: the parser reads
+        # the page's embedded JSON, never these.
+        await context.route("**/*", _async_media_route)
+    return context
+
+
+async def _async_media_route(route) -> None:
+    """proxy_pool.playwright_media_route for the ASYNC API, whose
+    route.abort()/continue_() are coroutines that must be awaited (the sync
+    handler would create them and drop them, so nothing would be decided)."""
+    if route.request.resource_type in BLOCKED_RESOURCE_TYPES:
+        await route.abort("blockedbyclient")
+    else:
+        await route.continue_()
 
 
 async def _close(page: Page, context: BrowserContext, *, reuse_default: bool) -> None:
@@ -199,10 +217,14 @@ class _PlaywrightEngine:
 
     def __init__(self, browser: Browser, args: argparse.Namespace, *, autosolve: bool, user_agent: Optional[str], client):
         self.browser, self.args, self.autosolve, self.user_agent, self.client = browser, args, autosolve, user_agent, client
+        self.media_blocked = False
 
     async def open(self, proxy) -> _PlaywrightSession:
         reuse_default = bool(self.args.cdp_endpoint)
-        context = await _new_context(self.browser, proxy, self.user_agent, reuse_default=reuse_default)
+        block = should_block_media(self.args, proxy_in_use=proxy is not None, over_cdp=reuse_default)
+        self.media_blocked = self.media_blocked or block
+        context = await _new_context(self.browser, proxy, self.user_agent, reuse_default=reuse_default,
+                                     block_media=block)
         page = await context.new_page()
         if self.autosolve:
             await _enable_scraping_browser_auto_solve(context, page)
@@ -242,6 +264,11 @@ async def run(args: argparse.Namespace) -> int:
         print(f"Error: {exc}", file=sys.stderr)
         return EXIT_BAD_USAGE
     proxy_pool = ProxyPool(proxies, shuffle=args.proxy_shuffle, block_retries=args.proxy_block_retries) if proxies else None
+    # Before any browser exists: a proxy that refuses its credential (407) or
+    # has no traffic left (402) would refuse every request the browser makes.
+    stop = page_flow.proxy_precheck(args, proxy_pool, urls=urls, engine_name=ENGINE_NAME, started_at=started_at)
+    if stop is not None:
+        return stop
 
     client = None
     if args.twocaptcha_key and (args.solve_captcha != "off" or args.fingerprint):