[jehrr]

Problems

1. Markup drift read as an empty list or as the end of the list. A served page with #hnmain and no tr.athing was classified empty even while it still carried its own More link, and a page that had rows but parsed to none only logged a warning. Measured before the change with the real news/news2 fixtures and athing renamed on page 2: status: complete, stop_reason: end_of_listing, 30 rows, exit 0. The same drift on page 1 gave exit 4.
2. cp .env.example .env broke every query flag. The example set HACKERNEWS_URL, env_config.apply filled args.url, and build_query then refused --feed show, --item 1 and --mode user --user dang with exit 2 in all four entry points.
3. Public files cited an unpublished instruction document by name and section number (about 95 lines in 13 files).
4. Lower-severity items: the inherited "credentials last about a day" claim (README, TROUBLESHOOTING, .env.example, the CDP 401 hint), Selenium's 403 → exit 5 undocumented, key masking limited to key= form, comments about other sites' columns, CI matrix stopping at 3.12.

Resulting behaviour

• A page with no parsed rows that still states it has content (listing: rows in the markup or a More link; item: a comment count above zero or no story row; user: the site's page without a parsable profile) is a failed page with stop_reason: parser_found_nothing and a saved <out>_pageN_debug.html. Page 1 → exit 5; later page → exit 6 (partial), earlier rows kept. The site's own short "nothing" sentences and a real page past the end (no rows, no More link, e.g. ?p=27) are unchanged: exit 4 / end_of_listing.
• HACKERNEWS_URL ships commented out. Each entry point records typed_url before .env is applied, and a URL from the environment is dropped when a query flag or --mode was typed. A typed --url with a query flag is still refused.
• _KEY_IN_TEXT_RE in scraper_api_client.py and fingerprint_client.py also masks clientKey: … and "key": "…".
• Citations replaced by the rule in plain words or removed; new check no shipped file cites the private instruction files or their sections (tokens assembled from pieces).
• README/TROUBLESHOOTING exit tables updated; diff_runs.UNTRACKED_FIELDS loses three names no row class has; tests.yml offline matrix is ["3.9", "3.14"]; canary's exit-4/5 messages describe the new parser-failure case.
• CHANGELOG [Unreleased] leads with a blockquote for the exit-code change.

Evidence

• Offline suite, core-only venv (Python 3.13): 662 passed, 0 failed, 8 engine groups skipped. Engine venvs (no browser download): playwright 672 passed / selenium 669 / puppeteer 669, each skipping only the two absent engines.
• scraper_api_client.parse_args with HACKERNEWS_URL exported: --feed show → listing/show, --mode user --user dang → user/dang, --item 1 → item/1, no flags → listing/news; typed --url … --feed show → exit 2.
• Planted-fault controls (PYTHONDONTWRITEBYTECODE=1, __pycache__ removed between runs), each RED on the named check, green after restore:
  • drift classifier returns False → page 2 drifted: exit 6, partial, parser_found_nothing got (0, 'complete', 'end_of_listing', []);
  • typed_url = args.url → an environment URL and a typed --feed: the flag wins fails;
  • a section mark planted in output_writer.py → no shipped file cites the private instruction files… names that file;
  • masking regex reverted to = only → three a key in the shape … is masked checks fail.
• The previous check that pinned exit 4 for "served with rows, parsed to none" was rewritten to expect exit 5: exit 4 told a pipeline the site had nothing, when this repo had failed to read it.

Canary (owner's request)

Before and after: daily, ungated (no secret needed; gh secret list shows none, and none is required), four jobs that run the real site with Playwright and go red on any exit other than 0 (3, 4, 5 and 6 each fail with ::error::). The asserted thresholds are parser floors (85/90 listing rows, 60 comments, 1 profile, non-null core fields, unique page+position, rank order). The last 10 scheduled runs were green with real rows (2026-10-14: 90 + 90 listing rows, 71 comments, 1 profile). Only the exit-4/5 messages were changed here.

Not changed

• Version bump, tag and release (no tag or release exists for 0.1.0; owner decision, audit F4).
• Inert --solve-captcha/--captcha-api/--min-score and warn-not-refuse for --proxy/--fingerprint with --cdp-endpoint (owner decision, F7).
• The item canary's exact comment-count equality (owner decision, F11).
• Selenium 403 classification itself (no real HN 403 body has been captured; documented instead, F6).
• The credential scan's skipping of ignored .env* files (F12): a design note, and the files are ignored by .gitignore.
• No captcha solve path exists in this repo, so there is no paid-solve budget to enforce.
• No GitHub releases exist, so no release notes needed editing. Repo description and topics already meet the org-profile audit.

🤖 Generated with Claude Code