Repository navigation
v1.0.1: fixes from the 2026-09-27 audit - #1
Merged
Merged
Conversation
… .env on --help From the third-party audit of 2026-09-27 and its re-check: - Publish JSON, CSV and sidecar atomically (temp file + fsync + os.replace, all staged, sidecar last). - Record the listing's own page numbers in logs, debug dumps and the sidecar; a run started on WS>0 named the wrong failed page. - List pages a concurrent run never attempted in `pages_unattempted`. - Refuse a missing --out directory before any fetch; a debug dump that cannot be written no longer turns exit 3 into exit 5. - Report internal errors (TypeError, KeyError, ...) as exit 1 with a masked traceback, keeping gathered rows, instead of as a transport failure. - Create raw page dumps 0600. - env_config.py and fingerprint_client.py parse argv before loading .env, so --help (run by CI and the offline suite) no longer reads it; the config report shows values by allowlist only. - Validate scraper_api_client.py's --timeout/--retries/--retry-delay. - Run the offline suite once in CI; fix two stale docstrings. Every fix is pinned by a smoke_test.py check that fails on 1.0.0. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the findings of the 2026-09-27 third-party audit that survived a re-check against the code, plus one the re-check found (
--helpreading.env). Details inCHANGELOG.md→ 1.0.1.Fixed
WS=120whose next page failed recordedpages_failed: [2]; now[3].pages_unattempted: pages a concurrent run's dead workers never fetched are now named, and make the run partial.--outdirectory checked before any fetch (exit 2). It used to turn a refused page into exit 5, and crash a successful run at the final write after all pages were paid for.internal_error, masked traceback, gathered rows kept) instead of a transport failure. Browser/driver errors unchanged.--helpno longer reads.env(env_config.py,fingerprint_client.py); the config report shows values by allowlist only.scraper_api_client.pyvalidates--timeout(1–120),--retries,--retry-delay.Deferred (not in this PR): FetchBackend / mobile API spike, schema v2,
src/layout, lockfiles, action SHA pinning, Docker digest + non-root.Verification
smoke_test.py: 359 PASS (Python 3.14 and 3.9);pytest: 1 passed;ci_checks.py --all: green.smoke_test.pyrun against 1.0.0 fails 25 checks, one or more per fix..envand any socket use: zero hits. A new check proves--helpof every CLI opens no.env(scratch copy + canary.env, so it works in CI too).WS=120→ pages 2–3, complete. Logs/outputs scanned for credential fragments: 0 hits.🤖 Generated with Claude Code