Repository navigation
0.1.4: fixes from the audit (offer queue, ssr filters, facets, diff sidecar, atomic writes) - #5
Merged
Merged
Conversation
…works in all three engines A third-party audit found that a good -> 404 -> good queue fetched two of its three offers, wrote one row, and reported `complete` with `stop_reason: no_new_products`. Reproduced live before touching anything. Running the same queue through all three engines afterwards found it was worse than the audit said, in four more places. The decision was re-spelt at every place an engine finishes a page, and the copies disagreed: * Playwright's sequential loop exempted `not_found` from the terminal test and then applied "this page added no new rows, so the listing ended" two lines later. The comment promised the queue would continue; the code stopped it. * Playwright's concurrent worker set the same "end of listing" event on any page with no rows, so a 404 stopped dispatch there too. * The FIRST page stopped the whole run on a 404, so a sitemap whose head had been taken down fetched nothing at all. * pyppeteer and Selenium had no exemption whatsoever. It is now one function, `page_flow.queue_decision`, called from all of them. In --mode offer every queued page is its own offer, so nothing one page says is evidence about the next; in a listing, a terminal state or a page that adds nothing still ends the run (pinned as the control). Found only because the queue was run through every engine, not just one: * Selenium and pyppeteer assigned the bare slug as the first address (`target = offer_slugs[0]`), so `--mode offer` over the sitemap navigated to a string like `acme-dev-warszawa-java`: exit 5 in Selenium, a traceback in pyppeteer. Playwright built the URL. README said "all three engines produce identical output", measured on --mode listings only. * pyppeteer called `pool.advance` without checking there was a pool. With no proxy `block_retries` is 1, not 0, so ANY navigation failure died with an AttributeError (exit 1) instead of reporting the failure (exit 5). The same class of defect v0.1.0 fixed one screen down. * In both parity engines the board-wide count request sat inside `if target != args.url:`, so it was skipped whenever --url named the endpoint, silently dropping the cap arithmetic from the sidecar. The sidecar now says how many queued offers were gone (`offers_not_found`, and which pages), because `pages_completed` counts pages that ANSWERED and could not otherwise distinguish "gone" from "read". Verified live, 3 engines x 3 queue shapes (404 first, middle, last): every cell requests all three offers, writes two rows, exits 0, `completed`. Five planted faults each turn the suite red on the check written for them. Not changed: a transient load failure on one offer still ends the run as `page_load_timeout`. That is a separate decision about retry budgets, not a 404, and the audit did not raise it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…before any request
An audit ran `--route ssr --experience senior --with-salary --keyword rust` and
found the fetched address was the bare all-locations page while the sidecar
recorded all three filters. Reproduced live: 23 of 50 rows senior, 28 of 50
priced, and `sort: salary` written beside rows in the site's own order.
What the rendered page applies was MEASURED rather than assumed, each
candidate judged by how many of its 50 rows satisfy it against a control
fetched the same minute:
applied the path (/warszawa, /remote, /remote/python: 50, 50, 49 of 50)
and ?keyword= (47 of 50 mention rust, against 0)
not applied with-salary, workplace-type, remoteWorkOptions, working-time,
sort and the API's own spellings of experience and employment
partly experience-level= is applied for junior (50 of 50, control 5)
and senior, but not for `manager` (2 of 50, control 1). A flag
that works for two values of six is the same defect on a
smaller surface, so it is not offered.
The route now takes exactly --category, --city, --remote and --keyword, and
`args_problem` refuses the rest with exit 2 before a request is made,
naming every offender at once. A --url is rebuilt from the merged filters
instead of fetched as given, because as given the flags beside it were
accepted and dropped. The sidecar records what ARRIVED: no sort, no order, no
radius, and the page size the site actually served.
`args_problem` replaces the filter-validation block that each of the three
engines carried a copy of, and gains the --retries rule: 0 made
`range(1, 0 + 1)` empty, so the page was never fetched and the run ended
`came back as unknown`, exit 4 — 'nothing found' for a fetch that never
happened. It is now exit 2.
Two more things found while measuring, neither in the audit:
* the scraper itself requested /api/candidate-api/offers/count on EVERY
listings run, --route ssr included, to compute the cap share. robots.txt
disallows /api/ and the README called the ssr route 'entirely within
robots.txt'. That request is skipped for ssr now; an offline check asserts
the route makes no /api/ request and that the api route still does, so the
assertion cannot pass on a fixture that never asks.
* the rendered payload changed shape under us. Its `meta` block (and with it
`totalItems`) is gone, the offers now sit in RSC props, and a page holds
50 rows rather than 100. The parser still finds all 50, but ssr runs can no
longer state a total, so they say nothing about the cap instead of
something fetched from the wrong place.
Six planted faults (count request restored, refusal disabled, --url fetched
verbatim, --retries 0 accepted, keyword dropped from the address, sidecar
claiming the endpoint's sort) each turn the suite red on the check written
for them.
Not changed: experience-level and employment-type stay unexposed on ssr
rather than half-supported; supporting them means measuring all six values of
each, which is a feature and not a fix.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ppeteer now ask for categories Two findings, one defect class. An audit traced that a failed request for the category counts is caught, logged as "the facet groups are still complete", and the run goes on as `complete` exit 0. Reproduced by making the request fail: 25 rows and a whole group gone, state still `content`. Running the mode through the other two engines then showed the same loss was not an outage but a permanent condition there. Selenium and pyppeteer imported `FACETS_JOIN` and `categories_api_url` and used neither, so every `--mode facets` run through them omitted the `categories` group: 57 rows against Playwright's 82, status `complete`. The group is the one the README's plan for reaching past the 10,000-offer cap is built from. The parity check that was meant to prevent this had only ever been run on --mode listings. Both engines now request the categories through the same `_fetch_body` shape Playwright has. And the verdict no longer depends on HOW a group was lost: `missing_facet_groups` compares the rows against the seven groups the board publishes, so a categories request that raised, a categories body that would not parse, and a group the site renamed all end the same way. A first version asked only whether the request had raised, which covers one of the three. A run missing a group stops with `facet_groups_missing`, which is not in COMPLETE_STOP_REASONS, so the family's own rule makes it status `partial`, exit 6 without a special case. The groups that did arrive are still written, and the sidecar names what is missing. The verdict is computed from the MERGED rows inside scrape(). An earlier draft set it inside the per-page fetch, which the offline tests replace, so the real logic would have been unreachable by them. Verified live: all three engines return 82 rows in 7 groups, `complete`. Five planted faults (categories never requested in Selenium, in pyppeteer; the partial verdict disabled; categories dropped from the expected set; the sidecar not naming the missing group) each turn the suite red on the check written for them. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…s one; every engine records its filters - diff_runs: a missing .meta.json removed every guard at once and the diff went ahead. Now a refusal, with --force as the explicit way round. An absent 'filters' key (did not say) is told apart from an empty one (no filters). - scraper_api_client: finish_run instead of save, so the row class follows the mode (facets CSV had the JobPosting header) and a sidecar is written; a facets run is partial (categories are a second endpoint). - engines: filters always written to the sidecar, even when empty. Each change has a planted-fault control that turns the suite red on the named check. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…good file write_json, write_csv and write_run_meta opened the target with 'w', which truncates before the first byte. They now write beside it and os.replace, with fsync, the temp file in the target's own directory, and the previous mode kept (NamedTemporaryFile is 0600 and a rename keeps it). Seven planted faults each turn the suite red on the named check; the newline one is structural because POSIX never translates newlines. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes from the third-party audit of
3222f2e, each verified by reproducing it first and each with a planted-fault control that turns the suite red on the named check.Fixed
page_flow.--mode offerworks in Selenium and pyppeteer; pyppeteer no longer crashes without a proxy pool.--route ssrrefuses filters the page ignores (exit 2, before any request) and records no sort/order/total it does not have. It makes no/api/request (robots.txt).--retries 0is refused.--mode facetsis partial (exit 6,missing_groups) when a group is missing; the parity engines now request the categories.diff_runs.pyrefuses a run with no sidecar (--forceto override); an absentfilterskey is told apart from an empty one.Not changed
_solve_budget, and the suite pins it.Verification
Suite: 776 checks green in the Playwright venv, 743 in the pyppeteer and Selenium venvs (the other engines' groups skip by design). Not verified here: the 0.1.4 CI run, and the sibling repos, which may share the diff_runs and atomic-write gaps.
🤖 Generated with Claude Code