diff --git a/CHANGELOG.md b/CHANGELOG.md index 88ad899..032e77e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,14 @@ burying it. ## [Unreleased] -> **Behaviour changes.** CSV output now prefixes a cell that starts with +## [0.1.4] + +> **Behaviour changes for existing users.** `--route ssr` now refuses, with +> exit 2, any filter other than category, city, remote and keyword (it used +> to accept them and ignore them). `diff_runs.py` now refuses a run with no +> sidecar unless `--force` is passed. `--retries 0` is refused. +> +> CSV output now prefixes a cell that starts with > `=`, `+`, `-`, `@`, a tab, CR or LF with an apostrophe (JSON is > unchanged), and the sidecar gains `csv_cells_escaped`. `.env.example` > no longer sets `JUSTJOIN_URL`; a `JUSTJOIN_URL` from the environment that @@ -20,6 +27,20 @@ burying it. ### Fixed +- **An offer queue no longer ends at the first delisted offer** (all three + engines). A 404 on one offer was read as the end of the run. +- **`--mode offer` works in Selenium and pyppeteer**, and pyppeteer no + longer crashes without a proxy pool. +- **`--route ssr` no longer accepts filters it ignores**, and no longer + records a sort, order or total the page does not carry. +- **`--mode facets` is partial when a group is missing**, and the parity + engines now request the categories. +- **`diff_runs.py` refuses runs without a sidecar**, and tells an absent + `filters` key from an empty one. Engines always record `filters`. +- **The Scraper API client writes a sidecar** and picks the row class by + mode (its facets CSV had the wrong header). +- **Output writes are atomic.** A failed write no longer replaces the + previous good JSON, CSV or sidecar with a truncated one. - **CSV cells are formula-escaped.** Every string in a row is written by an employer, and a spreadsheet runs a cell starting with `=`, `+`, `-` or `@` as a formula. Escaped in the CSV only, after list cells are joined; diff --git a/README.md b/README.md index 2ed8706..0f0423e 100644 --- a/README.md +++ b/README.md @@ -112,9 +112,18 @@ is where the offers endpoint lives. This scraper defaults to the endpoint, because it is what the site's own front end calls from the browser and it is the only route with pagination. That is stated here rather than left to be found. `--route ssr` reads the -rendered listing page instead: **one page of 100 rows** with a poorer -schema (no category, no languages, no skill levels), entirely within -robots.txt. +rendered listing page instead, entirely within robots.txt: it requests only +`/job-offers/...` and makes no `/api/` request (checked on 2026-10-07 by +logging every request of a run). What it gives up: + +- **one page of about 50 rows** (it was 100 until the page's payload changed + on 2026-10-07; the payload now carries no `meta` and no `totalItems`, so a + total is not recorded for this route); +- a poorer schema (no category, no languages, no skill levels); +- **four filters only**: `--category`, `--city`, `--remote` and `--keyword`. + Every other filter is refused with exit 2 before any request, naming the + flag, because the page ignores it and the run would otherwise return an + unfiltered page labelled as a filtered one. --- @@ -377,14 +386,32 @@ fetch that page, or the row fell outside this run's slice of a capped result set. `diff_runs.py` warns when either run was capped, and refuses a pair that used different modes, sorts, orderings or filters. +It also refuses a run that has **no `.meta.json` beside it**: everything +those checks read lives in that file, so its absence removes all of them at +once. `--force` diffs on trust. A sidecar written before filters were +recorded is compared, with a note that the selections could not be. + +A run that fetched an offer queue (`--mode offer`) does not end on an offer +the board has since delisted: that URL answers 404, is counted in the +sidecar (`offers_not_found`), and the queue goes on. `--retries 0` is +refused (exit 2) because it would mean no attempt at all. + +`--mode facets` is `partial` (exit 6) when any of its groups is missing from +the result, naming the groups in the sidecar's `missing_groups`. The Scraper +API client fetches a single URL, so its facets run is partial by +construction. + --- ## Engines Playwright is primary. Selenium and Puppeteer are parity engines — same -flags, same modes, same exit codes, same output. Verified on 2026-09-18: -all three produced **byte-identical rows** (modulo `scraped_at`) and -identical sidecars for the same command. +flags, same modes, same exit codes, same output. Verified per mode: on +2026-09-18 `--mode listings` produced **byte-identical rows** (modulo +`scraped_at`) and identical sidecars in all three engines. That measurement +covered listings only; `--mode offer` and the `facets` categories were broken +in the parity engines until 2026-10-07, when they were fixed and run live in +all three. ```bash pip install -r requirements.txt -r requirements-playwright.txt # primary diff --git a/diff_runs.py b/diff_runs.py index 61e2936..2064225 100644 --- a/diff_runs.py +++ b/diff_runs.py @@ -326,9 +326,10 @@ def _print_summary(result: dict) -> None: def _run_status(path: str) -> Tuple[Optional[str], Optional[dict]]: """Read the `.meta.json` sidecar beside a run's JSON output. - Returns (status, meta), or (None, None) when there is no sidecar — which - is the normal case for output written before run metadata existed, or by - `scraper_api_client.py` (single fetch, no pagination to cut short). + Returns (status, meta), or (None, None) when there is no sidecar — output + written before run metadata existed, or a sidecar that was deleted or + never copied beside the JSON. `_check_comparable` treats that as a reason + to refuse, not as a pass. """ meta_path = re.sub(r"\.json$", "", path) + ".meta.json" try: @@ -354,7 +355,25 @@ def _check_comparable(args) -> bool: for label, path in (("--old", args.old), ("--new", args.new)): status, meta = _run_status(path) if status is None: - continue # no sidecar: nothing to check, see _run_status + # No sidecar is NOT "nothing to check". Everything this function + # checks — whether the run was complete, which mode it was, how it + # was sorted and filtered, whether the result set was capped — + # lives in that file, so its absence removes every guard at once + # and the diff proceeds exactly as if all of them had passed. Two + # bare JSON files were accepted with exit 0, and an unfiltered run + # beside a filtered one reported hundreds of `removed`. + # + # Refused, with --force as the explicit way round. Output written + # by scraper_api_client.py used to be the reason for the leniency + # (it wrote no sidecar); it writes one now, so that reason is gone. + problems.append( + f"{label} ({path}) has no .meta.json beside it, so nothing " + f"says whether it is complete, which mode it was, or how it " + f"was sorted and filtered. Without that, every `removed` line " + f"may be an offer that was simply never fetched or fell " + f"outside a different selection. Re-run it with a current " + f"version, or pass --force to diff it on trust.") + continue mode = (meta or {}).get("mode") if mode: modes[label] = mode @@ -421,8 +440,12 @@ def _check_comparable(args) -> bool: sorts[label] = meta["sort"] if meta.get("order"): orders[label] = meta["order"] - if meta.get("mode") == "listings": - filters[label] = meta.get("filters") or {} + # Only when the run RECORDED its filters. A sidecar that says nothing + # about them (written before this field existed) is "unknown", which is + # not the same as "none" — treating the two alike either refuses a + # comparison that is fine or waves through one that is not. + if meta.get("mode") == "listings" and "filters" in meta: + filters[label] = meta["filters"] or {} if len(set(sorts.values())) > 1: problems.append( f"the two runs used different orderings ({sorts}). Every query " @@ -436,6 +459,14 @@ def _check_comparable(args) -> bool: f"the two runs used different sort directions ({orders}), which " f"on a capped result set means they hold opposite ends of the " f"same query.") + if len(filters) == 1: + # One side recorded its selection and the other did not, so it was not + # compared. Said out loud rather than skipped: the diff below is only + # as meaningful as the two selections being the same. + unknown = "--new" if "--old" in filters else "--old" + print(f"[i] {unknown} did not record which filters it used, so the two " + f"selections could not be compared. If they differ, `added` and " + f"`removed` describe that difference rather than the board.") if len(filters) == 2: old_f, new_f = filters["--old"], filters["--new"] if old_f != new_f: diff --git a/output_writer.py b/output_writer.py index eb0fa7f..d9a2441 100644 --- a/output_writer.py +++ b/output_writer.py @@ -106,8 +106,11 @@ empty CSV still gets the right header for the mode that produced it. """ +import contextlib import csv import json +import os +import tempfile from dataclasses import dataclass, asdict, field, fields from datetime import datetime, timezone from typing import Optional, List, Set, Sequence, Any, Tuple, Type @@ -515,8 +518,62 @@ def _csv_cell(v: Any) -> Tuple[Any, bool]: return v, False +def _target_mode(path: str) -> int: + """The permissions the finished file should have. + + `NamedTemporaryFile` creates 0600 and a rename keeps it, so without this + every output would come out readable by its owner only — a cron job + writing for another user, or a CI step uploading as a different one, + would find the file there and be unable to read it. Keep the mode the + previous file had; for a first write use what `open()` would have given + (0666 minus the umask). + """ + try: + return os.stat(path).st_mode & 0o7777 + except OSError: + mask = os.umask(0) + os.umask(mask) + return 0o666 & ~mask + + +@contextlib.contextmanager +def _atomic(path: str, newline: Optional[str] = None): + """Write to a temporary file beside `path`, then rename over it. + + Every write here replaces a file a previous run may have left, and the + invariant this module protects is that a bad run never destroys last + night's good data (`save` refuses to overwrite with an empty result for + the same reason). `open(path, "w")` truncates before the first byte is + written, so a kill, a full disk or a crash halfway through `json.dump` + leaves a SHORTER file where a complete one was — and the sidecar beside + it, which a consumer branches on, still describes the old good run. + + `os.replace` is atomic within one filesystem, so the temporary file is + created in the target's OWN directory; one under /tmp may be on another, + where a rename degrades to an interruptible copy. `fsync` before the + rename makes it hold after a power loss and not only after a crash. + """ + directory = os.path.dirname(os.path.abspath(path)) or "." + handle = tempfile.NamedTemporaryFile( + mode="w", encoding="utf-8", newline=newline, dir=directory, + prefix=os.path.basename(path) + ".", suffix=".tmp", delete=False) + try: + with handle: + yield handle + handle.flush() + os.fsync(handle.fileno()) + os.chmod(handle.name, _target_mode(path)) + os.replace(handle.name, path) + except BaseException: + try: + os.unlink(handle.name) + except OSError: + pass + raise + + def write_json(rows: Sequence[Any], path: str) -> None: - with open(path, "w", encoding="utf-8") as f: + with _atomic(path) as f: json.dump([asdict(r) for r in rows], f, ensure_ascii=False, indent=2) @@ -529,7 +586,7 @@ def write_csv(rows: Sequence[Any], path: str, row_cls: Type = JobPosting) -> int # The header comes from `row_cls`, not from the first row, so an empty # run still writes the columns of the mode that produced it. fieldnames = [f.name for f in fields(row_cls)] - with open(path, "w", encoding="utf-8", newline="") as f: + with _atomic(path, newline="") as f: writer = csv.DictWriter(f, fieldnames=fieldnames) writer.writeheader() escaped = 0 @@ -629,7 +686,7 @@ def write_run_meta(out_prefix: str, meta: dict) -> str: both complete, and between runs of different `mode`. """ path = f"{out_prefix}.meta.json" - with open(path, "w", encoding="utf-8") as f: + with _atomic(path) as f: json.dump(meta, f, ensure_ascii=False, indent=2) print(f"[+] Wrote run metadata -> {path} (status={meta.get('status')})") return path diff --git a/page_flow.py b/page_flow.py index 4a8930a..1291f76 100644 --- a/page_flow.py +++ b/page_flow.py @@ -250,6 +250,54 @@ def is_terminal(state: str) -> bool: return state in ("not_found", "cap_exceeded") +def pages_are_independent(mode: str, queued) -> bool: + """True when every queued page is its OWN offer rather than a slice of one + result set. + + In `--mode offer` a "page" is one offer fetched from an enumerated list, + so nothing one page says — empty, repeated, or 404 — is evidence about + the next. In a listing the opposite holds: a page that adds nothing new + IS evidence that the listing has ended. + + A single `--url` offer has no queue behind it, so it is not "independent" + in this sense: there is nothing to continue to, and its 404 should still + be the reason the run stopped. + """ + return mode == "offer" and bool(queued) + + +def queue_decision(state: Optional[str], fresh_rows: Optional[int], + independent: bool) -> Optional[str]: + """What a finished page says about the REST of the queue. + + Returns the `stop_reason` that ends the queue, or None to carry on. One + function, called from every place an engine finishes a page — the first + page, the sequential loop and the concurrent workers — because it used to + be re-spelt at each of them, and the copies disagreed in the way that + matters: + + * the Playwright loop skipped the terminal check for an offer's 404 and + then applied "this page added no new rows, so the listing ended" two + lines later, so one delisted offer silently cut the queue short and + the run reported `complete` with `stop_reason: no_new_products`; + * its concurrent worker set the same "end of listing" event on a 404; + * the first page stopped the whole run on a 404, so a sitemap whose + first slug had been taken down fetched nothing at all; + * the two parity engines had no skip at all. + + `fresh_rows=None` means "do not apply the exhausted-listing rule", which + is what the first page wants: its content decides how many pages there + are, not whether to stop. + """ + if independent: + return None + if state is not None and is_terminal(state): + return state + if fresh_rows is not None and fresh_rows == 0: + return "no_new_products" + return None + + # Whether a blocked page is worth re-fetching at all. # # True, and CONSULTED rather than merely documented — the engines read it, diff --git a/playwright_scraper.py b/playwright_scraper.py index ef225e2..61cfde8 100644 --- a/playwright_scraper.py +++ b/playwright_scraper.py @@ -126,6 +126,7 @@ sitemap_is_index, sitemap_locs, sitemap_offer_slugs, sku_from_url, user_agent_is_refused) +from product_parser import args_problem, missing_facet_groups from output_writer import (dedupe_by_key, finish_run, EXIT_API_ERROR, ROW_CLASS_BY_MODE, SOURCE_DEFAULT) import page_flow @@ -373,10 +374,9 @@ def _fetch_body(session, args, url: str) -> str: try: categories = _fetch_text(session, categories_api_url()) except (PWError, PWTimeout) as exc: - logger.warning("The category counts did not load (%s); the facet " - "groups are still complete. Categories will be " - "absent from this run rather than wrong.", - _mask_credentials(str(exc))[:120]) + logger.warning("The category counts did not load (%s); the run " + "will be partial rather than silently missing " + "them.", _mask_credentials(str(exc))[:120]) return body if categories.strip(): return body + FACETS_JOIN + categories @@ -1239,6 +1239,13 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome return outcome products, listing = parse_for_mode(body, session.page.url, args, page_num) + if args.mode == "facets": + gone = missing_facet_groups(products) + if gone: + logger.warning("The facet response is missing %s. The groups that " + "arrived are written, and the run is partial: " + "categories are what a plan for reaching past the " + "10,000 cap is built from.", ", ".join(gone)) logger.info("Parsed %d row(s) from page %d.", len(products), page_num) if listing is not None: @@ -1406,6 +1413,9 @@ def _fetch_pages_concurrently(args, pool, specs, concurrency: int): # 45 empty ones. Workers check it before taking more work, so at most # (concurrency - 1) extra pages are in flight when it trips. exhausted = threading.Event() + # Every queued page is its own offer in --mode offer, so a 404 or an empty + # one says nothing about the next — see page_flow.queue_decision. + independent = page_flow.pages_are_independent(args.mode, specs) def worker(index: int): name = f"worker-{index + 1}" @@ -1426,10 +1436,12 @@ def worker(index: int): page_num, url) with results_lock: results.append(outcome) - if outcome.ok and not outcome.products: - logger.info("[%s] page %d returned no rows — " - "treating that as the end of the listing " - "and stopping dispatch.", name, page_num) + if outcome.ok and page_flow.queue_decision( + outcome.state, len(outcome.products), + independent): + logger.info("[%s] page %d (%s) ends the listing — " + "stopping dispatch.", name, page_num, + outcome.state) exhausted.set() finally: session.close() @@ -1575,7 +1587,15 @@ def scrape(args) -> int: # says "complete" and one that says "complete, and a 52% # sample". Never fatal: a run without it is # still a good run, it just cannot state the share. - if args.mode == "listings": + # NOT for --route ssr. That route exists to stay inside robots.txt + # (which disallows /api/), and this request goes to + # /api/candidate-api/offers/count — so the scraper itself was + # fetching a disallowed path on the one route documented as + # robots-clean. The rendered page no longer states a total + # either (its `meta` block is gone, 2026-10-07), so an ssr + # sidecar says nothing about the cap rather than something fetched + # from the wrong place. + if args.mode == "listings" and args.route != "ssr": try: site_total = parse_count_response( _fetch_text(session, count_api_url())) @@ -1589,14 +1609,20 @@ def scrape(args) -> int: first = _fetch_one_page(session, args, pool, 1, target) outcomes.append(first) + independent = page_flow.pages_are_independent(args.mode, + offer_slugs) + first_verdict = page_flow.queue_decision(first.state, None, + independent) if not first.ok: stop_reason = ("page_load_timeout" if first.load_failed else f"blocked_{first.blocked_by}") blocked = first.blocked_by is not None - elif page_flow.is_terminal(first.state): + elif first_verdict: # Terminal by nature: another attempt at the same address - # cannot help, and neither is a block. - stop_reason = first.state + # cannot help, and neither is a block. NOT applied to an + # offer that heads a queue of other offers — one delisted + # slug must not cancel the run (`page_flow.queue_decision`). + stop_reason = first_verdict elif args.mode == "offer" and not offer_slugs: pass # `--url` named one offer; one page is the whole run else: @@ -1662,15 +1688,6 @@ def scrape(args) -> int: else f"blocked_{outcome.blocked_by}") blocked = outcome.blocked_by is not None break - if page_flow.is_terminal(outcome.state): - # `not_found` is ordinary in --mode offer: an - # offer taken down between the sitemap being - # read and its record being fetched. It ends - # THIS page, not the run. - if args.mode != "offer" or outcome.state != "not_found": - stop_reason = outcome.state - break - # Whether this page contributed anything not already # seen. Kept as a running check because the # condition is inherently sequential — "new" only @@ -1682,17 +1699,23 @@ def scrape(args) -> int: seen_keys.update(p.sku for p in outcome.products if p.sku is not None) - # A page past the first that contributes nothing new - # means the end of the results — or that pagination - # is looping back on itself. Either way there is - # nothing further to fetch, and this is the honest - # terminating condition: a property of the DATA, not - # of a CSS selector that may have been renamed. - if not fresh_count: - logger.info("Page %d added no rows not already seen " - "— treating that as the end of the " - "results.", page_num) - stop_reason = "no_new_products" + # ONE decision, shared with the first page and the + # concurrent workers, about whether this page ends the + # queue: a terminal state, or a page that adds nothing + # new (a property of the DATA, not of a selector). + # In --mode offer neither applies, + # because every page is a different offer: a 404 is an + # offer taken down between the sitemap and the fetch. + # This used to exempt not_found from the first test and + # then trip the second one two lines later, so a single + # delisted offer cut the run short and left it + # `complete`. + verdict = page_flow.queue_decision( + outcome.state, fresh_count, independent) + if verdict: + logger.info("Page %d (%s) ends the queue: %s.", + page_num, outcome.state, verdict) + stop_reason = verdict break if page_num - 1 < len(planned): @@ -1786,6 +1809,17 @@ def scrape(args) -> int: "" if done >= total_enumerated else " Pass --pages %d to cover the rest." % total_enumerated) + # A facets run that lacks a group is partial, not complete. `stop_reason` + # is deliberately NOT one of COMPLETE_STOP_REASONS, so finish_run reports + # status partial and exit 6 through the family's own rule + # rather than through a special case here. + # Judged on the MERGED rows, not on any one response, so the verdict does + # not depend on how a group went missing (see EXPECTED_FACET_GROUPS). + missing_groups = (missing_facet_groups(all_rows) + if args.mode == "facets" and all_rows else []) + if missing_groups: + stop_reason = "facet_groups_missing" + ok_pages = [o for o in outcomes if o.ok] failed_pages = [o.page_num for o in outcomes if not o.ok] final_url = (max(ok_pages, key=lambda o: o.page_num).final_url @@ -1802,20 +1836,48 @@ def scrape(args) -> int: if args.mode == "listings": extra.update(cap) extra["pages_available"] = pages_available - extra["items_per_page"] = args.per_page - extra["sort"] = args.sort - extra["order"] = args.order + if args.route == "ssr": + # The rendered page serves what the site chose to render — 50 rows + # on 2026-10-07, 100 a fortnight earlier — in the site's own order. + # `--per-page`, `--sort` and `--order` are refused for this route + # (`args_problem`), so recording them here would put three values + # in the sidecar that nothing applied. Record what ARRIVED instead. + extra["items_per_page"] = (len(outcomes[0].products) or None + if outcomes else None) + else: + extra["items_per_page"] = args.per_page + extra["sort"] = args.sort + extra["order"] = args.order # The ordering decides WHICH rows are in the file, not just their # order: the same query under `publishedAt` and under `salary` # overlapped on 5 of 100 ids (measured 2026-09-18), because each is # a different slice of a capped result set. So it is recorded, and # `diff_runs.py` refuses to compare two runs that disagree on it. filters = filters_from_args(args).as_params() - if filters: - extra["filters"] = filters + if args.route == "ssr": + # A radius is the endpoint's parameter; the page's own path takes + # a city and no radius, so claiming one was applied would be the + # same overstatement as claiming a filter it ignored. + filters.pop("cityRadius", None) + # ALWAYS recorded, even when empty. `diff_runs.py` reads the absence of + # this key as "this run did not say", which it must be able to tell + # apart from "this run used no filters". + extra["filters"] = filters + elif args.mode == "facets" and missing_groups: + extra["missing_groups"] = missing_groups elif args.mode == "offer": extra["offers_enumerated"] = total_enumerated or None extra["offers_fetched"] = len(ok_pages) + # Which pages of the queue were an offer the site no longer has. A 404 + # is ordinary here — the sitemap is generated ahead of the fetch — so + # it does not make the run partial, but it is not invisible either: + # `offers_fetched` counts pages that ANSWERED, and without this a + # consumer cannot tell an answer of "gone" from an offer that was read. + not_found = sorted(o.page_num for o in outcomes + if o.state == "not_found") + extra["offers_not_found"] = len(not_found) + if not_found: + extra["offers_not_found_pages"] = not_found return finish_run(all_rows, args.out, args.format, args.allow_empty, blocked=blocked, stop_reason=stop_reason, @@ -2122,23 +2184,12 @@ def parse_args(): p.error("--per-page must be between 1 and %d; justjoin.it answers " "HTTP 400 above that." % MAX_ITEMS_COUNT) - if args.mode != "listings": - # Refused rather than ignored, so a filtered `--mode offer` run - # cannot look like it narrowed the enumeration when it did not. - used = [name for name, value in ( - ("--category", args.category), ("--city", args.city), - ("--keyword", args.keyword), ("--remote", args.remote), - ("--experience", args.experience), ("--language", args.language), - ("--employment", args.employment), - ("--working-time", args.working_time), - ("--with-salary", args.with_salary)) if value] - if used: - p.error("%s filter%s the offers endpoint, which --mode %s does " - "not read. Use --mode listings to filter, then --mode " - "offer --slugs-file on its output to fetch those offers " - "in full." - % (", ".join(used), "" if len(used) == 1 else "s", - args.mode)) + # ONE shared validation (product_parser.args_problem) instead of a copy per + # engine: filters outside --mode listings, the filters the rendered route + # cannot apply, and --retries 0 (which skipped navigation entirely). + problem = args_problem(args) + if problem: + p.error(problem) # The sidecar's run label. `--category` is a real site filter here, so # unlike the repo this engine was ported from it is NOT reused as the diff --git a/product_parser.py b/product_parser.py index e60cbe7..7126f8c 100644 --- a/product_parser.py +++ b/product_parser.py @@ -1957,6 +1957,22 @@ def add(group: str, key: Any, count: Any) -> None: return out +# Every group the board publishes today: the six in `/offers/facets/count` plus +# the categories from their own endpoint. A run that returns fewer is not a +# smaller board, it is an incomplete read — and it is judged on the ROWS, not +# on how the missing group went missing, so a categories request that failed, +# a categories body that would not parse and a group the site renamed all +# land in the same place. (The first version of this check asked only whether +# the categories REQUEST had raised, which covers one of the three.) +EXPECTED_FACET_GROUPS = FACET_GROUPS + (CATEGORY_FACET_GROUP,) + + +def missing_facet_groups(rows: Sequence[Any]) -> List[str]: + """The expected facet groups that no row carries.""" + present = {getattr(r, "facet_group", None) for r in rows} + return [g for g in EXPECTED_FACET_GROUPS if g not in present] + + def parse_count_response(text: Optional[str]) -> Optional[int]: """The board-wide offer count from `/offers/count`.""" payload = parse_json(text) @@ -2067,6 +2083,123 @@ def filters_from_args(args) -> Filters: +# What the RENDERED listing page applies, measured 2026-10-07 by fetching it and +# counting how many of its 50 rows satisfy each candidate, against a control +# fetched in the same minute (a filter that is accepted and +# ignored looks exactly like one that works): +# +# APPLIED the path: /job-offers/{city}/{category}, /job-offers/remote and +# /job-offers/remote/{category} (warszawa: 50 of 50 in +# Warsaw; remote: 50 of 50 remote; remote/python: 49 of 50 python) +# ?keyword= 47 of 50 mention rust, against 0 of 50 +# NOT APPLIED with-salary / withSalary / salary 33 of 50 priced, control 30 +# workplace-type / remoteWorkOptions 19 remote, control 21 +# working-time / workingTimes 13 b2b_contract, control 12 +# sort / sortBy / orderBy=ascending first row not the oldest +# employmentTypes / experienceLevels the API's names; 9 and 30 +# of 50, control 9 and 24 +# PARTLY experience-level= junior 50 of 50 (control 5) and senior 50 of +# 50, but `manager` 2 of 50 (control 1): the page's value +# vocabulary is not the API's, and a flag that works for two +# values of six is the defect again with a smaller surface. +# employment-type=permanent 46 of 50 (control 9), unmeasured for +# the other five. +# +# So the route takes exactly these four, and refuses the rest before any +# request is made. `--route api` takes all of them. +SSR_FILTERS = ("category", "city", "remote", "keyword") + + +def ssr_url_for(filters: Filters) -> str: + """The rendered-listing address for a set of filters. + + Path for the location and category, `?keyword=` for a search — the four + things this page measurably applies. A city and `remote` are alternatives + in the path (there is no `/warszawa/remote`), so `args_problem` refuses + the pair rather than this function quietly dropping one. + """ + where = filters.city or ("remote" if filters.remote else None) + if not (where or filters.category): + url = DEFAULT_LISTING_URL + else: + url = listing_url(city=where, category=filters.category) + if filters.keyword: + url += "?" + urllib.parse.urlencode({"keyword": filters.keyword}) + return url + + +def args_problem(args) -> Optional[str]: + """The reason these arguments cannot be honoured, or None. + + ONE implementation, called from every engine's `parse_args`, replacing the + copy each of them carried. Everything here is a request this scraper + could otherwise accept and quietly not carry out — the failure already + found in the BBB scraper and again in the Binance one. + """ + retries = getattr(args, "retries", 1) + if retries is not None and retries < 1: + return ("--retries is the number of ATTEMPTS per page, so 0 means " + "never navigating at all. It used to be accepted and ended " + "in 'came back as unknown' and exit 4 on a perfectly healthy " + "site — a 'nothing found' for a fetch that never happened. " + "Use 1 for a single attempt.") + + used = [name for name, value in ( + ("--category", getattr(args, "category", None)), + ("--city", getattr(args, "city", None)), + ("--keyword", getattr(args, "keyword", None)), + ("--remote", getattr(args, "remote", False)), + ("--experience", getattr(args, "experience", None)), + ("--language", getattr(args, "language", None)), + ("--employment", getattr(args, "employment", None)), + ("--working-time", getattr(args, "working_time", None)), + ("--with-salary", getattr(args, "with_salary", False))) if value] + + if args.mode != "listings": + # Refused rather than ignored, so a filtered `--mode offer` run + # cannot look like it narrowed the enumeration when it did not. + if used: + return ("%s filter%s the offers endpoint, which --mode %s does " + "not read. Use --mode listings to filter, then --mode " + "offer --slugs-file on its output to fetch those offers " + "in full." + % (", ".join(used), "" if len(used) == 1 else "s", + args.mode)) + return None + + if getattr(args, "route", DEFAULT_ROUTE) != "ssr": + return None + + unsupported = [name for name in used + if name not in ("--category", "--city", "--keyword", + "--remote")] + if getattr(args, "city_radius", 30) not in (None, 30): + unsupported.append("--city-radius") + if getattr(args, "sort", DEFAULT_SORT) != DEFAULT_SORT: + unsupported.append("--sort") + if getattr(args, "order", DEFAULT_ORDER) != DEFAULT_ORDER: + unsupported.append("--order") + if getattr(args, "per_page", DEFAULT_ITEMS_COUNT) != DEFAULT_ITEMS_COUNT: + unsupported.append("--per-page") + if unsupported: + plural = len(unsupported) > 1 + return ("--route ssr reads the rendered listing page, which applies " + "only --category, --city, --remote and --keyword (measured " + "2026-10-07; see product_parser.SSR_FILTERS). It ignored %s " + "when tried, and a run that accepted %s would write %s into " + "its sidecar as if %s had narrowed the rows. Drop %s, or use " + "--route api, which takes every one of them." + % (", ".join(unsupported), "them" if plural else "it", + "them" if plural else "it", "they" if plural else "it", + "them" if plural else "it")) + if getattr(args, "city", None) and getattr(args, "remote", False): + return ("--city and --remote cannot be combined with --route ssr: the " + "page takes ONE location in its path (/job-offers/warszawa " + "or /job-offers/remote — there is no /warszawa/remote). Use " + "--route api to filter on both.") + return None + + def target_url(args) -> str: """The address this run actually fetches. @@ -2096,14 +2229,12 @@ def target_url(args) -> str: return api_url(items=1) if args.route == "ssr": - if args.url and route_of(args.url) == "listing": - return args.url - filters = filters_from_args(args) - if filters.category or filters.city or filters.remote: - return listing_url(city=filters.city or ( - "remote" if filters.remote else None), - category=filters.category) - return DEFAULT_LISTING_URL + # ALWAYS rebuilt from the merged filters, never returned as given. + # A user URL was fetched verbatim here while the flags beside it were + # accepted and ignored, so `--url .../warszawa --keyword rust` read + # an unfiltered Warsaw page and wrote `keywords: rust` into the + # sidecar. See `ssr_url_for` for what the page really applies. + return ssr_url_for(filters_from_args(args)) if args.url and route_of(args.url) == "api_offers": return args.url diff --git a/puppeteer_scraper.py b/puppeteer_scraper.py index 8eae366..4d6d323 100644 --- a/puppeteer_scraper.py +++ b/puppeteer_scraper.py @@ -68,6 +68,7 @@ parse_count_response, parse_for_mode, route_of, site_recaptcha_sitekey, split_facets_body, target_url) +from product_parser import args_problem, missing_facet_groups from output_writer import (dedupe_by_key, finish_run, EXIT_API_ERROR, ROW_CLASS_BY_MODE, SOURCE_DEFAULT) import page_flow @@ -527,6 +528,39 @@ def _fetch_page_source(session, url: str, timeout_ms: int = 60000) -> str: return session.bridge.run(session.page.content()) or "" +def _fetch_text(session, url: str, timeout_ms: int = 60000) -> str: + """Navigate and return the document's text — JSON exactly as served.""" + _fetch_page_source(session, url, timeout_ms) + return _snapshot(session, url) or "" + + +def _fetch_body(session, args, url: str) -> str: + """The document the parser is given for this fetch. + + `--mode facets` is TWO endpoints that are one answer — the facet groups, + and the category counts from their own route — joined with a separator the + parser undoes (`product_parser.FACETS_JOIN`). This engine had no such step + at all: it imported FACETS_JOIN and categories_api_url and never used + either, so every `--mode facets` run through it silently omitted the + `categories` group — 57 rows against Playwright's 82 — and reported + `complete`. The group it dropped is the one a plan for reaching past the + 10,000 cap is built from. Found by running the mode through this engine, + which the parity check had never done (it ran --mode listings only). + """ + body = _snapshot(session, url) or "" + if args.mode == "facets" and body.strip(): + try: + categories = _fetch_text(session, categories_api_url()) + except Exception as exc: # noqa: BLE001 — a driver raises many types + logger.warning("The category counts did not load (%s); the run " + "will be partial rather than silently missing " + "them.", _mask_credentials(str(exc))[:120]) + return body + if categories.strip(): + return body + FACETS_JOIN + categories + return body + + def handle_captcha_if_present(session, args) -> bool: """Detect and solve a challenge. True if something was solved. @@ -668,7 +702,12 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome args.retries, text[:120], pause) time.sleep(pause) - if load_failed and block_attempt < block_retries: + # `has_pool`, as in the Playwright engine. Without it, any navigation + # failure on a run with NO proxy — `block_retries` is 1 then, not 0 — + # went on to `pool.advance` on None and died with an AttributeError + # traceback (exit 1) instead of reporting the load failure (exit 5). + # Found by running `--mode offer` live, whose first address was bad. + if load_failed and has_pool and block_attempt < block_retries: pool.advance("unusable exit or repeated load failure") session.relaunch() bridge, page = session.bridge, session.page @@ -688,7 +727,7 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome logger.info("Not solving again on page %d: %d purchase(s) " "already made for it and SOLVES_PER_PAGE is %d.", page_num, solves_bought, page_flow.SOLVES_PER_PAGE) - html = _snapshot(session, url) or "" + html = _fetch_body(session, args, url) state = page_flow.classify(html, http_status, page.url, args.mode, args.route) @@ -710,7 +749,7 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome if found < need: logger.info("Still nothing after %.0fs (%d match(es) for %s).", wait_ms / 1000.0, found, sel) - html = _snapshot(session, url) or html + html = _fetch_body(session, args, url) or html state = page_flow.classify(html, http_status, page.url, args.mode, args.route) @@ -867,6 +906,13 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome return outcome products, listing = parse_for_mode(html, page.url, args, page_num) + if args.mode == "facets": + gone = missing_facet_groups(products) + if gone: + logger.warning("The facet response is missing %s. The groups that " + "arrived are written, and the run is partial: " + "categories are what a plan for reaching past the " + "10,000 cap is built from.", ", ".join(gone)) logger.info("Parsed %d row(s) from page %d.", len(products), page_num) if listing is not None: @@ -1032,36 +1078,57 @@ def scrape(args) -> int: mode=args.mode, source=SOURCE_DEFAULT, start_url=args.url or "", final_url="") total_enumerated = len(offer_slugs) - target = offer_slugs[0] - - if target != args.url: - logger.info("Fetching %s", target) - - # The board-wide total, for the cap arithmetic. One extra - # request, and it buys the difference between a sidecar that - # says "complete" and one that says "complete, and a 52% - # sample". Never fatal. - if args.mode == "listings": - try: - site_total = parse_count_response( - _fetch_page_source(session, count_api_url())) - except Exception as exc: # noqa: BLE001 - logger.info("The board-wide count did not load (%s); the " - "run continues without the cap share.", - _mask_credentials(str(exc))[:120]) + # A slug is not an address. Both parity engines assigned it bare, + # so `--mode offer` over the sitemap "navigated" to a string like + # `acme-dev-warszawa-java` and ended in exit 5 (Selenium) or a + # traceback (pyppeteer). Playwright built the URL; this does too. + first_url = detail_api_url(offer_slugs[0]) + if first_url: + target = first_url + + if args.url and target != args.url: + logger.info("Reading %s through %s — the endpoint is the " + "route with pagination, and the filters your URL " + "names are carried over to it.", args.url, target) + logger.info("Fetching %s", target) + + # The board-wide total, for the cap arithmetic. One extra + # request, and it buys the difference between a sidecar that + # says "complete" and one that says "complete, and a 52% + # sample". Never fatal. + # NOT for --route ssr. That route exists to stay inside robots.txt + # (which disallows /api/), and this request goes to + # /api/candidate-api/offers/count — so the scraper itself was + # fetching a disallowed path on the one route documented as + # robots-clean. The rendered page no longer states a total + # either (its `meta` block is gone, 2026-10-07), so an ssr + # sidecar says nothing about the cap rather than something fetched + # from the wrong place. + if args.mode == "listings" and args.route != "ssr": + try: + site_total = parse_count_response( + _fetch_page_source(session, count_api_url())) + except Exception as exc: # noqa: BLE001 + logger.info("The board-wide count did not load (%s); the " + "run continues without the cap share.", + _mask_credentials(str(exc))[:120]) first = _fetch_one_page(session, args, pool, 1, target) outcomes.append(first) + independent = page_flow.pages_are_independent(args.mode, offer_slugs) + first_verdict = page_flow.queue_decision(first.state, None, independent) if not first.ok: stop_reason = ("page_load_timeout" if first.load_failed else f"blocked_{first.blocked_by}") blocked = first.blocked_by is not None - elif page_flow.is_terminal(first.state): - # A 404, or an offset past the site's 10,000 ceiling. Another - # attempt at the same address cannot help and neither is a - # block, so the run stops and says which it was. - stop_reason = first.state + elif first_verdict: + # A 404, or an offset past the site's 10,000 ceiling: another + # attempt at the same address cannot help and neither is a block. + # Not applied to an offer heading a queue of OTHER offers — one + # delisted slug must not cancel the run + # (`page_flow.queue_decision`). + stop_reason = first_verdict elif args.mode == "offer" and not offer_slugs: pass # `--url` named one offer; one page is the whole run else: @@ -1094,11 +1161,17 @@ def scrape(args) -> int: if p.sku is None or p.sku not in seen_keys) seen_keys.update(p.sku for p in outcome.products if p.sku is not None) - if not fresh_count: - logger.info("Page %d added no rows not already seen — " - "treating that as the end of the listing.", - page_num) - stop_reason = "no_new_products" + # ONE decision, shared with the first page and with the other + # engines, about whether this page ends the queue. In --mode + # offer it never does: every page is a different offer, and a + # 404 is an offer taken down between the sitemap and the + # fetch. This engine had no such exemption at all. + verdict = page_flow.queue_decision(outcome.state, fresh_count, + independent) + if verdict: + logger.info("Page %d (%s) ends the queue: %s.", + page_num, outcome.state, verdict) + stop_reason = verdict break if index + 1 < len(planned): @@ -1179,6 +1252,17 @@ def scrape(args) -> int: "" if done >= total_enumerated else " Pass --pages %d to cover the rest." % total_enumerated) + # A facets run that lacks a group is partial, not complete. `stop_reason` + # is deliberately NOT one of COMPLETE_STOP_REASONS, so finish_run reports + # status partial and exit 6 through the family's own rule + # rather than through a special case here. + # Judged on the MERGED rows, not on any one response, so the verdict does + # not depend on how a group went missing (see EXPECTED_FACET_GROUPS). + missing_groups = (missing_facet_groups(all_rows) + if args.mode == "facets" and all_rows else []) + if missing_groups: + stop_reason = "facet_groups_missing" + ok_pages = [o for o in outcomes if o.ok] failed_pages = [o.page_num for o in outcomes if not o.ok] final_url = (max(ok_pages, key=lambda o: o.page_num).final_url @@ -1195,20 +1279,48 @@ def scrape(args) -> int: if args.mode == "listings": extra.update(cap) extra["pages_available"] = pages_available - extra["items_per_page"] = args.per_page - extra["sort"] = args.sort - extra["order"] = args.order + if args.route == "ssr": + # The rendered page serves what the site chose to render — 50 rows + # on 2026-10-07, 100 a fortnight earlier — in the site's own order. + # `--per-page`, `--sort` and `--order` are refused for this route + # (`args_problem`), so recording them here would put three values + # in the sidecar that nothing applied. Record what ARRIVED instead. + extra["items_per_page"] = (len(outcomes[0].products) or None + if outcomes else None) + else: + extra["items_per_page"] = args.per_page + extra["sort"] = args.sort + extra["order"] = args.order # The ordering decides WHICH rows are in the file, not just their # order: the same query under `publishedAt` and under `salary` # overlapped on 5 of 100 ids (measured 2026-09-18), because each is # a different slice of a capped result set. So it is recorded, and # `diff_runs.py` refuses to compare two runs that disagree on it. filters = filters_from_args(args).as_params() - if filters: - extra["filters"] = filters + if args.route == "ssr": + # A radius is the endpoint's parameter; the page's own path takes + # a city and no radius, so claiming one was applied would be the + # same overstatement as claiming a filter it ignored. + filters.pop("cityRadius", None) + # ALWAYS recorded, even when empty. `diff_runs.py` reads the absence of + # this key as "this run did not say", which it must be able to tell + # apart from "this run used no filters". + extra["filters"] = filters + elif args.mode == "facets" and missing_groups: + extra["missing_groups"] = missing_groups elif args.mode == "offer": extra["offers_enumerated"] = total_enumerated or None extra["offers_fetched"] = len(ok_pages) + # Which pages of the queue were an offer the site no longer has. A 404 + # is ordinary here — the sitemap is generated ahead of the fetch — so + # it does not make the run partial, but it is not invisible either: + # `offers_fetched` counts pages that ANSWERED, and without this a + # consumer cannot tell an answer of "gone" from an offer that was read. + not_found = sorted(o.page_num for o in outcomes + if o.state == "not_found") + extra["offers_not_found"] = len(not_found) + if not_found: + extra["offers_not_found_pages"] = not_found return finish_run(all_rows, args.out, args.format, args.allow_empty, blocked=blocked, stop_reason=stop_reason, @@ -1476,23 +1588,12 @@ def parse_args(): p.error("--per-page must be between 1 and %d; justjoin.it answers " "HTTP 400 above that." % MAX_ITEMS_COUNT) - if args.mode != "listings": - # Refused rather than ignored, so a filtered `--mode offer` run - # cannot look like it narrowed the enumeration when it did not. - used = [name for name, value in ( - ("--category", args.category), ("--city", args.city), - ("--keyword", args.keyword), ("--remote", args.remote), - ("--experience", args.experience), ("--language", args.language), - ("--employment", args.employment), - ("--working-time", args.working_time), - ("--with-salary", args.with_salary)) if value] - if used: - p.error("%s filter%s the offers endpoint, which --mode %s does " - "not read. Use --mode listings to filter, then --mode " - "offer --slugs-file on its output to fetch those offers " - "in full." - % (", ".join(used), "" if len(used) == 1 else "s", - args.mode)) + # ONE shared validation (product_parser.args_problem) instead of a copy per + # engine: filters outside --mode listings, the filters the rendered route + # cannot apply, and --retries 0 (which skipped navigation entirely). + problem = args_problem(args) + if problem: + p.error(problem) # The sidecar's run label. `--category` is a real site filter here, so # unlike the repo this engine was ported from it is NOT reused as the diff --git a/pyproject.toml b/pyproject.toml index 19b0eae..0201754 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "justjoin-scraper" -version = "0.1.3" +version = "0.1.4" description = "justjoin.it job scraper: the board with salaries, skills and seniority, one offer in full, or the board's own facet counts." readme = "README.md" license = "MIT" diff --git a/scraper_api_client.py b/scraper_api_client.py index e595234..a57e749 100644 --- a/scraper_api_client.py +++ b/scraper_api_client.py @@ -58,8 +58,8 @@ from product_parser import (BOT_CHALLENGE_MARKERS, MODES, DEFAULT_MODE, detect_bot_challenge, detect_page_state, parse_page, parse_detail_response, - ROUTES, DEFAULT_ROUTE) -from output_writer import save + ROUTES, DEFAULT_ROUTE, missing_facet_groups) +from output_writer import SOURCE_DEFAULT, finish_run import env_config import page_flow @@ -354,7 +354,31 @@ def _run_once(args, attempt: int = 1, attempts: int = 1) -> int: "you can see what actually came back.", dump) return 4 - return save(products, args.out, args.format, allow_empty=args.allow_empty) + # `finish_run`, not `save`: it picks the row class from the MODE (this + # client wrote a --mode facets CSV with the JobPosting header, which fails + # on the first Facet row) and it writes the `.meta.json` sidecar. With + # no sidecar a run said nothing about its mode, its route or whether it + # was complete, which is exactly the evidence `diff_runs.py` needs and + # used to wave through when it was absent. + stop_reason = "single_page_route" + extra = {"route": "scraper_api", "mode": args.mode} + if args.mode == "facets": + # One request, so the category counts (their own endpoint) are never + # part of this body: the run is partial by construction, and says so. + missing = missing_facet_groups(products) + if missing: + stop_reason = "facet_groups_missing" + extra["missing_groups"] = missing + logger.warning("The facet response is missing %s — the Scraper " + "API client fetches one URL, and the categories " + "are a second endpoint. Use a browser engine for " + "the full set; this run is written as partial.", + ", ".join(missing)) + return finish_run(products, args.out, args.format, args.allow_empty, + blocked=False, stop_reason=stop_reason, + pages_requested=1, pages_completed=1, mode=args.mode, + source=SOURCE_DEFAULT, start_url=args.url, + final_url=args.url, extra=extra) def parse_args(): diff --git a/selenium_scraper.py b/selenium_scraper.py index ba5a99d..5db63c8 100644 --- a/selenium_scraper.py +++ b/selenium_scraper.py @@ -60,6 +60,7 @@ parse_count_response, parse_for_mode, route_of, site_recaptcha_sitekey, split_facets_body, target_url) +from product_parser import args_problem, missing_facet_groups from output_writer import (dedupe_by_key, finish_run, EXIT_API_ERROR, ROW_CLASS_BY_MODE, SOURCE_DEFAULT) import page_flow @@ -420,6 +421,39 @@ def _fetch_page_source(session, url: str, timeout_ms: int = 60000) -> str: return session.driver.page_source or "" +def _fetch_text(session, url: str, timeout_ms: int = 60000) -> str: + """Navigate and return the document's text — JSON exactly as served.""" + _fetch_page_source(session, url, timeout_ms) + return _snapshot(session, url) or "" + + +def _fetch_body(session, args, url: str) -> str: + """The document the parser is given for this fetch. + + `--mode facets` is TWO endpoints that are one answer — the facet groups, + and the category counts from their own route — joined with a separator the + parser undoes (`product_parser.FACETS_JOIN`). This engine had no such step + at all: it imported FACETS_JOIN and categories_api_url and never used + either, so every `--mode facets` run through it silently omitted the + `categories` group — 57 rows against Playwright's 82 — and reported + `complete`. The group it dropped is the one a plan for reaching past the + 10,000 cap is built from. Found by running the mode through this engine, + which the parity check had never done (it ran --mode listings only). + """ + body = _snapshot(session, url) or "" + if args.mode == "facets" and body.strip(): + try: + categories = _fetch_text(session, categories_api_url()) + except Exception as exc: # noqa: BLE001 — a driver raises many types + logger.warning("The category counts did not load (%s); the run " + "will be partial rather than silently missing " + "them.", _mask_credentials(str(exc))[:120]) + return body + if categories.strip(): + return body + FACETS_JOIN + categories + return body + + def handle_captcha_if_present(session, args) -> bool: """Detect and solve a challenge. True if something was solved. @@ -584,7 +618,7 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome logger.info("Not solving again on page %d: %d purchase(s) " "already made for it and SOLVES_PER_PAGE is %d.", page_num, solves_bought, page_flow.SOLVES_PER_PAGE) - html = _snapshot(session, url) or "" + html = _fetch_body(session, args, url) state = page_flow.classify(html, http_status, d["current_url"](), args.mode, args.route) @@ -607,7 +641,7 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome if found < need: logger.info("Still nothing after %.0fs (%d match(es) for %s).", wait_ms / 1000.0, found, sel) - html = _snapshot(session, url) or html + html = _fetch_body(session, args, url) or html state = page_flow.classify(html, http_status, d["current_url"](), args.mode, args.route) @@ -762,6 +796,13 @@ def _fetch_one_page(session, args, pool, page_num: int, url: str) -> PageOutcome return outcome products, listing = parse_for_mode(html, final_url, args, page_num) + if args.mode == "facets": + gone = missing_facet_groups(products) + if gone: + logger.warning("The facet response is missing %s. The groups that " + "arrived are written, and the run is partial: " + "categories are what a plan for reaching past the " + "10,000 cap is built from.", ", ".join(gone)) logger.info("Parsed %d row(s) from page %d.", len(products), page_num) if listing is not None: @@ -921,36 +962,57 @@ def scrape(args) -> int: mode=args.mode, source=SOURCE_DEFAULT, start_url=args.url or "", final_url="") total_enumerated = len(offer_slugs) - target = offer_slugs[0] - - if target != args.url: - logger.info("Fetching %s", target) - - # The board-wide total, for the cap arithmetic. One extra - # request, and it buys the difference between a sidecar that - # says "complete" and one that says "complete, and a 52% - # sample". Never fatal. - if args.mode == "listings": - try: - site_total = parse_count_response( - _fetch_page_source(session, count_api_url())) - except Exception as exc: # noqa: BLE001 - logger.info("The board-wide count did not load (%s); the " - "run continues without the cap share.", - _mask_credentials(str(exc))[:120]) + # A slug is not an address. Both parity engines assigned it bare, + # so `--mode offer` over the sitemap "navigated" to a string like + # `acme-dev-warszawa-java` and ended in exit 5 (Selenium) or a + # traceback (pyppeteer). Playwright built the URL; this does too. + first_url = detail_api_url(offer_slugs[0]) + if first_url: + target = first_url + + if args.url and target != args.url: + logger.info("Reading %s through %s — the endpoint is the " + "route with pagination, and the filters your URL " + "names are carried over to it.", args.url, target) + logger.info("Fetching %s", target) + + # The board-wide total, for the cap arithmetic. One extra + # request, and it buys the difference between a sidecar that + # says "complete" and one that says "complete, and a 52% + # sample". Never fatal. + # NOT for --route ssr. That route exists to stay inside robots.txt + # (which disallows /api/), and this request goes to + # /api/candidate-api/offers/count — so the scraper itself was + # fetching a disallowed path on the one route documented as + # robots-clean. The rendered page no longer states a total + # either (its `meta` block is gone, 2026-10-07), so an ssr + # sidecar says nothing about the cap rather than something fetched + # from the wrong place. + if args.mode == "listings" and args.route != "ssr": + try: + site_total = parse_count_response( + _fetch_page_source(session, count_api_url())) + except Exception as exc: # noqa: BLE001 + logger.info("The board-wide count did not load (%s); the " + "run continues without the cap share.", + _mask_credentials(str(exc))[:120]) first = _fetch_one_page(session, args, pool, 1, target) outcomes.append(first) + independent = page_flow.pages_are_independent(args.mode, offer_slugs) + first_verdict = page_flow.queue_decision(first.state, None, independent) if not first.ok: stop_reason = ("page_load_timeout" if first.load_failed else f"blocked_{first.blocked_by}") blocked = first.blocked_by is not None - elif page_flow.is_terminal(first.state): - # A 404, or an offset past the site's 10,000 ceiling. Another - # attempt at the same address cannot help and neither is a - # block, so the run stops and says which it was. - stop_reason = first.state + elif first_verdict: + # A 404, or an offset past the site's 10,000 ceiling: another + # attempt at the same address cannot help and neither is a block. + # Not applied to an offer heading a queue of OTHER offers — one + # delisted slug must not cancel the run + # (`page_flow.queue_decision`). + stop_reason = first_verdict elif args.mode == "offer" and not offer_slugs: pass # `--url` named one offer; one page is the whole run else: @@ -983,11 +1045,17 @@ def scrape(args) -> int: if p.sku is None or p.sku not in seen_keys) seen_keys.update(p.sku for p in outcome.products if p.sku is not None) - if not fresh_count: - logger.info("Page %d added no rows not already seen — " - "treating that as the end of the listing.", - page_num) - stop_reason = "no_new_products" + # ONE decision, shared with the first page and with the other + # engines, about whether this page ends the queue. In --mode + # offer it never does: every page is a different offer, and a + # 404 is an offer taken down between the sitemap and the + # fetch. This engine had no such exemption at all. + verdict = page_flow.queue_decision(outcome.state, fresh_count, + independent) + if verdict: + logger.info("Page %d (%s) ends the queue: %s.", + page_num, outcome.state, verdict) + stop_reason = verdict break if index + 1 < len(planned): @@ -1067,6 +1135,17 @@ def scrape(args) -> int: "" if done >= total_enumerated else " Pass --pages %d to cover the rest." % total_enumerated) + # A facets run that lacks a group is partial, not complete. `stop_reason` + # is deliberately NOT one of COMPLETE_STOP_REASONS, so finish_run reports + # status partial and exit 6 through the family's own rule + # rather than through a special case here. + # Judged on the MERGED rows, not on any one response, so the verdict does + # not depend on how a group went missing (see EXPECTED_FACET_GROUPS). + missing_groups = (missing_facet_groups(all_rows) + if args.mode == "facets" and all_rows else []) + if missing_groups: + stop_reason = "facet_groups_missing" + ok_pages = [o for o in outcomes if o.ok] failed_pages = [o.page_num for o in outcomes if not o.ok] final_url = (max(ok_pages, key=lambda o: o.page_num).final_url @@ -1083,20 +1162,48 @@ def scrape(args) -> int: if args.mode == "listings": extra.update(cap) extra["pages_available"] = pages_available - extra["items_per_page"] = args.per_page - extra["sort"] = args.sort - extra["order"] = args.order + if args.route == "ssr": + # The rendered page serves what the site chose to render — 50 rows + # on 2026-10-07, 100 a fortnight earlier — in the site's own order. + # `--per-page`, `--sort` and `--order` are refused for this route + # (`args_problem`), so recording them here would put three values + # in the sidecar that nothing applied. Record what ARRIVED instead. + extra["items_per_page"] = (len(outcomes[0].products) or None + if outcomes else None) + else: + extra["items_per_page"] = args.per_page + extra["sort"] = args.sort + extra["order"] = args.order # The ordering decides WHICH rows are in the file, not just their # order: the same query under `publishedAt` and under `salary` # overlapped on 5 of 100 ids (measured 2026-09-18), because each is # a different slice of a capped result set. So it is recorded, and # `diff_runs.py` refuses to compare two runs that disagree on it. filters = filters_from_args(args).as_params() - if filters: - extra["filters"] = filters + if args.route == "ssr": + # A radius is the endpoint's parameter; the page's own path takes + # a city and no radius, so claiming one was applied would be the + # same overstatement as claiming a filter it ignored. + filters.pop("cityRadius", None) + # ALWAYS recorded, even when empty. `diff_runs.py` reads the absence of + # this key as "this run did not say", which it must be able to tell + # apart from "this run used no filters". + extra["filters"] = filters + elif args.mode == "facets" and missing_groups: + extra["missing_groups"] = missing_groups elif args.mode == "offer": extra["offers_enumerated"] = total_enumerated or None extra["offers_fetched"] = len(ok_pages) + # Which pages of the queue were an offer the site no longer has. A 404 + # is ordinary here — the sitemap is generated ahead of the fetch — so + # it does not make the run partial, but it is not invisible either: + # `offers_fetched` counts pages that ANSWERED, and without this a + # consumer cannot tell an answer of "gone" from an offer that was read. + not_found = sorted(o.page_num for o in outcomes + if o.state == "not_found") + extra["offers_not_found"] = len(not_found) + if not_found: + extra["offers_not_found_pages"] = not_found return finish_run(all_rows, args.out, args.format, args.allow_empty, blocked=blocked, stop_reason=stop_reason, @@ -1360,23 +1467,12 @@ def parse_args(): p.error("--per-page must be between 1 and %d; justjoin.it answers " "HTTP 400 above that." % MAX_ITEMS_COUNT) - if args.mode != "listings": - # Refused rather than ignored, so a filtered `--mode offer` run - # cannot look like it narrowed the enumeration when it did not. - used = [name for name, value in ( - ("--category", args.category), ("--city", args.city), - ("--keyword", args.keyword), ("--remote", args.remote), - ("--experience", args.experience), ("--language", args.language), - ("--employment", args.employment), - ("--working-time", args.working_time), - ("--with-salary", args.with_salary)) if value] - if used: - p.error("%s filter%s the offers endpoint, which --mode %s does " - "not read. Use --mode listings to filter, then --mode " - "offer --slugs-file on its output to fetch those offers " - "in full." - % (", ".join(used), "" if len(used) == 1 else "s", - args.mode)) + # ONE shared validation (product_parser.args_problem) instead of a copy per + # engine: filters outside --mode listings, the filters the rendered route + # cannot apply, and --retries 0 (which skipped navigation entirely). + problem = args_problem(args) + if problem: + p.error(problem) # The sidecar's run label. `--category` is a real site filter here, so # unlike the repo this engine was ported from it is NOT reused as the diff --git a/smoke_test.py b/smoke_test.py index 058a421..558977d 100644 --- a/smoke_test.py +++ b/smoke_test.py @@ -555,6 +555,1031 @@ def check_a_delisted_offer_is_read_from_the_body_when_there_is_no_status(): "a status is captured and then not used") +# --------------------------------------------------------------------------- +# The queue: one delisted offer must not end it +# --------------------------------------------------------------------------- +# +# Every earlier check of the 404 handling asserted that the CLASSIFIER named +# the state (`not_found`). None of them drove a loop over it, so a third-party +# audit found what 600 green checks had not: a good -> 404 -> good queue +# fetched two of its three offers, wrote one row, and reported `complete` with +# `stop_reason: no_new_products`. These checks run the real `scrape()` of every +# engine that is installed, with the driver and the per-page fetch stubbed. + +def _drive_scrape(engine, argv, outcome_for, side=None): + """Run `engine.scrape` on `argv` and return (exit, urls fetched, sidecar). + + `outcome_for(url)` -> (state, [skus]) says what each fetched URL "served". + `side`, if given, collects every request made outside the listing itself. + Only the browser session and the per-page fetch are replaced; the loop + that decides what a page means for the REST of the queue is the real one. + """ + import contextlib + import shutil + + class FakeSession: + def __init__(self, *a, **k): + self.pool = None + + def open(self): + return self + + def close(self): + pass + + def relaunch(self): + pass + + class FakeBridge: + def close(self): + pass + + class FakeCtx: + def __enter__(self): + return None + + def __exit__(self, *a): + return False + + fetched = [] + + def fake_fetch(session, args, pool, page_num, url): + fetched.append(url) + state, skus = outcome_for(url) + outcome = engine.PageOutcome(page_num=page_num, url=url) + outcome.state = state + outcome.final_url = url + outcome.pages_available = 10 + # A plain string is a sku (a JobPosting is built for it); anything + # else is a ready-made row, which is how --mode facets supplies Facets. + outcome.products = [ + row if not isinstance(row, str) + else JobPosting(sku=row, title=row, page=page_num, position=i + 1) + for i, row in enumerate(skus)] + return outcome + + def record(session, url, **kwargs): + # Every request the engine makes OUTSIDE the listing itself — the + # board count, a sitemap. `side` lets a check ask what they were. + if side is not None: + side.append(url) + return "" + + patches = {"_fetch_one_page": fake_fetch, + "_fetch_text": record, + "_fetch_page_source": record} + for name in ("_BrowserSession", "_Session"): + if hasattr(engine, name): + patches[name] = FakeSession + if hasattr(engine, "_AsyncBridge"): + patches["_AsyncBridge"] = FakeBridge + if hasattr(engine, "sync_playwright"): + patches["sync_playwright"] = lambda: FakeCtx() + + saved = {k: getattr(engine, k) for k in patches if hasattr(engine, k)} + saved_argv = sys.argv[:] + tmp = tempfile.mkdtemp(prefix="jj_queue_") + try: + for key, value in patches.items(): + setattr(engine, key, value) + out = os.path.join(tmp, "run") + sys.argv = [engine.__name__ + ".py"] + argv + ["--out", out, + "--delay", "0", + "--format", "json"] + args = engine.parse_args() + with contextlib.redirect_stdout(io.StringIO()): + code = engine.scrape(args) + meta_path = out + ".meta.json" + meta = (json.load(open(meta_path, encoding="utf-8")) + if os.path.exists(meta_path) else None) + return code, fetched, meta + finally: + sys.argv = saved_argv + for key, value in saved.items(): + setattr(engine, key, value) + shutil.rmtree(tmp, ignore_errors=True) + + +def _offer_queue_file(slugs): + handle = tempfile.NamedTemporaryFile("w", suffix=".txt", delete=False, + prefix="jj_slugs_") + handle.write("\n".join(slugs) + "\n") + handle.close() + return handle.name + + +def check_the_queue_decision_table(): + """The one function every loop now asks, pinned as a table.""" + qd = page_flow.queue_decision + # Independent pages (--mode offer with a queue): nothing ends the queue. + for state in ("content", "empty", "not_found", "cap_exceeded", None): + for fresh in (0, 5, None): + equal("independent page, state=%s fresh=%s" % (state, fresh), + qd(state, fresh, True), None) + # A listing: a terminal state or a page that adds nothing ends it. + equal("a listing 404 ends it, and says so", + qd("not_found", 0, False), "not_found") + equal("past the cap ends it, and says so", + qd("cap_exceeded", None, False), "cap_exceeded") + equal("a page with nothing new ends it", qd("content", 0, False), + "no_new_products") + equal("a page with something new does not", qd("content", 3, False), None) + equal("page 1 (fresh unknown) is not judged on freshness", + qd("content", None, False), None) + # What counts as independent. + equal("offer + a queue", page_flow.pages_are_independent("offer", ["a"]), + True) + equal("offer + NO queue (a single --url) is not", + page_flow.pages_are_independent("offer", []), False) + equal("a listing never is", + page_flow.pages_are_independent("listings", ["a", "b"]), False) + + +def check_a_delisted_offer_does_not_cut_the_queue_in_any_engine(): + """good/404/good, 404/good/good and good/good/404, through each engine's + own loop. All three offers must be REQUESTED, the two live ones written, + and the run must end `completed` — not `no_new_products`.""" + ran = 0 + shapes = { + "404 in the middle": ["a", "gone", "b"], + "404 FIRST (a sitemap whose head has been taken down)": + ["gone", "a", "b"], + "404 last": ["a", "b", "gone"], + } + for module in ENGINES: + engine = _import_engine(module) + if engine is None: + continue + for label, slugs in shapes.items(): + path = _offer_queue_file(slugs) + try: + def outcome_for(url): + if url.endswith("/gone"): + return "not_found", [] + return "content", [url.rsplit("/", 1)[-1]] + code, fetched, meta = _drive_scrape( + engine, ["--mode", "offer", "--slugs-file", path, + "--pages", "3"], outcome_for) + finally: + os.unlink(path) + ran += 1 + tag = "%s, %s" % (module, label) + equal("%s: every offer was requested" % tag, len(fetched), 3) + equal("%s: exit 0" % tag, code, 0) + check("%s: a sidecar was written" % tag, meta is not None) + if meta is None: + continue + equal("%s: the live offers were written" % tag, + meta["products"], 2) + equal("%s: the run ended because the queue did, not because a " + "page was empty" % tag, meta["stop_reason"], "completed") + equal("%s: the sidecar names the offer that was not found" % tag, + meta.get("offers_not_found"), 1) + if ran == 0: + skip("offer queue", "no engine library installed here") + + +def check_a_listing_that_repeats_itself_still_ends_the_run(): + """The control for the other direction. If `independent` were applied to + everything, a listing answering page 2 with page 1 again would be + re-collected for as long as it was asked to, and reported complete.""" + for module in ENGINES: + engine = _import_engine(module) + if engine is None: + continue + + def outcome_for(url): + return "content", ["same-offer"] + code, fetched, meta = _drive_scrape( + engine, ["--category", "python", "--pages", "3", + "--per-page", "1"], outcome_for) + equal("%s: page 3 was never requested" % module, len(fetched), 2) + check("%s: a sidecar was written" % module, meta is not None) + if meta is not None: + equal("%s: it says why it stopped" % module, + meta["stop_reason"], "no_new_products") + equal("%s: and holds the one row" % module, meta["products"], 1) + + +def check_the_concurrent_workers_agree_about_a_delisted_offer(): + """The Playwright engine's concurrent path set the same "end of listing" + event on a 404 that the sequential loop tripped over.""" + engine = _import_engine("playwright_scraper") + if engine is None: + return + import threading + + class Args: + delay = 0 + retries = 1 + retry_delay = 0 + out = "unused" + mode = "offer" + + seen = [] + lock = threading.Lock() + + def fake_fetch(session, args, pool, page_num, url): + with lock: + seen.append(page_num) + outcome = engine.PageOutcome(page_num=page_num, url=url) + gone = page_num == 3 + outcome.state = "not_found" if gone else "content" + outcome.products = [] if gone else [object()] + return outcome + + class FakeSession: + def __init__(self, *a, **k): + self.pool = None + + def open(self): + return self + + def close(self): + pass + + class FakeCtx: + def __enter__(self): + return None + + def __exit__(self, *a): + return False + + saved = (engine._fetch_one_page, engine._BrowserSession, + engine.sync_playwright) + engine._fetch_one_page = fake_fetch + engine._BrowserSession = FakeSession + engine.sync_playwright = lambda: FakeCtx() + try: + specs = [(n, "u%d" % n) for n in range(2, 9)] + results, unattempted, exhausted = engine._fetch_pages_concurrently( + Args(), None, specs, 3) + finally: + (engine._fetch_one_page, engine._BrowserSession, + engine.sync_playwright) = saved + equal("every queued offer was fetched, one of them a 404", + sorted(seen), list(range(2, 9))) + equal("nothing was left unattempted", unattempted, []) + equal("and dispatch was not stopped", exhausted, False) + + +def check_no_engine_touches_the_pool_without_asking_whether_there_is_one(): + """A run with no proxy has `pool = None`. Any `pool.` reached on that + path is an AttributeError traceback (exit 1) where the run should have + reported the failure it was handling (exit 5). + + It was reached twice in one family of code. v0.1.0 fixed `mask(pool.current)` + in the block-retry branch of the two parity engines, and the same engine + still had `pool.advance(...)` unguarded two screens up, on the path every + navigation failure takes: `block_retries` is 1 without a pool, not 0, so + the test `block_attempt < block_retries` was true exactly when there was + nothing to rotate to. A live `--mode offer` run found it, because its + first address was bad. No offline check could: the per-page fetch is the + thing a stub replaces. + + So assert it statically, on every engine: each `pool.` must sit under + something that has asked about `pool` or `has_pool` first — an `if`, a + conditional expression, or the left side of an `and`. + """ + names = ("pool", "has_pool") + + def mentions(node): + return any(isinstance(n, ast.Name) and n.id in names + for n in ast.walk(node)) + + for module in ENGINES: + path = os.path.join(HERE, module + ".py") + if not os.path.exists(path): + continue + tree = ast.parse(open(path, encoding="utf-8").read()) + parent = {} + for node in ast.walk(tree): + for child in ast.iter_child_nodes(node): + parent[child] = node + + accesses = [n for n in ast.walk(tree) + if isinstance(n, ast.Attribute) + and isinstance(n.value, ast.Name) and n.value.id == "pool"] + check("%s: the scan found pool accesses to judge" % module, + len(accesses) >= 5, "%d found" % len(accesses)) + + def enclosing_function(node): + while node in parent: + node = parent[node] + if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)): + return node + return None + + def falsy_when_no_pool(function): + """Names a function sets to a FALSY value when there is no pool: + `proxy = f(pool.current) if pool else None`. Testing such a name + is asking about the pool. `block_retries = ... if has_pool else 1` + is NOT one — its else is truthy, which is exactly the shape of the + defect this check exists for, so it must not count.""" + found = set() + for n in ast.walk(function): + if (isinstance(n, ast.Assign) and isinstance(n.value, ast.IfExp) + and mentions(n.value.test) + and isinstance(n.value.orelse, ast.Constant) + and not n.value.orelse.value): + found.update(t.id for t in n.targets + if isinstance(t, ast.Name)) + return found + + unguarded = [] + for access in accesses: + function = enclosing_function(access) + derived = falsy_when_no_pool(function) if function else set() + + def asks(test): + return mentions(test) or any( + isinstance(n, ast.Name) and n.id in derived + for n in ast.walk(test)) + + guarded, cur = False, access + # An early exit ahead of the access guards everything after it: + # `if not pool: return None`. + for stmt in (function.body if function else []): + if (isinstance(stmt, ast.If) and stmt.lineno < access.lineno + and mentions(stmt.test) and stmt.body + and isinstance(stmt.body[-1], (ast.Return, ast.Raise))): + guarded = True + while cur in parent and not guarded: + par = parent[cur] + if isinstance(par, (ast.If, ast.While)) and cur is not par.test: + guarded = asks(par.test) + elif isinstance(par, ast.IfExp) and cur is par.body: + guarded = asks(par.test) + elif isinstance(par, ast.BoolOp): + earlier = par.values[:par.values.index(cur)] + guarded = any(asks(v) for v in earlier) + cur = par + if not guarded: + unguarded.append("line %d: pool.%s" % (access.lineno, + access.attr)) + check("%s: every pool. is guarded" % module, not unguarded, + "; ".join(unguarded)) + + +# --------------------------------------------------------------------------- +# --route ssr: only what the rendered page applies, and no request to /api/ +# --------------------------------------------------------------------------- +# +# An audit ran `--route ssr --experience senior --with-salary --keyword rust` +# and found the fetched address was the bare all-locations page, with the +# sidecar recording all three filters as if they had narrowed it. Measured +# live afterwards: 23 of 50 rows were senior, 28 of 50 priced, and a sort the +# page also ignored was written into the sidecar beside them. + +import types as _types + + +def _ssr_args(**over): + base = dict(mode="listings", route="ssr", url=None, category=None, + city=None, remote=False, keyword=None, experience=None, + language=None, employment=None, working_time=None, + with_salary=False, city_radius=30, per_page=pp.DEFAULT_ITEMS_COUNT, + sort=pp.DEFAULT_SORT, order=pp.DEFAULT_ORDER, retries=3) + base.update(over) + return _types.SimpleNamespace(**base) + + +def check_the_ssr_address_carries_exactly_what_the_page_applies(): + cases = [ + ({}, "https://justjoin.it/job-offers/all-locations"), + ({"city": "warszawa"}, "https://justjoin.it/job-offers/warszawa"), + ({"category": "python"}, + "https://justjoin.it/job-offers/all-locations/python"), + ({"city": "krakow", "category": "python"}, + "https://justjoin.it/job-offers/krakow/python"), + ({"remote": True}, "https://justjoin.it/job-offers/remote"), + ({"remote": True, "category": "python"}, + "https://justjoin.it/job-offers/remote/python"), + ({"keyword": "rust"}, + "https://justjoin.it/job-offers/all-locations?keyword=rust"), + ({"city": "warszawa", "category": "python", "keyword": "django"}, + "https://justjoin.it/job-offers/warszawa/python?keyword=django"), + ] + for over, want in cases: + equal("ssr address for %s" % (over or "no filters"), + pp.target_url(_ssr_args(**over)), want) + + # A user URL is REBUILT, not fetched as given. As given, the flags beside + # it were accepted and dropped. + got = pp.target_url(_ssr_args( + url="https://justjoin.it/job-offers/warszawa", keyword="rust")) + equal("--url plus --keyword reaches the address", + got, "https://justjoin.it/job-offers/warszawa?keyword=rust") + got = pp.target_url(_ssr_args( + url="https://justjoin.it/job-offers/warszawa/python")) + equal("--url alone round-trips", got, + "https://justjoin.it/job-offers/warszawa/python") + equal("the supported set is the four the page applies", + pp.SSR_FILTERS, ("category", "city", "remote", "keyword")) + + +def check_filters_the_rendered_page_ignores_are_refused_before_any_request(): + problem = pp.args_problem + ok = _ssr_args(category="python", city="warszawa", keyword="rust") + equal("the four supported filters pass", problem(ok), None) + for flag, over in (("--experience", {"experience": ["senior"]}), + ("--with-salary", {"with_salary": True}), + ("--language", {"language": ["en"]}), + ("--employment", {"employment": ["b2b"]}), + ("--working-time", {"working_time": ["full_time"]}), + ("--sort", {"sort": "salary"}), + ("--order", {"order": "ascending"}), + ("--per-page", {"per_page": 20}), + ("--city-radius", {"city": "warszawa", + "city_radius": 50})): + msg = problem(_ssr_args(**over)) + check("--route ssr refuses %s" % flag, msg and flag in msg, msg) + msg = problem(_ssr_args(city="warszawa", remote=True)) + check("a city AND remote cannot share a path", + msg and "--city and --remote" in msg, msg) + # And it names every offender at once rather than one per attempt. + msg = problem(_ssr_args(experience=["senior"], with_salary=True, + sort="salary")) + check("it lists them all in one message", + msg and all(f in msg for f in ("--experience", "--with-salary", + "--sort")), msg) + # The same flags are fine on the route that applies them. + for over in ({"experience": ["senior"], "with_salary": True, + "sort": "salary", "per_page": 20, "language": ["en"]},): + equal("--route api accepts what ssr refuses", + problem(_ssr_args(route="api", **over)), None) + # Offer and facets read no query at all. + msg = problem(_ssr_args(mode="offer", route="api", category="python")) + check("--mode offer refuses a filter", msg and "--category" in msg, msg) + + +def check_retries_below_one_is_refused_because_it_means_no_navigation(): + """`--retries` is the number of ATTEMPTS. 0 made `range(1, 0 + 1)` empty, + so the page was never fetched, and the run said 'came back as unknown' and + exited 4 — 'nothing found' on a healthy site, for a fetch that never + happened (blocked, empty and failed are different answers).""" + for value in (0, -1): + msg = pp.args_problem(_ssr_args(route="api", retries=value)) + check("--retries %d is refused" % value, msg and "--retries" in msg, + msg) + equal("--retries 1 is the single-attempt spelling", + pp.args_problem(_ssr_args(route="api", retries=1)), None) + + ran = 0 + for module in ENGINES: + engine = _import_engine(module) + if engine is None: + continue + saved = sys.argv[:] + err = io.StringIO() + code = None + try: + sys.argv = [module + ".py", "--retries", "0"] + with contextlib_redirect_stderr(err): + try: + engine.parse_args() + except SystemExit as exc: + code = exc.code + finally: + sys.argv = saved + ran += 1 + equal("%s exits 2 on --retries 0" % module, code, 2) + check("%s says why" % module, "--retries" in err.getvalue(), + err.getvalue()[-160:]) + if ran == 0: + skip("--retries", "no engine library installed here") + + +def contextlib_redirect_stderr(stream): + import contextlib + return contextlib.redirect_stderr(stream) + + +def check_the_engines_refuse_unsupported_ssr_flags_through_the_shared_check(): + ran = 0 + for module in ENGINES: + engine = _import_engine(module) + if engine is None: + continue + saved = sys.argv[:] + err = io.StringIO() + code = None + try: + sys.argv = [module + ".py", "--route", "ssr", "--experience", + "senior", "--with-salary"] + with contextlib_redirect_stderr(err): + try: + engine.parse_args() + except SystemExit as exc: + code = exc.code + finally: + sys.argv = saved + ran += 1 + equal("%s exits 2" % module, code, 2) + check("%s names the flags the page ignores" % module, + "--experience" in err.getvalue() + and "--with-salary" in err.getvalue(), err.getvalue()[-200:]) + if ran == 0: + skip("ssr flags", "no engine library installed here") + + +def check_the_ssr_sidecar_records_what_arrived_not_what_was_asked(): + """`sort`, `order` and `items_per_page` describe a REQUEST the endpoint + honours. The rendered page honours none of them, so an ssr sidecar that + carried them would say a run was sorted by salary when the page had served + its own order. And `diff_runs.py` reads `sort` and `filters` to decide + whether two runs are comparable, so a wrong value there is not cosmetic.""" + ran = 0 + for module in ENGINES: + engine = _import_engine(module) + if engine is None: + continue + _, _, meta = _drive_scrape( + engine, ["--route", "ssr", "--city", "warszawa", "--keyword", + "rust", "--pages", "1"], + lambda url: ("content", ["a", "b", "c"])) + ran += 1 + check("%s: a sidecar was written" % module, meta is not None) + if meta is None: + continue + equal("%s: no sort is claimed for a page that has none" % module, + (meta.get("sort"), meta.get("order")), (None, None)) + equal("%s: the page size is what ARRIVED" % module, + meta.get("items_per_page"), 3) + applied = meta.get("filters") or {} + equal("%s: the keyword the page applied is recorded" % module, + applied.get("keywords"), "rust") + check("%s: no radius is claimed — the page's path takes none" % module, + "cityRadius" not in applied, str(applied)) + equal("%s: the route is recorded" % module, meta.get("route"), "ssr") + equal("%s: and so is the absence of any cap arithmetic" % module, + (meta.get("site_total"), meta.get("capped_by_site")), + (None, None)) + _, _, api_meta = _drive_scrape( + engine, ["--route", "api", "--pages", "1", "--per-page", "7"], + lambda url: ("content", ["a"])) + equal("%s: the api route still records the sort it sent" % module, + (api_meta.get("sort"), api_meta.get("items_per_page")), + (pp.DEFAULT_SORT, 7)) + if ran == 0: + skip("ssr sidecar", "no engine library installed here") + + +def check_the_ssr_route_never_requests_a_path_robots_txt_disallows(): + """`--route ssr` is documented as the robots-clean route, and the scraper + itself was fetching /api/candidate-api/offers/count on it to compute the + cap share — a path robots.txt disallows. Asserted on every request the + engine makes outside the listing itself, with the api route as the + control (it SHOULD ask for the count, or this check proves nothing).""" + ran = 0 + for module in ENGINES: + engine = _import_engine(module) + if engine is None: + continue + for route, expect_api in (("ssr", False), ("api", True)): + side = [] + _drive_scrape(engine, ["--route", route, "--pages", "1"], + lambda url: ("content", ["x"]), side=side) + ran += 1 + hit = [u for u in side if pp.API_PATH in u] + if expect_api: + check("%s: the api route DOES ask for the board count" + % module, hit, "side requests: %r" % side) + else: + equal("%s: the ssr route makes no /api/ request" % module, + hit, []) + if ran == 0: + skip("ssr robots", "no engine library installed here") + + +# --------------------------------------------------------------------------- +# --mode facets: a missing group is a partial run, not a complete one +# --------------------------------------------------------------------------- +# +# Two findings, one defect class. An audit traced that a failed request for +# the category counts is caught, logged, and the run continues as `complete`. +# Running the mode through the other engines then showed Selenium and +# pyppeteer never made that request at all: 57 rows against 82, every time. +# The missing group is the one the README's plan for reaching past the +# 10,000-offer cap is built from. + +def _facet_rows(with_categories=True, drop=()): + rows = pp.parse_facets_response( + FACETS_JSON, + categories_text=CATEGORIES_JSON if with_categories else None) + return [r for r in rows if r.facet_group not in drop] + + +def check_missing_facet_groups_are_judged_on_the_rows(): + equal("a full read misses nothing", + pp.missing_facet_groups(_facet_rows()), []) + equal("no categories request, no categories group", + pp.missing_facet_groups(_facet_rows(with_categories=False)), + ["categories"]) + equal("a group the site stopped publishing is named too", + pp.missing_facet_groups(_facet_rows(drop=("languages",))), + ["languages"]) + equal("several, in the order the groups are expected", + pp.missing_facet_groups( + _facet_rows(with_categories=False, drop=("workingTimes",))), + ["workingTimes", "categories"]) + # A categories body that arrives and will not parse is the same loss by a + # different road, and the judgement must not depend on which one it was. + broken = pp.parse_facets_response(FACETS_JSON, + categories_text="502") + equal("an unparseable categories body is a missing group, not a crash", + pp.missing_facet_groups(broken), ["categories"]) + equal("the expected set is the seven the board publishes", + sorted(pp.EXPECTED_FACET_GROUPS), + sorted(set(pp.FACET_GROUPS) | {pp.CATEGORY_FACET_GROUP})) + + +def check_a_facets_run_missing_a_group_is_partial_in_every_engine(): + ran = 0 + for module in ENGINES: + engine = _import_engine(module) + if engine is None: + continue + cases = [("complete read", _facet_rows(), 0, "complete", None), + ("no categories", _facet_rows(with_categories=False), 6, + "partial", ["categories"]), + ("a renamed group", _facet_rows(drop=("languages",)), 6, + "partial", ["languages"])] + for label, rows, want_exit, want_status, want_missing in cases: + code, fetched, meta = _drive_scrape( + engine, ["--mode", "facets"], lambda url, rows=rows: + ("content", rows)) + ran += 1 + tag = "%s, %s" % (module, label) + equal("%s: exit" % tag, code, want_exit) + check("%s: a sidecar was written" % tag, meta is not None) + if meta is None: + continue + equal("%s: status" % tag, meta["status"], want_status) + equal("%s: the sidecar names what is missing" % tag, + meta.get("missing_groups"), want_missing) + if want_missing: + equal("%s: and why the run stopped" % tag, + meta["stop_reason"], "facet_groups_missing") + equal("%s: the groups that DID arrive were still written" + % tag, meta["products"], len(rows)) + else: + equal("%s: a clean read stops as a one-page route" % tag, + meta["stop_reason"], "single_page_route") + if ran == 0: + skip("facets partial", "no engine library installed here") + + +def check_every_engine_asks_for_the_category_counts(): + """The Selenium and pyppeteer engines imported `FACETS_JOIN` and + `categories_api_url` and used neither. Driven through each engine's own + `_fetch_body` so an engine that forgets the second request fails here + instead of shipping 57 rows as a complete run.""" + ran = 0 + for module in ENGINES: + engine = _import_engine(module) + if engine is None: + continue + requested = [] + + def snapshot(*a, **k): + return FACETS_JSON + + def text_ok(session, url, **k): + requested.append(url) + return CATEGORIES_JSON + + def text_down(session, url, **k): + requested.append(url) + # The type each engine really catches: Playwright's own Error, + # and anything at all from the two whose drivers raise many. + raise getattr(engine, "PWError", RuntimeError)( + "net::ERR_CONNECTION_RESET") + + saved = (engine._snapshot, engine._fetch_text) + args = _types.SimpleNamespace(mode="facets") + session = _types.SimpleNamespace(page=None, driver=None) + try: + engine._snapshot = snapshot + engine._fetch_text = text_ok + joined = engine._fetch_body(session, args, "u") + engine._fetch_text = text_down + degraded = engine._fetch_body(session, args, "u") + engine._fetch_text = text_ok + other = engine._fetch_body( + session, _types.SimpleNamespace(mode="listings"), "u") + finally: + engine._snapshot, engine._fetch_text = saved + ran += 1 + check("%s: categories are requested from their own endpoint" % module, + any(pp.CATEGORIES_PATH in u for u in requested), str(requested)) + check("%s: and joined onto the facet body" % module, + pp.FACETS_JOIN in joined) + equal("%s: so a parsed run carries all seven groups" % module, + len({r.facet_group for r in pp.parse_for_mode( + joined, "", _types.SimpleNamespace( + mode="facets", route="api", per_page=1), 1)[0]}), 7) + equal("%s: a failed request degrades to the bare facet body" % module, + degraded, FACETS_JSON) + equal("%s: and no other mode asks at all" % module, other, + FACETS_JSON) + if ran == 0: + skip("facets body", "no engine library installed here") + + +# --------------------------------------------------------------------------- +# diff_runs: no sidecar is not a pass +# --------------------------------------------------------------------------- +# +# `_check_comparable` skipped a run whose sidecar was missing ("nothing to +# check"), but every check it makes — complete?, which mode?, sorted and +# filtered how?, capped? — lives in that sidecar. Its absence removed all the +# guards at once and the diff went ahead as if each had passed. The leniency +# had one real cause, which no longer exists: scraper_api_client.py wrote no +# sidecar. It does now. + +def _write_run(tmp, name, rows, meta=None): + path = os.path.join(tmp, name + ".json") + with open(path, "w", encoding="utf-8") as fh: + json.dump(rows, fh) + if meta is not None: + with open(os.path.join(tmp, name + ".meta.json"), "w", + encoding="utf-8") as fh: + json.dump(meta, fh) + return path + + +def _diff(old, new, *extra): + import contextlib + import diff_runs + saved = sys.argv[:] + out = io.StringIO() + try: + sys.argv = ["diff_runs.py", "--old", old, "--new", new] + list(extra) + with contextlib.redirect_stdout(out): + code = diff_runs.main() + finally: + sys.argv = saved + return code, out.getvalue() + + +def _meta(**over): + base = {"mode": "listings", "status": "complete", + "stop_reason": "completed", "pages_requested": 1, + "pages_completed": 1, "pages_failed": [], "sort": "publishedAt", + "order": "descending", "filters": {}} + base.update(over) + return base + + +def check_diff_runs_refuses_a_run_with_no_sidecar(): + rows = [{"sku": "a", "title": "x", "data_source": "api"}] + with tempfile.TemporaryDirectory() as tmp: + bare_old = _write_run(tmp, "bare_old", rows) + bare_new = _write_run(tmp, "bare_new", rows) + good = _write_run(tmp, "good", rows, _meta()) + + code, out = _diff(bare_old, bare_new) + equal("two bare JSON files are refused", code, 2) + check("and the reason says what is missing", + ".meta.json" in out and "--force" in out, out[-200:]) + code, out = _diff(bare_old, good) + equal("one bare file is refused too", code, 2) + code, out = _diff(good, bare_new) + equal("on either side", code, 2) + check("the message names the file that lacks it", + "bare_new" in out, out[-200:]) + code, _ = _diff(bare_old, bare_new, "--force") + equal("--force is the explicit way round", code, 0) + code, _ = _diff(good, good) + equal("two runs WITH sidecars still diff", code, 0) + + +def check_diff_runs_tells_unknown_filters_from_none(): + """An engine run that used no filters records `filters: {}`; a sidecar that + does not mention filters at all did not SAY. Those are different, and + treating them alike either refuses a comparison that is fine or waves + through one that is not.""" + rows = [{"sku": "a", "title": "x", "data_source": "api"}] + with tempfile.TemporaryDirectory() as tmp: + none_a = _write_run(tmp, "none_a", rows, _meta(filters={})) + none_b = _write_run(tmp, "none_b", rows, _meta(filters={})) + python = _write_run(tmp, "python", rows, + _meta(filters={"categories": "python"})) + silent = _write_run(tmp, "silent", rows, + {k: v for k, v in _meta().items() + if k != "filters"}) + code, _ = _diff(none_a, none_b) + equal("none against none is comparable", code, 0) + code, out = _diff(none_a, python) + equal("none against a filter is refused", code, 2) + check("naming the filters", "categories" in out, out[-250:]) + code, out = _diff(python, silent) + equal("a run that did not say is compared, not refused", code, 0) + check("and the diff says the selections could not be compared", + "did not record which filters" in out, out[-250:]) + + +def check_every_engine_records_its_filters_even_when_there_are_none(): + ran = 0 + for module in ENGINES: + engine = _import_engine(module) + if engine is None: + continue + _, _, meta = _drive_scrape(engine, ["--pages", "1"], + lambda url: ("content", ["a"])) + ran += 1 + check("%s: a sidecar was written" % module, meta is not None) + if meta is not None: + check("%s: an unfiltered run records `filters` as empty, not as " + "nothing" % module, meta.get("filters") == {}, + repr(meta.get("filters", ""))) + if ran == 0: + skip("recorded filters", "no engine library installed here") + + +# --------------------------------------------------------------------------- +# scraper_api_client: it writes a sidecar, and picks the row class by mode +# --------------------------------------------------------------------------- + +def _drive_client(url, mode, body, fmt="json"): + import contextlib + import shutil + import scraper_api_client as client + tmp = tempfile.mkdtemp(prefix="jj_client_") + saved_fetch, saved_argv = client.fetch_html, sys.argv[:] + try: + client.fetch_html = lambda args: (body, 200) + out = os.path.join(tmp, "run") + sys.argv = ["scraper_api_client.py", "--url", url, "--mode", mode, + "--key", "not-a-real-key", "--out", out, "--format", fmt] + with contextlib.redirect_stdout(io.StringIO()): + code = client.main() + meta_path = out + ".meta.json" + meta = (json.load(open(meta_path, encoding="utf-8")) + if os.path.exists(meta_path) else None) + files = sorted(os.listdir(tmp)) + csv_head = None + if os.path.exists(out + ".csv"): + csv_head = open(out + ".csv", encoding="utf-8").readline().strip() + return code, meta, files, csv_head + finally: + client.fetch_html, sys.argv = saved_fetch, saved_argv + shutil.rmtree(tmp, ignore_errors=True) + + +def check_the_scraper_api_client_writes_a_sidecar(): + code, meta, files, _ = _drive_client( + pp.api_url(items=7), "listings", LISTING_JSON) + equal("a listings run through the client exits 0", code, 0) + check("and leaves a sidecar beside it", meta is not None, str(files)) + if meta is not None: + equal("status", meta["status"], "complete") + equal("mode", meta["mode"], "listings") + equal("it says it went through the Scraper API", meta.get("route"), + "scraper_api") + equal("and how many rows", meta["products"], 7) + # What a consumer can then DO with it. + with tempfile.TemporaryDirectory() as tmp: + rows = [{"sku": "a", "title": "x", "data_source": "api"}] + a = _write_run(tmp, "a", rows, meta or _meta()) + b = _write_run(tmp, "b", rows, meta or _meta()) + code, _ = _diff(a, b) + equal("so diff_runs accepts a client run beside a client run", code, 0) + + +def check_the_scraper_api_client_marks_a_facets_run_partial_and_writes_csv(): + """One URL fetched, so the category counts — a second endpoint — are never + in the body. The run is partial by construction and must say so. Through + `save` it also wrote the CSV with the JobPosting header, which fails on + the first Facet row.""" + code, meta, files, csv_head = _drive_client( + output_writer_facets_url(), "facets", FACETS_JSON, fmt="both") + equal("a facets run through the client is partial", code, 6) + check("a sidecar was written", meta is not None, str(files)) + if meta is not None: + equal("status", meta["status"], "partial") + equal("it names the missing group", meta.get("missing_groups"), + ["categories"]) + equal("and why it stopped", meta["stop_reason"], + "facet_groups_missing") + check("the CSV has the Facet header, not the JobPosting one", + csv_head and csv_head.startswith("source,scraped_at,url,sku,title," + "facet_group"), str(csv_head)) + + +def output_writer_facets_url(): + return pp.facets_api_url() + + +# --------------------------------------------------------------------------- +# Output writes are atomic +# --------------------------------------------------------------------------- +# +# `open(path, "w")` truncates before a byte is written, so a crash halfway +# through `json.dump` replaced last night's good file with a shorter one. The +# sidecar is the part that matters most: it is the file a consumer branches on. + +def check_a_crash_during_a_write_leaves_the_previous_file_whole(): + import output_writer as ow + with tempfile.TemporaryDirectory() as tmp: + good_rows = [ow.JobPosting(source="s", scraped_at="t", url="u", + sku="a")] + paths = {"json": os.path.join(tmp, "o.json"), + "csv": os.path.join(tmp, "o.csv"), + "meta": os.path.join(tmp, "o.meta.json")} + ow.write_json(good_rows, paths["json"]) + ow.write_csv(good_rows, paths["csv"]) + ow.write_run_meta(os.path.join(tmp, "o"), {"status": "complete"}) + before = {k: open(p, "rb").read() for k, p in paths.items()} + + class Unserialisable: + pass + + def attempt(label, fn): + try: + fn() + except BaseException: + pass + else: + check("%s: the planted failure was raised" % label, False) + + attempt("json", lambda: ow.write_json([object()], paths["json"])) + attempt("csv", lambda: ow.write_csv([object()], paths["csv"])) + attempt("meta", lambda: ow.write_run_meta( + os.path.join(tmp, "o"), {"status": Unserialisable()})) + for k, p in paths.items(): + check("%s: the previous file is byte-identical after a failed " + "write" % k, open(p, "rb").read() == before[k]) + left = [n for n in os.listdir(tmp) if n.endswith(".tmp")] + check("and no temporary file is left behind", not left, str(left)) + + +def check_an_atomic_write_keeps_the_files_permissions(): + """The temporary file is created 0600 and a rename keeps that. Measured in + the sibling repos that adopted this writer: every output came out + owner-only.""" + import output_writer as ow + if not hasattr(os, "chmod") or os.name == "nt": + skip("permissions", "no POSIX modes here") + return + with tempfile.TemporaryDirectory() as tmp: + row = [ow.JobPosting(source="s", scraped_at="t", url="u", sku="a")] + fresh = os.path.join(tmp, "fresh.json") + ow.write_json(row, fresh) + mask = os.umask(0) + os.umask(mask) + equal("a new file gets what open() would have given", + os.stat(fresh).st_mode & 0o777, 0o666 & ~mask) + kept = os.path.join(tmp, "kept.json") + ow.write_json(row, kept) + os.chmod(kept, 0o640) + ow.write_json(row, kept) + equal("an existing file keeps its own mode", + os.stat(kept).st_mode & 0o777, 0o640) + + +def check_every_output_write_goes_through_the_atomic_helper(): + import ast + import inspect + import output_writer as ow + for name in ("write_json", "write_csv", "write_run_meta"): + tree = ast.parse(inspect.getsource(getattr(ow, name)).lstrip()) + opens = [n for n in ast.walk(tree) if isinstance(n, ast.Call) + and getattr(n.func, "id", None) == "open"] + atomics = [n for n in ast.walk(tree) if isinstance(n, ast.Call) + and getattr(n.func, "id", None) == "_atomic"] + check("%s writes through _atomic" % name, bool(atomics)) + check("and has no bare open()", not opens) + tree = ast.parse(inspect.getsource(ow._atomic).lstrip()) + attrs = {n.attr for n in ast.walk(tree) if isinstance(n, ast.Attribute)} + check("the temp file is created in the target's own directory", + "dir" in {k.arg for n in ast.walk(tree) if isinstance(n, ast.Call) + for k in n.keywords}) + check("it is fsynced and renamed", {"fsync", "replace"} <= attrs) + + +def check_the_csv_newline_survives_the_atomic_write(): + """Without `newline=""` the csv module's own \\r\\n gains a second CR per + row on Windows. POSIX never translates, so no run here can show the fault; + this reads the two places that carry it instead of pretending to measure + it.""" + import ast + import inspect + import output_writer as ow + tree = ast.parse(inspect.getsource(ow.write_csv).lstrip()) + calls = [n for n in ast.walk(tree) if isinstance(n, ast.Call) + and getattr(n.func, "id", None) == "_atomic"] + kw = {k.arg: k.value for c in calls for k in c.keywords} + check("write_csv asks _atomic for newline=''", + isinstance(kw.get("newline"), ast.Constant) + and kw["newline"].value == "", repr(kw)) + tree = ast.parse(inspect.getsource(ow._atomic).lstrip()) + passed = [k for n in ast.walk(tree) if isinstance(n, ast.Call) + for k in n.keywords if k.arg == "newline"] + check("and _atomic hands it to the temporary file", bool(passed)) + + def check_the_404_is_not_a_block(): """An offer taken down between the sitemap and the fetch is ordinary.""" equal("state", pp.detect_page_state(NOT_FOUND_JSON, 404, @@ -1482,7 +2507,8 @@ def check_policy_constants_have_a_consumer(): "should_retry", "should_solve", "counts_as_blocked", "should_parse", "concurrency_limit", "concurrency_for_mode", "pagination_is_addressable", "plan_page_urls", - "enumerate_offers", "needs_readiness_wait", "is_terminal"): + "enumerate_offers", "needs_readiness_wait", "queue_decision", + "pages_are_independent"): check("page_flow.%s has a caller outside its own module" % fn, fn in joined, "unused policy") @@ -1493,7 +2519,8 @@ def check_policy_constants_have_a_consumer(): # whether ANYTHING reads them, and the answer has to come from the # module's own source rather than from the engines'. own = open(os.path.join(HERE, "page_flow.py"), encoding="utf-8").read() - for fn in ("pages_to_plan", "max_offset_for", "mask_secrets"): + for fn in ("pages_to_plan", "max_offset_for", "mask_secrets", + "is_terminal"): calls = own.count(fn + "(") check("page_flow.%s is called, not merely defined" % fn, calls >= 2, "%d occurrence(s) — only its own def" % calls) @@ -1902,11 +2929,15 @@ def check_banned_and_removed_flags(): check("%s refuses a --url whose route disagrees with --mode" % module, "but --mode is" in source, "the refusal that makes --mode safe here is missing") - check("%s refuses a filter outside --mode listings" % module, - "does \n \"not read. Use --mode listings to filter" in source - or "not read. Use --mode listings to filter" in source, - "the refusal that keeps a filter from looking applied when it " - "is not is missing") + # The refusal itself now lives in ONE place — product_parser.args_problem + # — so the question per engine is whether it is CALLED, and from + # parse_args rather than from somewhere a run does not reach. + check("%s asks product_parser.args_problem" % module, + "args_problem(args)" in source, + "the shared refusal exists and this engine does not call it") + check("%s turns its answer into a usage error (exit 2)" % module, + "p.error(problem)" in source, + "the refusal would be computed and then ignored") def check_undefined_names_in_every_module():