From 74ba171e17536b7c61f885300952a5f54a72c098 Mon Sep 17 00:00:00 2001 From: Stefan S Date: Fri, 30 Oct 2026 17:10:29 +0100 Subject: [PATCH 01/55] doc: notice identity docs, README pointer, and CHANGELOG entry (issue #38) --- CHANGELOG.md | 3 ++ README.md | 2 +- doc/NOTICE_IDENTITY.md | 109 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 113 insertions(+), 1 deletion(-) create mode 100644 doc/NOTICE_IDENTITY.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 0a70277..d311ea6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added +- Opt-in daemon `notice_identity` with source-scoped PDF revision tracking, + nullable Download identity columns, and pinned pypdf. Original PDFs are kept. + (issue #38) - Opt-in daemon `replacement_calendar` action for bounded monthly replacement-calendar traversal (issue #37). Months come from the run-start date context (issue #26). The legacy batch runner refuses the action. diff --git a/README.md b/README.md index 6156ea9..14606cf 100644 --- a/README.md +++ b/README.md @@ -326,7 +326,7 @@ The BillCollector configuration for each web service from which you want to retr - One recipe per web portal identified by the yaml element `serviceName` and its filename `recipe-pw__`. - Each recipe is structured in numbered `steps`, executed in ascending `step` order. - Each step is a chain of `methods`: Playwright Page/Locator calls executed in order. Locator methods (`locator` for CSS selectors, `get_by_role`, `get_by_label`, `get_by_text`, `get_by_title`, `get_by_placeholder`, `get_by_test_id`) yield a locator that is chained into the following action (`click`, `fill`, `press`, `first`). Page-level methods include `goto` (open the start URL), `content_frame` (enter an iframe) and `close`. - - Downloads: a step with the `expect_download` method wraps the nested `steps` that trigger the download; the file is saved into the Downloads folder. A re-downloaded file that is byte-identical to one already tracked for the same task (same sha256) is recorded as a duplicate and the consumer hook is not re-run for it. + - Downloads: a step with the `expect_download` method wraps the nested `steps` that trigger the download; the file is saved into the Downloads folder. A re-downloaded file that is byte-identical to one already tracked for the same task (same sha256) is recorded as a duplicate and the consumer hook is not re-run for it. An opt-in `notice_identity` step option switches a download step from byte-hash deduplication to source-scoped identity plus a conservative PDF revision digest; see [notice identity](doc/NOTICE_IDENTITY.md). - Bounded pagination: use one page locator followed by `click_until_absent`. Its single options object requires `progress_selector`, `max_clicks`, and `timeout_ms`. `timeout_ms` is the total browser-action budget. Every click must increase the visible progress count or make the pager absent or hidden. This action supports cumulative "load more" controls. It rejects page-replacement pagination and virtualized lists because their visible item count does not increase. For monthly replacement calendars, use the daemon's [replacement_calendar action](doc/REPLACEMENT_CALENDARS.md). - Bounded collections: use one page locator followed by `download_all`. Its single options object requires `key_attribute`, `max_items`, `timeout_ms`, and `event_window_ms`. `timeout_ms` bounds the browser actions and observations for the whole collection. Persistence (`save_as`, hashing, consumers, and SQLite) remains synchronous and cannot be preempted by that browser deadline. `event_window_ms` is the full attribution window observed after each click, so `timeout_ms` must exceed the actual item count multiplied by `event_window_ms`, with additional time for browser actions. The engine reports the required and remaining budgets before its first click when the minimum attribution-window budget cannot fit. The key must be non-empty and unique for every item. The ordered key set must remain stable while each control produces exactly one download. `download_all` also rejects a suggested filename already saved earlier in the run instead of overwriting it. - Collection actions are terminal steps. They cannot be nested below `expect_download` or use `graceful: true`. Their explicit bounds prevent silent truncation and automatic retry after partial side effects. An operator-requested resume of `download_all` skips the stable item keys already persisted by that step and processes only the remaining items. diff --git a/doc/NOTICE_IDENTITY.md b/doc/NOTICE_IDENTITY.md new file mode 100644 index 0000000..328814c --- /dev/null +++ b/doc/NOTICE_IDENTITY.md @@ -0,0 +1,109 @@ +> Issue #38 implementation. Existing recipes retain the baseline byte-hash +> download path. The notice path builds on the #33 staged-publication and +> crash-recovery pipeline, which remains unchanged. + +# Notice identity and PDF revisions + +The collector owns repeat suppression. Paperless owns archive storage. A +download record is not an archive receipt. This feature does not change +delivery recovery, consumer error handling, run results, scheduling, or +financial import behavior. + +## Recipe interface + +Opt in on a daemon `expect_download`, `download_all`, or `replacement_calendar` +step. For example: + +```yaml +step: 4 +notice_identity: + query_parameters: [account, notice] +methods: + - method: expect_download +steps: + - step: 1 + methods: + - method: get_by_role + arguments: [{role: link}, {name: Download}] + - method: click +``` + +Use the provider's stable account, document kind and source notice identifier. +The example uses invented field names. It is not a Bourse Direct recipe. +The recipe must verify the selected account and document kind before download. +A date or filename alone is insufficient. Include a source discriminator when +several notices can exist on the same day. The task must represent one account. + +The source identity includes the HTTP(S) origin, URL path and declared query +fields. The database lookup scopes that identity to the task. Each field must occur exactly once with a nonblank value. Credentials +in the URL authority, blob URLs and missing or repeated fields are refused. +Undeclared query fields can include session tokens and do not affect identity. +The collector stores only a SHA-256 digest of this identity. It does not retain +the selected query values. Existing sanitized download URL retention still +applies. Do not use a query credential as an identity field. + +## Revision rule: pdf-graph-v1 + +The revision hashes the parsed PDF object graph with pypdf 6.21.2. It retains +page content, encoded streams, fonts, images, annotations, attachments, forms, +XMP, document metadata and local destination targets. Original bytes are never +rewritten. The following generation differences do not create a revision: + +- Trailer `/ID` and Info `/CreationDate` and `/ModDate`. +- Indirect object numbers, dictionary order and stream `/Length`. +- A generated alias of form `[A-Za-z]{3}_[A-Za-z0-9]{6}` for a single named + destination in a single-child name tree, its + matching `/Limits`, and matching local `/Dest` and `/GoTo` references. + Destination targets and outline titles remain part of the revision. External + links to that alias are outside this content-identity contract. + +Text equality is never a dedup key. A change to any other retained graph value +creates another revision. Recompression, XMP timestamp changes or a different +object-sharing layout can conservatively create another revision. This is not +a claim that all visually equivalent PDFs deduplicate. Changing this rule needs +a new fingerprint version and fresh acceptance evidence. + +A new source or revision publishes its untouched original through the existing +consumer path. An unchanged revision points to the first original and does not +publish or call consumers again. Download rows retain the raw hash and size of +each attempt plus `duplicate_of_id`. Regenerated duplicate bytes are discarded +from temporary staging. Earlier revisions remain available. Equal filenames +receive unique output paths; opted-in collection steps permit those names. +Default collection steps retain their existing collision guard. + +Encrypted, empty, oversized (>20 MiB), malformed or unsupported PDFs fail the +step without a revision record or publication. Traversal stops at 100,000 nodes +or depth 100. Parser diagnostics are suppressed and warnings reject identity. +The parser does not decode content streams. The existing runner failure/pause +contract applies; there is no text-only fallback. + +## Existing databases and recovery + +The schema adds nullable `source_identity` and `revision_sha256` columns and an +index. Existing rows and raw-SHA behavior remain intact. There is no automatic +historical conversion or private-file scan. The first opted-in acquisition of +an old notice can publish once because historical rows have no source identity. +A later acquisition of that revision is suppressed, including after DB reopen. +Different sources can publish identical bytes; different tasks never share the +notice-identity dedup key. A task/account reconfiguration needs its own history +and recovery decision. + +The existing publication/consumer/DB crash window remains unchanged. There is +no exactly-once consumer guarantee. Keep existing originals and database backup +procedures. An unavailable archive copy is not recovered by re-downloading an +unchanged revision; use the archive recovery procedure. + +Rollback: revert the code commit. Keep the additive columns; the old engine +ignores them. Restore a pre-change DB backup only under an independently approved +recovery plan. Never downgrade a live database automatically. + +## Isolated verification + +```bash +apps/.venv/bin/python -m pytest -q tests/test_notice_identity.py +``` + +Expected: all tests pass. The browser test uses intercepted loopback downloads, +original synthetic PDFs, equal filenames, three runs, one correction and no +external service. Other tests cover reloads, account/source separation, hashed +identities, invalid input, legacy migration and non-text corrections. From af5d542b32300e97364c4efb12c012091c9c66ea Mon Sep 17 00:00:00 2001 From: Stefan S Date: Fri, 30 Oct 2026 20:14:44 +0100 Subject: [PATCH 02/55] =?UTF-8?q?doc:=20plans=20=E2=80=94=20PR=20#41:=20DE?= =?UTF-8?q?LIVERED=20(S7=20complete:=20Gitea=20main=2074ba171,=20GitHub=20?= =?UTF-8?q?main=20e83c8e2,=20PR=20#41=20+=20issue=20#38=20closed),=20S7=20?= =?UTF-8?q?execution=20record=20+=20archive=20to=20completed/?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../plans/1792862814701-next-up-sequencing.md | 33 ++++++++++++++++--- .kilo/plans/README.md | 4 +-- ...793124431799-open-issues-plans-20261027.md | 0 ...93313075441-pr41-notice-identity-review.md | 0 4 files changed, 29 insertions(+), 8 deletions(-) rename .kilo/plans/{ => completed}/1793124431799-open-issues-plans-20261027.md (100%) rename .kilo/plans/{ => completed}/1793313075441-pr41-notice-identity-review.md (100%) diff --git a/.kilo/plans/1792862814701-next-up-sequencing.md b/.kilo/plans/1792862814701-next-up-sequencing.md index e9fb346..2e913c3 100644 --- a/.kilo/plans/1792862814701-next-up-sequencing.md +++ b/.kilo/plans/1792862814701-next-up-sequencing.md @@ -227,8 +227,8 @@ interleaved with the remaining Stage 1b portal sessions.** request. - New PRs #35/#39/#40/#41 arrived after the sequencing decision; they do not change #32's position (no known overlap with its 15 files) — a sequencing - addendum is a separate decision. **Addendum decided 2026-10-27** (plan - `1793124431799-open-issues-plans-20261027.md`): #34 → #35 → #42 → #39 → + addendum is a separate decision. **Addendum decided 2026-10-27** (plan + `completed/1793124431799-open-issues-plans-20261027.md`): #34 → #35 → #42 → #39 → #40, with #41 HELD on the #38 archive decision; PR #42 (issue #43) arrived 2026-10-27. - PR #35 S1+S2 done 2026-10-27 ("proceed with next pr" session): review plan @@ -308,9 +308,32 @@ interleaved with the remaining Stage 1b portal sessions.** commits `df3ec4b`/`d34e396`/`61defd3`, Gitea `main` `61defd3` (ff-only from `b693517`) / GitHub `main` `f26b2a9`; PR #40 closed without merge (comment 6509093900, 2026-10-29T21:48:00Z); issue #37 closed (comment - 6509221226, 2026-10-29T21:55:06Z); Discussion summary omitted at user - request). New PR #44 (C3 method-chain validation) arrived after the - addendum — separate sequencing decision. + 6509221226, 2026-10-29T21:55:06Z); Discussion summary omitted at user + request). New PR #44 (C3 method-chain validation) arrived after the + addendum — separate sequencing decision. + - PR #41 S1+S2 done 2026-10-29 ("continue PR #41" session): review plan + doc `completed/1793313075441-pr41-notice-identity-review.md` written, + verdict approve with maintainer changes (head `74421dd` stable; branch + re-commits the sanitized tip `d9362b3` as an orphaned + public-history commit, skipped; actual content = 10 files +828/−10; #38 + HOLD revised toward integration 2026-10-29 with F10/C8 as an accepted + risk; M1 contract / M2 schema / M3 #33 re-author as S3 decision + points; deployment-validation skip recorded), S3–S7 across three + "continue PR #41" sessions: S3 as-is, S4 selective integration (6 clean + files + db/recipes manual merge + downloads re-authored on the #33 + journal + 2 test adaptations + CHANGELOG fold), S5 all green on the + exact tree (14/14 unit, 647 + 53 subtests, harness PASS 7/7, 15/15 + recipes, ruff/bandit baseline; deployment skip user-confirmed), S6 docs + as-is (README pointer + NOTICE_IDENTITY header reframe, docs-only), + S7 — commits `14bab50`/`5c8216f`/`74ba171`, Gitea `main` `74ba171` + (ff-only from `61defd3`) / GitHub `main` `e83c8e2`; PR #41 closed + without merge (comment 6531164937, 2026-10-30T19:00:36Z); issue #38 + closed completed (comment 6531165804, 2026-10-30T19:00:39Z). + **The planned PR chain #34 → #35 → #42 → #39 → #40 → #41 is + complete.** Discussion summary text recorded in the review doc for + manual web-UI post (write API unavailable). Flowcool's archive-side + contract proposal on #38 (2026-10-30, comment 6518010065) awaits a + maintainer decision — a dedicated new issue if tracked. - The 1und1 portal iteration remains independent and can run in a live session at any time. diff --git a/.kilo/plans/README.md b/.kilo/plans/README.md index 7b784d3..0534353 100644 --- a/.kilo/plans/README.md +++ b/.kilo/plans/README.md @@ -18,6 +18,4 @@ paths, so a moved file gains the `completed/` prefix. | `1792793307574-recipe-repair-llm-options.md` | Public service recipe recovery — LLM-assisted repair options analysis (pre-step); Stage 0 triage done 2026-10-24, all 7 services classified | | `1792819039884-recipe-doctor-implementation.md` | Stage 1: offline LLM recipe doctor (`billcollector doctor`) implementation plan + pilot (buhl, datev) — both pilots green 2026-10-24 (runs 16/18, downloads succeeded); Stage 1b in progress (2026-10-25): 4/7 green (buhl, datev, nuernberger, winsim), remaining 1und1 (in progress), freenet_mobilfunk (not started), kabeldeutschland (Option D re-authoring) | | `1792861675256-issue27-maintainer-commitments.md` | Issue #27 product-audit triage (closed 2026-10-24): maintainer-side commitments C1-C8 (F04, F09, F08 contract, F06 contract, F02, F03, F05, F10) with acceptance criteria, dependencies and proposed order | - | `1792862814701-next-up-sequencing.md` | Next-up sequencing decision (2026-10-24), superseded in part (2026-10-25): C1–C3 done and pushed; current primary next-up is the PR continuation order #30 → #31 → #33 → #32 → #34 (one pr-review chain at a time), interleaved with Stage 1b portal sessions (next: 1und1); PR #30 + #31 DELIVERED 2026-10-25, PR #33 DELIVERED 2026-10-26, PR #32 DELIVERED 2026-10-26 (Gitea `main` `bde713c` / GitHub `93bfcc8`), PR #34 DELIVERED 2026-10-27 (Gitea `main` `53a5dff` / GitHub `6bb3348`), PR #35 DELIVERED 2026-10-28 (Gitea `main` `75b885b` / GitHub `484048d`), PR #42 DELIVERED 2026-10-28 (Gitea `main` `2a6e462` / GitHub `c2fb7f8`), PR #39 DELIVERED 2026-10-28 (Gitea `main` `b693517` / GitHub `ad18f7a`), PR #40 DELIVERED 2026-10-29 (Gitea `main` `61defd3` / GitHub `f26b2a9`) — the #34 → #35 → #42 → #39 → #40 chain is complete | - | `1793124431799-open-issues-plans-20261027.md` | All six open issues planned 2026-10-27 (#24 F10/C8 lane, #26 contract confirmed → PR #34 next, #36 + #37 agreed in principle → PRs #39/#40, #38 byte-hash kept + identity delegated to archive → PR #41 HELD, #43 fix assessed → PR #42); PR chain addendum: #34 → #35 → #42 → #39 → #40, #41 held (2026-10-29: #41 HOLD lifted, revised toward integration); decisions posted on all six issues; #34 DELIVERED 2026-10-27, #35 DELIVERED 2026-10-28, #42 DELIVERED 2026-10-28, #39 DELIVERED 2026-10-28, #40 DELIVERED 2026-10-29 — chain complete | - | `1793313075441-pr41-notice-identity-review.md` | PR #41 (issue #38) notice identities + PDF revisions — S1+S2 review done 2026-10-29, verdict approve with maintainer changes; #38 HOLD revised toward integration (F10/C8 accepted risk); next S3 proposal | + | `1792862814701-next-up-sequencing.md` | Next-up sequencing decision (2026-10-24), superseded in part (2026-10-25): C1–C3 done and pushed; current primary next-up is the PR continuation order #30 → #31 → #33 → #32 → #34 (one pr-review chain at a time), interleaved with Stage 1b portal sessions (next: 1und1); PR #30 + #31 DELIVERED 2026-10-25, PR #33 DELIVERED 2026-10-26, PR #32 DELIVERED 2026-10-26 (Gitea `main` `bde713c` / GitHub `93bfcc8`), PR #34 DELIVERED 2026-10-27 (Gitea `main` `53a5dff` / GitHub `6bb3348`), PR #35 DELIVERED 2026-10-28 (Gitea `main` `75b885b` / GitHub `484048d`), PR #42 DELIVERED 2026-10-28 (Gitea `main` `2a6e462` / GitHub `c2fb7f8`), PR #39 DELIVERED 2026-10-28 (Gitea `main` `b693517` / GitHub `ad18f7a`), PR #40 DELIVERED 2026-10-29 (Gitea `main` `61defd3` / GitHub `f26b2a9`), PR #41 DELIVERED 2026-10-30 (Gitea `main` `74ba171` / GitHub `e83c8e2`), issue #38 closed — the planned chain #34 → #35 → #42 → #39 → #40 → #41 is complete; flowcool's archive-side contract proposal on #38 (2026-10-30) awaits a maintainer decision | diff --git a/.kilo/plans/1793124431799-open-issues-plans-20261027.md b/.kilo/plans/completed/1793124431799-open-issues-plans-20261027.md similarity index 100% rename from .kilo/plans/1793124431799-open-issues-plans-20261027.md rename to .kilo/plans/completed/1793124431799-open-issues-plans-20261027.md diff --git a/.kilo/plans/1793313075441-pr41-notice-identity-review.md b/.kilo/plans/completed/1793313075441-pr41-notice-identity-review.md similarity index 100% rename from .kilo/plans/1793313075441-pr41-notice-identity-review.md rename to .kilo/plans/completed/1793313075441-pr41-notice-identity-review.md From 925223f13a1163ddaa44d6b3306da7e353da45c1 Mon Sep 17 00:00:00 2001 From: Stefan S Date: Fri, 30 Oct 2026 20:45:37 +0100 Subject: [PATCH 03/55] =?UTF-8?q?doc:=20plans=20=E2=80=94=20PR=20#41:=20fi?= =?UTF-8?q?nalize=20S7=20record=20(DELIVERED=20status=20+=20S7=20execution?= =?UTF-8?q?=20record=20in=20pr41=20plan=20doc,=20open-issues=20DELIVERED?= =?UTF-8?q?=20entry=20+=20completed/=20refs=20=E2=80=94=20final=20edits=20?= =?UTF-8?q?missed=20by=20interrupted=20commit=20af5d542)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...793124431799-open-issues-plans-20261027.md | 19 +++- ...93313075441-pr41-notice-identity-review.md | 89 ++++++++++++++++--- 2 files changed, 92 insertions(+), 16 deletions(-) diff --git a/.kilo/plans/completed/1793124431799-open-issues-plans-20261027.md b/.kilo/plans/completed/1793124431799-open-issues-plans-20261027.md index 8877acb..da5e642 100644 --- a/.kilo/plans/completed/1793124431799-open-issues-plans-20261027.md +++ b/.kilo/plans/completed/1793124431799-open-issues-plans-20261027.md @@ -94,7 +94,13 @@ review lane for each — plus the PR-chain addendum for the six open PRs `notice_identity` + PDF revision digest is accepted as the path forward (superseding the archive-delegation direction for the collector). F10/C8 stays a separate open lane and is carried as an accepted risk. See the - PR #41 review plan doc `1793313075441-pr41-notice-identity-review.md`. + PR #41 review plan doc `completed/1793313075441-pr41-notice-identity-review.md`. +- **DELIVERED 2026-10-30:** PR #41 integrated (commits `14bab50`/`5c8216f`/ + `74ba171`, Gitea `main` `74ba171` / GitHub `main` `e83c8e2`); PR #41 + closed without merge (comment 6531164937); issue #38 closed completed + (comment 6531165804). Flowcool's 2026-10-30 archive-side contract + proposal (comment 6518010065) remains an open question — a dedicated new + issue if tracked. ### #43 — runner startup, unwritable application directory @@ -110,7 +116,8 @@ review lane for each — plus the PR-chain addendum for the six open PRs decision" note for #35/#39/#40/#41; #42 arrived 2026-10-27) **Order: #34 → #35 → #42 → #39 → #40; #41 HELD (see #38 above).** All five -delivered as of 2026-10-29 (#40 last). +delivered as of 2026-10-29 (#40 last); #41 DELIVERED 2026-10-30 — all six +planned PRs delivered. Grounds: @@ -152,5 +159,11 @@ pre-image check against current dev (dev has diverged from `93bfcc8`). (#40 last: Gitea `main` `61defd3` / GitHub `main` `f26b2a9`, PR #40 closed without merge, issue #37 closed). Plan stays Active for the #38 decision path. 2026-10-29: PR #41 HOLD lifted (revised toward integration; under - pr-review, see `1793313075441-pr41-notice-identity-review.md`); archive this + pr-review, see `completed/1793313075441-pr41-notice-identity-review.md`); archive this plan to `completed/` once PR #41 reaches DELIVERED/REJECTED. +- 2026-10-30: PR #41 DELIVERED (S7 complete; Gitea `main` `74ba171` / + GitHub `main` `e83c8e2`; PR #41 closed without merge; issue #38 closed + completed). Archive trigger met — this plan moves to `completed/`. The one + open question left by this plan is flowcool's 2026-10-30 archive-side + contract proposal on #38 (comment 6518010065) — a dedicated new issue if + tracked. diff --git a/.kilo/plans/completed/1793313075441-pr41-notice-identity-review.md b/.kilo/plans/completed/1793313075441-pr41-notice-identity-review.md index e9b8959..400d7a2 100644 --- a/.kilo/plans/completed/1793313075441-pr41-notice-identity-review.md +++ b/.kilo/plans/completed/1793313075441-pr41-notice-identity-review.md @@ -1,14 +1,12 @@ # PR #41 review: source notice identities and PDF revisions (issue #38) -> Status: VALIDATED (2026-10-30) — S5 all green on the exact dirty tree (dev -> `e0cab50` + the 10 uncommitted S4 files): 14/14 unit, 647 passed / 10 -> skipped full regression, harness VERDICT: PASS (7/7 scenarios, 8 PDFs), -> 15/15 recipe validation, ruff clean, bandit baseline-identical, pypdf -> 6.21.2 verified in venv; deployment-process validation justified skip -> (no deploy-script change; pypdf via existing `Dockerfile:35` pip install) -> user-confirmed 2026-10-30. S6 documentation gate passed 2026-10-30 (doc set -> confirmed as-is; the two S6 edits are docs-only). Awaiting S7 -> (commit + push). +> Status: DELIVERED (2026-10-30) — S7 complete: dev commits +> `14bab50`/`5c8216f`/`74ba171` (feat/test/doc), Gitea `main` `74ba171` +> (ff-only from `61defd3`), GitHub `main` `e83c8e2` (mirrored from +> `f26b2a9`); PR #41 closed without merge (comment 6531164937, +> 2026-10-30T19:00:37Z), issue #38 closed completed (comment 6531165804, +> 2026-10-30T19:00:40Z); Discussion summary text recorded for manual web-UI +> post (write API unavailable). ## PR metadata @@ -42,7 +40,7 @@ are S3 decision points per issue #38. ## Decision history (issue #38) - **2026-10-27 (comment `6449978430`, recorded in - `1793124431799-open-issues-plans-20261027.md`):** collector keeps byte-hash + `completed/1793124431799-open-issues-plans-20261027.md`):** collector keeps byte-hash dedup; stable notice identity / revision handling **delegated to the archive layer**. PR #41 **HELD** until (1) #34 lands, (2) F10/C8 delivery-state exposure is scheduled/done, (3) identity scope + @@ -348,9 +346,74 @@ protected path and the stale PR #31 CHANGELOG line. - **Re-validation note (S7.0 prerequisite):** the two S6 edits touch docs-only files (README.md, doc/NOTICE_IDENTITY.md). No code or deployment-surface file changed after the S5 gate; no S5 check reads those - two files (`test_doc_recipe.py` reads only - `doc/examples/recipe-pw__synthetic.yaml`), so the S5 code checks and the - user-confirmed deployment-validation skip still cover the tree. + two files (`test_doc_recipe.py` reads only + `doc/examples/recipe-pw__synthetic.yaml`), so the S5 code checks and the + user-confirmed deployment-validation skip still cover the tree. + +## S7 execution record (2026-10-30) + +- **S7.0 prerequisite (exact tree):** `dev` == `origin/dev` == `27fd70b` + (doc-only S6 commit); `main` == `origin/main` == `61defd3`; PR head + `74421dd` unchanged; the 9 non-doc dirty files byte-match the S5 + fingerprints; README.md + doc/NOTICE_IDENTITY.md exactly the S6-confirmed + content — the S5 code checks and the user-confirmed deployment-validation + skip still cover the tree. +- **Commits (dev-commits, per-item confirmed):** `14bab50` `feat: opt-in + notice_identity with source-scoped identity and PDF revision digest + (issue #38)` (7 files, +308/−13); `5c8216f` `test: notice identity + revision-digest and runner adapter tests (issue #38)` (1 file, +326); + `74ba171` `doc: notice identity docs, README pointer, and CHANGELOG entry + (issue #38)` (3 files, +113/−1). Pushed `origin/dev` `27fd70b..74ba171`. +- **Promotion (user-gated):** Gitea `main` `61defd3` → `74ba171` ff-only + (7 commits: the 4 PR #41 plan-doc commits + the 3 above); pushed. No tag + (no release decided). +- **Mirror:** `bash .private/sync_github.sh` → GitHub `main` `f26b2a9` → + `e83c8e2` (sanitize-by-amend). Public gate green: no `.kilo`/`.vscode`/ + `.private`/`openwiki`/`AGENTS.md`/`.openwikiignore`; `.github` holds only + `ISSUE_TEMPLATE` (no `workflows/`); `notice.py`, + `tests/test_notice_identity.py`, `doc/NOTICE_IDENTITY.md` present. +- **PR #41 closed** (user confirmed the exact comment text): comment + 6531164937 (2026-10-30T19:00:36Z), closed without merge + 2026-10-30T19:00:37Z — content reached GitHub via the mirror. +- **Issue #38 closed** (user chose close + comment): comment 6531165804 + (2026-10-30T19:00:39Z), closed completed 2026-10-30T19:00:40Z. The + confirmed draft said "stays open"; only its final paragraph was rewritten + to the close wording ("I am closing it; if you want to track the + archive-side contract, please open a dedicated issue (or re-open this + one)") to match the confirmed close decision — all other text verbatim. +- **Open item handed off:** flowcool's 2026-10-30 archive-side contract + proposal on #38 (comment 6518010065) awaits a maintainer decision; if + tracked, it becomes a dedicated new issue. +- **Discussion summary (S7.6):** GitHub Discussion write API unavailable + from this environment (2026-10-25 finding); user chose manual web-UI + posting — text recorded below. + > **Public update: opt-in notice_identity download tracking (issue #38, + > PR #41)** + > + > `main` now includes the opt-in `notice_identity` step option for daemon + > download steps, contributed by @flowcool in PR #41: source-scoped + > identity plus a conservative PDF object-graph revision digest. A + > metadata-only regeneration of a notice no longer republishes or + > re-runs the consumer; a real correction preserves the original and + > publishes the new revision. + > + > Maintainer-side adaptations: the branch predated the staged-publication + > + crash-recovery pipeline (issue #33) on `main`, so the notice path was + > re-authored on that pipeline and the PR's pre-#33 publish helper was + > not shipped. The recipe contract and the additive `Download` columns + > were agreed and merged into the current schema; two test assertions + > were adapted to house style. + > + > Docs: `doc/NOTICE_IDENTITY.md` (contract + verification), README + > pointer, CHANGELOG entry. Validation on the promoted tree: 14/14 new + > unit tests, 647 passed full suite, regression harness PASS (7/7), 15/15 + > bundled recipes validate. + > + > @flowcool: the archive-side contract proposal on issue #38 remains open + > and under separate review — this delivery does not close that question. +- **Archive:** this plan doc + `completed/1793124431799-open-issues-plans-20261027.md` + (consumed per its own archive trigger: PR #41 reached DELIVERED) moved to + `completed/`; README + cross-references updated. ## Validation plan (S5, on the exact dev tree, from repo root, venv) From 8c9a17d5daf416eed2ed1d1535af979435dff357 Mon Sep 17 00:00:00 2001 From: Stefan S Date: Fri, 30 Oct 2026 22:59:30 +0100 Subject: [PATCH 04/55] =?UTF-8?q?doc:=20plans=20=E2=80=94=20PR=20#53:=20S1?= =?UTF-8?q?+S2=20review=20plan=20doc=20(verdict=20approve=20with=20M1=20CH?= =?UTF-8?q?ANGELOG=20placement;=20head=206b41040b=20stable,=204=20files=20?= =?UTF-8?q?3=20as-is=20+=201=20manual,=20#46=20before-claim=20verified),?= =?UTF-8?q?=20batch-2=20order=20#53=E2=86=92#54=E2=86=92#56=E2=86=92#55?= =?UTF-8?q?=E2=86=92#45=E2=86=92#44,=20sequencing=20+=20README=20updated?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../plans/1792862814701-next-up-sequencing.md | 40 +++++ ...1793397084822-pr53-cron-weekdays-review.md | 160 ++++++++++++++++++ .kilo/plans/README.md | 3 +- 3 files changed, 202 insertions(+), 1 deletion(-) create mode 100644 .kilo/plans/1793397084822-pr53-cron-weekdays-review.md diff --git a/.kilo/plans/1792862814701-next-up-sequencing.md b/.kilo/plans/1792862814701-next-up-sequencing.md index 2e913c3..e3092af 100644 --- a/.kilo/plans/1792862814701-next-up-sequencing.md +++ b/.kilo/plans/1792862814701-next-up-sequencing.md @@ -337,6 +337,46 @@ interleaved with the remaining Stage 1b portal sessions.** - The 1und1 portal iteration remains independent and can run in a live session at any time. +## PR batch #2 (2026-10-30) + +Six open PRs, all flowcool, all based on public `main` `e83c8e2` (the +post-#41 mirror tip): #53 (cron weekday numbering, #46), #54 (persistent +job-store rebuild, #47), #56 (terminal ntfy failures, #49), #55 (exact vault +item identity, #48), #45 (browser cache precedence, no issue), #44 (C3/F08 +recipe method-chain validation, contract agreed in #27). Open issues without +PRs: #50 (incremental download keys + single-PDF ZIP extraction), #51 +(Bourse Direct original-PDF adapter agreement), #52 (archive receipts after +notice identity — the pending #38 archive-side contract), plus the +maintainer lane #24. + +**Decision (user-confirmed 2026-10-30): #53 → #54 → #56 → #55 → #45 → #44, +one pr-review chain at a time.** + +Grounds: + +- #53 and #54 both touch `scheduler.py` + `test_bc_scheduler.py` + (non-overlapping hunks: #53 adds `cron_trigger` + the `_add_job_for` + guard; #54 changes `start()`/`resync()`). Kept adjacent so #54's S1 + pre-image check runs against the settled #53 tree. +- #56 (monitor/orchestrator/notify) and #55 (vault transport) are disjoint + from the scheduler and from each other; both single-commit PRs with S5 + fully local. +- #45 starts from the old sanitized tip `f26b2a9` (103 files / −22928 in the + diff = orphaned-tip STRIP artifact; the real change is ~10 files) and shows + `mergeable_state: dirty` — S1 must untangle the base before any verdict; + scheduled after the clean single-commit PRs. +- #44 (18 files +698/−90) extends the recipe-contract interface (agreed in + #27, cited in the PR body) and is the heaviest S1; it closes the batch. + +- PR #53 S1+S2 done 2026-10-30 ("continue last sessions" session): review + plan doc `1793397084822-pr53-cron-weekdays-review.md` — verdict approve + with maintainer changes (M1: the PR's new `### Fixed` header lands above + `### Added` and duplicates the existing `### Fixed` subsection; the entry + moves into the existing one); head `6b41040b` stable, merge-base + `e83c8e2`, single commit, 4 files — 3 as-is + 1 manual CHANGELOG; issue + #46 before-claim verified on the pinned APScheduler 3.11.3; deployment + validation recorded not deployment-relevant — Status: AWAITING-DECISION. + ## Explicitly not doing - No idling on the PRs: since 2026-10-25 they are handled through the diff --git a/.kilo/plans/1793397084822-pr53-cron-weekdays-review.md b/.kilo/plans/1793397084822-pr53-cron-weekdays-review.md new file mode 100644 index 0000000..108b949 --- /dev/null +++ b/.kilo/plans/1793397084822-pr53-cron-weekdays-review.md @@ -0,0 +1,160 @@ +# PR #53 review: crontab weekday numbering (issue #46) + +> Status: AWAITING-DECISION (2026-10-30) — S1+S2 done: verdict approve with one +> maintainer change (M1, CHANGELOG placement); 4 files, 3 as-is + 1 manual; +> next S3 gate. + +## PR metadata + +- PR: s-t-e-f-a-n/BillCollector#53 "fix: use crontab numbering for numeric + weekdays" (flowcool), branch `fix/upstream-cron-weekdays-20261030`, head + `6b41040b542618c3b59b98a84765baef54408336`, 1 commit, base `main` @ + `e83c8e2`. 4 files +84/−4. `mergeable_state: clean`. +- Issue: #46 (flowcool, 2026-10-30T20:00:20Z) — numeric Sunday schedules fire + on Monday because APScheduler 3.11.3 numbers Monday=0; `7` is refused. +- Created 2026-10-30T20:02:59Z (issue 2 minutes before the PR). +- Local state at review time: Gitea `dev` tip `925223f` (= `origin/dev`; PR #41 + plan record only, no public code changes since `main`), Gitea `main` + `74ba171`, GitHub `main` `e83c8e2`. +- AI disclosure present: "AI-assisted port, review and verification under my + responsibility." + +## Verdict + +**Approve with maintainer changes** — one placement fix (M1, CHANGELOG); all +code and tests apply as-is from head. + +## Verified facts + +1. Head stable: `git ls-remote github refs/pull/53/head` = `6b41040b...` = + fetched `refs/pr/53`. Single commit on merge-base `e83c8e2` (= current + GitHub `main`); no orphaned public-history commits (unlike #40/#41). +2. Issue #46 before-claim verified live on the pinned APScheduler 3.11.3: + `CronTrigger.from_crontab("0 0 * * 0")` → next fire Monday 2026-11-02; + `0 0 * * 7` → rejected ("the last value (7) is higher than the maximum + value (6)"). +3. Pre-image: all 4 PR files byte-identical between Gitea `dev` and the PR + base `e83c8e2` → hunks apply cleanly. Dev drift since the merge base is + only private agent-context paths (`.kilo/`, `.private/`, `.vscode/`, + `openwiki/`, `AGENTS.md`, `.openwikiignore`, `.github/workflows/`) — zero + overlap with the PR's paths. +4. `CronTrigger.from_crontab` has exactly two call sites in the codebase + (`scheduler.py:116` in `_add_job_for`, `ui/tasks.py:37` in `_valid_cron`); + the PR converts both to the shared `cron_trigger` → the issue's "one parser + for UI validation and scheduler construction" acceptance is fully covered. +5. All five #46 acceptance points are covered: single parser; 0/7 = Sunday and + 1 = Monday; lists, ascending ranges and steps keep cron numbering; named + weekdays retain the pinned parser's behavior; an invalid stored expression + skips only that task while the others start. +6. The documented ambiguous rejections (`1/2`, `mon-5`, `01`, `*/02`) are + implemented by the weekday regex and covered by the UI-level test + (`test_ui_and_scheduler_share_numeric_weekday_validation`). +7. `ZoneInfo("Europe/Zurich")` resolves in `apps/.venv` (pinned tzdata), so the + new Europe/Zurich test runs on the standard gate. +8. The PR's full-suite count (708 passed) was measured on the candidate head, + not on dev — S5 re-runs everything on the exact dev tree. + +## Code review findings + +`apps/billcollector/scheduler.py` — new `cron_trigger(expression, timezone)`: + +- Splits the five-field expression; the weekday field is processed per item: + items without digits (named) pass through unchanged; numeric items must + fullmatch `(\*|[0-7](?:-[0-7])?)(?:/([1-9][0-9]*))?` and are converted to + named days via `names[day]` (0/7 → "sun"), so APScheduler's Monday=0 never + sees a numeric weekday. +- Guards: non-5-field → ValueError; `*` → 0..6; ranges must ascend; a step on + a single day is refused (`1/2`); step `0`/`02` and `8` are refused by the + regex; unicode-digit input fails the regex → refused (safe). +- `dict.fromkeys` dedupes (`0,7` → `sun`); the rebuilt field is joined back + before `CronTrigger.from_crontab`; all other fields keep the pinned parser. +- `_add_job_for` wraps trigger construction in try/except ValueError → + `logger.error("Task %s not scheduled: invalid cron (%s)", task["id"], exc)` + and returns; the other tasks still schedule. The log carries the task id and + a fixed/field-level exception message — no cron expression, no secret. +- Behavior change: previously an invalid stored cron raised inside `resync()` + and propagated out of `start()` (daemon start failure); now that task is + skipped with an error log — exactly #46 acceptance point 5. + +`apps/billcollector/ui/tasks.py` — `_valid_cron` now calls the same +`cron_trigger`; the editor gate and the scheduler share one parser. + +`tests/test_bc_scheduler.py` — new `CronWeekdayTests` (3 methods, 15 subtests: +Sunday equality of `0`/`7`/`sun`/`0,7` from a Friday start in +Europe/Zurich; ranges/lists/steps numbering from a Sunday start; five +rejections) + one module-level pytest function for UI/scheduler parity (valid +set + the four documented ambiguous rejections). Every subtest expectation was +traced against the implementation — consistent. + +`CHANGELOG.md` — see M1 in the Integration plan. + +Nit (no action): the new tests are appended after the +`if __name__ == "__main__": unittest.main()` guard — they do not run under +direct script invocation, but they do under pytest (the repo gate). + +## Risks + +- Stored numeric weekday schedules shift their fire day after the upgrade + (they ran one day off under APScheduler's Monday=0). The CHANGELOG entry + tells users to review numeric schedules before upgrade and to use named + weekdays when rolling back to an older binary. Accepted: the fix restores + the documented crontab semantics the UI already advertises. +- Named pass-through depends on the pinned APScheduler (3.11.3) named-day + parsing; a future upgrade that changed it would be caught by the parity + test. +- Issue #46 opened 2 minutes before the PR (no maintainer pre-agreement + comment). The issue body itself specifies the full expected behavior, the + fix matches the documented crontab semantics, and no shared interface (DB + schema, control protocol, recipe contract, exit codes) changes. Accepted + with a note. + +## Integration plan (S3/S4, Gitea dev) + +| File | Action | +|---|---| +| `apps/billcollector/scheduler.py` | apply as-is from `refs/pr/53` (+42/−1) | +| `apps/billcollector/ui/tasks.py` | apply as-is from `refs/pr/53` (+2/−3) | +| `tests/test_bc_scheduler.py` | apply as-is from `refs/pr/53` (+34) | +| `CHANGELOG.md` | manual (M1): move the PR entry text into the existing `### Fixed` subsection of `## [Unreleased]` (prepended); drop the PR's new `### Fixed` header inserted above `### Added` (Keep a Changelog order + duplicate header) | + +No other maintainer-side changes. Expected S4 tree: 4 dirty files. + +M1 entry text (from the PR, kept verbatim): + +> - Parse numeric cron weekdays with crontab numbering (0/7 Sunday, 1 Monday) +> in both the task editor and scheduler (#46). Invalid stored schedules are +> skipped with their task id. Ambiguous legacy forms `1/2`, `mon-5`, `01` and +> `*/02` are rejected; use named weekdays before rolling back to an older +> binary. + +## Validation plan (S5, exact dev tree, repo root, venv) + +- PR tests: `apps/.venv/bin/python -m pytest tests/test_bc_scheduler.py + tests/test_bc_ui_task_detail.py -q` → expected 24 passed, 15 subtests (the + PR's claim). +- Full unit: `apps/.venv/bin/python -m pytest tests/ -q --tb=short` → all + green (dev baseline 647 + the new tests; S5 records the actual count). +- Full regression: `apps/.venv/bin/python tests/run_regression.py --ini + tests/bc_regression.ini` → exit 0 (7 scenarios / 8 PDFs, VERDICT: PASS). +- ruff (F821, E4/E7/E9) + bandit on the changed files → nothing new beyond the + baseline (four known medium-or-higher signatures on the apps tree; B608/B104 + baseline). + +## Deployment validation + +Not deployment-relevant: the PR touches only `scheduler.py`, `ui/tasks.py`, +tests, and CHANGELOG — no `Dockerfile`, `.dockerignore`, `BillCollector.sh`, +`install_docker-image.sh`, `deploy_remote.sh`, no INI keys or `.env` handling. +Block skipped; recorded here + user-confirmed at the S5 gate. + +## Out of scope + +- Persistent job-store startup ordering (issue #47, PR #54 — next in the + chain; #53 deliberately leaves numeric parsing out of that change). +- The legacy cron-driven batch path (OS cron, deprecated). +- Numeric parsing of the non-weekday fields (minute/hour/day-of-month/month + keep the pinned parser; only the weekday field was ambiguous). + +## Open questions + +- S3 gate: confirm M1 (CHANGELOG placement) as-is or with changes. diff --git a/.kilo/plans/README.md b/.kilo/plans/README.md index 0534353..5caaa3f 100644 --- a/.kilo/plans/README.md +++ b/.kilo/plans/README.md @@ -18,4 +18,5 @@ paths, so a moved file gains the `completed/` prefix. | `1792793307574-recipe-repair-llm-options.md` | Public service recipe recovery — LLM-assisted repair options analysis (pre-step); Stage 0 triage done 2026-10-24, all 7 services classified | | `1792819039884-recipe-doctor-implementation.md` | Stage 1: offline LLM recipe doctor (`billcollector doctor`) implementation plan + pilot (buhl, datev) — both pilots green 2026-10-24 (runs 16/18, downloads succeeded); Stage 1b in progress (2026-10-25): 4/7 green (buhl, datev, nuernberger, winsim), remaining 1und1 (in progress), freenet_mobilfunk (not started), kabeldeutschland (Option D re-authoring) | | `1792861675256-issue27-maintainer-commitments.md` | Issue #27 product-audit triage (closed 2026-10-24): maintainer-side commitments C1-C8 (F04, F09, F08 contract, F06 contract, F02, F03, F05, F10) with acceptance criteria, dependencies and proposed order | - | `1792862814701-next-up-sequencing.md` | Next-up sequencing decision (2026-10-24), superseded in part (2026-10-25): C1–C3 done and pushed; current primary next-up is the PR continuation order #30 → #31 → #33 → #32 → #34 (one pr-review chain at a time), interleaved with Stage 1b portal sessions (next: 1und1); PR #30 + #31 DELIVERED 2026-10-25, PR #33 DELIVERED 2026-10-26, PR #32 DELIVERED 2026-10-26 (Gitea `main` `bde713c` / GitHub `93bfcc8`), PR #34 DELIVERED 2026-10-27 (Gitea `main` `53a5dff` / GitHub `6bb3348`), PR #35 DELIVERED 2026-10-28 (Gitea `main` `75b885b` / GitHub `484048d`), PR #42 DELIVERED 2026-10-28 (Gitea `main` `2a6e462` / GitHub `c2fb7f8`), PR #39 DELIVERED 2026-10-28 (Gitea `main` `b693517` / GitHub `ad18f7a`), PR #40 DELIVERED 2026-10-29 (Gitea `main` `61defd3` / GitHub `f26b2a9`), PR #41 DELIVERED 2026-10-30 (Gitea `main` `74ba171` / GitHub `e83c8e2`), issue #38 closed — the planned chain #34 → #35 → #42 → #39 → #40 → #41 is complete; flowcool's archive-side contract proposal on #38 (2026-10-30) awaits a maintainer decision | + | `1793397084822-pr53-cron-weekdays-review.md` | PR #53 crontab weekday numbering (issue #46) review — S1+S2 done 2026-10-30, AWAITING-DECISION (S3 gate: M1 CHANGELOG placement) | +| `1792862814701-next-up-sequencing.md` | Next-up sequencing decision (2026-10-24), superseded in part (2026-10-25): C1–C3 done and pushed; current primary next-up is the PR continuation order #30 → #31 → #33 → #32 → #34 (one pr-review chain at a time), interleaved with Stage 1b portal sessions (next: 1und1); PR #30 + #31 DELIVERED 2026-10-25, PR #33 DELIVERED 2026-10-26, PR #32 DELIVERED 2026-10-26 (Gitea `main` `bde713c` / GitHub `93bfcc8`), PR #34 DELIVERED 2026-10-27 (Gitea `main` `53a5dff` / GitHub `6bb3348`), PR #35 DELIVERED 2026-10-28 (Gitea `main` `75b885b` / GitHub `484048d`), PR #42 DELIVERED 2026-10-28 (Gitea `main` `2a6e462` / GitHub `c2fb7f8`), PR #39 DELIVERED 2026-10-28 (Gitea `main` `b693517` / GitHub `ad18f7a`), PR #40 DELIVERED 2026-10-29 (Gitea `main` `61defd3` / GitHub `f26b2a9`), PR #41 DELIVERED 2026-10-30 (Gitea `main` `74ba171` / GitHub `e83c8e2`), issue #38 closed — the planned chain #34 → #35 → #42 → #39 → #40 → #41 is complete; flowcool's archive-side contract proposal on #38 (2026-10-30) awaits a maintainer decision | From f2bb4ee5208f7dede432478c682adf8830f7a9d9 Mon Sep 17 00:00:00 2001 From: Stefan S Date: Fri, 30 Oct 2026 23:01:57 +0100 Subject: [PATCH 05/55] =?UTF-8?q?doc:=20plans=20=E2=80=94=20PR=20#53:=20S3?= =?UTF-8?q?=20gate=20passed=20(M1=20as-is)=20+=20S4=20integration=20applie?= =?UTF-8?q?d=20(preflight=20green=20on=20dev=208c9a17d,=203=20files=20byte?= =?UTF-8?q?-identical=20to=20head=206b41040b=20+=20CHANGELOG=20M1=20placem?= =?UTF-8?q?ent,=20tree=20dirty=20for=20S5),=20status=20INTEGRATING?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...1793397084822-pr53-cron-weekdays-review.md | 24 +++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/.kilo/plans/1793397084822-pr53-cron-weekdays-review.md b/.kilo/plans/1793397084822-pr53-cron-weekdays-review.md index 108b949..1632ad4 100644 --- a/.kilo/plans/1793397084822-pr53-cron-weekdays-review.md +++ b/.kilo/plans/1793397084822-pr53-cron-weekdays-review.md @@ -1,8 +1,8 @@ # PR #53 review: crontab weekday numbering (issue #46) -> Status: AWAITING-DECISION (2026-10-30) — S1+S2 done: verdict approve with one -> maintainer change (M1, CHANGELOG placement); 4 files, 3 as-is + 1 manual; -> next S3 gate. +> Status: INTEGRATING (2026-10-30) — S3 gate passed (M1 confirmed as-is) + +> S4 integration applied (3 files as-is from head `6b41040b` + M1 CHANGELOG +> placement); tree dirty for S5; next S5 validation. ## PR metadata @@ -157,4 +157,20 @@ Block skipped; recorded here + user-confirmed at the S5 gate. ## Open questions -- S3 gate: confirm M1 (CHANGELOG placement) as-is or with changes. +- None. S3 gate passed 2026-10-30 (M1 confirmed as-is). + +## Execution record + +- S1+S2 (2026-10-30, "continue last sessions" session): plan doc created, + verdict approve with M1; commit `8c9a17d` (plan doc + sequencing + README), + pushed to `origin/dev`. +- S3 (2026-10-30, same session): user confirmed the proposal as-is — M1 + CHANGELOG placement accepted, no changes to the as-is file set. +- S4 (2026-10-30, same session): preflight green on dev `8c9a17d` (clean + tree; head `6b41040b` unchanged via `git ls-remote`; pre-image re-check: + all 4 files still dev-identical to the PR base). Applied: + `scheduler.py` / `ui/tasks.py` / `test_bc_scheduler.py` as-is from + `refs/pr/53` (verified byte-identical to head), CHANGELOG M1 — entry + text verbatim into the existing `### Fixed` subsection (prepended), the + PR's duplicate `### Fixed`-above-`### Added` insertion dropped. Tree left + dirty (4 files) for S5; no commit of the integrated content. From 4fddfbc37e5c779228737da15272a9b138b3bfae Mon Sep 17 00:00:00 2001 From: Stefan S Date: Fri, 30 Oct 2026 23:26:58 +0100 Subject: [PATCH 06/55] =?UTF-8?q?doc:=20plans=20=E2=80=94=20PR=20#53:=20S5?= =?UTF-8?q?=20validation=20all=20green=20(PR=20tests=2024p/15=20subtests,?= =?UTF-8?q?=20unit=20651p,=20regression=20VERDICT:=20PASS,=20ruff/bandit?= =?UTF-8?q?=20clean);=20deployment=20recorded=20skip=20(not=20deployment-r?= =?UTF-8?q?elevant);=20status=20VALIDATED,=20S5=20gate=20passed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...1793397084822-pr53-cron-weekdays-review.md | 29 ++++++++++++++++--- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/.kilo/plans/1793397084822-pr53-cron-weekdays-review.md b/.kilo/plans/1793397084822-pr53-cron-weekdays-review.md index 1632ad4..fbe51f5 100644 --- a/.kilo/plans/1793397084822-pr53-cron-weekdays-review.md +++ b/.kilo/plans/1793397084822-pr53-cron-weekdays-review.md @@ -1,8 +1,9 @@ # PR #53 review: crontab weekday numbering (issue #46) -> Status: INTEGRATING (2026-10-30) — S3 gate passed (M1 confirmed as-is) + -> S4 integration applied (3 files as-is from head `6b41040b` + M1 CHANGELOG -> placement); tree dirty for S5; next S5 validation. +> Status: VALIDATED (2026-10-30) — S5 validation all green on the exact dev +> tree (PR tests 24 passed / 15 subtests, unit 651 passed, regression +> VERDICT: PASS, ruff/bandit clean); deployment block recorded skip (not +> deployment-relevant); awaiting the S5 user gate before S6. ## PR metadata @@ -157,7 +158,8 @@ Block skipped; recorded here + user-confirmed at the S5 gate. ## Open questions -- None. S3 gate passed 2026-10-30 (M1 confirmed as-is). +- None. S3 gate passed 2026-10-30 (M1 confirmed as-is); S5 gate passed + 2026-10-30 (validation results + deployment skip confirmed). ## Execution record @@ -174,3 +176,22 @@ Block skipped; recorded here + user-confirmed at the S5 gate. text verbatim into the existing `### Fixed` subsection (prepended), the PR's duplicate `### Fixed`-above-`### Added` insertion dropped. Tree left dirty (4 files) for S5; no commit of the integrated content. +- S5 (2026-10-30, "continue PR #53" session): preflight re-run green — head + still `6b41040b` via `git ls-remote`; dev tip `f2bb4ee` (= `origin/dev`); + dirty tree unchanged (3 code files verified byte-identical to + `refs/pr/53`, CHANGELOG shows the M1 placement). On the exact dev tree + (repo root, `apps/.venv`): + - PR tests: `pytest tests/test_bc_scheduler.py + tests/test_bc_ui_task_detail.py -q` → 24 passed, 15 subtests passed + (matches the PR's claim). + - Full unit: `pytest tests/ -q --tb=short` → 651 passed, 10 skipped, 68 + subtests passed (dev baseline 647 + the 4 new tests). + - Full regression: `tests/run_regression.py --ini + tests/bc_regression.ini` → exit 0, VERDICT: PASS (7 scenarios / 8 PDFs, + all expectations matched). + - ruff (F821, E4/E7/E9) on the 3 changed code files → all checks passed. + - bandit on the 2 changed app files → no findings (nothing new beyond the + known baseline). + - Deployment validation: not deployment-relevant (see Deployment + validation section) — recorded skip, user-confirmed at the S5 gate. + - No scope adjustments needed; tree left dirty (4 files) for S6/S7. From e730206e211efebd124136fa43767af5c4d3b44f Mon Sep 17 00:00:00 2001 From: Stefan S Date: Fri, 30 Oct 2026 23:35:51 +0100 Subject: [PATCH 07/55] =?UTF-8?q?doc:=20plans=20=E2=80=94=20PR=20#53:=20S6?= =?UTF-8?q?=20docs=20verified=20(CHANGELOG=20M1=20in=20place,=20no=20READM?= =?UTF-8?q?E/CONTRIBUTING=20changes);=20status=20VALIDATED,=20S6=20gate=20?= =?UTF-8?q?pending?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../plans/1793397084822-pr53-cron-weekdays-review.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/.kilo/plans/1793397084822-pr53-cron-weekdays-review.md b/.kilo/plans/1793397084822-pr53-cron-weekdays-review.md index fbe51f5..f8fd014 100644 --- a/.kilo/plans/1793397084822-pr53-cron-weekdays-review.md +++ b/.kilo/plans/1793397084822-pr53-cron-weekdays-review.md @@ -3,7 +3,8 @@ > Status: VALIDATED (2026-10-30) — S5 validation all green on the exact dev > tree (PR tests 24 passed / 15 subtests, unit 651 passed, regression > VERDICT: PASS, ruff/bandit clean); deployment block recorded skip (not -> deployment-relevant); awaiting the S5 user gate before S6. +> deployment-relevant); S6 docs verified (CHANGELOG entry in place, no +> README/CONTRIBUTING changes); awaiting the S6 user gate before S7. ## PR metadata @@ -195,3 +196,12 @@ Block skipped; recorded here + user-confirmed at the S5 gate. - Deployment validation: not deployment-relevant (see Deployment validation section) — recorded skip, user-confirmed at the S5 gate. - No scope adjustments needed; tree left dirty (4 files) for S6/S7. +- S6 (2026-10-30, "continue PR #53" session): doc review on the dirty tree — + CHANGELOG `## [Unreleased]` entry confirmed in place (M1, verbatim from + the PR, prepended to the existing `### Fixed` subsection; Keep a Changelog + order; sanitized, public-safe); no README change (README documents cron + only as "a 5-field crontab expression" with a day-of-month example — the + fix restores the semantics it already advertises; no doc anywhere states + weekday numbering); no CONTRIBUTING change (no contributor-rule change); + validation results + integration details already recorded (S4/S5 entries). + Tree left dirty (4 files) for S7. Awaiting the S6 user gate. From 9ccb1ffb535e20d88f9d3eb6d3444257d9b761d2 Mon Sep 17 00:00:00 2001 From: Stefan S Date: Fri, 30 Oct 2026 23:38:22 +0100 Subject: [PATCH 08/55] fix: parse numeric cron weekdays with crontab numbering in scheduler and UI (#46) --- apps/billcollector/scheduler.py | 42 ++++++++++++++++++++++++++++++++- apps/billcollector/ui/tasks.py | 5 ++-- 2 files changed, 43 insertions(+), 4 deletions(-) diff --git a/apps/billcollector/scheduler.py b/apps/billcollector/scheduler.py index f02a8b9..d264667 100644 --- a/apps/billcollector/scheduler.py +++ b/apps/billcollector/scheduler.py @@ -22,6 +22,7 @@ """ import logging +import re from apscheduler.jobstores.sqlalchemy import SQLAlchemyJobStore from apscheduler.schedulers.asyncio import AsyncIOScheduler @@ -33,6 +34,40 @@ logger = logging.getLogger("billcollector.scheduler") +def cron_trigger(expression: str, timezone=None) -> CronTrigger: + """Use cron's numeric weekdays (0/7 Sunday), not APScheduler's Monday=0. + + Convert numeric lists/ranges/steps before the pinned APScheduler parser. + Other fields and named weekday expressions retain that parser's semantics. + """ + fields = expression.split() + if len(fields) != 5: + raise ValueError("Cron requires five fields") + names = ("sun", "mon", "tue", "wed", "thu", "fri", "sat", "sun") + weekdays = [] + for item in fields[4].split(","): + if not any(c.isdigit() for c in item): + weekdays.append(item) + continue + match = re.fullmatch(r"(\*|[0-7](?:-[0-7])?)(?:/([1-9][0-9]*))?", item) + if not match: + raise ValueError("Invalid numeric cron weekday") + base, step = match.groups() + if base == "*": + start, end = 0, 6 + elif "-" in base: + start, end = map(int, base.split("-")) + else: + if step is not None: + raise ValueError("Cron weekday steps require a range or wildcard") + start = end = int(base) + if start > end: + raise ValueError("Cron weekday range must ascend") + weekdays.extend(names[day] for day in range(start, end + 1, int(step or 1))) + fields[4] = ",".join(dict.fromkeys(weekdays)) + return CronTrigger.from_crontab(" ".join(fields), timezone=timezone) + + def job_id_for(task_id: int) -> str: """Stable APScheduler job id for one task (one cron job per task).""" return f"task-{task_id}" @@ -111,9 +146,14 @@ def sync_task(self, task_id: int) -> None: def _add_job_for(self, task) -> None: if not task["enabled"] or not task["cron"]: return + try: + trigger = cron_trigger(task["cron"], timezone=self._timezone) + except ValueError as exc: + logger.error("Task %s not scheduled: invalid cron (%s)", task["id"], exc) + return self._scheduler.add_job( _task_job, - trigger=CronTrigger.from_crontab(task["cron"], timezone=self._timezone), + trigger=trigger, args=[task["id"]], id=job_id_for(task["id"]), replace_existing=True) diff --git a/apps/billcollector/ui/tasks.py b/apps/billcollector/ui/tasks.py index 48a8b99..3c2d6d4 100644 --- a/apps/billcollector/ui/tasks.py +++ b/apps/billcollector/ui/tasks.py @@ -19,12 +19,11 @@ import math -from apscheduler.triggers.cron import CronTrigger from nicegui import ui from .. import config from ..db import Database -from ..scheduler import TaskScheduler +from ..scheduler import TaskScheduler, cron_trigger # The cron a new task gets (locked decision 9). DEFAULT_CRON = "0 0 1 * *" @@ -34,7 +33,7 @@ def _valid_cron(cron: str) -> bool: """True when the string parses as a 5-field crontab expression (the same parser APScheduler uses for the job trigger).""" try: - CronTrigger.from_crontab(cron) + cron_trigger(cron) return True except ValueError: return False From 44c693af8568330c99e21ba4b4487daf1e2843e2 Mon Sep 17 00:00:00 2001 From: Stefan S Date: Fri, 30 Oct 2026 23:38:22 +0100 Subject: [PATCH 09/55] test: cover crontab weekday numbering and UI/scheduler parity (#46) --- tests/test_bc_scheduler.py | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/tests/test_bc_scheduler.py b/tests/test_bc_scheduler.py index e8f9cef..9c22d04 100644 --- a/tests/test_bc_scheduler.py +++ b/tests/test_bc_scheduler.py @@ -11,6 +11,8 @@ import sys import tempfile import unittest +from datetime import datetime +from zoneinfo import ZoneInfo from pathlib import Path sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "apps")) @@ -156,3 +158,35 @@ def test_task_job_dispatches_to_the_run_callback(self): if __name__ == "__main__": unittest.main() + + +class CronWeekdayTests(unittest.TestCase): + def test_numeric_sunday_and_named_sunday_have_the_same_next_fire(self): + zone = ZoneInfo("Europe/Zurich") + start = datetime(2026, 10, 30, tzinfo=zone) + expected = datetime(2026, 11, 1, 4, 17, tzinfo=zone) + for day in ("0", "7", "sun", "0,7"): + with self.subTest(day=day): + trigger = scheduler.cron_trigger(f"17 4 * * {day}", zone) + self.assertEqual(trigger.get_next_fire_time(None, start), expected) + + def test_numeric_ranges_lists_and_steps_follow_cron_numbering(self): + zone = ZoneInfo("UTC") + sunday = datetime(2026, 11, 1, tzinfo=zone) + for day, expected in (("1-5", 2), ("1,3,5", 2), ("0-6/2", 1), ("*/2", 1), ("6-7", 1), ("sun,1", 1)): + with self.subTest(day=day): + trigger = scheduler.cron_trigger(f"0 0 * * {day}", zone) + self.assertEqual(trigger.get_next_fire_time(None, sunday).day, expected) + + def test_invalid_numeric_weekdays_are_rejected(self): + for day in ("8", "7-1", "1/2", "*/0", "1-5/0"): + with self.subTest(day=day), self.assertRaises(ValueError): + scheduler.cron_trigger(f"0 0 * * {day}") + + +def test_ui_and_scheduler_share_numeric_weekday_validation(): + from billcollector.ui.tasks import _valid_cron + for day in ('0', '7', 'sun', '1-5', '*/2'): + assert _valid_cron(f'17 4 * * {day}') + for day in ('8', '1/2', 'mon-5', '01', '*/02'): + assert not _valid_cron(f'17 4 * * {day}') From fa1ade5f1e979f64ce9579d95b97956e9fc0f479 Mon Sep 17 00:00:00 2001 From: Stefan S Date: Fri, 30 Oct 2026 23:38:22 +0100 Subject: [PATCH 10/55] =?UTF-8?q?doc:=20changelog=20=E2=80=94=20crontab=20?= =?UTF-8?q?weekday=20numbering=20fix=20(#46)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CHANGELOG.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index d311ea6..8970d5e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -233,6 +233,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- Parse numeric cron weekdays with crontab numbering (0/7 Sunday, 1 Monday) + in both the task editor and scheduler (#46). Invalid stored schedules are + skipped with their task id. Ambiguous legacy forms `1/2`, `mon-5`, `01` and + `*/02` are rejected; use named weekdays before rolling back to an older + binary. + - Keep residual daemon exception messages and tracebacks out of per-run logs and SQLite notes. Retain fixed method/category diagnostics for scan, OTP, screenshot, browser-launch and unexpected runner failures. From 35b4e18c4b81118ed42f0e73bc36077abd5d9134 Mon Sep 17 00:00:00 2001 From: Stefan S Date: Fri, 30 Oct 2026 23:47:59 +0100 Subject: [PATCH 11/55] =?UTF-8?q?doc:=20plans=20=E2=80=94=20PR=20#53:=20S7?= =?UTF-8?q?=20complete=20(DELIVERED:=20Gitea=20main=20fa1ade5,=20GitHub=20?= =?UTF-8?q?main=20a24668a,=20PR=20#53=20+=20issue=20#46=20closed);=20plan?= =?UTF-8?q?=20doc=20archived=20to=20completed/=20+=20README=20+=20sequenci?= =?UTF-8?q?ng=20refs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../plans/1792862814701-next-up-sequencing.md | 8 +++ .kilo/plans/README.md | 3 +- ...1793397084822-pr53-cron-weekdays-review.md | 64 +++++++++++++++++-- 3 files changed, 67 insertions(+), 8 deletions(-) rename .kilo/plans/{ => completed}/1793397084822-pr53-cron-weekdays-review.md (76%) diff --git a/.kilo/plans/1792862814701-next-up-sequencing.md b/.kilo/plans/1792862814701-next-up-sequencing.md index e3092af..18b17de 100644 --- a/.kilo/plans/1792862814701-next-up-sequencing.md +++ b/.kilo/plans/1792862814701-next-up-sequencing.md @@ -376,6 +376,14 @@ Grounds: `e83c8e2`, single commit, 4 files — 3 as-is + 1 manual CHANGELOG; issue #46 before-claim verified on the pinned APScheduler 3.11.3; deployment validation recorded not deployment-relevant — Status: AWAITING-DECISION. +- PR #53 DELIVERED 2026-10-30 (S3-S7 in follow-up sessions): dev commits + `9ccb1ff`/`44c693a`/`fa1ade5` (fix/test/doc); Gitea `main` `fa1ade5` + (ff-only from `74ba171`); GitHub `main` `a24668a` (mirrored from + `e83c8e2`); PR #53 closed without merge, issue #46 closed completed; + Discussion summary text recorded for manual web-UI post. Plan doc archived + to `completed/`. Next in the batch-2 chain: **#54** (persistent job-store + rebuild, #47) — its S1 pre-image check runs against the settled #53 + `scheduler.py`/`test_bc_scheduler.py` tree. ## Explicitly not doing diff --git a/.kilo/plans/README.md b/.kilo/plans/README.md index 5caaa3f..09bfd41 100644 --- a/.kilo/plans/README.md +++ b/.kilo/plans/README.md @@ -18,5 +18,4 @@ paths, so a moved file gains the `completed/` prefix. | `1792793307574-recipe-repair-llm-options.md` | Public service recipe recovery — LLM-assisted repair options analysis (pre-step); Stage 0 triage done 2026-10-24, all 7 services classified | | `1792819039884-recipe-doctor-implementation.md` | Stage 1: offline LLM recipe doctor (`billcollector doctor`) implementation plan + pilot (buhl, datev) — both pilots green 2026-10-24 (runs 16/18, downloads succeeded); Stage 1b in progress (2026-10-25): 4/7 green (buhl, datev, nuernberger, winsim), remaining 1und1 (in progress), freenet_mobilfunk (not started), kabeldeutschland (Option D re-authoring) | | `1792861675256-issue27-maintainer-commitments.md` | Issue #27 product-audit triage (closed 2026-10-24): maintainer-side commitments C1-C8 (F04, F09, F08 contract, F06 contract, F02, F03, F05, F10) with acceptance criteria, dependencies and proposed order | - | `1793397084822-pr53-cron-weekdays-review.md` | PR #53 crontab weekday numbering (issue #46) review — S1+S2 done 2026-10-30, AWAITING-DECISION (S3 gate: M1 CHANGELOG placement) | -| `1792862814701-next-up-sequencing.md` | Next-up sequencing decision (2026-10-24), superseded in part (2026-10-25): C1–C3 done and pushed; current primary next-up is the PR continuation order #30 → #31 → #33 → #32 → #34 (one pr-review chain at a time), interleaved with Stage 1b portal sessions (next: 1und1); PR #30 + #31 DELIVERED 2026-10-25, PR #33 DELIVERED 2026-10-26, PR #32 DELIVERED 2026-10-26 (Gitea `main` `bde713c` / GitHub `93bfcc8`), PR #34 DELIVERED 2026-10-27 (Gitea `main` `53a5dff` / GitHub `6bb3348`), PR #35 DELIVERED 2026-10-28 (Gitea `main` `75b885b` / GitHub `484048d`), PR #42 DELIVERED 2026-10-28 (Gitea `main` `2a6e462` / GitHub `c2fb7f8`), PR #39 DELIVERED 2026-10-28 (Gitea `main` `b693517` / GitHub `ad18f7a`), PR #40 DELIVERED 2026-10-29 (Gitea `main` `61defd3` / GitHub `f26b2a9`), PR #41 DELIVERED 2026-10-30 (Gitea `main` `74ba171` / GitHub `e83c8e2`), issue #38 closed — the planned chain #34 → #35 → #42 → #39 → #40 → #41 is complete; flowcool's archive-side contract proposal on #38 (2026-10-30) awaits a maintainer decision | +| `1792862814701-next-up-sequencing.md` | Next-up sequencing decision (2026-10-24), superseded in part (2026-10-25): C1–C3 done and pushed; current primary next-up is the PR continuation order #30 → #31 → #33 → #32 → #34 (one pr-review chain at a time), interleaved with Stage 1b portal sessions (next: 1und1); PR #30 + #31 DELIVERED 2026-10-25, PR #33 DELIVERED 2026-10-26, PR #32 DELIVERED 2026-10-26 (Gitea `main` `bde713c` / GitHub `93bfcc8`), PR #34 DELIVERED 2026-10-27 (Gitea `main` `53a5dff` / GitHub `6bb3348`), PR #35 DELIVERED 2026-10-28 (Gitea `main` `75b885b` / GitHub `484048d`), PR #42 DELIVERED 2026-10-28 (Gitea `main` `2a6e462` / GitHub `c2fb7f8`), PR #39 DELIVERED 2026-10-28 (Gitea `main` `b693517` / GitHub `ad18f7a`), PR #40 DELIVERED 2026-10-29 (Gitea `main` `61defd3` / GitHub `f26b2a9`), PR #41 DELIVERED 2026-10-30 (Gitea `main` `74ba171` / GitHub `e83c8e2`), issue #38 closed — the planned chain #34 → #35 → #42 → #39 → #40 → #41 is complete; flowcool's archive-side contract proposal on #38 (2026-10-30) awaits a maintainer decision; PR #53 DELIVERED 2026-10-30 (Gitea `main` `fa1ade5` / GitHub `a24668a`), issue #46 closed — batch-2 chain next: #54 | diff --git a/.kilo/plans/1793397084822-pr53-cron-weekdays-review.md b/.kilo/plans/completed/1793397084822-pr53-cron-weekdays-review.md similarity index 76% rename from .kilo/plans/1793397084822-pr53-cron-weekdays-review.md rename to .kilo/plans/completed/1793397084822-pr53-cron-weekdays-review.md index f8fd014..120d768 100644 --- a/.kilo/plans/1793397084822-pr53-cron-weekdays-review.md +++ b/.kilo/plans/completed/1793397084822-pr53-cron-weekdays-review.md @@ -1,10 +1,11 @@ # PR #53 review: crontab weekday numbering (issue #46) -> Status: VALIDATED (2026-10-30) — S5 validation all green on the exact dev -> tree (PR tests 24 passed / 15 subtests, unit 651 passed, regression -> VERDICT: PASS, ruff/bandit clean); deployment block recorded skip (not -> deployment-relevant); S6 docs verified (CHANGELOG entry in place, no -> README/CONTRIBUTING changes); awaiting the S6 user gate before S7. +> Status: DELIVERED (2026-10-30) — S7 complete: dev commits `9ccb1ff` / +> `44c693a` / `fa1ade5` (fix/test/doc), Gitea `main` `fa1ade5` (ff-only from +> `74ba171`), GitHub `main` `a24668a` (mirrored from `e83c8e2`); PR #53 closed +> without merge (comment 6535059059), issue #46 closed completed (comment +> 6535067954); Discussion summary text recorded for manual web-UI post +> (write API unavailable). ## PR metadata @@ -160,7 +161,9 @@ Block skipped; recorded here + user-confirmed at the S5 gate. ## Open questions - None. S3 gate passed 2026-10-30 (M1 confirmed as-is); S5 gate passed - 2026-10-30 (validation results + deployment skip confirmed). + 2026-10-30 (validation results + deployment skip confirmed); S6 gate passed + 2026-10-30 (doc state confirmed); S7 public texts confirmed 2026-10-30 (PR + close comment, issue #46 comment, Discussion summary). ## Execution record @@ -205,3 +208,52 @@ Block skipped; recorded here + user-confirmed at the S5 gate. weekday numbering); no CONTRIBUTING change (no contributor-rule change); validation results + integration details already recorded (S4/S5 entries). Tree left dirty (4 files) for S7. Awaiting the S6 user gate. + - S6 gate passed (user confirmed the doc state). +- S7 (2026-10-30, "continue PR #53" session): prerequisite re-checked — the + S5-validated code content is byte-identical on the promoted tree (only the + private plan-doc commit `e730206` moved since S5). + - **Commits (dev-commits, per-item confirmed):** `9ccb1ff` `fix: parse + numeric cron weekdays with crontab numbering in scheduler and UI (#46)` + (2 files, +43/−4); `44c693a` `test: cover crontab weekday numbering and + UI/scheduler parity (#46)` (1 file, +34); `fa1ade5` `doc: changelog — + crontab weekday numbering fix (#46)` (1 file, +6). Pushed `origin/dev` + `e730206..fa1ade5`. + - **Promotion (user-gated at the S6 gate):** Gitea `main` `74ba171` → + `fa1ade5` ff-only; pushed. No tag (no release decided). Verified + `git diff main dev` empty, tree clean. + - **Mirror:** `bash .private/sync_github.sh` → GitHub `main` `e83c8e2` → + `a24668a` (sanitize-by-amend). Public gate green: no `.kilo`/`.vscode`/ + `.private`/`openwiki`/`AGENTS.md`/`.openwikiignore`; `.github` holds only + `ISSUE_TEMPLATE` (no `workflows/`); `cron_trigger` present in + `scheduler.py` on GitHub `main`. + - **PR #53 closed** (user confirmed the exact comment text): comment + 6535059059 (2026-10-30T22:43:12Z), closed without merge + 2026-10-30T22:43:23Z — content reached GitHub via the mirror. + - **Issue #46 closed** (user confirmed the exact comment text): comment + 6535067954 (2026-10-30T22:43:51Z), closed completed + 2026-10-30T22:43:51Z. + - **Discussion summary (S7.6):** GitHub Discussion write API unavailable + from this environment (2026-10-25 finding); user chose manual web-UI + posting — text recorded below. + > **Public update: crontab weekday numbering for cron schedules (issue + > #46, PR #53)** + > + > `main` now parses numeric cron weekdays with crontab numbering (0/7 + > Sunday, 1 Monday) in both the task editor and the scheduler, contributed + > by @flowcool in PR #53. One shared `cron_trigger` parser validates + > editor input and builds the scheduler trigger; an invalid stored + > schedule now skips only that task (logged with its task id) instead of + > failing daemon start. + > + > Maintainer-side adaptation: the CHANGELOG entry was folded into the + > existing `Unreleased`/`Fixed` subsection (entry text verbatim). + > + > Validation on the promoted tree: 24 PR tests (15 subtests), 651 passed + > full unit suite, regression harness PASS (7 scenarios / 8 PDFs), + > ruff/bandit clean. + > + > User-facing note: stored numeric weekday schedules now fire on the day + > they intended (under the old parser they ran one day off); when rolling + > back to an older binary, use named weekdays. + - **Archive:** this plan doc moved to `completed/`; README + sequencing + cross-references updated. From 30839f4e86544e4f9e8098894e62bdcd94be0725 Mon Sep 17 00:00:00 2001 From: Stefan S Date: Sat, 31 Oct 2026 00:18:23 +0100 Subject: [PATCH 12/55] =?UTF-8?q?doc:=20plans=20=E2=80=94=20PR=20#54:=20S1?= =?UTF-8?q?+S2=20review=20(verdict=20approve=20with=20M1=20CHANGELOG=20Fix?= =?UTF-8?q?ed-placement;=203-file=20delta=20vs=20branch=20point=20e83c8e2,?= =?UTF-8?q?=20101-file=20GitHub=20diff=20is=20orphaned-tip=20STRIP=20artif?= =?UTF-8?q?act);=20plan=20doc=20+=20README=20+=20sequencing;=20status=20AW?= =?UTF-8?q?AITING-DECISION?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../plans/1792862814701-next-up-sequencing.md | 12 ++ ...81422-pr54-persistent-job-resync-review.md | 197 ++++++++++++++++++ .kilo/plans/README.md | 3 +- 3 files changed, 211 insertions(+), 1 deletion(-) create mode 100644 .kilo/plans/1793401881422-pr54-persistent-job-resync-review.md diff --git a/.kilo/plans/1792862814701-next-up-sequencing.md b/.kilo/plans/1792862814701-next-up-sequencing.md index 18b17de..3ac9a07 100644 --- a/.kilo/plans/1792862814701-next-up-sequencing.md +++ b/.kilo/plans/1792862814701-next-up-sequencing.md @@ -384,6 +384,18 @@ Grounds: to `completed/`. Next in the batch-2 chain: **#54** (persistent job-store rebuild, #47) — its S1 pre-image check runs against the settled #53 `scheduler.py`/`test_bc_scheduler.py` tree. +- PR #54 S1+S2 done 2026-10-30 ("continue pr 54" session): review plan doc + `1793401881422-pr54-persistent-job-resync-review.md` — verdict approve with + maintainer changes (M1: same CHANGELOG `### Fixed`-placement fix as #53; the + PR's header lands above `### Added` and duplicates the existing `### Fixed` + subsection); head `7f6cb02` stable, branch point = orphaned public tip + `e83c8e2` (the 101-file GitHub "Files changed" view is a STRIP artifact, + real content 3 files +42/−4), both code hunks apply cleanly on the post-#53 + dev tree (verified `git apply --check`, no 3way); issue #47 before-claim + verified on the pinned APScheduler 3.11.3 (`Float(25)` next_run_time); no + scheduler overlap with open #44/#45/#55/#56; deployment validation recorded + not deployment-relevant — Status: AWAITING-DECISION, S2 gate passed, S3 + next. ## Explicitly not doing diff --git a/.kilo/plans/1793401881422-pr54-persistent-job-resync-review.md b/.kilo/plans/1793401881422-pr54-persistent-job-resync-review.md new file mode 100644 index 0000000..e5d13c0 --- /dev/null +++ b/.kilo/plans/1793401881422-pr54-persistent-job-resync-review.md @@ -0,0 +1,197 @@ +# PR #54 review: rebuild persistent jobs before scheduler resume (issue #47) + +> Status: AWAITING-DECISION (2026-10-30) — S1+S2 done: verdict approve with +> maintainer changes (M1, CHANGELOG placement); code/tests apply as-is. S2 gate +> passed 2026-10-30 (verdict + doc-only commit confirmed); S3 proposal next. + +## PR metadata + +- PR: s-t-e-f-a-n/BillCollector#54 "fix: rebuild persistent jobs before + scheduler resume" (flowcool), branch `fix/upstream-persistent-jobstore-20261030`, + head `7f6cb02f3c39869403d526372201f5940b6f8743`. Two commits in the fetched + ref (orphaned public tip `e83c8e2` + the fix commit `7f6cb02`); the real + content is **3 files +42/−4** against the branch point `e83c8e2`. + `mergeable_state: clean` (vs base `main` @ `a24668a`). +- Issue: #47 (flowcool, 2026-10-30T20:00:47Z) — `TaskScheduler.start()` runs + the startup resync before APScheduler loads its persistent SQLAlchemy job + store, so cached jobs of disabled/deleted tasks survive and can fire. +- Created 2026-10-30T20:04:42Z (issue 4 minutes before the PR). +- Local state at review time: Gitea `dev` tip `35b4e18` (= `origin/dev`; PR #53 + plan record only, no code since `main`), Gitea `main` `fa1ade5`, GitHub `main` + `a24668a`. +- AI disclosure present: "AI-assisted focused port, source review and synthetic + verification under my responsibility." + +## Verdict + +**Approve with maintainer changes** — one placement fix (M1, CHANGELOG); all +code and tests apply as-is from head. + +## Verified facts + +1. Head stable: fetched `refs/pr/54` = `7f6cb02f...`; PR `updated_at` + 2026-10-30T22:39:18Z predates the fetch. S4 preflight re-checks via + `git ls-remote github refs/pull/54/head`. +2. Branch point: the fork starts at the orphaned public tip `e83c8e2` (GitHub + `main` at the PR #41 delivery). `e83c8e2` is **not** an ancestor of the + current GitHub `main` `a24668a` (each mirror sync re-pushes the Gitea + lineage plus one amended sanitized tip, orphaning the previous public tip). + The GitHub "Files changed" view (101 files, +155/−23388, showing deletions of + `.kilo/`, `openwiki/`, `AGENTS.md`, …) is an orphaned-tip STRIP artifact, + not PR content — the same pattern as #45's 103-file diff. Integration + applies the 3-file delta against `e83c8e2` only. +3. Issue #47 before-claim verified against the pinned APScheduler 3.11.3: in + the old `start()`, `resync()` ran **before** `_scheduler.start()`, so + `get_jobs()` saw an empty in-memory table; the persistent + `apscheduler_jobs` rows are loaded by the jobstore **at** scheduler start, + i.e. after the rebuild → stale jobs of disabled/deleted tasks survived. + `next_run_time` is a `Float(25)` UTC timestamp in the pinned jobstore, so + the test's `UPDATE apscheduler_jobs SET next_run_time=0` is a valid + overdue state (not an ISO-string parse hazard). +4. Pre-image vs Gitea `dev` (drifted from `e83c8e2` by the PR #53 weekday + fix): `scheduler.py` diverges in regions PR #54 does not touch (`import re`, + module-level `cron_trigger()`, the `_add_job_for` try/except wrapper); + `tests/test_bc_scheduler.py` diverges in the imports and the trailing + `CronWeekdayTests` block. `git apply --check` of both PR hunks on a scratch + worktree of `dev` → **clean, no 3way needed**. +5. The new test uses the public `scheduler` property (present on + `TaskScheduler`) — `fresh.scheduler.get_jobs()` is valid; stub recording + callback, temporary SQLite DB, no real collection. The callback-routing + subtlety is correct: the fresh constructor re-points the module-global + `_run_callback`, so a stale fire would land in the test's `calls` list, not + the base `self.calls`. +6. `ui/app.py:142` calls `scheduler.start()` unguarded; the old code also + propagated `resync()` failures out of `start()` (daemon startup failure) → + failure-path semantics unchanged; the new code additionally shuts down the + owned scheduler on failure (issue #47: "Shutdown the owned scheduler if + resync fails"). +7. No shared-interface change: no DB schema migration (the + `apscheduler_jobs` table is APScheduler's own cache, structure unchanged), + no control protocol, no recipe contract, no exit codes. The issue states + "Task data remains authoritative; no schema change"; the PR implements + exactly that. +8. Cross-PR: open PRs #44, #45, #55, #56 (all flowcool, batch 2, same base). + None touches `scheduler.py` or `test_bc_scheduler.py` (heads checked against + the merge base). Only shared file is CHANGELOG (manual merge regardless). + Sequencing doc: #54 is next in the batch-2 chain, after #53. +9. The PR's test-count claim (10 passed) matches the head file: 6 + `JobSyncTests` + 4 `PersistenceTests`. + +## Code review findings + +`apps/billcollector/scheduler.py`: + +- `start()`: now `_scheduler.start(paused=True)` → `resync()` in a try/except + (on failure: `shutdown(wait=False)` + re-raise) → `resume()`. The persistent + jobstore loads at scheduler start, **before** the rebuild, while paused — + nothing can fire mid-rebuild — and the rebuild drops the loaded stale jobs. + Docstring ("Re-sync all jobs from the Task table and start the scheduler.") + remains accurate. +- `resync()`: `for job in list(get_jobs()): job.remove()` → + `remove_all_jobs()`. Removes every job from memory **and** the store + (deletes all `apscheduler_jobs` rows). This also fixes the + resync-when-not-started path: the old code removed only in-memory jobs, so + persisted rows survived a not-started `resync()` (the existing + `PersistenceTests` passed trivially there); the new one actually clears the + store. +- Unchanged: `stop()`, `sync_task()`, `_add_job_for()`, `_remove_job()`, + `_task_job` target, module docstring. + +`tests/test_bc_scheduler.py` — new +`PersistenceTests.test_start_drops_persisted_disabled_and_deleted_jobs_before_firing`: +creates two enabled cron tasks, one start/stop cycle (jobs persist), disables +one + deletes the other, forces the persisted rows overdue +(`next_run_time=0`), then runs a fresh `TaskScheduler` with a recording +callback through a real `start()` and asserts: in-memory job list empty, store +row count 0, and no callback invocation after a short sleep. Deterministic, +isolated (temp SQLite, stub callback), and mirrors the issue's reproduction 1:1. + +`CHANGELOG.md` — see M1. + +Nit (no action): the `asyncio.sleep(0.01)` after resume — the primary +assertions (jobs empty, store empty) do not depend on timing; `calls == []` +is a belt-and-braces firing check. + +## Risks + +- A `resync()` failure during `start()` now shuts down the scheduler it + started before propagating — strictly cleaner than before; daemon startup + still fails (unchanged semantics at `ui/app.py:142`). +- `remove_all_jobs()` drops **all** store rows, including any row not tied to + a Task (orphaned rows) — that is the module docstring's intent ("The + jobstore is a cache — the Task table always wins"). +- Issue #47 opened 4 minutes before the PR (no maintainer pre-agreement + comment). The issue body itself specifies the full expected behavior ("Load + the job store while paused, rebuild it from Task, then resume. Task remains + authoritative. Shutdown the owned scheduler if resync fails. No + task/schema migration or change to configured cron semantics."), and the PR + implements it exactly. Accepted with a note (same pattern as #46). + +## Integration plan (S3/S4, Gitea dev) + +| File | Action | +|---|---| +| `apps/billcollector/scheduler.py` | apply as-is from `refs/pr/54` (+8/−4) | +| `tests/test_bc_scheduler.py` | apply as-is from `refs/pr/54` (+29) | +| `CHANGELOG.md` | manual (M1): move the PR entry text into the existing `### Fixed` subsection of `## [Unreleased]` (prepended); drop the PR's new `### Fixed` header inserted above `### Added` (Keep a Changelog order + duplicate header) | + +Critical: apply the 3-file delta against `e83c8e2` +(`git diff e83c8e2..refs/pr/54`), **not** the 101-file merge-base diff (which +"deletes" maintainer-private paths on the dev side). + +No other maintainer-side changes. Expected S4 tree: 3 dirty files. + +M1 entry text (from the PR, kept verbatim): + +> - Load the persistent scheduler job store while paused, rebuild it from Task, +> then resume (#47). Disabled or deleted tasks cannot fire stale overdue +> jobs. + +## Validation plan (S5, exact dev tree, repo root, venv) + +- PR tests: `apps/.venv/bin/python -m pytest tests/test_bc_scheduler.py -q` + → expected 10 passed (the PR's claim). +- Before/after proof: the new test must **fail** against the pre-integration + dev `scheduler.py` (S5 records it on a scratch worktree — proves the fix, + not just the test). +- Full unit: `apps/.venv/bin/python -m pytest tests/ -q --tb=short` → all + green (dev baseline 651 passed / 10 skipped / 68 subtests per the PR #53 S5 + record + the 1 new test; S5 records the actual count). +- Full regression: `apps/.venv/bin/python tests/run_regression.py --ini + tests/bc_regression.ini` → exit 0 (7 scenarios / 8 PDFs, VERDICT: PASS). +- ruff (F821, E4/E7/E9) + bandit on the changed files → nothing new beyond the + known baseline (B608 dynamic SQL table names, B104 UI bind). + +## Deployment validation + +Not deployment-relevant: the PR touches only `scheduler.py`, tests, and +CHANGELOG — no `Dockerfile`, `.dockerignore`, `BillCollector.sh`, +`install_docker-image.sh`, `deploy_remote.sh`, no INI keys or `.env` +handling. Block skipped; recorded here + user-confirmed at the S5 gate. + +## Out of scope + +- Numeric weekday parsing (delivered in #46/PR #53; issue #47 keeps it + separate explicitly). +- The legacy cron-driven batch path (OS cron, deprecated). +- APScheduler upgrades / non-persistent jobstore setups (pinned 3.11.3, single + default jobstore). + +## Open questions + +- None. S2 gate passed 2026-10-30 (verdict approve with M1 confirmed, + doc-only commit + push approved). + +## Execution record + +- S1+S2 (2026-10-30, "continue pr 54" session): no prior plan doc existed + (PR #53 had just been DELIVERED) → fresh S1 review. Head fetched to + `refs/pr/54` (`7f6cb02f...`); merge-base analysis untangled the orphaned-tip + base (`e83c8e2`, not an ancestor of `a24668a`); real content = 3 files; + pre-image check green (hunks apply cleanly on `dev` scratch worktree); + issue #47 before-claim verified against pinned APScheduler 3.11.3 + (`Float(25)` next_run_time); cross-PR check: no scheduler overlap with + #44/#45/#55/#56. Verdict: approve with M1. Plan doc created + (Status: AWAITING-DECISION); S2 gate passed — doc-only commit (plan doc + + README + sequencing) pushed to `origin/dev`; the commit hash is recorded in + the S3 entry. diff --git a/.kilo/plans/README.md b/.kilo/plans/README.md index 09bfd41..33354d0 100644 --- a/.kilo/plans/README.md +++ b/.kilo/plans/README.md @@ -18,4 +18,5 @@ paths, so a moved file gains the `completed/` prefix. | `1792793307574-recipe-repair-llm-options.md` | Public service recipe recovery — LLM-assisted repair options analysis (pre-step); Stage 0 triage done 2026-10-24, all 7 services classified | | `1792819039884-recipe-doctor-implementation.md` | Stage 1: offline LLM recipe doctor (`billcollector doctor`) implementation plan + pilot (buhl, datev) — both pilots green 2026-10-24 (runs 16/18, downloads succeeded); Stage 1b in progress (2026-10-25): 4/7 green (buhl, datev, nuernberger, winsim), remaining 1und1 (in progress), freenet_mobilfunk (not started), kabeldeutschland (Option D re-authoring) | | `1792861675256-issue27-maintainer-commitments.md` | Issue #27 product-audit triage (closed 2026-10-24): maintainer-side commitments C1-C8 (F04, F09, F08 contract, F06 contract, F02, F03, F05, F10) with acceptance criteria, dependencies and proposed order | -| `1792862814701-next-up-sequencing.md` | Next-up sequencing decision (2026-10-24), superseded in part (2026-10-25): C1–C3 done and pushed; current primary next-up is the PR continuation order #30 → #31 → #33 → #32 → #34 (one pr-review chain at a time), interleaved with Stage 1b portal sessions (next: 1und1); PR #30 + #31 DELIVERED 2026-10-25, PR #33 DELIVERED 2026-10-26, PR #32 DELIVERED 2026-10-26 (Gitea `main` `bde713c` / GitHub `93bfcc8`), PR #34 DELIVERED 2026-10-27 (Gitea `main` `53a5dff` / GitHub `6bb3348`), PR #35 DELIVERED 2026-10-28 (Gitea `main` `75b885b` / GitHub `484048d`), PR #42 DELIVERED 2026-10-28 (Gitea `main` `2a6e462` / GitHub `c2fb7f8`), PR #39 DELIVERED 2026-10-28 (Gitea `main` `b693517` / GitHub `ad18f7a`), PR #40 DELIVERED 2026-10-29 (Gitea `main` `61defd3` / GitHub `f26b2a9`), PR #41 DELIVERED 2026-10-30 (Gitea `main` `74ba171` / GitHub `e83c8e2`), issue #38 closed — the planned chain #34 → #35 → #42 → #39 → #40 → #41 is complete; flowcool's archive-side contract proposal on #38 (2026-10-30) awaits a maintainer decision; PR #53 DELIVERED 2026-10-30 (Gitea `main` `fa1ade5` / GitHub `a24668a`), issue #46 closed — batch-2 chain next: #54 | +| `1792862814701-next-up-sequencing.md` | Next-up sequencing decision (2026-10-24), superseded in part (2026-10-25): C1–C3 done and pushed; current primary next-up is the PR continuation order #30 → #31 → #33 → #32 → #34 (one pr-review chain at a time), interleaved with Stage 1b portal sessions (next: 1und1); PR #30 + #31 DELIVERED 2026-10-25, PR #33 DELIVERED 2026-10-26, PR #32 DELIVERED 2026-10-26 (Gitea `main` `bde713c` / GitHub `93bfcc8`), PR #34 DELIVERED 2026-10-27 (Gitea `main` `53a5dff` / GitHub `6bb3348`), PR #35 DELIVERED 2026-10-28 (Gitea `main` `75b885b` / GitHub `484048d`), PR #42 DELIVERED 2026-10-28 (Gitea `main` `2a6e462` / GitHub `c2fb7f8`), PR #39 DELIVERED 2026-10-28 (Gitea `main` `b693517` / GitHub `ad18f7a`), PR #40 DELIVERED 2026-10-29 (Gitea `main` `61defd3` / GitHub `f26b2a9`), PR #41 DELIVERED 2026-10-30 (Gitea `main` `74ba171` / GitHub `e83c8e2`), issue #38 closed — the planned chain #34 → #35 → #42 → #39 → #40 → #41 is complete; flowcool's archive-side contract proposal on #38 (2026-10-30) awaits a maintainer decision; PR #53 DELIVERED 2026-10-30 (Gitea `main` `fa1ade5` / GitHub `a24668a`), issue #46 closed — batch-2 chain now at #54 (S1+S2 done 2026-10-30, verdict approve with M1) | +| `1793401881422-pr54-persistent-job-resync-review.md` | PR #54 review (batch-2 chain): rebuild persistent jobs before scheduler resume (issue #47) — S1+S2 done 2026-10-30: real content 3 files vs branch point `e83c8e2` (101-file GitHub diff = orphaned-tip STRIP artifact), verdict approve with M1 (CHANGELOG `### Fixed` placement), AWAITING-DECISION, S3 next | From fb82e95799a88b4d41d4889df50b6d84be0e2507 Mon Sep 17 00:00:00 2001 From: Stefan S Date: Sat, 31 Oct 2026 00:28:13 +0100 Subject: [PATCH 13/55] =?UTF-8?q?doc:=20plans=20=E2=80=94=20PR=20#54:=20S3?= =?UTF-8?q?=20gate=20passed=20(proposal=20as-is)=20+=20S4=20integration=20?= =?UTF-8?q?applied=20(code=20hunks=20clean=20on=20dev=2030839f4,=20M1=20CH?= =?UTF-8?q?ANGELOG=20Fixed-placement,=20tree=20dirty=20for=20S5),=20status?= =?UTF-8?q?=20INTEGRATING?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...81422-pr54-persistent-job-resync-review.md | 30 ++++++++++++++----- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/.kilo/plans/1793401881422-pr54-persistent-job-resync-review.md b/.kilo/plans/1793401881422-pr54-persistent-job-resync-review.md index e5d13c0..64a1d6e 100644 --- a/.kilo/plans/1793401881422-pr54-persistent-job-resync-review.md +++ b/.kilo/plans/1793401881422-pr54-persistent-job-resync-review.md @@ -1,8 +1,9 @@ # PR #54 review: rebuild persistent jobs before scheduler resume (issue #47) -> Status: AWAITING-DECISION (2026-10-30) — S1+S2 done: verdict approve with -> maintainer changes (M1, CHANGELOG placement); code/tests apply as-is. S2 gate -> passed 2026-10-30 (verdict + doc-only commit confirmed); S3 proposal next. +> Status: INTEGRATING (2026-10-30) — S1+S2 done: verdict approve with +> maintainer changes (M1, CHANGELOG placement); code/tests apply as-is. S3 gate +> passed + S4 integration applied (code hunks clean on dev, M1 CHANGELOG +> Fixed-placement in place, tree dirty for S5). ## PR metadata @@ -179,8 +180,8 @@ handling. Block skipped; recorded here + user-confirmed at the S5 gate. ## Open questions -- None. S2 gate passed 2026-10-30 (verdict approve with M1 confirmed, - doc-only commit + push approved). +- None. S3 gate passed 2026-10-30 (proposal confirmed as-is); S4 integration + applied, tree left dirty for S5 validation. ## Execution record @@ -192,6 +193,19 @@ handling. Block skipped; recorded here + user-confirmed at the S5 gate. issue #47 before-claim verified against pinned APScheduler 3.11.3 (`Float(25)` next_run_time); cross-PR check: no scheduler overlap with #44/#45/#55/#56. Verdict: approve with M1. Plan doc created - (Status: AWAITING-DECISION); S2 gate passed — doc-only commit (plan doc + - README + sequencing) pushed to `origin/dev`; the commit hash is recorded in - the S3 entry. + (Status: AWAITING-DECISION); S2 gate passed — doc-only commit (plan doc + + README + sequencing) pushed to `origin/dev`; the commit hash is recorded in + the S3 entry. + +- S3+S4 (2026-10-30, "continue pr 54" session): S3 proposal presented (2 code + files as-is + M1 CHANGELOG Fixed-placement + 3 commits) and confirmed as-is. + S4 preflight: clean tree on `dev`; PR head unchanged (`7f6cb02f`, matches + local `refs/pr/54`); dev had moved only by the doc-only S2 commit + (`30839f4`, no code to the 3 files) so the pre-image checks still held. + Applied the 3-file delta as **hunks** (not full-file checkout — the PR head + files are based on the orphaned tip `e83c8e2` and lack the #53 weekday + content): `git apply` of `git diff e83c8e2..refs/pr/54 -- scheduler.py + test_bc_scheduler.py` → clean (scheduler +35/+2 offset, expected). M1 + applied by hand: PR entry prepended as the first item of the existing + `### Fixed` subsection (line 234), no stray `### Fixed` above `### Added`. + Tree left dirty (3 files) for S5. Status: INTEGRATING. From f5be955b77b250cafc12e534b34d766d1da9776b Mon Sep 17 00:00:00 2001 From: Stefan S Date: Sat, 31 Oct 2026 07:20:48 +0100 Subject: [PATCH 14/55] =?UTF-8?q?doc:=20plans=20=E2=80=94=20PR=20#54:=20S5?= =?UTF-8?q?=20validated=20(PR=20module=2014=20passed,=20before/after=20pro?= =?UTF-8?q?of=20on=20pre-integration=20scheduler,=20unit=20652/10/68,=20re?= =?UTF-8?q?gression=20PASS,=20ruff+bandit=20clean=20vs=20baseline,=20deplo?= =?UTF-8?q?yment=20recorded-skip),=20S5=20gate=20passed,=20status=20VALIDA?= =?UTF-8?q?TED?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../plans/1792862814701-next-up-sequencing.md | 10 +++++ ...81422-pr54-persistent-job-resync-review.md | 45 ++++++++++++++++--- .kilo/plans/README.md | 4 +- 3 files changed, 50 insertions(+), 9 deletions(-) diff --git a/.kilo/plans/1792862814701-next-up-sequencing.md b/.kilo/plans/1792862814701-next-up-sequencing.md index 3ac9a07..06eb774 100644 --- a/.kilo/plans/1792862814701-next-up-sequencing.md +++ b/.kilo/plans/1792862814701-next-up-sequencing.md @@ -396,6 +396,16 @@ Grounds: scheduler overlap with open #44/#45/#55/#56; deployment validation recorded not deployment-relevant — Status: AWAITING-DECISION, S2 gate passed, S3 next. +- PR #54 S3-S5 done 2026-10-31 ("continue pr 54" session): S3 proposal + confirmed as-is (2 code files as-is + M1 CHANGELOG Fixed-placement); S4 + integration applied as hunks on dev (`git apply` of + `git diff e83c8e2..refs/pr/54` for scheduler.py + test — clean; M1 by + hand), tree left dirty for S5; S5 validation green on the exact dev tree — + PR module 14 passed / 15 subtests, before/after proof (new test fails on + the pre-integration `scheduler.py`: 2 stale jobs survive `start()`), full + unit 652 / 10 skipped / 68 subtests, regression VERDICT: PASS (7 scenarios + / 8 PDFs), ruff + bandit clean vs baseline; deployment skip recorded (not + deployment-relevant) — Status: VALIDATED, S5 gate passed, S6 next. ## Explicitly not doing diff --git a/.kilo/plans/1793401881422-pr54-persistent-job-resync-review.md b/.kilo/plans/1793401881422-pr54-persistent-job-resync-review.md index 64a1d6e..510861a 100644 --- a/.kilo/plans/1793401881422-pr54-persistent-job-resync-review.md +++ b/.kilo/plans/1793401881422-pr54-persistent-job-resync-review.md @@ -1,9 +1,11 @@ # PR #54 review: rebuild persistent jobs before scheduler resume (issue #47) -> Status: INTEGRATING (2026-10-30) — S1+S2 done: verdict approve with -> maintainer changes (M1, CHANGELOG placement); code/tests apply as-is. S3 gate -> passed + S4 integration applied (code hunks clean on dev, M1 CHANGELOG -> Fixed-placement in place, tree dirty for S5). +> Status: VALIDATED (2026-10-31) — S5 validation green on the exact dev tree: +> PR module 14 passed / 15 subtests, before/after proof (new test fails on +> pre-integration scheduler), full unit 652/10 skipped/68 subtests, regression +> VERDICT: PASS (7 scenarios / 8 PDFs), ruff + bandit clean vs baseline, +> deployment block recorded-skip (not deployment-relevant). S5 gate passed +> 2026-10-31; ready for S6. ## PR metadata @@ -181,7 +183,8 @@ handling. Block skipped; recorded here + user-confirmed at the S5 gate. ## Open questions - None. S3 gate passed 2026-10-30 (proposal confirmed as-is); S4 integration - applied, tree left dirty for S5 validation. + applied; S5 validation green + S5 gate passed 2026-10-31 (validation results + confirmed incl. the recorded deployment skip); next: S6 documentation. ## Execution record @@ -207,5 +210,33 @@ handling. Block skipped; recorded here + user-confirmed at the S5 gate. content): `git apply` of `git diff e83c8e2..refs/pr/54 -- scheduler.py test_bc_scheduler.py` → clean (scheduler +35/+2 offset, expected). M1 applied by hand: PR entry prepended as the first item of the existing - `### Fixed` subsection (line 234), no stray `### Fixed` above `### Added`. - Tree left dirty (3 files) for S5. Status: INTEGRATING. + `### Fixed` subsection (line 234), no stray `### Fixed` above `### Added`. + Tree left dirty (3 files) for S5. Status: INTEGRATING. + +- S5 (2026-10-31, "continue pr 54" session): preflight re-run green — head + still `7f6cb02f` via `git ls-remote github refs/pull/54/head`; dev tip + `fb82e95` (= `origin/dev`, doc-only since S4); dirty tree unchanged (3 + files: scheduler.py +12/−4, test +29, CHANGELOG +3). On the exact dev tree + (repo root, `apps/.venv`): + - PR module: `pytest tests/test_bc_scheduler.py -q` → 14 passed, 15 subtests + passed (the PR's 10 tests + the 4 #53 weekday tests already on dev; the + PR's "10 passed" claim was against the head file). + - Before/after proof: scratch worktree at `fb82e95` (pre-integration code) + with only the PR's test hunk applied → the new test fails as expected + (`AssertionError` — 2 stale jobs `task-1`/`task-2` survive `start()`); + scratch worktree removed. Proves the fix, not just the test. + - Full unit: `pytest tests/ -q --tb=short` → 652 passed, 10 skipped, 68 + subtests passed (dev baseline 651 + the 1 new test). + - Full regression: `tests/run_regression.py --ini tests/bc_regression.ini` + → exit 0, VERDICT: PASS (7 scenarios / 8 PDFs, all expectations matched). + - ruff (F821, E4/E7/E9) on both code files → all checks passed; bandit → + only the 2 pre-existing B101 asserts in the #53 weekday tests + (`test_bc_scheduler.py:219/221`, not in the PR-added lines); nothing new + beyond the known baseline. + - Deployment validation: not deployment-relevant (no Dockerfile / + .dockerignore / BillCollector.sh / install_docker-image.sh / + deploy_remote.sh / INI keys / .env handling) — recorded skip, + user-confirmed at the S5 gate. + - No scope adjustments needed; tree left dirty (3 files) for S6/S7. + S5 gate passed 2026-10-31 (user confirmed the validation results incl. + the recorded deployment skip). Status: VALIDATED. diff --git a/.kilo/plans/README.md b/.kilo/plans/README.md index 33354d0..ede2700 100644 --- a/.kilo/plans/README.md +++ b/.kilo/plans/README.md @@ -18,5 +18,5 @@ paths, so a moved file gains the `completed/` prefix. | `1792793307574-recipe-repair-llm-options.md` | Public service recipe recovery — LLM-assisted repair options analysis (pre-step); Stage 0 triage done 2026-10-24, all 7 services classified | | `1792819039884-recipe-doctor-implementation.md` | Stage 1: offline LLM recipe doctor (`billcollector doctor`) implementation plan + pilot (buhl, datev) — both pilots green 2026-10-24 (runs 16/18, downloads succeeded); Stage 1b in progress (2026-10-25): 4/7 green (buhl, datev, nuernberger, winsim), remaining 1und1 (in progress), freenet_mobilfunk (not started), kabeldeutschland (Option D re-authoring) | | `1792861675256-issue27-maintainer-commitments.md` | Issue #27 product-audit triage (closed 2026-10-24): maintainer-side commitments C1-C8 (F04, F09, F08 contract, F06 contract, F02, F03, F05, F10) with acceptance criteria, dependencies and proposed order | -| `1792862814701-next-up-sequencing.md` | Next-up sequencing decision (2026-10-24), superseded in part (2026-10-25): C1–C3 done and pushed; current primary next-up is the PR continuation order #30 → #31 → #33 → #32 → #34 (one pr-review chain at a time), interleaved with Stage 1b portal sessions (next: 1und1); PR #30 + #31 DELIVERED 2026-10-25, PR #33 DELIVERED 2026-10-26, PR #32 DELIVERED 2026-10-26 (Gitea `main` `bde713c` / GitHub `93bfcc8`), PR #34 DELIVERED 2026-10-27 (Gitea `main` `53a5dff` / GitHub `6bb3348`), PR #35 DELIVERED 2026-10-28 (Gitea `main` `75b885b` / GitHub `484048d`), PR #42 DELIVERED 2026-10-28 (Gitea `main` `2a6e462` / GitHub `c2fb7f8`), PR #39 DELIVERED 2026-10-28 (Gitea `main` `b693517` / GitHub `ad18f7a`), PR #40 DELIVERED 2026-10-29 (Gitea `main` `61defd3` / GitHub `f26b2a9`), PR #41 DELIVERED 2026-10-30 (Gitea `main` `74ba171` / GitHub `e83c8e2`), issue #38 closed — the planned chain #34 → #35 → #42 → #39 → #40 → #41 is complete; flowcool's archive-side contract proposal on #38 (2026-10-30) awaits a maintainer decision; PR #53 DELIVERED 2026-10-30 (Gitea `main` `fa1ade5` / GitHub `a24668a`), issue #46 closed — batch-2 chain now at #54 (S1+S2 done 2026-10-30, verdict approve with M1) | -| `1793401881422-pr54-persistent-job-resync-review.md` | PR #54 review (batch-2 chain): rebuild persistent jobs before scheduler resume (issue #47) — S1+S2 done 2026-10-30: real content 3 files vs branch point `e83c8e2` (101-file GitHub diff = orphaned-tip STRIP artifact), verdict approve with M1 (CHANGELOG `### Fixed` placement), AWAITING-DECISION, S3 next | +| `1792862814701-next-up-sequencing.md` | Next-up sequencing decision (2026-10-24), superseded in part (2026-10-25): C1–C3 done and pushed; current primary next-up is the PR continuation order #30 → #31 → #33 → #32 → #34 (one pr-review chain at a time), interleaved with Stage 1b portal sessions (next: 1und1); PR #30 + #31 DELIVERED 2026-10-25, PR #33 DELIVERED 2026-10-26, PR #32 DELIVERED 2026-10-26 (Gitea `main` `bde713c` / GitHub `93bfcc8`), PR #34 DELIVERED 2026-10-27 (Gitea `main` `53a5dff` / GitHub `6bb3348`), PR #35 DELIVERED 2026-10-28 (Gitea `main` `75b885b` / GitHub `484048d`), PR #42 DELIVERED 2026-10-28 (Gitea `main` `2a6e462` / GitHub `c2fb7f8`), PR #39 DELIVERED 2026-10-28 (Gitea `main` `b693517` / GitHub `ad18f7a`), PR #40 DELIVERED 2026-10-29 (Gitea `main` `61defd3` / GitHub `f26b2a9`), PR #41 DELIVERED 2026-10-30 (Gitea `main` `74ba171` / GitHub `e83c8e2`), issue #38 closed — the planned chain #34 → #35 → #42 → #39 → #40 → #41 is complete; flowcool's archive-side contract proposal on #38 (2026-10-30) awaits a maintainer decision; PR #53 DELIVERED 2026-10-30 (Gitea `main` `fa1ade5` / GitHub `a24668a`), issue #46 closed — batch-2 chain now at #54 (S1-S5 done, VALIDATED 2026-10-31, S6 next) | +| `1793401881422-pr54-persistent-job-resync-review.md` | PR #54 review (batch-2 chain): rebuild persistent jobs before scheduler resume (issue #47) — S1+S2 done 2026-10-30: real content 3 files vs branch point `e83c8e2` (101-file GitHub diff = orphaned-tip STRIP artifact), verdict approve with M1 (CHANGELOG `### Fixed` placement); S3-S5 done 2026-10-31: proposal confirmed as-is, integration applied (hunks clean on post-#53 dev), validation green (PR module 14 passed, before/after proof, unit 652/10/68, regression PASS, ruff+bandit clean vs baseline, deployment recorded-skip) — VALIDATED, S6 next | From e61f2df4493895d7e5732abae9b9ea563dc75a65 Mon Sep 17 00:00:00 2001 From: Stefan S Date: Sat, 31 Oct 2026 07:28:48 +0100 Subject: [PATCH 15/55] =?UTF-8?q?doc:=20plans=20=E2=80=94=20PR=20#54:=20S6?= =?UTF-8?q?=20docs=20confirmed=20(CHANGELOG=20M1=20entry=20stands=20under?= =?UTF-8?q?=20[Unreleased]=20Fixed,=20sanitized;=20no=20README/CONTRIBUTIN?= =?UTF-8?q?G=20changes),=20S6=20gate=20passed,=20status=20VALIDATED;=20rea?= =?UTF-8?q?dy=20for=20S7?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../plans/1792862814701-next-up-sequencing.md | 14 +++++-- ...81422-pr54-persistent-job-resync-review.md | 39 ++++++++++++++----- .kilo/plans/README.md | 4 +- 3 files changed, 42 insertions(+), 15 deletions(-) diff --git a/.kilo/plans/1792862814701-next-up-sequencing.md b/.kilo/plans/1792862814701-next-up-sequencing.md index 06eb774..cba39e0 100644 --- a/.kilo/plans/1792862814701-next-up-sequencing.md +++ b/.kilo/plans/1792862814701-next-up-sequencing.md @@ -403,9 +403,17 @@ Grounds: hand), tree left dirty for S5; S5 validation green on the exact dev tree — PR module 14 passed / 15 subtests, before/after proof (new test fails on the pre-integration `scheduler.py`: 2 stale jobs survive `start()`), full - unit 652 / 10 skipped / 68 subtests, regression VERDICT: PASS (7 scenarios - / 8 PDFs), ruff + bandit clean vs baseline; deployment skip recorded (not - deployment-relevant) — Status: VALIDATED, S5 gate passed, S6 next. + unit 652 / 10 skipped / 68 subtests, regression VERDICT: PASS (7 scenarios + / 8 PDFs), ruff + bandit clean vs baseline; deployment skip recorded (not + deployment-relevant) — Status: VALIDATED, S5 gate passed, S6 next. +- PR #54 S6 done 2026-10-31 ("continue pr 54" session): documentation + confirmed — the CHANGELOG M1 entry stands as the first item of + `[Unreleased]` → `### Fixed` (sanitized, references #47 only); no README + change (line 233 already documents the Task-table-is-authoritative rebuild + behavior the fix makes true); no CONTRIBUTING change (no rule change) — + Status: VALIDATED, S6 gate passed, S7 next (commit + push: Gitea dev → + main promotion → GitHub mirror → PR close + issue #47 closure + Discussion + summary). ## Explicitly not doing diff --git a/.kilo/plans/1793401881422-pr54-persistent-job-resync-review.md b/.kilo/plans/1793401881422-pr54-persistent-job-resync-review.md index 510861a..9709dd9 100644 --- a/.kilo/plans/1793401881422-pr54-persistent-job-resync-review.md +++ b/.kilo/plans/1793401881422-pr54-persistent-job-resync-review.md @@ -1,11 +1,14 @@ # PR #54 review: rebuild persistent jobs before scheduler resume (issue #47) -> Status: VALIDATED (2026-10-31) — S5 validation green on the exact dev tree: -> PR module 14 passed / 15 subtests, before/after proof (new test fails on -> pre-integration scheduler), full unit 652/10 skipped/68 subtests, regression -> VERDICT: PASS (7 scenarios / 8 PDFs), ruff + bandit clean vs baseline, -> deployment block recorded-skip (not deployment-relevant). S5 gate passed -> 2026-10-31; ready for S6. +> Status: VALIDATED (2026-10-31) — S6 docs confirmed: CHANGELOG M1 entry in +> place under [Unreleased] Fixed (sanitized, references #47 only), no README/ +> CONTRIBUTING changes (README:233 wording stays accurate, no rule changes). +> S5 validation green on the exact dev tree: PR module 14 passed / 15 subtests, +> before/after proof (new test fails on pre-integration scheduler), full unit +> 652/10 skipped/68 subtests, regression VERDICT: PASS (7 scenarios / 8 PDFs), +> ruff + bandit clean vs baseline, deployment block recorded-skip (not +> deployment-relevant). S5 + S6 gates passed 2026-10-31; ready for S7 +> (commit + push). ## PR metadata @@ -184,7 +187,10 @@ handling. Block skipped; recorded here + user-confirmed at the S5 gate. - None. S3 gate passed 2026-10-30 (proposal confirmed as-is); S4 integration applied; S5 validation green + S5 gate passed 2026-10-31 (validation results - confirmed incl. the recorded deployment skip); next: S6 documentation. + confirmed incl. the recorded deployment skip); S6 docs confirmed + S6 gate + passed 2026-10-31 (CHANGELOG entry as the only public artifact; no README/ + CONTRIBUTING changes); next: S7 commit + push (Gitea dev → main promotion → + GitHub mirror → PR close). ## Execution record @@ -237,6 +243,19 @@ handling. Block skipped; recorded here + user-confirmed at the S5 gate. .dockerignore / BillCollector.sh / install_docker-image.sh / deploy_remote.sh / INI keys / .env handling) — recorded skip, user-confirmed at the S5 gate. - - No scope adjustments needed; tree left dirty (3 files) for S6/S7. - S5 gate passed 2026-10-31 (user confirmed the validation results incl. - the recorded deployment skip). Status: VALIDATED. + - No scope adjustments needed; tree left dirty (3 files) for S6/S7. + S5 gate passed 2026-10-31 (user confirmed the validation results incl. + the recorded deployment skip). Status: VALIDATED. + +- S6 (2026-10-31, "continue pr 54" session): documentation checked and + confirmed. + - CHANGELOG: the M1 entry stands as the first item of `## [Unreleased]` → + `### Fixed` (line 236), sanitized and public-safe (references #47 only, + no private paths); wording unchanged from the S3-confirmed proposal. + - README: line 233 already documents "the Task table in the database is the + source of truth, and the daemon rebuilds its scheduler jobs from it on + every start" — the fix makes that statement true; no user-visible behavior + or setup change → no README edit. + - CONTRIBUTING: no contributor-rule change → no edit. + S6 gate passed 2026-10-31 (user confirmed the documentation). Tree still + dirty (3 integration files) for S7. Status: VALIDATED. diff --git a/.kilo/plans/README.md b/.kilo/plans/README.md index ede2700..186b395 100644 --- a/.kilo/plans/README.md +++ b/.kilo/plans/README.md @@ -18,5 +18,5 @@ paths, so a moved file gains the `completed/` prefix. | `1792793307574-recipe-repair-llm-options.md` | Public service recipe recovery — LLM-assisted repair options analysis (pre-step); Stage 0 triage done 2026-10-24, all 7 services classified | | `1792819039884-recipe-doctor-implementation.md` | Stage 1: offline LLM recipe doctor (`billcollector doctor`) implementation plan + pilot (buhl, datev) — both pilots green 2026-10-24 (runs 16/18, downloads succeeded); Stage 1b in progress (2026-10-25): 4/7 green (buhl, datev, nuernberger, winsim), remaining 1und1 (in progress), freenet_mobilfunk (not started), kabeldeutschland (Option D re-authoring) | | `1792861675256-issue27-maintainer-commitments.md` | Issue #27 product-audit triage (closed 2026-10-24): maintainer-side commitments C1-C8 (F04, F09, F08 contract, F06 contract, F02, F03, F05, F10) with acceptance criteria, dependencies and proposed order | -| `1792862814701-next-up-sequencing.md` | Next-up sequencing decision (2026-10-24), superseded in part (2026-10-25): C1–C3 done and pushed; current primary next-up is the PR continuation order #30 → #31 → #33 → #32 → #34 (one pr-review chain at a time), interleaved with Stage 1b portal sessions (next: 1und1); PR #30 + #31 DELIVERED 2026-10-25, PR #33 DELIVERED 2026-10-26, PR #32 DELIVERED 2026-10-26 (Gitea `main` `bde713c` / GitHub `93bfcc8`), PR #34 DELIVERED 2026-10-27 (Gitea `main` `53a5dff` / GitHub `6bb3348`), PR #35 DELIVERED 2026-10-28 (Gitea `main` `75b885b` / GitHub `484048d`), PR #42 DELIVERED 2026-10-28 (Gitea `main` `2a6e462` / GitHub `c2fb7f8`), PR #39 DELIVERED 2026-10-28 (Gitea `main` `b693517` / GitHub `ad18f7a`), PR #40 DELIVERED 2026-10-29 (Gitea `main` `61defd3` / GitHub `f26b2a9`), PR #41 DELIVERED 2026-10-30 (Gitea `main` `74ba171` / GitHub `e83c8e2`), issue #38 closed — the planned chain #34 → #35 → #42 → #39 → #40 → #41 is complete; flowcool's archive-side contract proposal on #38 (2026-10-30) awaits a maintainer decision; PR #53 DELIVERED 2026-10-30 (Gitea `main` `fa1ade5` / GitHub `a24668a`), issue #46 closed — batch-2 chain now at #54 (S1-S5 done, VALIDATED 2026-10-31, S6 next) | -| `1793401881422-pr54-persistent-job-resync-review.md` | PR #54 review (batch-2 chain): rebuild persistent jobs before scheduler resume (issue #47) — S1+S2 done 2026-10-30: real content 3 files vs branch point `e83c8e2` (101-file GitHub diff = orphaned-tip STRIP artifact), verdict approve with M1 (CHANGELOG `### Fixed` placement); S3-S5 done 2026-10-31: proposal confirmed as-is, integration applied (hunks clean on post-#53 dev), validation green (PR module 14 passed, before/after proof, unit 652/10/68, regression PASS, ruff+bandit clean vs baseline, deployment recorded-skip) — VALIDATED, S6 next | +| `1792862814701-next-up-sequencing.md` | Next-up sequencing decision (2026-10-24), superseded in part (2026-10-25): C1–C3 done and pushed; current primary next-up is the PR continuation order #30 → #31 → #33 → #32 → #34 (one pr-review chain at a time), interleaved with Stage 1b portal sessions (next: 1und1); PR #30 + #31 DELIVERED 2026-10-25, PR #33 DELIVERED 2026-10-26, PR #32 DELIVERED 2026-10-26 (Gitea `main` `bde713c` / GitHub `93bfcc8`), PR #34 DELIVERED 2026-10-27 (Gitea `main` `53a5dff` / GitHub `6bb3348`), PR #35 DELIVERED 2026-10-28 (Gitea `main` `75b885b` / GitHub `484048d`), PR #42 DELIVERED 2026-10-28 (Gitea `main` `2a6e462` / GitHub `c2fb7f8`), PR #39 DELIVERED 2026-10-28 (Gitea `main` `b693517` / GitHub `ad18f7a`), PR #40 DELIVERED 2026-10-29 (Gitea `main` `61defd3` / GitHub `f26b2a9`), PR #41 DELIVERED 2026-10-30 (Gitea `main` `74ba171` / GitHub `e83c8e2`), issue #38 closed — the planned chain #34 → #35 → #42 → #39 → #40 → #41 is complete; flowcool's archive-side contract proposal on #38 (2026-10-30) awaits a maintainer decision; PR #53 DELIVERED 2026-10-30 (Gitea `main` `fa1ade5` / GitHub `a24668a`), issue #46 closed — batch-2 chain now at #54 (S1-S6 done, VALIDATED 2026-10-31, S7 next) | +| `1793401881422-pr54-persistent-job-resync-review.md` | PR #54 review (batch-2 chain): rebuild persistent jobs before scheduler resume (issue #47) — S1+S2 done 2026-10-30: real content 3 files vs branch point `e83c8e2` (101-file GitHub diff = orphaned-tip STRIP artifact), verdict approve with M1 (CHANGELOG `### Fixed` placement); S3-S5 done 2026-10-31: proposal confirmed as-is, integration applied (hunks clean on post-#53 dev), validation green (PR module 14 passed, before/after proof, unit 652/10/68, regression PASS, ruff+bandit clean vs baseline, deployment recorded-skip); S6 docs confirmed 2026-10-31 (CHANGELOG M1 entry stands, no README/CONTRIBUTING changes) — VALIDATED, S7 next | From f68a6633dff9b3b9927316646e00112238b21f16 Mon Sep 17 00:00:00 2001 From: Stefan S Date: Sat, 31 Oct 2026 07:32:42 +0100 Subject: [PATCH 16/55] fix: rebuild persistent scheduler jobs from Task before resume (#47) --- apps/billcollector/scheduler.py | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/apps/billcollector/scheduler.py b/apps/billcollector/scheduler.py index d264667..4b98803 100644 --- a/apps/billcollector/scheduler.py +++ b/apps/billcollector/scheduler.py @@ -110,8 +110,13 @@ def scheduler(self) -> AsyncIOScheduler: def start(self) -> None: """Re-sync all jobs from the Task table and start the scheduler.""" - self.resync() - self._scheduler.start() + self._scheduler.start(paused=True) + try: + self.resync() + except Exception: + self._scheduler.shutdown(wait=False) + raise + self._scheduler.resume() logger.info("Scheduler started (%d job(s))", len(self._scheduler.get_jobs())) @@ -131,8 +136,7 @@ def stop(self) -> None: def resync(self) -> None: """Rebuild the job table from the Task table (startup hook): drop every stored job and re-add one per enabled task with a cron.""" - for job in list(self._scheduler.get_jobs()): - job.remove() + self._scheduler.remove_all_jobs() for task in self._db.list_tasks(): self._add_job_for(task) From f163369522223d9951c3fb2bc7b18cecca990ccd Mon Sep 17 00:00:00 2001 From: Stefan S Date: Sat, 31 Oct 2026 07:32:46 +0100 Subject: [PATCH 17/55] test: cover start() dropping persisted disabled and deleted jobs (#47) --- tests/test_bc_scheduler.py | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/tests/test_bc_scheduler.py b/tests/test_bc_scheduler.py index 9c22d04..82ba555 100644 --- a/tests/test_bc_scheduler.py +++ b/tests/test_bc_scheduler.py @@ -155,6 +155,35 @@ def test_task_job_dispatches_to_the_run_callback(self): asyncio.run(scheduler._task_job(task_id)) self.assertEqual(self.calls, [task_id]) + def test_start_drops_persisted_disabled_and_deleted_jobs_before_firing(self): + disabled = self.db.create_task("disabled", "u", enabled=True, cron="0 0 1 * *") + deleted = self.db.create_task("deleted", "u", enabled=True, cron="0 0 1 * *") + self._start_stop_cycle() + self.db.update_task(disabled, enabled=False) + self.db.conn.execute("DELETE FROM Task WHERE id=?", (deleted,)) + # Simulate jobs that became due while the daemon was stopped. + self.db.conn.execute("UPDATE apscheduler_jobs SET next_run_time=0") + self.db.conn.commit() + calls = [] + + async def callback(task_id): + calls.append(task_id) + + fresh = TaskScheduler(self.db, callback) + + async def cycle(): + fresh.start() + try: + await asyncio.sleep(0.01) + self.assertEqual(fresh.scheduler.get_jobs(), []) + self.assertEqual(self.db.conn.execute( + "SELECT COUNT(*) FROM apscheduler_jobs").fetchone()[0], 0) + self.assertEqual(calls, []) + finally: + fresh.stop() + + asyncio.run(cycle()) + if __name__ == "__main__": unittest.main() From a75861fbf1c43757bb6f932f65846f5aa35d102f Mon Sep 17 00:00:00 2001 From: Stefan S Date: Sat, 31 Oct 2026 07:32:51 +0100 Subject: [PATCH 18/55] =?UTF-8?q?doc:=20changelog=20=E2=80=94=20persistent?= =?UTF-8?q?=20scheduler=20job=20rebuild=20fix=20(#47)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CHANGELOG.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8970d5e..ccdfe26 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -233,6 +233,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Fixed +- Load the persistent scheduler job store while paused, rebuild it from Task, + then resume (#47). Disabled or deleted tasks cannot fire stale overdue jobs. + - Parse numeric cron weekdays with crontab numbering (0/7 Sunday, 1 Monday) in both the task editor and scheduler (#46). Invalid stored schedules are skipped with their task id. Ambiguous legacy forms `1/2`, `mon-5`, `01` and From 497dbdb74426b4d7debe9cde3bac1a3b90113f90 Mon Sep 17 00:00:00 2001 From: Stefan S Date: Sat, 31 Oct 2026 07:50:06 +0100 Subject: [PATCH 19/55] =?UTF-8?q?doc:=20plans=20=E2=80=94=20PR=20#54:=20S7?= =?UTF-8?q?=20complete=20(DELIVERED:=20Gitea=20main=20a75861f,=20GitHub=20?= =?UTF-8?q?main=20ebf20bb,=20PR=20#54=20+=20issue=20#47=20closed);=20plan?= =?UTF-8?q?=20doc=20archived=20to=20completed/=20+=20README=20+=20sequenci?= =?UTF-8?q?ng=20refs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../plans/1792862814701-next-up-sequencing.md | 10 +++ .kilo/plans/README.md | 3 +- ...81422-pr54-persistent-job-resync-review.md | 77 +++++++++++++++---- 3 files changed, 73 insertions(+), 17 deletions(-) rename .kilo/plans/{ => completed}/1793401881422-pr54-persistent-job-resync-review.md (78%) diff --git a/.kilo/plans/1792862814701-next-up-sequencing.md b/.kilo/plans/1792862814701-next-up-sequencing.md index cba39e0..9e8f62d 100644 --- a/.kilo/plans/1792862814701-next-up-sequencing.md +++ b/.kilo/plans/1792862814701-next-up-sequencing.md @@ -414,6 +414,16 @@ Grounds: Status: VALIDATED, S6 gate passed, S7 next (commit + push: Gitea dev → main promotion → GitHub mirror → PR close + issue #47 closure + Discussion summary). +- PR #54 S7 complete 2026-10-31 ("continue pr 54" session): DELIVERED — + commits `f68a663`/`f163369`/`a75861f` (fix/test/doc) pushed to `origin/dev`; + Gitea `main` `fa1ade5` → `a75861f` (ff-only, user-gated, no tag); GitHub + `main` `a24668a` → `ebf20bb` (sanitize-by-amend, public gate green: no + private paths at the tip, fix present); PR #54 closed without merge + (comment 6540230969, 2026-10-31T06:45:07Z), issue #47 closed completed + (comment 6540231280, 2026-10-31T06:45:09Z); Discussion summary text + recorded for manual web-UI post (write API unavailable). Plan doc archived + to `completed/`. Next in the batch-2 chain: **#56** (terminal ntfy + failures, #49) — its S1 runs against the settled `a75861f` tree. ## Explicitly not doing diff --git a/.kilo/plans/README.md b/.kilo/plans/README.md index 186b395..b113510 100644 --- a/.kilo/plans/README.md +++ b/.kilo/plans/README.md @@ -18,5 +18,4 @@ paths, so a moved file gains the `completed/` prefix. | `1792793307574-recipe-repair-llm-options.md` | Public service recipe recovery — LLM-assisted repair options analysis (pre-step); Stage 0 triage done 2026-10-24, all 7 services classified | | `1792819039884-recipe-doctor-implementation.md` | Stage 1: offline LLM recipe doctor (`billcollector doctor`) implementation plan + pilot (buhl, datev) — both pilots green 2026-10-24 (runs 16/18, downloads succeeded); Stage 1b in progress (2026-10-25): 4/7 green (buhl, datev, nuernberger, winsim), remaining 1und1 (in progress), freenet_mobilfunk (not started), kabeldeutschland (Option D re-authoring) | | `1792861675256-issue27-maintainer-commitments.md` | Issue #27 product-audit triage (closed 2026-10-24): maintainer-side commitments C1-C8 (F04, F09, F08 contract, F06 contract, F02, F03, F05, F10) with acceptance criteria, dependencies and proposed order | -| `1792862814701-next-up-sequencing.md` | Next-up sequencing decision (2026-10-24), superseded in part (2026-10-25): C1–C3 done and pushed; current primary next-up is the PR continuation order #30 → #31 → #33 → #32 → #34 (one pr-review chain at a time), interleaved with Stage 1b portal sessions (next: 1und1); PR #30 + #31 DELIVERED 2026-10-25, PR #33 DELIVERED 2026-10-26, PR #32 DELIVERED 2026-10-26 (Gitea `main` `bde713c` / GitHub `93bfcc8`), PR #34 DELIVERED 2026-10-27 (Gitea `main` `53a5dff` / GitHub `6bb3348`), PR #35 DELIVERED 2026-10-28 (Gitea `main` `75b885b` / GitHub `484048d`), PR #42 DELIVERED 2026-10-28 (Gitea `main` `2a6e462` / GitHub `c2fb7f8`), PR #39 DELIVERED 2026-10-28 (Gitea `main` `b693517` / GitHub `ad18f7a`), PR #40 DELIVERED 2026-10-29 (Gitea `main` `61defd3` / GitHub `f26b2a9`), PR #41 DELIVERED 2026-10-30 (Gitea `main` `74ba171` / GitHub `e83c8e2`), issue #38 closed — the planned chain #34 → #35 → #42 → #39 → #40 → #41 is complete; flowcool's archive-side contract proposal on #38 (2026-10-30) awaits a maintainer decision; PR #53 DELIVERED 2026-10-30 (Gitea `main` `fa1ade5` / GitHub `a24668a`), issue #46 closed — batch-2 chain now at #54 (S1-S6 done, VALIDATED 2026-10-31, S7 next) | -| `1793401881422-pr54-persistent-job-resync-review.md` | PR #54 review (batch-2 chain): rebuild persistent jobs before scheduler resume (issue #47) — S1+S2 done 2026-10-30: real content 3 files vs branch point `e83c8e2` (101-file GitHub diff = orphaned-tip STRIP artifact), verdict approve with M1 (CHANGELOG `### Fixed` placement); S3-S5 done 2026-10-31: proposal confirmed as-is, integration applied (hunks clean on post-#53 dev), validation green (PR module 14 passed, before/after proof, unit 652/10/68, regression PASS, ruff+bandit clean vs baseline, deployment recorded-skip); S6 docs confirmed 2026-10-31 (CHANGELOG M1 entry stands, no README/CONTRIBUTING changes) — VALIDATED, S7 next | +| `1792862814701-next-up-sequencing.md` | Next-up sequencing decision (2026-10-24), superseded in part (2026-10-25): C1–C3 done and pushed; current primary next-up is the PR continuation order #30 → #31 → #33 → #32 → #34 (one pr-review chain at a time), interleaved with Stage 1b portal sessions (next: 1und1); PR #30 + #31 DELIVERED 2026-10-25, PR #33 DELIVERED 2026-10-26, PR #32 DELIVERED 2026-10-26 (Gitea `main` `bde713c` / GitHub `93bfcc8`), PR #34 DELIVERED 2026-10-27 (Gitea `main` `53a5dff` / GitHub `6bb3348`), PR #35 DELIVERED 2026-10-28 (Gitea `main` `75b885b` / GitHub `484048d`), PR #42 DELIVERED 2026-10-28 (Gitea `main` `2a6e462` / GitHub `c2fb7f8`), PR #39 DELIVERED 2026-10-28 (Gitea `main` `b693517` / GitHub `ad18f7a`), PR #40 DELIVERED 2026-10-29 (Gitea `main` `61defd3` / GitHub `f26b2a9`), PR #41 DELIVERED 2026-10-30 (Gitea `main` `74ba171` / GitHub `e83c8e2`), issue #38 closed — the planned chain #34 → #35 → #42 → #39 → #40 → #41 is complete; flowcool's archive-side contract proposal on #38 (2026-10-30) awaits a maintainer decision; PR #53 DELIVERED 2026-10-30 (Gitea `main` `fa1ade5` / GitHub `a24668a`), issue #46 closed, PR #54 DELIVERED 2026-10-31 (Gitea `main` `a75861f` / GitHub `ebf20bb`), issue #47 closed — batch-2 chain now at #56 (terminal ntfy failures, #49) | diff --git a/.kilo/plans/1793401881422-pr54-persistent-job-resync-review.md b/.kilo/plans/completed/1793401881422-pr54-persistent-job-resync-review.md similarity index 78% rename from .kilo/plans/1793401881422-pr54-persistent-job-resync-review.md rename to .kilo/plans/completed/1793401881422-pr54-persistent-job-resync-review.md index 9709dd9..7e432e2 100644 --- a/.kilo/plans/1793401881422-pr54-persistent-job-resync-review.md +++ b/.kilo/plans/completed/1793401881422-pr54-persistent-job-resync-review.md @@ -1,14 +1,11 @@ # PR #54 review: rebuild persistent jobs before scheduler resume (issue #47) -> Status: VALIDATED (2026-10-31) — S6 docs confirmed: CHANGELOG M1 entry in -> place under [Unreleased] Fixed (sanitized, references #47 only), no README/ -> CONTRIBUTING changes (README:233 wording stays accurate, no rule changes). -> S5 validation green on the exact dev tree: PR module 14 passed / 15 subtests, -> before/after proof (new test fails on pre-integration scheduler), full unit -> 652/10 skipped/68 subtests, regression VERDICT: PASS (7 scenarios / 8 PDFs), -> ruff + bandit clean vs baseline, deployment block recorded-skip (not -> deployment-relevant). S5 + S6 gates passed 2026-10-31; ready for S7 -> (commit + push). +> Status: DELIVERED (2026-10-31) — S7 complete: dev commits `f68a663` / +> `f163369` / `a75861f` (fix/test/doc), Gitea `main` `a75861f` (ff-only from +> `fa1ade5`), GitHub `main` `ebf20bb` (mirrored from `a75861f`); PR #54 closed +> without merge (comment 6540230969, 2026-10-31T06:45:07Z), issue #47 closed +> completed (comment 6540231280, 2026-10-31T06:45:09Z); Discussion summary +> text recorded for manual web-UI post (write API unavailable). ## PR metadata @@ -185,12 +182,11 @@ handling. Block skipped; recorded here + user-confirmed at the S5 gate. ## Open questions -- None. S3 gate passed 2026-10-30 (proposal confirmed as-is); S4 integration - applied; S5 validation green + S5 gate passed 2026-10-31 (validation results - confirmed incl. the recorded deployment skip); S6 docs confirmed + S6 gate - passed 2026-10-31 (CHANGELOG entry as the only public artifact; no README/ - CONTRIBUTING changes); next: S7 commit + push (Gitea dev → main promotion → - GitHub mirror → PR close). +- None. All gates passed: S3 (2026-10-30, proposal confirmed as-is), S5 + (2026-10-31, validation results incl. the recorded deployment skip), S6 + (2026-10-31, CHANGELOG entry as the only public artifact, no README/ + CONTRIBUTING changes), S7 (2026-10-31, 3 commits as-is, promotion, mirror, + public texts). Delivered 2026-10-31; plan doc archived to `completed/`. ## Execution record @@ -259,3 +255,54 @@ handling. Block skipped; recorded here + user-confirmed at the S5 gate. - CONTRIBUTING: no contributor-rule change → no edit. S6 gate passed 2026-10-31 (user confirmed the documentation). Tree still dirty (3 integration files) for S7. Status: VALIDATED. + +- S7 (2026-10-31, "continue pr 54" session): prerequisite re-checked — the + S5-validated content unchanged on the dev tree (dirty 3 files byte- + identical to the S5 diffstat; only the S6 doc-only commit `e61f2df` moved + since S5; dev == origin/dev; deployment recorded-skip still covers the + promoted tree). + - **Commits (dev-commits, all 3 + push confirmed):** `f68a663` `fix: + rebuild persistent scheduler jobs from Task before resume (#47)` + (scheduler.py, +8/−4); `f163369` `test: cover start() dropping persisted + disabled and deleted jobs (#47)` (test, +29); `a75861f` `doc: changelog — + persistent scheduler job rebuild fix (#47)` (CHANGELOG, +3). Pushed + `origin/dev` `e61f2df..a75861f`. + - **Promotion (user-gated):** Gitea `main` `fa1ade5` → `a75861f` ff-only; + pushed. No tag (no release decided). Verified `git diff main dev` empty, + status clean. + - **Mirror:** `bash .private/sync_github.sh` → GitHub `main` `a24668a` → + `ebf20bb` (sanitize-by-amend). Public gate green: no `.kilo`/`.vscode`/ + `.private`/`openwiki`/`AGENTS.md`/`.openwikiignore` at the tip; `.github` + holds only `ISSUE_TEMPLATE` (no `workflows/`); the `start()`/`resync()` + fix present in `scheduler.py` on GitHub `main`. + - **PR #54 closed** (user confirmed the exact comment text): comment + 6540230969 (2026-10-31T06:45:06Z), closed without merge + 2026-10-31T06:45:07Z — the content reached GitHub via the mirror. + - **Issue #47 closed** (user confirmed the exact comment text): comment + 6540231280 (2026-10-31T06:45:08Z), closed completed + 2026-10-31T06:45:09Z. + - **Discussion summary (S7.6):** GitHub Discussion write API unavailable + from this environment (2026-10-25 finding); user chose manual web-UI + posting — text recorded below. + > **Public update: persistent scheduler job-store rebuild at startup + > (issue #47, PR #54)** + > + > `main` now starts the scheduler paused, rebuilds the job table from the + > Task table while the persistent job store is loaded, and only then + > resumes — contributed by @flowcool in PR #54. `resync()` now uses + > `remove_all_jobs()`, which also clears the persistent store; a failed + > rebuild shuts the scheduler down before the error propagates. + > + > Maintainer-side adaptation: the CHANGELOG entry was folded into the + > existing `Unreleased`/`Fixed` subsection (entry text verbatim). + > + > Validation on the promoted tree: 14 scheduler-module tests (15 + > subtests), 652 passed full unit suite, regression harness PASS (7 + > scenarios / 8 PDFs), ruff/bandit clean. + > + > User-facing note: after a daemon restart, jobs the persistent job store + > still held for disabled or deleted tasks no longer fire (previously + > they could fire if their next run was due while the daemon was + > stopped). + - **Archive:** this plan doc moved to `completed/`; README + sequencing + cross-references updated. From a96fabe483a610c3f445b4c385d324823417d5a2 Mon Sep 17 00:00:00 2001 From: Stefan S Date: Sat, 31 Oct 2026 08:25:15 +0100 Subject: [PATCH 20/55] =?UTF-8?q?doc:=20plans=20=E2=80=94=20PR=20#56:=20S1?= =?UTF-8?q?+S2=20review=20(verdict=20approve=20with=20M1=20CHANGELOG=20pla?= =?UTF-8?q?cement;=205-file=20delta=20vs=20branch=20point=20e83c8e2,=20103?= =?UTF-8?q?-file=20GitHub=20diff=20is=20orphaned-tip=20STRIP=20artifact,?= =?UTF-8?q?=204/5=20byte-identical=20pre-images=20on=20dev=20497dbdb,=20no?= =?UTF-8?q?=20overlap=20with=20open=20#44/#45/#55=20besides=20CHANGELOG,?= =?UTF-8?q?=20deployment=20not=20deployment-relevant);=20plan=20doc=20+=20?= =?UTF-8?q?README=20+=20sequencing;=20status=20AWAITING-DECISION?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../plans/1792862814701-next-up-sequencing.md | 12 ++ ...7736-pr56-terminal-ntfy-failures-review.md | 179 ++++++++++++++++++ .kilo/plans/README.md | 3 +- 3 files changed, 193 insertions(+), 1 deletion(-) create mode 100644 .kilo/plans/1793430417736-pr56-terminal-ntfy-failures-review.md diff --git a/.kilo/plans/1792862814701-next-up-sequencing.md b/.kilo/plans/1792862814701-next-up-sequencing.md index 9e8f62d..6172eef 100644 --- a/.kilo/plans/1792862814701-next-up-sequencing.md +++ b/.kilo/plans/1792862814701-next-up-sequencing.md @@ -424,6 +424,18 @@ Grounds: recorded for manual web-UI post (write API unavailable). Plan doc archived to `completed/`. Next in the batch-2 chain: **#56** (terminal ntfy failures, #49) — its S1 runs against the settled `a75861f` tree. +- PR #56 S1+S2 done 2026-10-31 ("continue pr 56" session): review plan doc + `1793430417736-pr56-terminal-ntfy-failures-review.md` — verdict approve with + maintainer changes (M1: the PR's top-level `## Terminal run notifications + (#49)` section breaks the Keep a Changelog structure; the entry moves under + `[Unreleased] → ### Added` in house style); head `95b11de` stable, branch + point = orphaned public tip `e83c8e2` (the 103-file GitHub "Files changed" + view is a STRIP artifact, real content 5 files +215/−1), 4 of 5 files + byte-identical pre-images on dev `497dbdb` (clean apply), only `CHANGELOG.md` + manual; no overlap with #53/#54 code, no overlap with open #44/#45/#55 + besides CHANGELOG; deployment validation recorded not deployment-relevant — + Status: AWAITING-DECISION, S2 gate pending (user review of verdict + plan), + S3 next. ## Explicitly not doing diff --git a/.kilo/plans/1793430417736-pr56-terminal-ntfy-failures-review.md b/.kilo/plans/1793430417736-pr56-terminal-ntfy-failures-review.md new file mode 100644 index 0000000..8c48613 --- /dev/null +++ b/.kilo/plans/1793430417736-pr56-terminal-ntfy-failures-review.md @@ -0,0 +1,179 @@ +# PR #56 review: terminal daemon failures through opt-in ntfy (issue #49) + +> Status: AWAITING-DECISION (2026-10-31) — S1+S2 review complete; verdict +> approve with one maintainer change (M1, CHANGELOG placement); 4 of 5 files +> apply byte-identical from head, tree clean on dev `497dbdb`. + +## PR metadata + +- PR: s-t-e-f-a-n/BillCollector#56 "feat(notify): report terminal daemon + failures through opt-in ntfy" (flowcool), branch + `feature/upstream-terminal-notifications-20261030`, head + `95b11de63548df3e8cb1a94ab8b05e99a9d0bd74`. Two commits in the fetched ref + (orphaned public tip `e83c8e2` + the feature commit `95b11de`); the real + content is **5 files +215/−1** against the branch point `e83c8e2`. + `mergeable_state: clean` (vs base `main` @ `ebf20bb`, the current GitHub + tip). No PR or issue comments. +- Issue: #49 (flowcool, open) — terminal failure, interruption and attention + timeout must reach the operator through the existing opt-in ntfy topic; + fixed account-free messages; success/cancellation silent; notification + failures never change the collector outcome; no double-send across + finalization/Stop/shutdown; no SQLite connections in workers. +- Created 2026-10-30T20:13:33Z; last updated 2026-10-31T06:34:55Z (before + this review's fetch). +- Local state at review time: Gitea `dev` = `origin/dev` = `497dbdb`, Gitea + `main` = `origin/main` = `a75861f`, GitHub `main` = `ebf20bb`. Working tree + clean. +- AI disclosure present: "AI-assisted focused port and synthetic verification + under my responsibility. Branch starts directly at public main `e83c8e2`." + +## Verdict + +**Approve with maintainer changes** — one placement fix (M1, CHANGELOG +structure); all code and tests apply as-is from head (byte-identical +pre-images on `dev`). + +## Verified facts + +1. Head stable: fetched `refs/pr/56` = `95b11de6...`; PR `updated_at` + 2026-10-31T06:34:55Z predates the fetch. S4 preflight re-checks via + `git ls-remote github refs/pull/56/head`. +2. Branch point: the fork starts at the orphaned public tip `e83c8e2` (GitHub + `main` at the PR #41 delivery; Gitea-side equivalent `74ba171`). `e83c8e2` + is **not** an ancestor of the current GitHub `main` `ebf20bb` (each mirror + sync re-pushes the Gitea lineage plus one amended sanitized tip, orphaning + the previous public tip — same pattern as #53/#54). The GitHub "Files + changed" view (103 files, +328/−23385; the `e83c8e2`-vs-merge-base part is + 99 files −23384 of STRIPped private content, `openwiki/` et al.) is an + orphaned-tip STRIP artifact, not PR content. Integration applies the + 5-file delta against `e83c8e2` only. +3. Issue #49 before-claim plausible and S5-verifiable: on the pre-integration + tree the new `RunNotificationTests::test_failed_crashed_and_early_failed_ + runs_send_once` must fail (no terminal notification path exists in + `notify.py` / `orchestrator.py` at `e83c8e2`); after integration the PR's + stated command passes: 49 passed, 15 subtests. +4. Pre-image vs Gitea `dev` `497dbdb` (drifted from the branch point only by + the PR #54 scheduler fix + plan docs): `notify.py`, `orchestrator.py`, + `tests/test_bc_notify.py`, `tests/test_bc_orchestrator.py` are + **byte-identical** to the PR base → hunks apply cleanly. `CHANGELOG.md` + diverges (PR #53/#54 entries added under `[Unreleased]`) → manual merge. + Dev files changed since the Gitea branch point `74ba171`: `CHANGELOG.md`, + `apps/billcollector/scheduler.py`, `apps/billcollector/ui/tasks.py`, + `tests/test_bc_scheduler.py` — only `CHANGELOG.md` overlaps the PR. +5. Code review (per call site), behavior preserved vs changed: + - `notify.py` +47: new module constant `_RUN_MESSAGES` (fixed strings for + `failure` / `interrupted` / `needs_attention`; no portal URL, account, + document, run id or exception text) and `notify_run_result(db, result, + *, dispatch=None)` following the existing `notify_login_locked` pattern: + Settings read on the caller thread (worker receives only immutable + strings — no SQLite connection), blank/absent topic → no dispatch, + unknown/success/cancelled result → silent before any Settings read, all + failures (Settings, dispatch, HTTP) caught and logged as exception class + name or HTTP status only (endpoint/topic/exception text never logged). + New `_send_run_result` mirrors `_send_login_locked` (Title + "BillCollector: run needs attention", Priority 4, Tags warning, + `NTFY_TIMEOUT`, no retry on non-200). Docstring updated to "pause, login + lock and daemon run outcomes". + - `orchestrator.py` +8, three call sites, all through the existing + `_dispatch_notification` worker: + (a) `cleanup_orphans` first loop — `interrupted` for each recovered + orphan row (row then finalized → a second cleanup at the next startup + cannot re-send; the second, attempt-only loop is untouched). + (b) `_finalize_run` no-row path (early failure before the Run row + exists) — `exit_code != 0` → `interrupted` for a signal-killed runner + (`< 0`) else `failure`; exit 0 stays silent. + (c) `_finalize_run` row path (every terminal run) — + `updates.get("result", row["result"])`: runner-finalized rows reuse the + runner's result (success/cancelled/needs_attention/interrupted/failure), + crash-finalized rows use the monitor's derived result. + The two `_finalize_run` paths are mutually exclusive (early return), and + the orphan cleanup only sees `result IS NULL` rows → single owner per + terminal outcome, no double-send. `shutdown()` awaits the monitors + (which call `_finalize_run`) and then gathers `self._notifications`, so + sends complete before the daemon exits — "including after shutdown" + holds. + - Test review: `tests/test_bc_notify.py` +67 (`NotifyRunResultTests`, + 6 methods: fixed payloads once, absent/blank topic never dispatches, + other results silent + no Settings read, exception text/endpoint + redaction in logs, HTTP rejection logged without retry, Settings and + dispatch errors contained). `tests/test_bc_orchestrator.py` +89 + (`RunNotificationTests`, 7 methods: fail/crash/earlyfail send once with + the fixed payload and release the task, no topic → no send, send error + does not change result/release and leaks nothing, success silent, + shutdown sends once, dashboard Stop sends one interruption push, orphan + recovery notifies only on the first cleanup). All base helpers + (`_orch(mode, sleep)`, `earlyfail` stub mode, `create_run`, + `set_setting`) exist in the byte-identical dev tree. No synthetic + markers, no secrets (`"private secret"` strings are redaction + assertions only). +6. Process check (CONTRIBUTING.md): issue referenced (#49); intended behavior + + verification stated in the PR body; proof shipped (before/after pytest + with expected counts); AI assistance disclosed; no secrets in code/tests; + no new network exposure (existing `NTFY_HOST` env + `ntfy_topic` + setting, opt-in); no shared-interface change (no setting, result value, + exit code, DB schema, control protocol or recipe contract change — + confirmed in the diff). +7. Cross-PR: open PRs are #55 (vault item identity), #45 (Playwright browser + cache), #44 (recipe method-chain validation). No file overlap with #56 + besides `CHANGELOG.md` (established per-PR merge pattern). Sequencing: + batch-2 chain `#53 → #54 → #56 → #55 → #45 → #44` (user-confirmed + 2026-10-30); #53/#54 DELIVERED → #56 is next in order. + +## Risks + +- Low overall: additive, best-effort notification; run results, slot release, + exit codes and the daemon's control flow are unchanged; a dead ntfy + endpoint only adds one logged warning per terminal outcome. +- Missed-notification window (accepted, by design): if the daemon crashes + after the runner finalized the row but before the monitor's send, that + outcome is never notified (startup cleanup does not re-send finalized + rows). Consistent with issue #49's best-effort framing. +- M1 as-shipped breaks the Keep a Changelog structure (a stray top-level + `## Terminal run notifications (#49)` section before the `[Unreleased]` + section) → manual placement fix required. + +## Integration plan (S4, on Gitea `dev`) + +| File | Action | Note | +|---|---|---| +| `apps/billcollector/notify.py` | apply from head | pre-image byte-identical to dev | +| `apps/billcollector/orchestrator.py` | apply from head | pre-image byte-identical to dev | +| `tests/test_bc_notify.py` | apply from head | pre-image byte-identical to dev | +| `tests/test_bc_orchestrator.py` | apply from head | pre-image byte-identical to dev | +| `CHANGELOG.md` | **manual (M1)** | drop the author's top-level `## Terminal run notifications (#49)` section; add under `## [Unreleased] → ### Added` in the house style: `- Terminal daemon run notifications: account-free failure, interruption and attention notices through the existing opt-in ntfy topic. Successful and cancelled runs stay silent, delivery is best-effort, and the monitor sends each terminal notice once. (issue #49)` | + +Commits (dev-commits, one concern each, proposed in S7): +1. `feat: notify terminal daemon run failures through the opt-in ntfy topic (#49)` — notify.py + orchestrator.py +2. `test: cover terminal run notifications (once-only, redaction, silent paths) (#49)` — both test files +3. `doc: changelog — terminal run notifications (#49)` — CHANGELOG.md (M1 placement) + +## Validation plan (S5, on the exact dev tree, `apps/.venv/bin/python`) + +1. Before/after proof: `tests.test_bc_notify.NotifyRunResultTests` and + `tests.test_bc_orchestrator.RunNotificationTests` fail on the + pre-integration tree, pass after (PR's stated evidence: + `python -m pytest tests/test_bc_notify.py tests/test_bc_orchestrator.py -q` + → 49 passed, 15 subtests; unittest equivalent expected). +2. PR unit tests: `apps/.venv/bin/python -m unittest + tests.test_bc_notify tests.test_bc_orchestrator -v` → all OK, 15 subtests. +3. Full regression: `apps/.venv/bin/python tests/run_regression.py --ini + tests/bc_regression.ini` → exit 0, VERDICT: PASS (7 scenarios / 8 PDFs). +4. `ruff` (F821, E4/E7/E9) + `bandit` on the changed files → nothing new + beyond the known baseline (B608 dynamic SQL table names, B104 UI bind). + +## Deployment validation + +Not deployment-relevant: no `Dockerfile`, `.dockerignore`, `BillCollector.sh`, +`install_docker-image.sh`, `deploy_remote.sh`, INI key or `.env`-handling +change (the `NTFY_HOST` env var and `ntfy_topic` setting pre-date this PR). +Recorded skip; user confirmation recorded at the S5 gate. + +## Out of scope + +- PRs #55, #45, #44 (later in the batch-2 chain). +- The missed-notification window (best-effort by issue #49's framing). +- Pause and login-lock notification behavior (unchanged). + +## Open questions + +None at S2. diff --git a/.kilo/plans/README.md b/.kilo/plans/README.md index b113510..da8a32d 100644 --- a/.kilo/plans/README.md +++ b/.kilo/plans/README.md @@ -18,4 +18,5 @@ paths, so a moved file gains the `completed/` prefix. | `1792793307574-recipe-repair-llm-options.md` | Public service recipe recovery — LLM-assisted repair options analysis (pre-step); Stage 0 triage done 2026-10-24, all 7 services classified | | `1792819039884-recipe-doctor-implementation.md` | Stage 1: offline LLM recipe doctor (`billcollector doctor`) implementation plan + pilot (buhl, datev) — both pilots green 2026-10-24 (runs 16/18, downloads succeeded); Stage 1b in progress (2026-10-25): 4/7 green (buhl, datev, nuernberger, winsim), remaining 1und1 (in progress), freenet_mobilfunk (not started), kabeldeutschland (Option D re-authoring) | | `1792861675256-issue27-maintainer-commitments.md` | Issue #27 product-audit triage (closed 2026-10-24): maintainer-side commitments C1-C8 (F04, F09, F08 contract, F06 contract, F02, F03, F05, F10) with acceptance criteria, dependencies and proposed order | -| `1792862814701-next-up-sequencing.md` | Next-up sequencing decision (2026-10-24), superseded in part (2026-10-25): C1–C3 done and pushed; current primary next-up is the PR continuation order #30 → #31 → #33 → #32 → #34 (one pr-review chain at a time), interleaved with Stage 1b portal sessions (next: 1und1); PR #30 + #31 DELIVERED 2026-10-25, PR #33 DELIVERED 2026-10-26, PR #32 DELIVERED 2026-10-26 (Gitea `main` `bde713c` / GitHub `93bfcc8`), PR #34 DELIVERED 2026-10-27 (Gitea `main` `53a5dff` / GitHub `6bb3348`), PR #35 DELIVERED 2026-10-28 (Gitea `main` `75b885b` / GitHub `484048d`), PR #42 DELIVERED 2026-10-28 (Gitea `main` `2a6e462` / GitHub `c2fb7f8`), PR #39 DELIVERED 2026-10-28 (Gitea `main` `b693517` / GitHub `ad18f7a`), PR #40 DELIVERED 2026-10-29 (Gitea `main` `61defd3` / GitHub `f26b2a9`), PR #41 DELIVERED 2026-10-30 (Gitea `main` `74ba171` / GitHub `e83c8e2`), issue #38 closed — the planned chain #34 → #35 → #42 → #39 → #40 → #41 is complete; flowcool's archive-side contract proposal on #38 (2026-10-30) awaits a maintainer decision; PR #53 DELIVERED 2026-10-30 (Gitea `main` `fa1ade5` / GitHub `a24668a`), issue #46 closed, PR #54 DELIVERED 2026-10-31 (Gitea `main` `a75861f` / GitHub `ebf20bb`), issue #47 closed — batch-2 chain now at #56 (terminal ntfy failures, #49) | +| `1792862814701-next-up-sequencing.md` | Next-up sequencing decision (2026-10-24), superseded in part (2026-10-25): C1–C3 done and pushed; current primary next-up is the PR continuation order #30 → #31 → #33 → #32 → #34 (one pr-review chain at a time), interleaved with Stage 1b portal sessions (next: 1und1); PR #30 + #31 DELIVERED 2026-10-25, PR #33 DELIVERED 2026-10-26, PR #32 DELIVERED 2026-10-26 (Gitea `main` `bde713c` / GitHub `93bfcc8`), PR #34 DELIVERED 2026-10-27 (Gitea `main` `53a5dff` / GitHub `6bb3348`), PR #35 DELIVERED 2026-10-28 (Gitea `main` `75b885b` / GitHub `484048d`), PR #42 DELIVERED 2026-10-28 (Gitea `main` `2a6e462` / GitHub `c2fb7f8`), PR #39 DELIVERED 2026-10-28 (Gitea `main` `b693517` / GitHub `ad18f7a`), PR #40 DELIVERED 2026-10-29 (Gitea `main` `61defd3` / GitHub `f26b2a9`), PR #41 DELIVERED 2026-10-30 (Gitea `main` `74ba171` / GitHub `e83c8e2`), issue #38 closed — the planned chain #34 → #35 → #42 → #39 → #40 → #41 is complete; flowcool's archive-side contract proposal on #38 (2026-10-30) awaits a maintainer decision; PR #53 DELIVERED 2026-10-30 (Gitea `main` `fa1ade5` / GitHub `a24668a`), issue #46 closed, PR #54 DELIVERED 2026-10-31 (Gitea `main` `a75861f` / GitHub `ebf20bb`), issue #47 closed — batch-2 chain now at #56 (terminal ntfy failures, #49); PR #56 S1+S2 done 2026-10-31 (approve with M1 CHANGELOG placement, 4/5 files byte-identical from head, AWAITING-DECISION) | +| `1793430417736-pr56-terminal-ntfy-failures-review.md` | PR #56 review (terminal ntfy failures, issue #49) — S1+S2 done 2026-10-31: verdict approve with M1 CHANGELOG placement, 4/5 files apply byte-identical from head `95b11de` (orphaned-tip branch point `e83c8e2`), Status: AWAITING-DECISION | From e3844d44c0be7ebe9d52867d60b0e901a4424b5a Mon Sep 17 00:00:00 2001 From: Stefan S Date: Sat, 31 Oct 2026 13:20:43 +0100 Subject: [PATCH 21/55] =?UTF-8?q?doc:=20plans=20=E2=80=94=20PR=20#56:=20S5?= =?UTF-8?q?=20results=20(M2=20test=5Fbc=5Flogin=20post-count=20adaptation,?= =?UTF-8?q?=20full=20unit=20665=20passed)=20+=20S6=20docs=20confirmed=20(C?= =?UTF-8?q?HANGELOG=20M1=20entry=20stands=20under=20[Unreleased]=20Added,?= =?UTF-8?q?=20sanitized;=20README=20ntfy=20section=20renamed=20+=20extende?= =?UTF-8?q?d),=20S6=20gate=20passed,=20status=20VALIDATED;=20ready=20for?= =?UTF-8?q?=20S7?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../plans/1792862814701-next-up-sequencing.md | 22 ++++++ ...7736-pr56-terminal-ntfy-failures-review.md | 77 +++++++++++++++---- 2 files changed, 83 insertions(+), 16 deletions(-) diff --git a/.kilo/plans/1792862814701-next-up-sequencing.md b/.kilo/plans/1792862814701-next-up-sequencing.md index 6172eef..cb2086c 100644 --- a/.kilo/plans/1792862814701-next-up-sequencing.md +++ b/.kilo/plans/1792862814701-next-up-sequencing.md @@ -436,6 +436,28 @@ Grounds: besides CHANGELOG; deployment validation recorded not deployment-relevant — Status: AWAITING-DECISION, S2 gate pending (user review of verdict + plan), S3 next. +- PR #56 S3-S5 done 2026-10-31 (same session): S2 gate passed (verdict + confirmed, doc commit `a96fabe` pushed); S3 proposal confirmed as-is; S4 + applied on dev (4 files byte-identical from head `95b11de` + M1 CHANGELOG + `[Unreleased] → ### Added` first entry, tree left dirty); S5 green on the + exact dev tree — before/after proof (21 new tests fail pre-integration, + 49 OK after), full unit 665 passed / 10 skipped / 83 subtests, regression + VERDICT: PASS (7 scenarios / 8 PDFs, exit 0), ruff clean, bandit 0 new on + app files; deployment recorded skip (not deployment-relevant) — S5 found one + extra maintainer change, M2: `tests/test_bc_login.py` + `test_slow_login_notification_keeps_daemon_loop_responsive` pinned one + `requests.post` per path, but the `finalize`/`orphan` paths now also send + the terminal run notification (the PR's scoped evidence never ran that + module); adapted to path-dependent post counts (2/2/1/1) — S5 gate passed + (same session); Status: VALIDATED, S6 next. +- PR #56 S6 done 2026-10-31 (next session): S6 docs in tree — CHANGELOG M1 + entry stands under `[Unreleased] → ### Added` (sanitized, first item); + README section `### Pause notifications (ntfy)` → `### Notifications (ntfy)` + (TOC anchor, opt-in sentence extended to terminal run outcomes, new + terminal-notices paragraph, both `NTFY_HOST` descriptions); CONTRIBUTING + unchanged — S6 gate passed (user confirmed as-is) — Status: VALIDATED, S7 + next (dev-commits proposal, push `origin/dev`, ff-only promotion, GitHub + mirror, close PR #56 + issue #49). ## Explicitly not doing diff --git a/.kilo/plans/1793430417736-pr56-terminal-ntfy-failures-review.md b/.kilo/plans/1793430417736-pr56-terminal-ntfy-failures-review.md index 8c48613..38c4578 100644 --- a/.kilo/plans/1793430417736-pr56-terminal-ntfy-failures-review.md +++ b/.kilo/plans/1793430417736-pr56-terminal-ntfy-failures-review.md @@ -1,8 +1,14 @@ # PR #56 review: terminal daemon failures through opt-in ntfy (issue #49) -> Status: AWAITING-DECISION (2026-10-31) — S1+S2 review complete; verdict -> approve with one maintainer change (M1, CHANGELOG placement); 4 of 5 files -> apply byte-identical from head, tree clean on dev `497dbdb`. +> Status: VALIDATED (2026-10-31) — S5 green on the exact dev tree: before/after +> proof (21 new tests fail on the pre-integration code, all 49 pass after), PR +> unit 49 OK / 15 subtests, full unit 665 passed / 10 skipped / 83 subtests +> (after the M2 test adaptation), regression VERDICT: PASS (7 scenarios / 8 +> PDFs, exit 0), ruff clean, bandit 0 findings on the changed app files; +> deployment validation recorded skip (not deployment-relevant), S5 gate +> passed; S6 docs confirmed (CHANGELOG M1 entry stands under [Unreleased] +> Added, sanitized; README ntfy section renamed + extended), S6 gate passed; +> ready for S7. ## PR metadata @@ -29,8 +35,9 @@ ## Verdict -**Approve with maintainer changes** — one placement fix (M1, CHANGELOG -structure); all code and tests apply as-is from head (byte-identical +**Approve with maintainer changes** — M1 (CHANGELOG structure) and M2 +(`tests/test_bc_login.py` post-count adaptation, discovered by the S5 full +unit suite); all PR code and tests apply as-is from head (byte-identical pre-images on `dev`). ## Verified facts @@ -141,32 +148,70 @@ pre-images on `dev`). | `tests/test_bc_notify.py` | apply from head | pre-image byte-identical to dev | | `tests/test_bc_orchestrator.py` | apply from head | pre-image byte-identical to dev | | `CHANGELOG.md` | **manual (M1)** | drop the author's top-level `## Terminal run notifications (#49)` section; add under `## [Unreleased] → ### Added` in the house style: `- Terminal daemon run notifications: account-free failure, interruption and attention notices through the existing opt-in ntfy topic. Successful and cancelled runs stay silent, delivery is best-effort, and the monitor sends each terminal notice once. (issue #49)` | +| `tests/test_bc_login.py` | **manual (M2)** | S5 discovery: `test_slow_login_notification_keeps_daemon_loop_responsive` pinned exactly one `requests.post` per path; the `finalize` and `orphan` paths now legitimately send the terminal run notification in addition to the login lock one. Adapted the tail assertions to a path-dependent count (`expected_posts = 2 if path in ("finalize", "orphan") else 1`), the worker-thread check to all workers off-loop, and the call-count assert to match. The loop-responsiveness intent is unchanged for every send. | Commits (dev-commits, one concern each, proposed in S7): 1. `feat: notify terminal daemon run failures through the opt-in ntfy topic (#49)` — notify.py + orchestrator.py -2. `test: cover terminal run notifications (once-only, redaction, silent paths) (#49)` — both test files +2. `test: cover terminal run notifications and adapt the slow-login post-count expectations (#49)` — test_bc_notify.py, test_bc_orchestrator.py (from head) + test_bc_login.py (M2) 3. `doc: changelog — terminal run notifications (#49)` — CHANGELOG.md (M1 placement) ## Validation plan (S5, on the exact dev tree, `apps/.venv/bin/python`) -1. Before/after proof: `tests.test_bc_notify.NotifyRunResultTests` and - `tests.test_bc_orchestrator.RunNotificationTests` fail on the - pre-integration tree, pass after (PR's stated evidence: - `python -m pytest tests/test_bc_notify.py tests/test_bc_orchestrator.py -q` - → 49 passed, 15 subtests; unittest equivalent expected). +1. Before/after proof: the new `NotifyRunResultTests` / `RunNotificationTests` + fail on the pre-integration tree, pass after (PR's stated evidence: 49 + passed, 15 subtests). 2. PR unit tests: `apps/.venv/bin/python -m unittest - tests.test_bc_notify tests.test_bc_orchestrator -v` → all OK, 15 subtests. -3. Full regression: `apps/.venv/bin/python tests/run_regression.py --ini + tests.test_bc_notify tests.test_bc_orchestrator -v` → all OK. +3. Full unit suite: `apps/.venv/bin/python -m pytest tests/ -q` → all pass. +4. Full regression: `apps/.venv/bin/python tests/run_regression.py --ini tests/bc_regression.ini` → exit 0, VERDICT: PASS (7 scenarios / 8 PDFs). -4. `ruff` (F821, E4/E7/E9) + `bandit` on the changed files → nothing new - beyond the known baseline (B608 dynamic SQL table names, B104 UI bind). +5. `ruff` + `bandit` on the changed files → nothing new beyond the known + baseline. + +### S5 results (2026-10-31, tree = dev `a96fabe` + the 5-file integration) + +1. Before-proof: with the PR code files stashed (new tests kept), `unittest + tests.test_bc_notify tests.test_bc_orchestrator` → 49 run, FAILED + (failures=8, errors=13) — exactly the 21 new notification tests fail + (`notify_run_result` absent / no terminal sends); the pre-existing 28 pass. + After: the same command → 49 OK. The PR's 49-passed / 15-subtests claim + reproduces. +2. Full unit suite (first run): 2 failed, 663 passed, 10 skipped, 83 + subtests — `test_bc_login.py::test_slow_login_notification_keeps_daemon_ + loop_responsive[finalize|orphan]` pinned exactly one `requests.post` per + path (the PR's scoped evidence never ran that module). M2 applied → + re-run: **665 passed, 10 skipped, 83 subtests**. +3. Regression: VERDICT: PASS — 7/7 pairs matched (3× success, 1 graceful, + 1 badlogin failure, 1 empty failure, 1 dlfail failure), 8 expected PDFs, + exit 0. +4. `ruff check` (repo ruff.toml) on all 5 changed files: all checks passed. + `bandit` on the changed app files (notify.py, orchestrator.py): 0 findings + (same as the base versions); the test-file findings are the pre-existing + `assert` baseline, unchanged. ## Deployment validation Not deployment-relevant: no `Dockerfile`, `.dockerignore`, `BillCollector.sh`, `install_docker-image.sh`, `deploy_remote.sh`, INI key or `.env`-handling change (the `NTFY_HOST` env var and `ntfy_topic` setting pre-date this PR). -Recorded skip; user confirmation recorded at the S5 gate. +Recorded skip on 2026-10-31 (S5); user confirmation at the S5 gate. +Validated tree: dev `a96fabe` + the 5-file integration (working tree). + +## S6 documentation (2026-10-31) + +- `CHANGELOG.md`: the M1 entry (first item of `[Unreleased] → ### Added`, + house style, `(issue #49)`) stands as placed in S4 — sanitized, no private + paths or hosts. +- `README.md` (user-visible behavior changed — a new class of pushes): + section `### Pause notifications (ntfy)` → `### Notifications (ntfy)` + (TOC anchor included); the opt-in sentence now covers "a paused run, a + blocked login, or a terminal run outcome"; one new sentence documents the + terminal notices (fixed account-free messages, which outcomes send, + success/cancelled silent, best-effort); both `NTFY_HOST` descriptions now + read "push server for the ntfy notifications". +- `CONTRIBUTING.md`: no change (no contributor-rule change). +- S6 gate: user confirmed the docs as-is on 2026-10-31; status VALIDATED, + ready for S7 (code/test/doc commits per the plan above, Gitea first). ## Out of scope From da01d5bf266204d1b20e7efaf39a0337d68ea4ba Mon Sep 17 00:00:00 2001 From: Stefan S Date: Sat, 31 Oct 2026 14:53:23 +0100 Subject: [PATCH 22/55] feat: notify terminal daemon run failures through the opt-in ntfy topic (#49) --- apps/billcollector/notify.py | 47 +++++++++++++++++++++++++++++- apps/billcollector/orchestrator.py | 8 +++++ 2 files changed, 54 insertions(+), 1 deletion(-) diff --git a/apps/billcollector/notify.py b/apps/billcollector/notify.py index 9cdc82b..5d9e6fa 100644 --- a/apps/billcollector/notify.py +++ b/apps/billcollector/notify.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -"""ntfy push notifications for pause events (decision 13). +"""ntfy push notifications for pause, login lock and daemon run outcomes. Every pause pushes with a deep link to the session page (``/runs/``) so the operator can act from a phone @@ -75,3 +75,48 @@ def notify_pause(db, run, task): # The exception can include the URL (and topic); keep the class name only. logger.warning(f"ntfy pause push failed: {type(e).__name__}") return False + + +_RUN_MESSAGES = { + "failure": "A collector run failed. Check the BillCollector dashboard.", + "interrupted": "A collector run was interrupted. Check the BillCollector dashboard.", + "needs_attention": "A collector run timed out waiting for attention. Check the BillCollector dashboard.", +} + + +def notify_run_result(db, result, *, dispatch=None): + """Best-effort daemon outcome push with a fixed, account-free payload. + + Read Settings before dispatch: the worker receives only immutable strings, + never the SQLite connection. Successful and cancelled runs stay silent. + """ + try: + message = _RUN_MESSAGES.get(result) + if message is None: + return False + topic = (db.get_setting("ntfy_topic") or "").strip() + if not topic: + return False + url = f"{_ntfy_host()}/{topic}" + if dispatch is not None: + return dispatch(_send_run_result, url, message) + return _send_run_result(url, message) + except Exception as e: + # Settings/dispatch failures must not affect finalization either. + logger.warning("ntfy run result push failed: %s", type(e).__name__) + return False + + +def _send_run_result(url, message): + try: + response = requests.post( + url, data=message.encode("utf-8"), + headers={"Title": "BillCollector: run needs attention", "Priority": "4", + "Tags": "warning"}, timeout=NTFY_TIMEOUT) + if response.status_code == 200: + return True + logger.warning("ntfy run result push failed: HTTP %s", response.status_code) + except Exception as e: + # Exception text can contain the endpoint or topic. Never log it. + logger.warning("ntfy run result push failed: %s", type(e).__name__) + return False diff --git a/apps/billcollector/orchestrator.py b/apps/billcollector/orchestrator.py index 97f63fd..6420159 100644 --- a/apps/billcollector/orchestrator.py +++ b/apps/billcollector/orchestrator.py @@ -83,6 +83,7 @@ from . import config, display from .db import Database, now from .login import settle_attempt +from .notify import notify_run_result from .variables import DateContext logger = logging.getLogger("billcollector.orchestrator") @@ -211,6 +212,7 @@ def cleanup_orphans(self) -> int: if run["display"] is not None: display.remove_display_lock(run["display"]) self._db.release_task_run(run["task_id"]) + notify_run_result(self._db, "interrupted", dispatch=self._dispatch_notification) count += 1 logger.info("orphan cleanup: run %s (task %s, pid %s) marked interrupted", run["id"], run["task_id"], run["pid"]) @@ -491,6 +493,9 @@ def _finalize_run(self, task_id: int, pid: int, exit_code: int, tail) -> None: logger.warning("start_run: no Run row for task %s (pid %s, exit %s); " "runner output tail: %s", task_id, pid, exit_code, "".join(tail).strip() or "(none)") + if exit_code != 0: + notify_run_result(self._db, "interrupted" if exit_code < 0 else "failure", + dispatch=self._dispatch_notification) return run_id = row["id"] settle_attempt(self._db, task_id, run_id, @@ -524,3 +529,6 @@ def _finalize_run(self, task_id: int, pid: int, exit_code: int, tail) -> None: self._db.release_task_run(task_id) logger.info("runner for task %s (run %s) exited with code %s", task_id, run_id, exit_code) + # Only the monitor sends terminal notifications, including after shutdown. + notify_run_result(self._db, updates.get("result", row["result"]), + dispatch=self._dispatch_notification) From 32140f7cb87b2bd9c18d6539536a6984bd18c7c4 Mon Sep 17 00:00:00 2001 From: Stefan S Date: Sat, 31 Oct 2026 14:53:23 +0100 Subject: [PATCH 23/55] test: cover terminal run notifications and adapt the slow-login post-count expectations (#49) --- tests/test_bc_login.py | 9 ++-- tests/test_bc_notify.py | 67 ++++++++++++++++++++++++++ tests/test_bc_orchestrator.py | 89 +++++++++++++++++++++++++++++++++++ 3 files changed, 162 insertions(+), 3 deletions(-) diff --git a/tests/test_bc_login.py b/tests/test_bc_login.py index ac91eb3..e368abb 100644 --- a/tests/test_bc_login.py +++ b/tests/test_bc_login.py @@ -453,10 +453,13 @@ def slow_post(*args, **kwargs): await shutdown else: await orch.shutdown() + # The finalize and orphan paths also send the terminal run + # notification (issue #49) alongside the login lock one. + expected_posts = 2 if path in ("finalize", "orphan") else 1 assert finished.is_set() - assert len(worker_threads) == 1 - assert worker_threads[0] != loop_thread - post.assert_called_once() + assert len(worker_threads) == expected_posts + assert all(t != loop_thread for t in worker_threads) + assert post.call_count == expected_posts assert not orch._notifications diff --git a/tests/test_bc_notify.py b/tests/test_bc_notify.py index 95b5f06..98cb443 100644 --- a/tests/test_bc_notify.py +++ b/tests/test_bc_notify.py @@ -98,5 +98,72 @@ def test_transport_error_is_swallowed(self): self.assertFalse(notify.notify_pause(self.db, self.run, self.task)) +class NotifyRunResultTests(unittest.TestCase): + def setUp(self): + self.db = mock.Mock() + self.db.get_setting.return_value = "bc-topic" + self.env = mock.patch.dict(os.environ, {"NTFY_HOST": "http://ntfy.test:7979/"}) + self.env.start() + self.addCleanup(self.env.stop) + + def test_outcomes_send_fixed_payloads_once(self): + for result, message in { + "failure": "A collector run failed. Check the BillCollector dashboard.", + "interrupted": "A collector run was interrupted. Check the BillCollector dashboard.", + "needs_attention": "A collector run timed out waiting for attention. Check the BillCollector dashboard.", + }.items(): + with self.subTest(result=result), mock.patch.object( + notify.requests, "post", return_value=mock.Mock(status_code=200)) as post: + self.assertTrue(notify.notify_run_result(self.db, result)) + post.assert_called_once_with( + "http://ntfy.test:7979/bc-topic", data=message.encode("utf-8"), + headers={"Title": "BillCollector: run needs attention", "Priority": "4", + "Tags": "warning"}, timeout=notify.NTFY_TIMEOUT) + + def test_absent_or_blank_topic_never_dispatches(self): + for topic in (None, "", " "): + self.db.get_setting.return_value = topic + with self.subTest(topic=topic), mock.patch.object(notify.requests, "post") as post: + dispatch = mock.Mock() + self.assertFalse(notify.notify_run_result(self.db, "failure", dispatch=dispatch)) + post.assert_not_called() + dispatch.assert_not_called() + + def test_other_results_are_silent(self): + for result in (None, "success", "cancelled", "https://private.test/secret"): + with self.subTest(result=result), mock.patch.object(notify.requests, "post") as post: + self.assertFalse(notify.notify_run_result(self.db, result)) + post.assert_not_called() + self.db.get_setting.assert_not_called() + + def test_send_failure_is_logged_without_endpoint_or_exception_text(self): + for error in (requests.exceptions.ConnectionError("http://private.test/secret"), + ValueError("private account and secret")): + with self.subTest(error=type(error).__name__), mock.patch.object( + notify.requests, "post", side_effect=error) as post, \ + self.assertLogs(notify.logger, level="WARNING") as logs: + self.assertFalse(notify.notify_run_result(self.db, "failure")) + post.assert_called_once() + self.assertEqual(logs.output, [ + f"WARNING:billcollector.notify:ntfy run result push failed: {type(error).__name__}"]) + + def test_http_rejection_is_logged_without_retry(self): + with mock.patch.object(notify.requests, "post", return_value=mock.Mock(status_code=503)) as post, \ + self.assertLogs(notify.logger, level="WARNING") as logs: + self.assertFalse(notify.notify_run_result(self.db, "failure")) + post.assert_called_once() + self.assertEqual(logs.output, [ + "WARNING:billcollector.notify:ntfy run result push failed: HTTP 503"]) + + def test_settings_and_dispatch_errors_do_not_escape(self): + self.db.get_setting.side_effect = RuntimeError("secret") + with self.assertLogs(notify.logger, level="WARNING"): + self.assertFalse(notify.notify_run_result(self.db, "failure")) + self.db.get_setting.side_effect = None + with self.assertLogs(notify.logger, level="WARNING"): + self.assertFalse(notify.notify_run_result( + self.db, "failure", dispatch=mock.Mock(side_effect=RuntimeError("secret")))) + + if __name__ == "__main__": unittest.main() diff --git a/tests/test_bc_orchestrator.py b/tests/test_bc_orchestrator.py index f54e278..7e8b562 100644 --- a/tests/test_bc_orchestrator.py +++ b/tests/test_bc_orchestrator.py @@ -28,6 +28,7 @@ import tempfile import time import unittest +from unittest import mock from pathlib import Path sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "apps")) @@ -257,6 +258,94 @@ def test_early_runner_failure_releases_the_claim(self): self.assertEqual(self.db.get_task(task_id)["status"], "idle") +class RunNotificationTests(OrchestratorTestBase): + def setUp(self): + super().setUp() + self.db.set_setting("ntfy_topic", "bc-topic") + patcher = mock.patch("billcollector.notify.requests.post", + return_value=mock.Mock(status_code=200)) + self.post = patcher.start() + self.addCleanup(patcher.stop) + self.task_id = self.db.create_task("private-service", "private-account", enabled=True) + + def _run_and_drain(self, orch): + async def scenario(): + self.assertTrue(await orch.start_run(self.task_id)) + code = await orch.monitors[self.task_id] + await orch.shutdown() # drains notification workers, never sends twice + return code + return asyncio.run(scenario()) + + def test_failed_crashed_and_early_failed_runs_send_once(self): + for mode in ("fail", "crash", "earlyfail"): + with self.subTest(mode=mode): + self.post.reset_mock() + self.assertEqual(self._run_and_drain(self._orch(mode)), 1) + self.post.assert_called_once() + self.assertEqual(self.post.call_args.kwargs["data"], + b"A collector run failed. Check the BillCollector dashboard.") + self.assertEqual(self.db.get_task(self.task_id)["status"], "idle") + + def test_no_topic_means_no_send(self): + self.db.set_setting("ntfy_topic", "") + self.assertEqual(self._run_and_drain(self._orch("fail")), 1) + self.post.assert_not_called() + + def test_send_error_does_not_change_result_or_task_release(self): + self.post.side_effect = RuntimeError("private secret") + with self.assertLogs("billcollector.notify", level="WARNING") as logs: + self.assertEqual(self._run_and_drain(self._orch("fail")), 1) + self.post.assert_called_once() + self.assertEqual(self.db.list_runs(self.task_id)[0]["result"], "failure") + self.assertEqual(self.db.get_task(self.task_id)["status"], "idle") + self.assertNotIn("private secret", str(logs.output)) + + def test_success_does_not_send(self): + self.assertEqual(self._run_and_drain(self._orch("ok")), 0) + self.post.assert_not_called() + + def test_interrupted_shutdown_sends_once(self): + orch = self._orch("crash", sleep=30) + async def scenario(): + self.assertTrue(await orch.start_run(self.task_id)) + for _ in range(400): + if self.db.list_runs(self.task_id): + break + await asyncio.sleep(0.01) + self.assertTrue(self.db.list_runs(self.task_id)) + await orch.shutdown() + asyncio.run(scenario()) + self.post.assert_called_once() + self.assertEqual(self.post.call_args.kwargs["data"], + b"A collector run was interrupted. Check the BillCollector dashboard.") + self.assertEqual(self.db.list_runs(self.task_id)[0]["result"], "interrupted") + + def test_dashboard_stop_sends_one_interruption_push(self): + orch = self._orch("crash", sleep=30) + async def scenario(): + self.assertTrue(await orch.start_run(self.task_id)) + monitor = orch.monitors[self.task_id] + for _ in range(400): + if self.db.list_runs(self.task_id): + break + await asyncio.sleep(0.01) + self.assertTrue(self.db.list_runs(self.task_id)) + self.assertTrue(await orch.stop_run(self.task_id)) + await monitor + await orch.shutdown() + asyncio.run(scenario()) + self.post.assert_called_once() + self.assertEqual(self.post.call_args.kwargs["data"], + b"A collector run was interrupted. Check the BillCollector dashboard.") + + def test_orphan_recovery_notifies_only_on_first_cleanup(self): + run_id = self.db.create_run(self.task_id) + self.assertEqual(self.orch.cleanup_orphans(), 1) + self.assertEqual(self.orch.cleanup_orphans(), 0) + self.post.assert_called_once() + self.assertEqual(self.db.get_run(run_id)["result"], "interrupted") + + class SkipTests(OrchestratorTestBase): """start_run does not spawn when it cannot claim the task.""" From 2bf53d810ca17eff9a0f11eabf0992a652c4918e Mon Sep 17 00:00:00 2001 From: Stefan S Date: Sat, 31 Oct 2026 14:53:23 +0100 Subject: [PATCH 24/55] =?UTF-8?q?doc:=20changelog=20+=20README=20=E2=80=94?= =?UTF-8?q?=20terminal=20run=20notifications=20(#49)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CHANGELOG.md | 4 ++++ README.md | 12 +++++++----- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ccdfe26..9f2cbcc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Added +- Terminal daemon run notifications: account-free failure, interruption and + attention notices through the existing opt-in ntfy topic. Successful and + cancelled runs stay silent, delivery is best-effort, and the monitor sends + each terminal notice once. (issue #49) - Opt-in daemon `notice_identity` with source-scoped PDF revision tracking, nullable Download identity columns, and pinned pypdf. Original PDFs are kept. (issue #38) diff --git a/README.md b/README.md index 14606cf..3277918 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ Compose service includes a pinned seccomp profile. See - [Vault of Secrets](#vault-of-secrets) - [Enabling DNS and HTTPS with Let's Encrypt certs](#enabling-dns-and-https-with-lets-encrypt-certs) - [Installing the Daemon](#installing-the-daemon) - - [Pause notifications (ntfy)](#pause-notifications-ntfy) + - [Notifications (ntfy)](#notifications-ntfy) - [Daemon Mode](#daemon-mode) - [Environment Variables](#environment-variables) - [Container Semantics](#container-semantics) @@ -192,7 +192,7 @@ Now that we have done a good job installing all the prerequisites, we are focusi and set `BW_API_URL` to the HTTP URL of your local `bw serve` API (e.g., `http://:8087`; it must resolve only to loopback/private addresses, HTTPS is refused). See [Vaultwarden and Bitwarden](#vaultwarden-and-bitwarden). -4. Optionally, create a root `.env` with the compose variables - all of them have defaults, the file is optional: `TZ` (daemon calendar dates and cron schedules, default `UTC`), `BC_UI_PORT` (published UI port, default `8766`), `NTFY_HOST` (push server for pause notifications, default `https://ntfy.sh`), `NTFY_BASE_URL` (LAN-absolute deep links, with the bundled ntfy), `APP_UID`/`APP_GID` (container user). +4. Optionally, create a root `.env` with the compose variables - all of them have defaults, the file is optional: `TZ` (daemon calendar dates and cron schedules, default `UTC`), `BC_UI_PORT` (published UI port, default `8766`), `NTFY_HOST` (push server for the ntfy notifications, default `https://ntfy.sh`), `NTFY_BASE_URL` (LAN-absolute deep links, with the bundled ntfy), `APP_UID`/`APP_GID` (container user). Set `TZ=Europe/Zurich`, for example, to use local dates and schedules. Compose reads `TZ` from the shell first, then the root `.env`; an unset @@ -217,9 +217,11 @@ Now that we have done a good job installing all the prerequisites, we are focusi 8. Collect your first document: on the **Dashboard** press **Run now** for an enabled task - watch the run live (session page, noVNC tab) until it reaches `success`; the document appears on the Downloads page and in the consumer folder. Every step is driven from the browser, no shell access to the daemon needed. -### Pause notifications (ntfy) +### Notifications (ntfy) -A fresh installation sends no notifications: the `ntfy topic` setting starts empty, and a paused run (or a blocked login) only pushes when you set a topic. To opt in, open **Settings** and set an **ntfy topic** - choose your own topic name, it is the install-specific destination the pushes are sent to. +A fresh installation sends no notifications: the `ntfy topic` setting starts empty, and a paused run, a blocked login, or a terminal run outcome only pushes when you set a topic. To opt in, open **Settings** and set an **ntfy topic** - choose your own topic name, it is the install-specific destination the pushes are sent to. + +Terminal run notices are fixed and account-free: a failed, interrupted, or attention-timeout run sends one push each (including a run the daemon reclaims at startup), while successful and cancelled runs stay silent. Delivery is best-effort - a failed push never changes the run outcome. The push server defaults to the public [ntfy.sh](https://ntfy.sh). To run your own push server in the same stack instead: @@ -240,7 +242,7 @@ On the first start (while the task list is still empty), the `[Playwright]` sect - `BW_API_HOST=127.0.0.1:8087` (optional): Host header for a `bw serve` in another container - `BC_UI_HOST`: daemon UI bind host (default `127.0.0.1`; set `0.0.0.0` inside the container) - `BC_UI_PASSWORD` (optional): seeds the UI password hash on the first start (a hash set via the UI is never overwritten) -- `NTFY_HOST`: push server for the pause notifications (default `https://ntfy.sh`) +- `NTFY_HOST`: push server for the ntfy notifications (default `https://ntfy.sh`) - `BC_CHROMIUM_ARGS=--disable-software-rasterizer`: dev-host launch workaround only; leave unset for stock behavior - `VAULT_HOST` is no longer used, see [doc/vault_transport.md](doc/vault_transport.md) From 19c4164c3e1b7d30f29a3e60b97e14b629b6546c Mon Sep 17 00:00:00 2001 From: Stefan S Date: Sat, 31 Oct 2026 16:26:44 +0100 Subject: [PATCH 25/55] =?UTF-8?q?doc:=20plans=20=E2=80=94=20PR=20#56:=20S7?= =?UTF-8?q?=20complete=20(DELIVERED:=20Gitea=20main=202bf53d8,=20GitHub=20?= =?UTF-8?q?main=204265ddb,=20PR=20#56=20+=20issue=20#49=20closed);=20plan?= =?UTF-8?q?=20doc=20archived=20to=20completed/=20+=20README=20+=20sequenci?= =?UTF-8?q?ng=20refs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../plans/1792862814701-next-up-sequencing.md | 17 ++- .kilo/plans/README.md | 3 +- ...7736-pr56-terminal-ntfy-failures-review.md | 119 ++++++++++++++++-- 3 files changed, 124 insertions(+), 15 deletions(-) rename .kilo/plans/{ => completed}/1793430417736-pr56-terminal-ntfy-failures-review.md (66%) diff --git a/.kilo/plans/1792862814701-next-up-sequencing.md b/.kilo/plans/1792862814701-next-up-sequencing.md index cb2086c..62288d0 100644 --- a/.kilo/plans/1792862814701-next-up-sequencing.md +++ b/.kilo/plans/1792862814701-next-up-sequencing.md @@ -455,9 +455,20 @@ Grounds: README section `### Pause notifications (ntfy)` → `### Notifications (ntfy)` (TOC anchor, opt-in sentence extended to terminal run outcomes, new terminal-notices paragraph, both `NTFY_HOST` descriptions); CONTRIBUTING - unchanged — S6 gate passed (user confirmed as-is) — Status: VALIDATED, S7 - next (dev-commits proposal, push `origin/dev`, ff-only promotion, GitHub - mirror, close PR #56 + issue #49). + unchanged — S6 gate passed (user confirmed as-is) — Status: VALIDATED, S7 + next (dev-commits proposal, push `origin/dev`, ff-only promotion, GitHub + mirror, close PR #56 + issue #49). +- PR #56 S7 complete 2026-10-31 (follow-up "continue pr 56" session): + DELIVERED — commits `da01d5b`/`32140f7`/`2bf53d8` (feat/test/doc) pushed to + `origin/dev`; Gitea `main` `a75861f` → `2bf53d8` (ff-only, user-gated, no + tag); GitHub `main` `ebf20bb` → `4265ddb` (sanitize-by-amend, public gate + green: no private paths at the tip, the #49 CHANGELOG entry present); PR + #56 closed without merge (comment 6544710707, 2026-10-31T14:29:39Z), issue + #49 closed completed (comment 6544711134, 2026-10-31T14:29:42Z); Discussion + summary text recorded in the review doc for manual web-UI post (write API + unavailable). Plan doc archived to `completed/` (this session). Next in the + batch-2 chain: **#55** (exact vault item identity, #48) — its S1 runs + against the settled `2bf53d8` tree. ## Explicitly not doing diff --git a/.kilo/plans/README.md b/.kilo/plans/README.md index da8a32d..740b50d 100644 --- a/.kilo/plans/README.md +++ b/.kilo/plans/README.md @@ -18,5 +18,4 @@ paths, so a moved file gains the `completed/` prefix. | `1792793307574-recipe-repair-llm-options.md` | Public service recipe recovery — LLM-assisted repair options analysis (pre-step); Stage 0 triage done 2026-10-24, all 7 services classified | | `1792819039884-recipe-doctor-implementation.md` | Stage 1: offline LLM recipe doctor (`billcollector doctor`) implementation plan + pilot (buhl, datev) — both pilots green 2026-10-24 (runs 16/18, downloads succeeded); Stage 1b in progress (2026-10-25): 4/7 green (buhl, datev, nuernberger, winsim), remaining 1und1 (in progress), freenet_mobilfunk (not started), kabeldeutschland (Option D re-authoring) | | `1792861675256-issue27-maintainer-commitments.md` | Issue #27 product-audit triage (closed 2026-10-24): maintainer-side commitments C1-C8 (F04, F09, F08 contract, F06 contract, F02, F03, F05, F10) with acceptance criteria, dependencies and proposed order | -| `1792862814701-next-up-sequencing.md` | Next-up sequencing decision (2026-10-24), superseded in part (2026-10-25): C1–C3 done and pushed; current primary next-up is the PR continuation order #30 → #31 → #33 → #32 → #34 (one pr-review chain at a time), interleaved with Stage 1b portal sessions (next: 1und1); PR #30 + #31 DELIVERED 2026-10-25, PR #33 DELIVERED 2026-10-26, PR #32 DELIVERED 2026-10-26 (Gitea `main` `bde713c` / GitHub `93bfcc8`), PR #34 DELIVERED 2026-10-27 (Gitea `main` `53a5dff` / GitHub `6bb3348`), PR #35 DELIVERED 2026-10-28 (Gitea `main` `75b885b` / GitHub `484048d`), PR #42 DELIVERED 2026-10-28 (Gitea `main` `2a6e462` / GitHub `c2fb7f8`), PR #39 DELIVERED 2026-10-28 (Gitea `main` `b693517` / GitHub `ad18f7a`), PR #40 DELIVERED 2026-10-29 (Gitea `main` `61defd3` / GitHub `f26b2a9`), PR #41 DELIVERED 2026-10-30 (Gitea `main` `74ba171` / GitHub `e83c8e2`), issue #38 closed — the planned chain #34 → #35 → #42 → #39 → #40 → #41 is complete; flowcool's archive-side contract proposal on #38 (2026-10-30) awaits a maintainer decision; PR #53 DELIVERED 2026-10-30 (Gitea `main` `fa1ade5` / GitHub `a24668a`), issue #46 closed, PR #54 DELIVERED 2026-10-31 (Gitea `main` `a75861f` / GitHub `ebf20bb`), issue #47 closed — batch-2 chain now at #56 (terminal ntfy failures, #49); PR #56 S1+S2 done 2026-10-31 (approve with M1 CHANGELOG placement, 4/5 files byte-identical from head, AWAITING-DECISION) | -| `1793430417736-pr56-terminal-ntfy-failures-review.md` | PR #56 review (terminal ntfy failures, issue #49) — S1+S2 done 2026-10-31: verdict approve with M1 CHANGELOG placement, 4/5 files apply byte-identical from head `95b11de` (orphaned-tip branch point `e83c8e2`), Status: AWAITING-DECISION | +| `1792862814701-next-up-sequencing.md` | Next-up sequencing decision (2026-10-24), superseded in part (2026-10-25): C1–C3 done and pushed; current primary next-up is the PR continuation order #30 → #31 → #33 → #32 → #34 (one pr-review chain at a time), interleaved with Stage 1b portal sessions (next: 1und1); PR #30 + #31 DELIVERED 2026-10-25, PR #33 DELIVERED 2026-10-26, PR #32 DELIVERED 2026-10-26 (Gitea `main` `bde713c` / GitHub `93bfcc8`), PR #34 DELIVERED 2026-10-27 (Gitea `main` `53a5dff` / GitHub `6bb3348`), PR #35 DELIVERED 2026-10-28 (Gitea `main` `75b885b` / GitHub `484048d`), PR #42 DELIVERED 2026-10-28 (Gitea `main` `2a6e462` / GitHub `c2fb7f8`), PR #39 DELIVERED 2026-10-28 (Gitea `main` `b693517` / GitHub `ad18f7a`), PR #40 DELIVERED 2026-10-29 (Gitea `main` `61defd3` / GitHub `f26b2a9`), PR #41 DELIVERED 2026-10-30 (Gitea `main` `74ba171` / GitHub `e83c8e2`), issue #38 closed — the planned chain #34 → #35 → #42 → #39 → #40 → #41 is complete; flowcool's archive-side contract proposal on #38 (2026-10-30) awaits a maintainer decision; PR #53 DELIVERED 2026-10-30 (Gitea `main` `fa1ade5` / GitHub `a24668a`), issue #46 closed, PR #54 DELIVERED 2026-10-31 (Gitea `main` `a75861f` / GitHub `ebf20bb`), issue #47 closed; PR #56 DELIVERED 2026-10-31 (Gitea `main` `2bf53d8` / GitHub `4265ddb`), issue #49 closed — batch-2 chain now at #55 (exact vault item identity, #48) | diff --git a/.kilo/plans/1793430417736-pr56-terminal-ntfy-failures-review.md b/.kilo/plans/completed/1793430417736-pr56-terminal-ntfy-failures-review.md similarity index 66% rename from .kilo/plans/1793430417736-pr56-terminal-ntfy-failures-review.md rename to .kilo/plans/completed/1793430417736-pr56-terminal-ntfy-failures-review.md index 38c4578..5817c46 100644 --- a/.kilo/plans/1793430417736-pr56-terminal-ntfy-failures-review.md +++ b/.kilo/plans/completed/1793430417736-pr56-terminal-ntfy-failures-review.md @@ -1,14 +1,11 @@ # PR #56 review: terminal daemon failures through opt-in ntfy (issue #49) -> Status: VALIDATED (2026-10-31) — S5 green on the exact dev tree: before/after -> proof (21 new tests fail on the pre-integration code, all 49 pass after), PR -> unit 49 OK / 15 subtests, full unit 665 passed / 10 skipped / 83 subtests -> (after the M2 test adaptation), regression VERDICT: PASS (7 scenarios / 8 -> PDFs, exit 0), ruff clean, bandit 0 findings on the changed app files; -> deployment validation recorded skip (not deployment-relevant), S5 gate -> passed; S6 docs confirmed (CHANGELOG M1 entry stands under [Unreleased] -> Added, sanitized; README ntfy section renamed + extended), S6 gate passed; -> ready for S7. +> Status: DELIVERED (2026-10-31) — S7 complete: dev commits `da01d5b` / +> `32140f7` / `2bf53d8` (feat/test/doc), Gitea `main` `2bf53d8` (ff-only from +> `a75861f`), GitHub `main` `4265ddb` (mirrored from `2bf53d8`); PR #56 closed +> without merge (comment 6544710707, 2026-10-31T14:29:39Z), issue #49 closed +> completed (comment 6544711134, 2026-10-31T14:29:42Z); Discussion summary +> text recorded below for manual web-UI post (write API unavailable). ## PR metadata @@ -221,4 +218,106 @@ Validated tree: dev `a96fabe` + the 5-file integration (working tree). ## Open questions -None at S2. +- None. All gates passed: S3 (2026-10-30, proposal confirmed as-is), S5 + (2026-10-31, validation results incl. the recorded deployment skip), S6 + (2026-10-31, CHANGELOG + README docs confirmed), S7 (2026-10-31, 3 commits + as proposed, promotion, mirror, public texts). Delivered 2026-10-31; plan + doc archived to `completed/`. + +## Execution record + +- S1+S2 (2026-10-30): fresh S1 review (PR #54 had just been DELIVERED). Head + fetched to `refs/pr/56` (`95b11de6...`); merge-base analysis untangled the + orphaned-tip base (`e83c8e2`, not an ancestor of GitHub `main` `ebf20bb`); + real content = 5 files +215/−1 against `e83c8e2` (the 103-file GitHub diff + is an orphaned-tip STRIP artifact). Pre-image vs Gitea `dev` `497dbdb`: + 4/5 byte-identical (clean hunks), CHANGELOG diverged (manual M1). Code review + per call site (notify.py `notify_run_result` + `_send_run_result`; + orchestrator.py 3 `_finalize_run`/orphan call sites; single-owner + no-double-send; `shutdown()` gathers the notifications). Verdict: approve + with M1 (CHANGELOG placement). Plan doc created (AWAITING-DECISION); S2 gate + passed — doc-only commit pushed to `origin/dev`. + +- S3 (2026-10-30): proposal (2 code files + 2 test files as-is from head, M1 + CHANGELOG `Unreleased → Added` placement, 3 commits) confirmed as-is. + +- S4 (2026-10-30): preflight green (clean tree, head unchanged `95b11de6`, dev + doc-only since S3). Applied the 5-file delta (4 as-is from head + M1 + hand-placed) on `dev`; tree left dirty for S5. Status: INTEGRATING. + +- S5 (2026-10-31): on the exact dev tree (repo root, `apps/.venv`): + - Before/after proof: PR code stashed (new tests kept) → the 21 new + notification tests fail (no terminal path); after → 49 OK. The PR's + 49-passed / 15-subtests claim reproduces. + - Full unit (first run): 2 failed, 663 passed — `test_bc_login.py:: + test_slow_login_notification_keeps_daemon_loop_responsive[finalize|orphan]` + pinned one `requests.post`/path; M2 applied (path-dependent count) → + re-run: 665 passed, 10 skipped, 83 subtests. + - Regression: VERDICT: PASS (7 scenarios / 8 PDFs, exit 0). + - ruff on all 5 changed files: clean; bandit on the changed app files: 0 + findings. + - Deployment validation: not deployment-relevant (no Dockerfile / + .dockerignore / BillCollector.sh / install_docker-image.sh / + deploy_remote.sh / INI keys / .env handling; NTFY_HOST + ntfy_topic + pre-date the PR) — recorded skip, user-confirmed at the S5 gate. + S5 gate passed 2026-10-31. Status: VALIDATED. + +- S6 (2026-10-31): CHANGELOG M1 entry confirmed as the first item of + `[Unreleased] → Added` (sanitized); README "Pause notifications (ntfy)" → + "Notifications (ntfy)" (TOC + body) with the terminal-notice sentence and the + two `NTFY_HOST` descriptions updated; CONTRIBUTING unchanged. S6 gate passed. + Status: VALIDATED. + +- S7 (2026-10-31): prerequisite re-checked — S5-validated content unchanged on + the dev tree (the 5 code/test files byte-identical to the S5 diffstat; only + the S5/S6 doc-only plan commit `e3844d4` + the S6 README doc moved since S5; + dev == origin/dev; deployment recorded-skip still covers the promoted tree). + - **Commits (dev-commits, 3 confirmed + push):** `da01d5b` `feat: notify + terminal daemon run failures through the opt-in ntfy topic (#49)` + (notify.py + orchestrator.py); `32140f7` `test: cover terminal run + notifications and adapt the slow-login post-count expectations (#49)` + (test_bc_notify.py, test_bc_orchestrator.py from head + test_bc_login.py + M2); `2bf53d8` `doc: changelog + README — terminal run notifications (#49)` + (CHANGELOG M1 + README). Pushed `origin/dev` `e3844d4..2bf53d8`. + - **Promotion (user-gated):** Gitea `main` `a75861f` → `2bf53d8` ff-only + (carries the 3 integration commits + 3 doc-only plan commits; the plan + docs are STRIP'd from the mirror); pushed. No tag (no release). Verified + `git diff main dev` empty, status clean. + - **Mirror:** `bash .private/sync_github.sh` → GitHub `main` `ebf20bb` → + `4265ddb` (sanitize-by-amend). Public gate green: no `.kilo`/`.vscode`/ + `.private`/`openwiki`/`AGENTS.md`/`.openwikiignore` at the tip; `.github` + holds only `ISSUE_TEMPLATE` (no `workflows/`); the #49 CHANGELOG entry + present on GitHub `main`. + - **PR #56 closed** (user confirmed the exact comment text): comment + 6544710707 (2026-10-31T14:29:39Z), closed without merge + 2026-10-31T14:29:40Z — the content reached GitHub via the mirror. + - **Issue #49 closed** (user confirmed the exact comment text): comment + 6544711134 (2026-10-31T14:29:42Z), closed completed + 2026-10-31T14:29:42Z. + - **Discussion summary (S7.6):** GitHub Discussion write API unavailable + from this environment (2026-10-25 finding); user chose manual web-UI + posting — text recorded below. + > **Public update: terminal daemon run notifications (issue #49, PR #56)** + > + > `main` now sends a fixed, account-free push for each terminal daemon run + > outcome — failure, interruption (including a run the daemon reclaims at + > startup), and attention timeout — through the existing opt-in ntfy topic; + > successful and cancelled runs stay silent. Contributed by @flowcool in + > PR #56. A failed push never changes the run outcome (best-effort by + > design). + > + > Maintainer-side adaptations: the CHANGELOG entry was placed under + > `Unreleased/Added` (the PR shipped it as a top-level section); + > `tests/test_bc_login.py`'s slow-login post-count expectation was made + > path-dependent (the `finalize`/`orphan` paths now also send the terminal + > notice); the README ntfy section was renamed to "Notifications (ntfy)" + > and extended. + > + > Validation on the promoted tree: 49 notification tests (15 subtests), 665 + > passed full unit suite, regression PASS (7 scenarios / 8 PDFs), + > ruff/bandit clean. + > + > User-facing note: this only pushes if you have set an ntfy topic in + > Settings; a fresh install sends nothing. + - **Archive:** this plan doc moved to `completed/`; README + sequencing + cross-references updated. From 53bed6b70515f9fb22a581c7e6744a1b188116f3 Mon Sep 17 00:00:00 2001 From: Stefan S Date: Sat, 31 Oct 2026 16:56:59 +0100 Subject: [PATCH 26/55] =?UTF-8?q?doc:=20plans=20=E2=80=94=20PR=20#55:=20S1?= =?UTF-8?q?+S2=20review=20(verdict=20approve=20with=20M1=20CHANGELOG=20pla?= =?UTF-8?q?cement;=206-file=20delta=20vs=20branch=20point=20e83c8e2,=20104?= =?UTF-8?q?-file=20GitHub=20diff=20is=20orphaned-tip=20STRIP=20artifact,?= =?UTF-8?q?=204/5=20byte-identical=20pre-images=20on=20dev=2019c4164=20+?= =?UTF-8?q?=20new=20test=20file=20clean=20add,=20no=20overlap=20with=20ope?= =?UTF-8?q?n=20#44/#45=20besides=20runner.py=20+=20CHANGELOG,=20deployment?= =?UTF-8?q?=20not=20deployment-relevant);=20plan=20doc=20+=20README=20+=20?= =?UTF-8?q?sequencing;=20status=20AWAITING-DECISION?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../plans/1792862814701-next-up-sequencing.md | 13 + ...1-pr55-vault-exact-item-identity-review.md | 236 ++++++++++++++++++ .kilo/plans/README.md | 3 +- 3 files changed, 251 insertions(+), 1 deletion(-) create mode 100644 .kilo/plans/1793461722991-pr55-vault-exact-item-identity-review.md diff --git a/.kilo/plans/1792862814701-next-up-sequencing.md b/.kilo/plans/1792862814701-next-up-sequencing.md index 62288d0..52f520d 100644 --- a/.kilo/plans/1792862814701-next-up-sequencing.md +++ b/.kilo/plans/1792862814701-next-up-sequencing.md @@ -469,6 +469,19 @@ Grounds: unavailable). Plan doc archived to `completed/` (this session). Next in the batch-2 chain: **#55** (exact vault item identity, #48) — its S1 runs against the settled `2bf53d8` tree. +- PR #55 S1+S2 done 2026-10-31 ("continue pr 55" session): review plan doc + `1793461722991-pr55-vault-exact-item-identity-review.md` — verdict approve + with maintainer changes (M1: the PR's `### Fixed` section is inserted + before `### Added`, breaking the Keep a Changelog order; the entry moves + to the top of the existing `[Unreleased] → ### Fixed` section); head + `397b0b6` stable, branch point = orphaned public tip `e83c8e2` (the 104-file + GitHub "Files changed" view is a STRIP artifact, real content 6 files + +177/−11), 4 of 5 existing files byte-identical pre-images on dev + `19c4164` (clean apply) + the new `tests/test_bc_vault_exact.py` a clean + add, only `CHANGELOG.md` manual; overlap with open #44/#45 limited to + `runner.py` (different area) + CHANGELOG; deployment validation recorded + not deployment-relevant — Status: AWAITING-DECISION, S2 gate pending + (user review of verdict + plan), S3 next. ## Explicitly not doing diff --git a/.kilo/plans/1793461722991-pr55-vault-exact-item-identity-review.md b/.kilo/plans/1793461722991-pr55-vault-exact-item-identity-review.md new file mode 100644 index 0000000..7a7e368 --- /dev/null +++ b/.kilo/plans/1793461722991-pr55-vault-exact-item-identity-review.md @@ -0,0 +1,236 @@ +# PR #55 review: exact vault item identity before credential reads (issue #48) + +> Status: AWAITING-DECISION (2026-10-31) — S1+S2 complete: verdict approve +> with one maintainer change (M1 CHANGELOG placement); 5/6 files clean +> pre-images on dev `19c4164` (4 byte-identical + 1 new-file clean add); S3 +> proposal next. + +## PR metadata + +- PR: s-t-e-f-a-n/BillCollector#55 "fix(vault): resolve exact item identity + before credential reads" (flowcool), head + `397b0b617aaa0eaaf174d0a93897b244e492fc06` (fetched to `refs/pr/55`, + matches `git ls-remote github`). Two commits in the fetched ref (orphaned + public tip `e83c8e2` + the fix commit `397b0b6`); the real content is + **6 files +177/−11** against the branch point `e83c8e2`. The GitHub "Files + changed" view (104 files, +290/−23395) is an orphaned-tip STRIP artifact: + the merge-base `5c8216f` carries 32 `openwiki/` files, the PR head has 0. + Base ref `main` @ `4265ddb` (current GitHub tip); the branch actually + starts at the orphaned public tip `e83c8e2` (GitHub `main` at the PR #41 + delivery; Gitea-side equivalent `74ba171`). +- Issue: #48 (flowcool, open) — `fetch_credentials()` retrieves + `/object/item/` and `/object/totp/`; the CLI's name resolution + can select a case-insensitive or ambiguous match, and the fresh-TOTP + lookup repeats name selection. Required: list/search, filter on the exact + full name, exactly one match, item and TOTP by the validated id; optional + TOTP and existing retries preserved; item naming unchanged; the pinned + local transport must allow the endpoint search query while continuing to + reject queried base URLs, userinfo, fragments, redirects and non-local + targets; errors/logs must not include names, query values, credentials or + raw endpoint bodies; no schema, recipe, control-protocol or exit-code + change. +- Created 2026-10-30T20:11:14Z; last updated 2026-10-31T13:59:37Z (before + this review's fetch). +- Local state at review time: Gitea `dev` = `origin/dev` = `19c4164`, Gitea + `main` = `origin/main` = `2bf53d8`, GitHub `main` = `4265ddb`. Working + tree clean. +- AI disclosure present: "AI-assisted focused port and synthetic + verification under my responsibility. Branch starts directly at public + main `e83c8e2`." + +## Verdict + +**Approve with maintainer changes** — M1 (CHANGELOG placement) only; all PR +code and tests apply as-is from head (byte-identical pre-images on `dev`, +the new test file is a clean add). + +## Verified facts + +1. Head stable: fetched `refs/pr/55` = `397b0b61...`; PR `updated_at` + 2026-10-31T13:59:37Z predates the fetch. S4 preflight re-checks via + `git ls-remote github refs/pull/55/head`. +2. Branch point: the fork starts at the orphaned public tip `e83c8e2` + (GitHub `main` at the PR #41 delivery; Gitea-side equivalent `74ba171`). + `e83c8e2` is **not** an ancestor of the current GitHub `main` `4265ddb` + (each mirror sync re-pushes the Gitea lineage plus one amended sanitized + tip, orphaning the previous public tip — same pattern as #53/#54/#56). + The GitHub "Files changed" view (104 files, +290/−23395; the + merge-base-`5c8216f`-to-`e83c8e2` part is 99 files −23384 of STRIPped + private content, `openwiki/` et al.) is an orphaned-tip STRIP artifact, + not PR content. Integration applies the 6-file delta against `e83c8e2` + only. +3. Issue #48 before-claim plausible and S5-verifiable: on the pre-integration + tree the new `test_case_variants_resolve_exact_id_and_totp_by_id` must + fail (the base `fetch_credentials` resolves by name, so the mocked + search response is misread and `user`/`password` come back None); after + integration the PR's stated command passes: 130 passed, 30 subtests. +4. Pre-image vs Gitea `dev` `19c4164`: `apps/billcollector/vault.py`, + `apps/billcollector/runner.py`, `apps/vault_transport.py` and + `tests/test_vault_transport.py` are **byte-identical** to the PR base → + hunks apply cleanly. `tests/test_bc_vault_exact.py` is a new file (absent + in both base and dev) → clean add. `CHANGELOG.md` diverges (PR #53/#54/ + #56 entries under `[Unreleased]`) → manual merge (M1). Dev files changed + since the Gitea branch point `74ba171`: `CHANGELOG.md`, `README.md`, + `apps/billcollector/notify.py`, `apps/billcollector/orchestrator.py`, + `apps/billcollector/scheduler.py`, `apps/billcollector/ui/tasks.py`, + `tests/test_bc_login.py`, `tests/test_bc_notify.py`, + `tests/test_bc_orchestrator.py`, `tests/test_bc_scheduler.py` — only + `CHANGELOG.md` overlaps the PR. +5. Code review (per file), behavior preserved vs changed: + - `vault.py` `fetch_credentials` (the core fix): search + `/list/object/items?search=` → `json.loads` → require + `success is True` + list `data.data` → **exact case-sensitive** name + match (`item.get("name") == item_name`) → **exactly one** match, else + `VaultAPIError("Expected one exact vault item match")` → `item_id` + validated against `[A-Za-z0-9_-]{1,128}` → item fetched **by id** + `/object/item/` → detail re-validated (`success is True`, + `data.id == item_id`, `data.name == item_name`) → TOTP fetched **by id** + (optional TOTP preserved: `get_totp` still treats 400 as "no TOTP"). + Return dict gains an additive `item_id`; the `user`/`password`/`otp`/ + `url` extraction is unchanged. Error handling: + `ValueError/TypeError/KeyError/AttributeError` from parsing → + `VaultAPIError("Invalid vault item response")` **`from None`** (no + names, queries, credentials or bodies leak); `VaultAPIError` itself + (e.g. a real 4xx from the search) is not in the catch tuple and + propagates with its status intact, as does the "exactly one match" + error. Retry behavior preserved (unchanged `vault_request` policy). + - `runner.py` (2 hunks): `_fresh_otp` now resolves the TOTP target as + `getattr(bcs, "vault_item_id", None) or bcs.vault_item_name` — the + run-start credential re-fetch targets the validated id when present + and falls back to the name (backwards-compatible with any context that + sets only the name). `_RunContext.__init__` stores + `self.bcs.vault_item_id = credentials.get("item_id")` alongside the + existing `vault_item_name`/`otp_fallback` attributes. + - `vault_transport.py` (1 hunk pair): `pinned_api_urls(url, *, + allow_query=True)` — the rejection clause changes from `parsed.query` + to `(parsed.query and not allow_query)`, so request URLs may carry a + query while every other authority check (scheme, hostname, userinfo, + fragment, port, local-only DNS resolution, sorted numeric targets) is + untouched. `pinned_api_url` (the configured-base validator) now calls + `pinned_api_urls(url, allow_query=False)` → **queried base URLs stay + rejected**. `vault_http_request` (the request path) uses the default + `allow_query=True`, preserving path+query in every pinned target + (`parsed._replace(netloc=...)`). Redirect refusal, Host-header ASCII + check and exception sanitization are unchanged. + - Test review: `tests/test_bc_vault_exact.py` (new, 115 lines, pytest — + consistent with the repo's 21 pytest test modules + `pytest.ini`): + case-variant exact selection + item/TOTP by id (asserts the exact + request URLs); missing/ambiguous/invalid-id (`../SECRET_TOKEN`) all + fail before TOTP with no "SECRET_TOKEN"/"Genevois" in the error; + changed item identity rejected; `_fresh_otp` uses the stored id; and a + real loopback `ThreadingHTTPServer` preflight test (status, escaped + name `CC Genevois & #/?`, exact item, optional TOTP 400 — only config + injected, DNS/transport/HTTP stay real). `tests/test_vault_transport.py` + +24: the search query is preserved in every pinned target (IPv4+IPv6), + more unsafe forms rejected for the base validator, and a query never + bypasses the authority/fragment/port/public-IP checks. Synthetic + markers only, no secrets, no network beyond 127.0.0.1. +6. Process check (CONTRIBUTING.md): issue referenced (#48); intended + behavior + verification stated in the PR body; proof shipped + (before/after pytest with expected counts + ruff/bandit/whitespace + results); AI assistance disclosed; no secrets in code/tests/artifacts; no + new network exposure (local pinning preserved; only the search query on + request URLs is new — URL-quoted and local-only); no shared-interface + change (no DB schema, control protocol, recipe contract or exit code — + confirmed in the diff; the `fetch_credentials` return dict gains an + additive `item_id` key read by its only caller). +7. Cross-PR: open PRs are #55 (this), #45 (Playwright browser cache), #44 + (recipe method-chain validation) — all flowcool, all base `main`. + Batch-2 chain `#53 → #54 → #56 → #55 → #45 → #44` (user-confirmed + 2026-10-30); #53/#54/#56 DELIVERED → #55 is next in order. File overlap + with #45/#44 besides `CHANGELOG.md`: `apps/billcollector/runner.py` + (their real content also touches runner.py; #55's two hunks sit in + `_fresh_otp`/`_RunContext`, a different area — S4/S5 pre-image re-checks + cover any movement). #55's `vault.py`, `vault_transport.py` and both + test files are unique to this PR. + +## Risks + +- Low overall: the change narrows credential resolution (name → validated + id) behind the existing transport pinning; new failure modes are + `VaultAPIError`s the runner already records as run failures; no behavior + change for a vault with exactly one correctly-cased item. +- New hard dependency on the `/list/object/items?search=` endpoint in the + operator's bw serve build; if absent, runs fail with a clear 4xx + `VaultAPIError` (final, no retry) instead of silently resolving the wrong + item — a deliberate behavior change per issue #48. +- M1 as-shipped breaks the Keep a Changelog ordering (a new `### Fixed` + section inserted before `### Added`) → manual placement fix required. + +## Integration plan (S4, on Gitea `dev`) + +| File | Action | Note | +|---|---|---| +| `apps/billcollector/vault.py` | apply from head | pre-image byte-identical to dev | +| `apps/billcollector/runner.py` | apply from head | pre-image byte-identical to dev | +| `apps/vault_transport.py` | apply from head | pre-image byte-identical to dev | +| `tests/test_bc_vault_exact.py` | apply from head (new file) | absent in base and dev | +| `tests/test_vault_transport.py` | apply from head | pre-image byte-identical to dev | +| `CHANGELOG.md` | **manual (M1)** | drop the author's misplaced `### Fixed` section (inserted before `### Added`); add the entry at the top of the existing `## [Unreleased] → ### Fixed` section in house style: `- Resolve one exact full vault item name, then fetch credentials and fresh TOTP by its validated id (#48). The pinned local HTTP transport preserves endpoint queries while continuing to refuse queried base URLs and unsafe authorities.` | + +Commits (dev-commits, one concern each, proposed in S7): +1. `fix: resolve the exact vault item id before credential reads (#48)` — + vault.py + runner.py + vault_transport.py +2. `test: cover exact vault item identity and the query-preserving transport + pin (#48)` — test_bc_vault_exact.py + test_vault_transport.py +3. `doc: changelog — exact vault item identity (#48)` — CHANGELOG.md (M1 + placement) + +## Validation plan (S5, on the exact dev tree, `apps/.venv/bin/python`) + +1. Before/after proof: the new `tests/test_bc_vault_exact.py` fails on the + pre-integration tree (name-based resolution), passes after (PR's stated + evidence: 130 passed, 30 subtests across + `test_vault_transport.py test_bc_vault_exact.py test_bc_login.py + test_bc_runner.py`; plus 18 passed, 14 subtests for + `test_vault_diagnostics.py test_bc_vault_settings.py`). +2. PR unit tests: `apps/.venv/bin/python -m pytest + tests/test_bc_vault_exact.py tests/test_vault_transport.py -v` → all + pass. +3. Full unit suite: `apps/.venv/bin/python -m pytest tests/ -q` → all pass + (watch for pinned behavior assertions in other modules, as in PR #56's + M2). +4. Full regression: `apps/.venv/bin/python tests/run_regression.py --ini + tests/bc_regression.ini` → exit 0, VERDICT: PASS (7 scenarios / 8 PDFs). +5. `ruff` + `bandit` on the changed files → nothing new beyond the known + baseline (PR claim: ruff + whitespace pass, bandit retains the same four + baseline findings). + +## Deployment validation + +Not deployment-relevant: no `Dockerfile`, `.dockerignore`, +`BillCollector.sh`, `install_docker-image.sh`, `deploy_remote.sh`, INI key or +`.env`-handling change (the change is pure Python inside the existing +runtime). To be recorded with the S5 run; user confirmation at the S5 gate. + +## Out of scope + +- PRs #45, #44 (later in the batch-2 chain). +- The bw serve search-endpoint availability (operator environment, not + shipped by this PR). +- Any vault-side data change (item naming unchanged per issue #48). + +## Open questions + +- None at S2: the S3 proposal follows (5 files as-is from head, M1 CHANGELOG + placement, 3 commits). + +## Execution record + +- S1 (2026-10-31, "continue pr 55" session): fresh S1 review — no plan doc + existed for #55 (plans jumped #54 → #56). Head fetched to `refs/pr/55` + (`397b0b61...`, matches `ls-remote`); PR metadata via the GitHub API + (issue #48, open PRs #55/#45/#44). Merge-base analysis untangled the + orphaned-tip base (`e83c8e2`, not an ancestor of GitHub `main` + `4265ddb`; merge-base `5c8216f`); real content = 6 files +177/−11 against + `e83c8e2` (the 104-file GitHub diff is an orphaned-tip STRIP artifact). + Pre-image vs Gitea `dev` `19c4164`: 4/5 existing files byte-identical + (clean hunks), the new test file a clean add, CHANGELOG diverged (manual + M1). Code review per file (vault.py exact-id resolution + error + sanitization, runner.py fresh-OTP id, vault_transport.py `allow_query` + scoping); process check PASS; cross-PR overlap limited to runner.py + + CHANGELOG. Verdict: approve with M1 (CHANGELOG placement). S1 gate passed + (user "proceed, yes", 2026-10-31). +- S2 (2026-10-31, same session): plan doc created (this file, + AWAITING-DECISION); README + sequencing cross-references updated; doc-only + commit pushed to `origin/dev`. S3 next. diff --git a/.kilo/plans/README.md b/.kilo/plans/README.md index 740b50d..0cb59e1 100644 --- a/.kilo/plans/README.md +++ b/.kilo/plans/README.md @@ -18,4 +18,5 @@ paths, so a moved file gains the `completed/` prefix. | `1792793307574-recipe-repair-llm-options.md` | Public service recipe recovery — LLM-assisted repair options analysis (pre-step); Stage 0 triage done 2026-10-24, all 7 services classified | | `1792819039884-recipe-doctor-implementation.md` | Stage 1: offline LLM recipe doctor (`billcollector doctor`) implementation plan + pilot (buhl, datev) — both pilots green 2026-10-24 (runs 16/18, downloads succeeded); Stage 1b in progress (2026-10-25): 4/7 green (buhl, datev, nuernberger, winsim), remaining 1und1 (in progress), freenet_mobilfunk (not started), kabeldeutschland (Option D re-authoring) | | `1792861675256-issue27-maintainer-commitments.md` | Issue #27 product-audit triage (closed 2026-10-24): maintainer-side commitments C1-C8 (F04, F09, F08 contract, F06 contract, F02, F03, F05, F10) with acceptance criteria, dependencies and proposed order | -| `1792862814701-next-up-sequencing.md` | Next-up sequencing decision (2026-10-24), superseded in part (2026-10-25): C1–C3 done and pushed; current primary next-up is the PR continuation order #30 → #31 → #33 → #32 → #34 (one pr-review chain at a time), interleaved with Stage 1b portal sessions (next: 1und1); PR #30 + #31 DELIVERED 2026-10-25, PR #33 DELIVERED 2026-10-26, PR #32 DELIVERED 2026-10-26 (Gitea `main` `bde713c` / GitHub `93bfcc8`), PR #34 DELIVERED 2026-10-27 (Gitea `main` `53a5dff` / GitHub `6bb3348`), PR #35 DELIVERED 2026-10-28 (Gitea `main` `75b885b` / GitHub `484048d`), PR #42 DELIVERED 2026-10-28 (Gitea `main` `2a6e462` / GitHub `c2fb7f8`), PR #39 DELIVERED 2026-10-28 (Gitea `main` `b693517` / GitHub `ad18f7a`), PR #40 DELIVERED 2026-10-29 (Gitea `main` `61defd3` / GitHub `f26b2a9`), PR #41 DELIVERED 2026-10-30 (Gitea `main` `74ba171` / GitHub `e83c8e2`), issue #38 closed — the planned chain #34 → #35 → #42 → #39 → #40 → #41 is complete; flowcool's archive-side contract proposal on #38 (2026-10-30) awaits a maintainer decision; PR #53 DELIVERED 2026-10-30 (Gitea `main` `fa1ade5` / GitHub `a24668a`), issue #46 closed, PR #54 DELIVERED 2026-10-31 (Gitea `main` `a75861f` / GitHub `ebf20bb`), issue #47 closed; PR #56 DELIVERED 2026-10-31 (Gitea `main` `2bf53d8` / GitHub `4265ddb`), issue #49 closed — batch-2 chain now at #55 (exact vault item identity, #48) | +| `1792862814701-next-up-sequencing.md` | Next-up sequencing decision (2026-10-24), superseded in part (2026-10-25): C1–C3 done and pushed; current primary next-up is the PR continuation order #30 → #31 → #33 → #32 → #34 (one pr-review chain at a time), interleaved with Stage 1b portal sessions (next: 1und1); PR #30 + #31 DELIVERED 2026-10-25, PR #33 DELIVERED 2026-10-26, PR #32 DELIVERED 2026-10-26 (Gitea `main` `bde713c` / GitHub `93bfcc8`), PR #34 DELIVERED 2026-10-27 (Gitea `main` `53a5dff` / GitHub `6bb3348`), PR #35 DELIVERED 2026-10-28 (Gitea `main` `75b885b` / GitHub `484048d`), PR #42 DELIVERED 2026-10-28 (Gitea `main` `2a6e462` / GitHub `c2fb7f8`), PR #39 DELIVERED 2026-10-28 (Gitea `main` `b693517` / GitHub `ad18f7a`), PR #40 DELIVERED 2026-10-29 (Gitea `main` `61defd3` / GitHub `f26b2a9`), PR #41 DELIVERED 2026-10-30 (Gitea `main` `74ba171` / GitHub `e83c8e2`), issue #38 closed — the planned chain #34 → #35 → #42 → #39 → #40 → #41 is complete; flowcool's archive-side contract proposal on #38 (2026-10-30) awaits a maintainer decision; PR #53 DELIVERED 2026-10-30 (Gitea `main` `fa1ade5` / GitHub `a24668a`), issue #46 closed, PR #54 DELIVERED 2026-10-31 (Gitea `main` `a75861f` / GitHub `ebf20bb`), issue #47 closed; PR #56 DELIVERED 2026-10-31 (Gitea `main` `2bf53d8` / GitHub `4265ddb`), issue #49 closed — batch-2 chain now at #55 (exact vault item identity, #48); PR #55 S1+S2 done 2026-10-31 (approve with M1 CHANGELOG placement, 5/6 files clean from head `397b0b6`, AWAITING-DECISION) | +| `1793461722991-pr55-vault-exact-item-identity-review.md` | PR #55 review (exact vault item identity, issue #48) — S1+S2 done 2026-10-31: verdict approve with M1 CHANGELOG placement, 4/5 existing files byte-identical from head `397b0b6` (orphaned-tip branch point `e83c8e2`) + new test file, Status: AWAITING-DECISION | From eedbed7993182e2b5a4822ee758b0df952e3f48d Mon Sep 17 00:00:00 2001 From: Stefan S Date: Sat, 31 Oct 2026 20:04:58 +0100 Subject: [PATCH 27/55] =?UTF-8?q?doc:=20plans=20=E2=80=94=20PR=20#55:=20S3?= =?UTF-8?q?=20confirmed=20as-is=20+=20S4=20integration=20applied=20(5/6=20?= =?UTF-8?q?files=20byte-identical=20to=20head=20397b0b61,=20M1=20CHANGELOG?= =?UTF-8?q?=20entry=20relocated=20to=20existing=20[Unreleased]=20Fixed=20s?= =?UTF-8?q?ection,=20tree=20left=20dirty=20for=20S5);=20status=20INTEGRATI?= =?UTF-8?q?NG?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...1-pr55-vault-exact-item-identity-review.md | 22 +++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/.kilo/plans/1793461722991-pr55-vault-exact-item-identity-review.md b/.kilo/plans/1793461722991-pr55-vault-exact-item-identity-review.md index 7a7e368..c81c4b6 100644 --- a/.kilo/plans/1793461722991-pr55-vault-exact-item-identity-review.md +++ b/.kilo/plans/1793461722991-pr55-vault-exact-item-identity-review.md @@ -1,9 +1,10 @@ # PR #55 review: exact vault item identity before credential reads (issue #48) -> Status: AWAITING-DECISION (2026-10-31) — S1+S2 complete: verdict approve -> with one maintainer change (M1 CHANGELOG placement); 5/6 files clean -> pre-images on dev `19c4164` (4 byte-identical + 1 new-file clean add); S3 -> proposal next. +> Status: INTEGRATING (2026-10-31) — S3 proposal confirmed as-is by the user; +> S4 applied: 5/6 files byte-identical to PR head `397b0b61` on dev +> (4 byte-identical pre-images + 1 new-file clean add), M1 CHANGELOG entry +> placed at the top of the existing `[Unreleased] → Fixed` section (author's +> misplaced block dropped); tree left dirty for S5 validation. ## PR metadata @@ -234,3 +235,16 @@ runtime). To be recorded with the S5 run; user confirmation at the S5 gate. - S2 (2026-10-31, same session): plan doc created (this file, AWAITING-DECISION); README + sequencing cross-references updated; doc-only commit pushed to `origin/dev`. S3 next. +- S3 (2026-10-31, "continue pr 55" session): proposal presented (5 files + as-is from head, M1 CHANGELOG placement, 3 commits); user confirmed as-is. +- S4 (2026-10-31, same session): preflight re-verified (dev = origin/dev = + `53bed6b` doc-only atop the reviewed `19c4164`; PR head still `397b0b61`; + 4 files byte-identical to the PR base, new test file absent). Applied the + 5 files from `refs/pr/55` to the worktree (worktree-only, index untouched) + and verified byte-identity: the 4 modified files share the PR delta's + patch-id `034ddf9b...`, the new file's blob hash `f5995304...` matches the + head. M1 applied: the author's misplaced `### Fixed` block (before + `### Added`) was NOT taken; the identical entry text was inserted at the + top of the existing `[Unreleased] → ### Fixed` section (above the #47 + entry). Working tree left dirty (5 code/test files + CHANGELOG) for S5. + Status: INTEGRATING. From da1aa40e631f3b84091c533c92282f7a56e664c0 Mon Sep 17 00:00:00 2001 From: Stefan S Date: Sat, 31 Oct 2026 20:39:57 +0100 Subject: [PATCH 28/55] =?UTF-8?q?doc:=20plans=20=E2=80=94=20PR=20#55:=20S5?= =?UTF-8?q?=20all=20green=20on=20exact=20dev=20tree=20(before/after=208/8,?= =?UTF-8?q?=20PR=20commands=20130/30=20+=2018/14=20exact,=20full=20unit=20?= =?UTF-8?q?675/90=20subtests,=20regression=20PASS,=20ruff=20clean,=20bandi?= =?UTF-8?q?t=20zero-new=20on=20production=20files),=20deployment=20skip=20?= =?UTF-8?q?recorded=20+=20user-confirmed;=20status=20VALIDATED,=20S6=20doc?= =?UTF-8?q?s=20drafted?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...1-pr55-vault-exact-item-identity-review.md | 62 ++++++++++++++++--- 1 file changed, 53 insertions(+), 9 deletions(-) diff --git a/.kilo/plans/1793461722991-pr55-vault-exact-item-identity-review.md b/.kilo/plans/1793461722991-pr55-vault-exact-item-identity-review.md index c81c4b6..4e629f4 100644 --- a/.kilo/plans/1793461722991-pr55-vault-exact-item-identity-review.md +++ b/.kilo/plans/1793461722991-pr55-vault-exact-item-identity-review.md @@ -1,10 +1,13 @@ # PR #55 review: exact vault item identity before credential reads (issue #48) -> Status: INTEGRATING (2026-10-31) — S3 proposal confirmed as-is by the user; -> S4 applied: 5/6 files byte-identical to PR head `397b0b61` on dev -> (4 byte-identical pre-images + 1 new-file clean add), M1 CHANGELOG entry -> placed at the top of the existing `[Unreleased] → Fixed` section (author's -> misplaced block dropped); tree left dirty for S5 validation. +> Status: VALIDATED (2026-10-31) — S5 all green on the exact dev tree: +> before/after proof (8/8 new tests fail pre-integration, pass after), PR +> commands exact (130 passed/30 subtests; 18/14), full unit 675 passed/10 +> skipped/90 subtests, regression VERDICT: PASS exit 0 (7 scenarios/8 PDFs), +> ruff clean, bandit zero-new on production files (test file: B101 asserts + +> B105 synthetic only); deployment skip recorded + user-confirmed at the S5 +> gate. S6 docs pending gate (CHANGELOG M1 in tree + vault_transport.md +> correction). ## PR metadata @@ -199,10 +202,12 @@ Commits (dev-commits, one concern each, proposed in S7): ## Deployment validation -Not deployment-relevant: no `Dockerfile`, `.dockerignore`, -`BillCollector.sh`, `install_docker-image.sh`, `deploy_remote.sh`, INI key or -`.env`-handling change (the change is pure Python inside the existing -runtime). To be recorded with the S5 run; user confirmation at the S5 gate. +**Recorded skip, user-confirmed at the S5 gate (2026-10-31).** Not +deployment-relevant: no `Dockerfile`, `.dockerignore`, `BillCollector.sh`, +`install_docker-image.sh`, `deploy_remote.sh`, INI key or `.env`-handling +change (the change is pure Python inside the existing runtime). Validated tree +basis for S7: dev `53bed6b` + the dirty integration (5 PR files + M1 +CHANGELOG); if the tree moves after S6, re-validate per the S7 prerequisite. ## Out of scope @@ -248,3 +253,42 @@ runtime). To be recorded with the S5 run; user confirmation at the S5 gate. top of the existing `[Unreleased] → ### Fixed` section (above the #47 entry). Working tree left dirty (5 code/test files + CHANGELOG) for S5. Status: INTEGRATING. +- S5 (2026-10-31, same session): all checks green on the exact dev tree + (`53bed6b` + dirty integration), from repo root with the venv: + - Before/after proof: with the 3 production files stashed to base, the new + `test_bc_vault_exact.py` fails **8/8** (name-based resolution); with the + integration, all pass. + - PR unit tests (2 files): 30 passed, 30 subtests. + - PR-stated 4-file command (`test_vault_transport.py test_bc_vault_exact.py + test_bc_login.py test_bc_runner.py`): **130 passed, 30 subtests** — exact + match to the PR's stated evidence. + - PR-stated 2-file command (`test_vault_diagnostics.py + test_bc_vault_settings.py`): 18 passed, 14 subtests — exact match. + - Full unit suite: 675 passed, 10 skipped, 90 subtests (Δ +10 vs PR #56's + 665 = exactly the 8 new exact-identity tests + 2 new transport tests). + - Full regression: `run_regression.py --ini tests/bc_regression.ini` → + exit 0, VERDICT: PASS (7 scenarios / 8 PDFs). + - ruff (F821, E4/E7/E9) on all 5 changed files: all checks passed. + - bandit per-file vs the pre-image: vault.py / vault_transport.py / + test_vault_transport.py 0 findings (unchanged); runner.py 1×B110 + (pre-existing, identical to base); new test file 11×B101 (asserts) + + 3×B105 (`synthetic-password`) — test-file baseline only, no secrets. + Production files: zero new. + - Deployment validation: **recorded skip** — not deployment-relevant (no + Dockerfile / .dockerignore / BillCollector.sh / install_docker-image.sh / + deploy_remote.sh / INI key / .env change; pure Python inside the existing + runtime). + S5 gate passed: user confirmed the deployment skip and proceeded + (2026-10-31). Status: VALIDATED. +- S6 (2026-10-31, same session): CHANGELOG M1 entry stands as-is in the + `[Unreleased] → Fixed` section (sanitized, public-safe). No README or + CONTRIBUTING change: the operator-visible contract (`BW_API_URL`, item + naming, optional TOTP) is unchanged, only the internal resolution path + narrows. `doc/vault_transport.md` corrected (two edits, in tree, pending + the S6 gate): (a) the "URL credentials, query strings, fragments, HTTPS and + public targets are refused" line now distinguishes the base URL (still + refuses query strings) from request URLs (endpoint query preserved, e.g. the + item search query); (b) the stale "Exact item resolution and stable account + identity are separate work" + "The diagnostics fix in upstream PR #11 is + also needed" sentences (PR #11 already delivered; #48 now landed) replaced + with the exact-id resolution description. S6 gate pending. From 47e1c81f4b384140fb502c6129bcc0ba9193a3c5 Mon Sep 17 00:00:00 2001 From: Stefan S Date: Sat, 31 Oct 2026 20:43:08 +0100 Subject: [PATCH 29/55] =?UTF-8?q?doc:=20plans=20=E2=80=94=20PR=20#55:=20S6?= =?UTF-8?q?=20docs=20confirmed=20(CHANGELOG=20M1=20stands,=20vault=5Ftrans?= =?UTF-8?q?port.md=20correction,=20no=20README/CONTRIBUTING=20change);=20s?= =?UTF-8?q?tatus=20VALIDATED,=20S7=20next?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...1-pr55-vault-exact-item-identity-review.md | 20 ++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/.kilo/plans/1793461722991-pr55-vault-exact-item-identity-review.md b/.kilo/plans/1793461722991-pr55-vault-exact-item-identity-review.md index 4e629f4..60c08fb 100644 --- a/.kilo/plans/1793461722991-pr55-vault-exact-item-identity-review.md +++ b/.kilo/plans/1793461722991-pr55-vault-exact-item-identity-review.md @@ -6,8 +6,8 @@ > skipped/90 subtests, regression VERDICT: PASS exit 0 (7 scenarios/8 PDFs), > ruff clean, bandit zero-new on production files (test file: B101 asserts + > B105 synthetic only); deployment skip recorded + user-confirmed at the S5 -> gate. S6 docs pending gate (CHANGELOG M1 in tree + vault_transport.md -> correction). +> gate; S6 docs confirmed (CHANGELOG M1 stands + vault_transport.md +> correction, no README/CONTRIBUTING change). S7 next. ## PR metadata @@ -177,8 +177,9 @@ Commits (dev-commits, one concern each, proposed in S7): vault.py + runner.py + vault_transport.py 2. `test: cover exact vault item identity and the query-preserving transport pin (#48)` — test_bc_vault_exact.py + test_vault_transport.py -3. `doc: changelog — exact vault item identity (#48)` — CHANGELOG.md (M1 - placement) +3. `doc: changelog + vault transport doc — exact vault item identity (#48)` + — CHANGELOG.md (M1 placement) + doc/vault_transport.md (S6 correction: + base/request query distinction, exact-id resolution description) ## Validation plan (S5, on the exact dev tree, `apps/.venv/bin/python`) @@ -218,8 +219,9 @@ CHANGELOG); if the tree moves after S6, re-validate per the S7 prerequisite. ## Open questions -- None at S2: the S3 proposal follows (5 files as-is from head, M1 CHANGELOG - placement, 3 commits). +- None: S3 confirmed, S4 applied, S5 validated (deployment skip + user-confirmed), S6 docs confirmed. S7 (commit + push, promote, mirror, + close) is the only remaining stage. ## Execution record @@ -291,4 +293,8 @@ CHANGELOG); if the tree moves after S6, re-validate per the S7 prerequisite. item search query); (b) the stale "Exact item resolution and stable account identity are separate work" + "The diagnostics fix in upstream PR #11 is also needed" sentences (PR #11 already delivered; #48 now landed) replaced - with the exact-id resolution description. S6 gate pending. + with the exact-id resolution description. S6 gate passed: user confirmed + the docs (2026-10-31). S7 next: dev-commits (3 commits: code / test / + doc — the doc commit now covers CHANGELOG.md + doc/vault_transport.md), + push origin/dev, ff-only promote to main, sync_github.sh mirror, PR + + issue close comments per the publication routine. From f4f1a05faa8830b13e362e7d79c62138010683e7 Mon Sep 17 00:00:00 2001 From: Stefan S Date: Sat, 31 Oct 2026 20:52:41 +0100 Subject: [PATCH 30/55] fix: resolve the exact vault item id before credential reads (#48) --- apps/billcollector/runner.py | 4 +++- apps/billcollector/vault.py | 28 +++++++++++++++++++++++++--- apps/vault_transport.py | 11 ++++++----- 3 files changed, 34 insertions(+), 9 deletions(-) diff --git a/apps/billcollector/runner.py b/apps/billcollector/runner.py index c5b4f55..5bd6e11 100644 --- a/apps/billcollector/runner.py +++ b/apps/billcollector/runner.py @@ -166,7 +166,8 @@ def _fresh_otp(bcs): run-start credential when the item has no TOTP or the fetch fails. """ try: - totp_item = get_totp(f"{bcs.api_url}/object/totp/{bcs.vault_item_name}") + item = getattr(bcs, "vault_item_id", None) or bcs.vault_item_name + totp_item = get_totp(f"{bcs.api_url}/object/totp/{item}") except VaultAPIError as e: logger.warning("OTP re-fetch failed (%s); using run-start credential", failure_record("get_totp", e)) @@ -207,6 +208,7 @@ def __init__(self, task, run_id, db, debug, credentials, api_url): # Extra attributes used by _fresh_otp at substitution time. self.bcs.api_url = api_url self.bcs.vault_item_name = f"{task['service']} {task['user']}".strip() + self.bcs.vault_item_id = credentials.get("item_id") self.bcs.otp_fallback = credentials["otp"] # Pause state (C5): the pause loop and the Copilot context handler # read these off the context. diff --git a/apps/billcollector/vault.py b/apps/billcollector/vault.py index df08254..ffea8cc 100644 --- a/apps/billcollector/vault.py +++ b/apps/billcollector/vault.py @@ -16,7 +16,9 @@ import json import logging +import re import time +from urllib.parse import quote import requests from flatten_json import flatten @@ -158,15 +160,35 @@ def fetch_credentials(api_url, service, user): """Resolve the portal credentials of one task from the vault. The item name is the name-based link `` `` (decision 7). - Returns ``{"user", "password", "otp", "url"}``; ``otp`` is None when the + Returns ``{"item_id", "user", "password", "otp", "url"}``; ``otp`` is None when the item has no TOTP entry, ``url`` is the item's first login URI (informational - the recipe's goto step holds the URL the engine visits). Raises VaultAPIError on any real failure. """ item_name = f"{service} {user}".strip() - item = get_json(f"{api_url}/object/item/{item_name}") - totp_item = get_totp(f"{api_url}/object/totp/{item_name}") + try: + payload = json.loads(get_json( + f"{api_url}/list/object/items?search={quote(item_name, safe='')}")) + items = payload["data"]["data"] + if payload.get("success") is not True or not isinstance(items, list): + raise ValueError + matches = [item for item in items + if isinstance(item, dict) and item.get("name") == item_name] + if len(matches) != 1: + raise VaultAPIError("Expected one exact vault item match") + item_id = matches[0]["id"] + if not isinstance(item_id, str) or not re.fullmatch(r"[A-Za-z0-9_-]{1,128}", item_id): + raise ValueError + item = get_json(f"{api_url}/object/item/{item_id}") + detail = json.loads(item) + if (detail.get("success") is not True or detail["data"].get("id") != item_id + or detail["data"].get("name") != item_name): + raise ValueError + except (ValueError, TypeError, KeyError, AttributeError): + raise VaultAPIError("Invalid vault item response") from None + totp_item = get_totp(f"{api_url}/object/totp/{item_id}") return { + "item_id": item_id, "user": get_json_property_value(item, "data_login_username"), "password": get_json_property_value(item, "data_login_password"), "otp": get_json_property_value(totp_item, "data_data") if totp_item is not None else None, diff --git a/apps/vault_transport.py b/apps/vault_transport.py index f062883..8c084f2 100644 --- a/apps/vault_transport.py +++ b/apps/vault_transport.py @@ -21,12 +21,13 @@ class VaultTransportError(ValueError): """Invalid API configuration or an unsafe response; do not retry it.""" -def pinned_api_urls(url): - """Return all vetted numeric local targets and the original Host header.""" +def pinned_api_urls(url, *, allow_query=True): + """Vet the HTTP authority, preserving the request path and query when pinning.""" try: parsed = urlsplit(url) if (parsed.scheme != "http" or not parsed.hostname or parsed.username is not None - or parsed.password is not None or parsed.query or parsed.fragment): + or parsed.password is not None or parsed.fragment + or (parsed.query and not allow_query)): raise ValueError("invalid API URL") port = parsed.port if parsed.port is not None else 80 if port == 0: @@ -51,10 +52,10 @@ def pinned_api_urls(url): def pinned_api_url(url, attempts=3, backoff=2): - """Validate preflight DNS with the caller's request retry policy.""" + """Validate a query-free base API URL with the caller's DNS retry policy.""" for attempt in range(1, attempts + 1): try: - targets, host = pinned_api_urls(url) + targets, host = pinned_api_urls(url, allow_query=False) return targets[0], host except requests.exceptions.ConnectionError: if attempt == attempts: From 79d22b1e043a91cba56d6aa59eadac485781c3d9 Mon Sep 17 00:00:00 2001 From: Stefan S Date: Sat, 31 Oct 2026 20:52:41 +0100 Subject: [PATCH 31/55] test: cover exact vault item identity and the query-preserving transport pin (#48) --- tests/test_bc_vault_exact.py | 115 ++++++++++++++++++++++++++++++++++ tests/test_vault_transport.py | 24 ++++++- 2 files changed, 137 insertions(+), 2 deletions(-) create mode 100644 tests/test_bc_vault_exact.py diff --git a/tests/test_bc_vault_exact.py b/tests/test_bc_vault_exact.py new file mode 100644 index 0000000..f599530 --- /dev/null +++ b/tests/test_bc_vault_exact.py @@ -0,0 +1,115 @@ +"""Exact-name vault lookup avoids case-insensitive CLI ambiguity.""" +import json +from unittest import mock + +import pytest + +from billcollector import vault + + +def response(data): + return json.dumps({"success": True, "data": data}) + + +def test_case_variants_resolve_exact_id_and_totp_by_id(): + chosen = {"id": "exact-id", "name": "CC Genevois"} + detail = {**chosen, "login": {"username": "synthetic-user", "password": "synthetic-password", + "uris": [{"uri": "https://portal.invalid"}]}} + with mock.patch.object(vault, "get_json", side_effect=[ + response({"data": [{"id": "other-id", "name": "CC genevois"}, chosen]}), + response(detail)]) as get, mock.patch.object(vault, "get_totp", return_value=None) as totp: + result = vault.fetch_credentials("http://127.0.0.1:8087", "CC", "Genevois") + assert result["item_id"] == "exact-id" + assert result["user"] == "synthetic-user" + assert result["password"] == "synthetic-password" + assert result["otp"] is None + assert get.call_args_list == [ + mock.call("http://127.0.0.1:8087/list/object/items?search=CC%20Genevois"), + mock.call("http://127.0.0.1:8087/object/item/exact-id")] + totp.assert_called_once_with("http://127.0.0.1:8087/object/totp/exact-id") + + +@pytest.mark.parametrize("items", [[], [{"id": "one", "name": "CC genevois"}], + [{"id": "a", "name": "CC Genevois"}, {"id": "b", "name": "CC Genevois"}], + [{"id": "../SECRET_TOKEN", "name": "CC Genevois"}]]) +def test_missing_ambiguous_or_invalid_id_fails_without_totp_or_secret_output(items): + with mock.patch.object(vault, "get_json", return_value=response({"data": items})) as get, \ + mock.patch.object(vault, "get_totp") as totp: + with pytest.raises(vault.VaultAPIError) as error: + vault.fetch_credentials("http://127.0.0.1:8087", "CC", "Genevois") + assert "SECRET_TOKEN" not in str(error.value) + assert "Genevois" not in str(error.value) + assert get.call_count == 1 + totp.assert_not_called() + + +def test_changed_item_identity_is_rejected(): + with mock.patch.object(vault, "get_json", side_effect=[ + response({"data": [{"id": "id", "name": "CC Genevois"}]}), + response({"id": "id", "name": "other"})]), mock.patch.object(vault, "get_totp") as totp: + with pytest.raises(vault.VaultAPIError): + vault.fetch_credentials("http://127.0.0.1:8087", "CC", "Genevois") + totp.assert_not_called() + + +def test_fresh_totp_uses_exact_id_after_initial_lookup(): + from types import SimpleNamespace + from billcollector import runner + bcs = SimpleNamespace(api_url="http://127.0.0.1:8087", vault_item_id="exact-id", + vault_item_name="CC Genevois", otp_fallback=None) + with mock.patch.object(runner, "get_totp", return_value=response({"data":"synthetic-otp"})) as get: + assert runner._fresh_otp(bcs) == "synthetic-otp" + get.assert_called_once_with("http://127.0.0.1:8087/object/totp/exact-id") + + +def test_daemon_preflight_fetches_credentials_through_real_transport(monkeypatch): + """Exercise status, exact lookup, item and optional TOTP over real HTTP.""" + import threading + from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + from billcollector import runner + + received = [] + item_name = "CC Genevois & #/?" + item = {"id": "exact-id", "name": item_name, "login": { + "username": "synthetic-user", "password": "synthetic-password", + "uris": [{"uri": "https://portal.invalid"}]}} + search_path = "/list/object/items?search=CC%20Genevois%20%26%20%23%2F%3F" + routes = { + "/status": (200, {"template": {"status": "unlocked"}}), + search_path: (200, {"data": [{"id": "other-id", "name": item_name.lower()}, item]}), + "/object/item/exact-id": (200, item), + "/object/totp/exact-id": (400, {}), + } + + class Handler(BaseHTTPRequestHandler): + def do_GET(self): + received.append((self.path, self.headers.get("Host"))) + status, data = routes.get(self.path, (404, {})) + body = response(data).encode() + self.send_response(status) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def log_message(self, *args): + pass + + server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + api_url = f"http://127.0.0.1:{server.server_port}" + # Only inject configuration; preflight, DNS, transport and HTTP stay real. + monkeypatch.setattr(runner.config, "vault_settings", lambda db: (api_url, "")) + monkeypatch.setenv("BW_API_URL", api_url) + monkeypatch.setenv("BW_API_HOST", "") + try: + result = runner._vault_preflight({"service": "CC", "user": "Genevois & #/?"}, None) + finally: + server.shutdown() + server.server_close() + thread.join() + assert result == {"item_id": "exact-id", "user": "synthetic-user", + "password": "synthetic-password", "otp": None, + "url": "https://portal.invalid"} + assert received == [(path, f"127.0.0.1:{server.server_port}") for path in routes] diff --git a/tests/test_vault_transport.py b/tests/test_vault_transport.py index 5d5f749..8f1fba3 100644 --- a/tests/test_vault_transport.py +++ b/tests/test_vault_transport.py @@ -1,4 +1,4 @@ -"""Synthetic vault transport checks: all DNS and HTTP operations are mocked.""" +"""Synthetic vault transport checks with mocked DNS and loopback HTTP fixtures.""" import importlib from contextlib import nullcontext @@ -40,6 +40,13 @@ def test_local_ranges_and_default_port(self): self.assertEqual(host, "bw.local") resolve.assert_called_once_with("bw.local", 80, type=socket.SOCK_STREAM) + def test_list_search_query_is_preserved_in_every_pinned_target(self): + suffix = "/list/object/items?search=CC%20Genevois%26%23%2F%3F" + with patch.object(transport.socket, "getaddrinfo", return_value=dns("127.0.0.1", "::1")): + targets, host = transport.pinned_api_urls(f"http://bw.local:8087{suffix}") + self.assertEqual(targets, [f"http://127.0.0.1:8087{suffix}", f"http://[::1]:8087{suffix}"]) + self.assertEqual(host, "bw.local:8087") + def test_public_mixed_empty_and_link_local_answers_are_rejected(self): for answers in (dns("8.8.8.8"), dns("127.0.0.1", "8.8.8.8"), [], dns("169.254.169.254")): with self.subTest(answers=answers), \ @@ -51,13 +58,26 @@ def test_public_mixed_empty_and_link_local_answers_are_rejected(self): def test_unsafe_url_forms_are_rejected_before_dns(self): for url in (None, "https://bw.local/status", "http://user:password@bw.local/", - "http://bw.local:0/", "http://bw.local:99999/", "http://bw.local/?token=synthetic", + "http://bw.local:0/", "http://bw.local:99999/", "http://@bw.local/", + "http://user@bw.local/", "http://bw.local:invalid/", + "http://bw.local/?token=synthetic", "http://bw.local/#synthetic", "file:///synthetic", "not-a-url"): with self.subTest(url=url), patch.object(transport.socket, "getaddrinfo") as resolve: with self.assertRaises(transport.VaultTransportError): transport.pinned_api_url(url) resolve.assert_not_called() + def test_request_query_does_not_bypass_authority_or_fragment_checks(self): + for url in ("https://bw.local/list?search=x", "http://user@bw.local/list?search=x", + "http://bw.local:0/list?search=x", "http://bw.local/list?search=x#fragment"): + with self.subTest(url=url), patch.object(transport.socket, "getaddrinfo") as resolve: + with self.assertRaises(transport.VaultTransportError): + transport.pinned_api_urls(url) + resolve.assert_not_called() + with patch.object(transport.socket, "getaddrinfo", return_value=dns("8.8.8.8")): + with self.assertRaises(transport.VaultTransportError): + transport.pinned_api_urls("http://public.invalid/list?search=x") + def test_dns_failure_is_a_bounded_configuration_error(self): with patch.object(transport.socket, "getaddrinfo", side_effect=OSError("SYNTHETIC_PRIVATE_MARKER")): with self.assertRaises(transport.VaultTransportError) as error: From 3c950d7a9f80b34269800d4d524b5ef60a532c78 Mon Sep 17 00:00:00 2001 From: Stefan S Date: Sat, 31 Oct 2026 20:52:41 +0100 Subject: [PATCH 32/55] =?UTF-8?q?doc:=20changelog=20+=20vault=20transport?= =?UTF-8?q?=20doc=20=E2=80=94=20exact=20vault=20item=20identity=20(#48)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .github/workflows/openwiki-update.yml | 89 - .kilo/kilo.jsonc | 3 - ...1792793307574-recipe-repair-llm-options.md | 171 -- ...2819039884-recipe-doctor-implementation.md | 402 ----- ...61675256-issue27-maintainer-commitments.md | 243 --- .../plans/1792862814701-next-up-sequencing.md | 499 ------ ...1-pr55-vault-exact-item-identity-review.md | 300 ---- .kilo/plans/README.md | 22 - ...9661685-billcollector-daemon-webui-plan.md | 331 ---- ...74061-billcollector-nicegui-daemon-plan.md | 1463 ----------------- ...1790371260966-ubuntu-2404-wsl-migration.md | 245 --- ...0374270-ubuntu-2404-migration-remainder.md | 135 -- .../1790462762560-logger-replace-prints.md | 168 -- .../1790465908381-nicegui-manual-run-ui.md | 129 -- ...790468502492-gui-concept-dashboard-spec.md | 330 ---- ...0539917809-gitea-github-branch-strategy.md | 194 --- .../1790625192832-flowcool-fork-extensions.md | 343 ---- ...58680-local-regression-test-web-service.md | 702 -------- .../1790891025465-deploy-skills-plan.md | 314 ---- .../1790893135799-plan-files-audit.md | 55 - ...1791145017911-single-dockerfile-nonroot.md | 229 --- ...237156540-pr11-vault-diagnostics-review.md | 171 -- .../1791396615979-pr-review-skill.md | 297 ---- ...791399933135-pr12-docker-context-review.md | 332 ---- ...91407447879-deployment-layer-reuse-plan.md | 159 -- ...91411641871-pr13-vault-transport-review.md | 542 ------ ...33220-pr14-16-flowcool-triage-checklist.md | 189 --- .../1791497917898-github-issue-replies.md | 156 -- ...91620921558-chromium-warm-profile-crash.md | 482 ------ .../1791705823133-m2-ui-core-plan.md | 792 --------- .../1791829791901-m2-d64-exit-pass-guide.md | 353 ---- ...1998985158-m3-interactive-sessions-plan.md | 728 -------- .../1792126455141-m3-wave1-exit-pass-guide.md | 638 ------- ...245102-plan-audit-and-concessions-merge.md | 274 --- ...335541451-private-ai-artifacts-boundary.md | 186 --- ...792337230912-openwiki-boundary-followup.md | 165 -- .../1792360635843-m3-wave1-closeout.md | 137 -- .../1792384312843-wave1-nas-deploy.md | 299 ---- ...-pr17-browser-evidence-redaction-review.md | 186 --- ...34-pr17-evidence-selfhealing-refinement.md | 401 ----- .../1792530797544-plan-cleanup-finalize.md | 102 -- .../1792533823395-m4-adoption-plan.md | 292 ---- .../1792533823395-m4-exit-pass-guide.md | 261 --- ...1792622454541-pr23-residual-logs-review.md | 256 --- .../1792708132014-pr22-safe-login-review.md | 443 ----- ...84993002-pr19-bounded-collection-review.md | 636 ------- ...1475-pr30-migration-cron-example-review.md | 144 -- ...1-onboarding-switch-independence-review.md | 251 --- ...629-pr33-download-crash-recovery-review.md | 371 ----- ...3044794581-pr32-chromium-sandbox-review.md | 611 ------- ...793124431799-open-issues-plans-20261027.md | 169 -- ...793127186466-pr34-date-variables-review.md | 599 ------- ...38983439-pr35-contributor-guides-review.md | 492 ------ ...-pr42-daemon-runtime-directories-review.md | 333 ---- ...75507-pr39-input-triggered-login-review.md | 398 ----- ...064786-pr40-replacement-calendar-review.md | 347 ---- ...93313075441-pr41-notice-identity-review.md | 470 ------ ...1793397084822-pr53-cron-weekdays-review.md | 259 --- ...81422-pr54-persistent-job-resync-review.md | 308 ---- ...7736-pr56-terminal-ntfy-failures-review.md | 323 ---- .kilo/skills/deploy/SKILL.md | 164 -- .kilo/skills/dev-commits/SKILL.md | 36 - .kilo/skills/pr-review/SKILL.md | 244 --- .openwikiignore | 8 - .private/sync_github.sh | 48 - .vscode/launch.json | 18 - .vscode/settings.json | 5 - AGENTS.md | 46 - CHANGELOG.md | 4 + doc/vault_transport.md | 15 +- openwiki/.claims/architecture/overview.json | 442 ----- openwiki/.claims/architecture/roadmap.json | 226 --- .../architecture/shared-interfaces.json | 320 ---- openwiki/.claims/concepts/recipes.json | 232 --- .../.claims/concepts/vault-transport.json | 202 --- openwiki/.claims/operations/deployment.json | 244 --- openwiki/.claims/quickstart.json | 202 --- openwiki/.claims/testing/verification.json | 190 --- openwiki/.claims/workflows/run-lifecycle.json | 276 ---- openwiki/.last-update.json | 8 - openwiki/.page-manifest.json | 68 - openwiki/INSTRUCTIONS.md | 1 - openwiki/architecture/index.md | 5 - openwiki/architecture/overview.md | 174 -- openwiki/architecture/roadmap.md | 78 - openwiki/architecture/shared-interfaces.md | 264 --- openwiki/concepts/index.md | 4 - openwiki/concepts/recipes.md | 249 --- openwiki/concepts/vault-transport.md | 94 -- openwiki/guides/index.md | 1 - openwiki/index.md | 18 - openwiki/modules/index.md | 1 - openwiki/operations/deployment.md | 238 --- openwiki/operations/index.md | 3 - openwiki/quickstart.md | 132 -- openwiki/testing/index.md | 3 - openwiki/testing/verification.md | 196 --- openwiki/ui/index.md | 1 - openwiki/ui/web-ui.md | 133 -- openwiki/workflows/index.md | 3 - openwiki/workflows/pause-repair.md | 104 -- openwiki/workflows/run-lifecycle.md | 217 --- 102 files changed, 14 insertions(+), 24822 deletions(-) delete mode 100644 .github/workflows/openwiki-update.yml delete mode 100644 .kilo/kilo.jsonc delete mode 100644 .kilo/plans/1792793307574-recipe-repair-llm-options.md delete mode 100644 .kilo/plans/1792819039884-recipe-doctor-implementation.md delete mode 100644 .kilo/plans/1792861675256-issue27-maintainer-commitments.md delete mode 100644 .kilo/plans/1792862814701-next-up-sequencing.md delete mode 100644 .kilo/plans/1793461722991-pr55-vault-exact-item-identity-review.md delete mode 100644 .kilo/plans/README.md delete mode 100644 .kilo/plans/completed/1790289661685-billcollector-daemon-webui-plan.md delete mode 100644 .kilo/plans/completed/1790364274061-billcollector-nicegui-daemon-plan.md delete mode 100644 .kilo/plans/completed/1790371260966-ubuntu-2404-wsl-migration.md delete mode 100644 .kilo/plans/completed/1790380374270-ubuntu-2404-migration-remainder.md delete mode 100644 .kilo/plans/completed/1790462762560-logger-replace-prints.md delete mode 100644 .kilo/plans/completed/1790465908381-nicegui-manual-run-ui.md delete mode 100644 .kilo/plans/completed/1790468502492-gui-concept-dashboard-spec.md delete mode 100644 .kilo/plans/completed/1790539917809-gitea-github-branch-strategy.md delete mode 100644 .kilo/plans/completed/1790625192832-flowcool-fork-extensions.md delete mode 100644 .kilo/plans/completed/1790714158680-local-regression-test-web-service.md delete mode 100644 .kilo/plans/completed/1790891025465-deploy-skills-plan.md delete mode 100644 .kilo/plans/completed/1790893135799-plan-files-audit.md delete mode 100644 .kilo/plans/completed/1791145017911-single-dockerfile-nonroot.md delete mode 100644 .kilo/plans/completed/1791237156540-pr11-vault-diagnostics-review.md delete mode 100644 .kilo/plans/completed/1791396615979-pr-review-skill.md delete mode 100644 .kilo/plans/completed/1791399933135-pr12-docker-context-review.md delete mode 100644 .kilo/plans/completed/1791407447879-deployment-layer-reuse-plan.md delete mode 100644 .kilo/plans/completed/1791411641871-pr13-vault-transport-review.md delete mode 100644 .kilo/plans/completed/1791489933220-pr14-16-flowcool-triage-checklist.md delete mode 100644 .kilo/plans/completed/1791497917898-github-issue-replies.md delete mode 100644 .kilo/plans/completed/1791620921558-chromium-warm-profile-crash.md delete mode 100644 .kilo/plans/completed/1791705823133-m2-ui-core-plan.md delete mode 100644 .kilo/plans/completed/1791829791901-m2-d64-exit-pass-guide.md delete mode 100644 .kilo/plans/completed/1791998985158-m3-interactive-sessions-plan.md delete mode 100644 .kilo/plans/completed/1792126455141-m3-wave1-exit-pass-guide.md delete mode 100644 .kilo/plans/completed/1792189245102-plan-audit-and-concessions-merge.md delete mode 100644 .kilo/plans/completed/1792335541451-private-ai-artifacts-boundary.md delete mode 100644 .kilo/plans/completed/1792337230912-openwiki-boundary-followup.md delete mode 100644 .kilo/plans/completed/1792360635843-m3-wave1-closeout.md delete mode 100644 .kilo/plans/completed/1792384312843-wave1-nas-deploy.md delete mode 100644 .kilo/plans/completed/1792475813001-pr17-browser-evidence-redaction-review.md delete mode 100644 .kilo/plans/completed/1792521924834-pr17-evidence-selfhealing-refinement.md delete mode 100644 .kilo/plans/completed/1792530797544-plan-cleanup-finalize.md delete mode 100644 .kilo/plans/completed/1792533823395-m4-adoption-plan.md delete mode 100644 .kilo/plans/completed/1792533823395-m4-exit-pass-guide.md delete mode 100644 .kilo/plans/completed/1792622454541-pr23-residual-logs-review.md delete mode 100644 .kilo/plans/completed/1792708132014-pr22-safe-login-review.md delete mode 100644 .kilo/plans/completed/1792784993002-pr19-bounded-collection-review.md delete mode 100644 .kilo/plans/completed/1792916051475-pr30-migration-cron-example-review.md delete mode 100644 .kilo/plans/completed/1792918289483-pr31-onboarding-switch-independence-review.md delete mode 100644 .kilo/plans/completed/1792965931629-pr33-download-crash-recovery-review.md delete mode 100644 .kilo/plans/completed/1793044794581-pr32-chromium-sandbox-review.md delete mode 100644 .kilo/plans/completed/1793124431799-open-issues-plans-20261027.md delete mode 100644 .kilo/plans/completed/1793127186466-pr34-date-variables-review.md delete mode 100644 .kilo/plans/completed/1793138983439-pr35-contributor-guides-review.md delete mode 100644 .kilo/plans/completed/1793207695728-pr42-daemon-runtime-directories-review.md delete mode 100644 .kilo/plans/completed/1793217175507-pr39-input-triggered-login-review.md delete mode 100644 .kilo/plans/completed/1793228064786-pr40-replacement-calendar-review.md delete mode 100644 .kilo/plans/completed/1793313075441-pr41-notice-identity-review.md delete mode 100644 .kilo/plans/completed/1793397084822-pr53-cron-weekdays-review.md delete mode 100644 .kilo/plans/completed/1793401881422-pr54-persistent-job-resync-review.md delete mode 100644 .kilo/plans/completed/1793430417736-pr56-terminal-ntfy-failures-review.md delete mode 100644 .kilo/skills/deploy/SKILL.md delete mode 100644 .kilo/skills/dev-commits/SKILL.md delete mode 100644 .kilo/skills/pr-review/SKILL.md delete mode 100644 .openwikiignore delete mode 100755 .private/sync_github.sh delete mode 100644 .vscode/launch.json delete mode 100644 .vscode/settings.json delete mode 100644 AGENTS.md delete mode 100644 openwiki/.claims/architecture/overview.json delete mode 100644 openwiki/.claims/architecture/roadmap.json delete mode 100644 openwiki/.claims/architecture/shared-interfaces.json delete mode 100644 openwiki/.claims/concepts/recipes.json delete mode 100644 openwiki/.claims/concepts/vault-transport.json delete mode 100644 openwiki/.claims/operations/deployment.json delete mode 100644 openwiki/.claims/quickstart.json delete mode 100644 openwiki/.claims/testing/verification.json delete mode 100644 openwiki/.claims/workflows/run-lifecycle.json delete mode 100644 openwiki/.last-update.json delete mode 100644 openwiki/.page-manifest.json delete mode 100644 openwiki/INSTRUCTIONS.md delete mode 100644 openwiki/architecture/index.md delete mode 100644 openwiki/architecture/overview.md delete mode 100644 openwiki/architecture/roadmap.md delete mode 100644 openwiki/architecture/shared-interfaces.md delete mode 100644 openwiki/concepts/index.md delete mode 100644 openwiki/concepts/recipes.md delete mode 100644 openwiki/concepts/vault-transport.md delete mode 100644 openwiki/guides/index.md delete mode 100644 openwiki/index.md delete mode 100644 openwiki/modules/index.md delete mode 100644 openwiki/operations/deployment.md delete mode 100644 openwiki/operations/index.md delete mode 100644 openwiki/quickstart.md delete mode 100644 openwiki/testing/index.md delete mode 100644 openwiki/testing/verification.md delete mode 100644 openwiki/ui/index.md delete mode 100644 openwiki/ui/web-ui.md delete mode 100644 openwiki/workflows/index.md delete mode 100644 openwiki/workflows/pause-repair.md delete mode 100644 openwiki/workflows/run-lifecycle.md diff --git a/.github/workflows/openwiki-update.yml b/.github/workflows/openwiki-update.yml deleted file mode 100644 index e18b90b..0000000 --- a/.github/workflows/openwiki-update.yml +++ /dev/null @@ -1,89 +0,0 @@ -name: OpenWiki Update - -on: - workflow_dispatch: - schedule: - - cron: "0 8 * * *" - -permissions: - contents: write - pull-requests: write - -jobs: - update: - runs-on: ubuntu-latest - steps: - - name: Check out repository - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 - with: - # Full history so `openwiki code --update` can diff HEAD against the - # commit it last documented; a shallow clone hides that commit and the - # update runs against an empty change summary. - fetch-depth: 0 - - - name: Set up Node.js - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4 - with: - node-version: "22" - - - name: Install OpenWiki - # mermaid + jsdom are optional; they add high-fidelity validation of Mermaid diagrams. Remove if your wiki has none. - run: npm install --global openwiki@0.7.2 mermaid@11.16.0 jsdom@29.1.1 - - - name: Run OpenWiki - id: openwiki - continue-on-error: true - run: openwiki code --update --print - env: - OPENWIKI_PROVIDER: openai-compatible - OPENAI_COMPATIBLE_API_KEY: ${{ secrets.OPENAI_COMPATIBLE_API_KEY }} - OPENAI_COMPATIBLE_BASE_URL: ${{ vars.OPENAI_COMPATIBLE_BASE_URL }} - OPENWIKI_MODEL_ID: "solar-llm" - # Required for the LangSmith connector's code-mode pull to authenticate. - # For extra workspaces, add OPENWIKI_LANGSMITH_API_KEY_2, _3, ... as repo - # secrets and env entries here. - OPENWIKI_LANGSMITH_API_KEY: ${{ secrets.OPENWIKI_LANGSMITH_API_KEY }} - # Optional: also trace this workflow's own OpenWiki run to LangSmith. - LANGSMITH_API_KEY: ${{ secrets.LANGSMITH_API_KEY }} - LANGCHAIN_PROJECT: openwiki - LANGCHAIN_TRACING_V2: "true" - - - name: Remove transient OpenWiki run state - if: ${{ !cancelled() }} - run: rm -f -- openwiki/.run.json - - - name: List OpenWiki update paths - id: paths - if: ${{ !cancelled() }} - # CLAUDE.md is listed only when present: git add fails on a missing path - # and then stages nothing. - run: | - paths=openwiki,AGENTS.md,.github/workflows/openwiki-update.yml - if [ -e CLAUDE.md ]; then paths="$paths,CLAUDE.md"; fi - echo "list=$paths" >> "$GITHUB_OUTPUT" - - - name: Create OpenWiki update pull request - id: create-pr - if: ${{ !cancelled() }} - uses: peter-evans/create-pull-request@22a9089034f40e5a961c8808d113e2c98fb63676 # v7 - with: - add-paths: ${{ steps.paths.outputs.list }} - branch: openwiki/update - commit-message: "docs: update OpenWiki" - title: "docs: update OpenWiki" - body: | - Automated OpenWiki documentation update. - - OpenWiki result: ${{ steps.openwiki.outcome }} - - When the result is `failure`, this PR intentionally preserves only the - pages completed before the failure. Merge it to make that progress the - baseline for the next scheduled run. - - - name: Annotate OpenWiki update pull request - if: ${{ !cancelled() && steps.create-pr.outputs.pull-request-url != '' }} - run: echo "::notice title=OpenWiki update pull request::${{ steps.create-pr.outputs.pull-request-url }}" - - - name: Propagate OpenWiki failure - if: ${{ steps.openwiki.outcome == 'failure' }} - run: exit 1 diff --git a/.kilo/kilo.jsonc b/.kilo/kilo.jsonc deleted file mode 100644 index 571a15b..0000000 --- a/.kilo/kilo.jsonc +++ /dev/null @@ -1,3 +0,0 @@ -{ - "$schema": "https://app.kilo.ai/config.json" -} diff --git a/.kilo/plans/1792793307574-recipe-repair-llm-options.md b/.kilo/plans/1792793307574-recipe-repair-llm-options.md deleted file mode 100644 index 210d1c8..0000000 --- a/.kilo/plans/1792793307574-recipe-repair-llm-options.md +++ /dev/null @@ -1,171 +0,0 @@ -# Public service recipe recovery - options for LLM-assisted repair (pre-step) - -Goal: make the 7 public service recipes (`1und1`, `buhl`, `datev`, -`freenet_mobilfunk`, `kabeldeutschland`, `nuernberger`, `winsim`) fully -operational again (run to end, PDF downloaded). The recipes are months to a -year old; the portals changed in the meantime. This document is the **pre-step -options analysis** for a follow-up implementation plan. - -## Verified context - -- **Recipes** live in `apps/recipes_playwright/recipe-pw__.yaml`, - validated against `recipe-pw-schema.yaml`. Method chains: locators - (`locator`, `get_by_role/label/text/title/...`) into actions - (`click`, `fill`, `press`), page-level `goto`/`close`, `expect_download` - with nested steps, daemon-only `wait_user`, `graceful` steps, and the - candidate phased-`login` contract (`login.py`, `doc/SAFE_LOGIN.md`). -- **No LLM code exists today.** No module, setting, or protocol action - references a local LLM (no ollama/LLM endpoint in `config.py` - `DEFAULT_SETTINGS`, the control protocol, or the UI). -- **The pause context is the designed LLM input contract** (decision 20, - `apps/billcollector/pause.py:91-164`): complete, stable, small, versioned - via `control.py`. Per pause it persists on `Run.pause_context`: - `reason`, `prompt`, `url`, `page_title`, `error`, `step_number`, the - failing step's locator-action JSON, **screenshot** (path + base64), - **numbered interactive elements** (role/name/rect/candidate - locator/actionTypes from the `pagestate.py` DOM scan), `page_text` - (<= 20 KB), `deadline_at`. The pause columns survive resume/timeout, so a - pause context stays queryable after the run ends. -- **Roadmap states explicitly**: "AI-driven intervention is designed for but - not implemented" (`openwiki/architecture/roadmap.md`); "a future AI - assistant is a second client of the same protocol, **gated by human - approval**" (daemon plan, decision line). -- **Existing repair machinery (works today, no new code):** - - Run -> non-graceful step fails 3x -> **pause** (`step_failed` / - `download_failed` / `login_failed` / `wait_user`) -> ntfy deep link. - - Session page `/runs/`: **Copilot** tab (live screenshot + numbered - overlay + click/fill/press/drag + Resume/Cancel) and **noVNC** tab - (full browser on the run's Xvfb display, websockify port `6080 + - (display - display_start)`). - - **Repair recipe** link -> editor failure panel (screenshot, failed step, - expected locator chain, candidate rows with **Use** -> - `apply_candidate_to_step` at `recipes.py:213`) -> Validate -> atomic - Save -> optional git auto-commit. - - **One-off vs permanent split:** a Copilot action (click the right - element, navigate) rescues the *current* run on resume; the recipe edit - on disk takes effect on the **next** run (the recipe is loaded once at - run start, `runner.py:906-908` - resume re-executes the in-memory step, - it does not re-read the file). - - **Verification channel:** step screenshots strip + noVNC tab while the - run is live; PDF lands on the Downloads page / consumer folder. - -## Options - -### Option A - Manual repair loop (no LLM), as-is - -Per service: **Run now** -> wait for pause -> read Copilot screenshot + -candidate list -> **Use** the matching candidate (or hand-edit the step in -the editor) -> Save -> **Run now** again (fresh run picks up the fixed -recipe) -> watch (noVNC / step strip) -> confirm PDF on Downloads. Repeat -per failing step. - -- Pros: zero code; the designed path; every pause produced becomes a real - `pause_context` corpus for later LLM prompt design. -- Cons: human must judge every step across 7 services; slow; flow-level - breaks (new dialogs, changed navigation, changed download mechanism) - still require hand-authoring YAML; one-off Copilot rescues re-occur every - run unless the recipe is also fixed. - -### Option B - Offline "recipe doctor" (first LLM integration, dev-side) - -A new dev-side CLI (new `billcollector` subcommand, e.g. `doctor `, -or a `helpers/` script following the `BillCollector*` pattern) that takes: - -1. the current recipe YAML (`recipe_path_for(service)`), and -2. one or more `pause_context` payloads - exported from the NAS DB - (`Run.pause_context` rows) or freshly captured locally with a short - Playwright session against the live portal - - -and prompts the local LLM (OpenAI-compatible / Ollama HTTP endpoint) to -return the corrected step or corrected recipe YAML (strict output format). -The tool passes the result through the existing `validate_recipe` gate -(`recipes.py:28`), prints a diff, and the operator applies it through the -existing editor **Use/Save/commit** path (or `save_recipe`). Verification -is the standard Run-now + watch + PDF check. - -- Pros: no daemon change, no protocol version bump, no NAS risk; consumes - the *exact live failure state* (pause context includes the post-login - page); reuses the existing validation/save/commit gate; text-only LLM - input (elements + step + error + page text) works with any local model, - no vision requirement; the human-approval gate is inherent (editor). -- Cons: two hops (export context -> apply fix); no live feedback loop; - fresh local captures of login-gated pages still need a manual login or - the exported post-login context. - -### Option C - In-daemon "LLM copilot" at pause time (follow-up) - -LLM suggestion inside the pause experience: on `step_failed`, the UI (or a -new `suggest` control-socket action = protocol version bump) sends the live -pause context (screenshot if the model is vision-capable, else elements + -step + page text) to the configured local LLM endpoint; the suggestion -renders as an extra candidate row in the failure panel / pause banner with -the existing **Use** button. Human approval stays (project decision: AI is -a second client of the same protocol, gated by human approval); noVNC stays -the fallback. - -- Pros: suggestion at the exact moment of failure against the live page; - one click to apply; becomes the durable product feature. -- Cons: largest surface - new Settings entry (`llm_url`/`llm_model`), - network reachability from the NAS container to the LLM, prompt - engineering, LLM-down must not break the pause (degrade to the current - UI), and either a `control.py` protocol version bump (runner-side) or a - UI-side HTTP call. - -### Option D - LLM-assisted re-authoring (for flow-level breaks) - -For portals whose flow changed fundamentally (new onboarding/consent -dialogs, restructured invoice area, new download mechanism): re-record the -new flow with the codegen helper (`helpers/BillCollectorCreateRecipe_pw.py`, -`playwright codegen` against the live portal) and let the LLM translate the -recorded Python into recipe YAML - the existing `-t` translator is -mechanical; the LLM improves locator quality, wraps the download in -`expect_download`, adds `graceful` banner steps, and keeps the -`{{USERNAME}}/{{PASSWORD}}/{{OTP}}` placeholders. Verify with Run now. - -- Pros: handles breaks beyond locator drift; reuses the authoring pipeline. -- Cons: codegen needs a logged-in session (manual login in the codegen - browser, or reuse of the daemon's per-task profile); most effort per - service; use only where A/B show a flow break, not locator drift. - -## How the existing pause/noVNC tools support the objective - -1. **Pause context = LLM input contract.** Complete, stable, small, - versioned, and persisted - an LLM can be invoked on demand, even after - the run ended (pause columns survive). -2. **Element scan already produces locator candidates** - (`pagestate.py` -> `get_by_label`/`get_by_role`/`get_by_text`/ - `get_by_title` with rects). The LLM's value-add is *selection and - chain-building against the step's intent* (description text, expected - locator, error), not DOM scanning. -3. **The editor's Use-candidate + Validate/Save/commit is the sanctioned - apply path** for any LLM output - human approval, git history, schema - gate, no new write path. -4. **noVNC + step strip = verification channel.** After applying a fix, - Run now and watch the browser live until the PDF lands; when the LLM is - uncertain, the operator drives the page directly. -5. **Copilot click/fill/press/drag rescues the current paused run - one-off**, so PDF collection can continue while the permanent recipe - fix is being worked out. - -## Recommended staging - -- **Stage 0 (no code):** Option A triage of all 7 services. Classify each - failure (locator drift vs flow break vs download break vs login break), - collect real `pause_context`s. Unblocks immediate PDF collection and - produces the corpus for prompt design. -- **Stage 1:** Option B (offline recipe doctor) - first LLM integration, - dev-side, reusing the validation gate; pilot on one service. -- **Stage 2 (separate plan, optional):** Option C (in-daemon LLM copilot) - once the prompt is proven in Stage 1; Option D per service as needed for - flow-level breaks. - -## Open questions (resolve before the implementation plan) - -1. **LLM stack:** engine (Ollama?), model(s), vision-capable?, and where it - runs (NAS host reachable from the daemon container vs dev machine) with - the network path. Determines prompt input (screenshot vs text) and - whether Stage 2/C is feasible on the NAS at all. -2. **Autonomy:** confirm human approval stays required for every LLM - proposed change (recommended; consistent with the project decision). -3. **Pilot service:** recommend `datev` (TOTP login, most complex) - or - start with the easiest to win a quick success. diff --git a/.kilo/plans/1792819039884-recipe-doctor-implementation.md b/.kilo/plans/1792819039884-recipe-doctor-implementation.md deleted file mode 100644 index c9bb6a5..0000000 --- a/.kilo/plans/1792819039884-recipe-doctor-implementation.md +++ /dev/null @@ -1,402 +0,0 @@ -# Stage 1: offline LLM recipe doctor — implementation plan - -Extended 2026-10-24: Stage 1b continuation for the remaining five -services (see below). - -Goal: implement Option B from -`1792793307574-recipe-repair-llm-options.md` — a dev-side -`billcollector doctor` subcommand that proposes a repaired recipe step -for a captured pause context, gated by the existing -`validate_recipe`/`save_recipe` path and explicit human approval. -Then pilot it on `buhl` (the easiest failure) and `datev`. - -## Stage 0 results (input to this plan) - -All 7 services run locally (display shim for the WSL2 X-socket quirk), -all paused `step_failed`, pause contexts in `/tmp/kilo/triage/`: - -| Service | Step | Failing step | Error | Class | Expected fix | -|---|---|---|---|---|---| -| buhl | 5 | `get_by_role(textbox,"E-Mail-Adresse")` + `press(value:"Tab")` | TypeError → `other` | **recipe bug**: Playwright `press()` takes `key`, not `value` (kwargs dispatch, runner.py:418-424) | `value` → `key` | -| datev | 8 | `get_by_role(button,"Login").click` | strict-mode, resolved 2 | locator ambiguity | `locator` with `#login` or `.first` | -| 1und1 | 4 | `get_by_role(button,"Click to reveal",exact).click` | timeout | stale step (Keycloak changed) | drop/replace step | -| winsim | 8 | `get_by_role(link,"Dialog Schließen").click` | timeout | stale step (dialog gone, rect 0,0,0,0) | drop step | -| freenet_mobilfunk | 3 | `get_by_role(link,"Jetzt einloggen").click` | timeout | element visible, click not actionable (overlay?) | locator tweak / force | -| nuernberger | 7 | `get_by_role(link,"Postfach").click` | timeout | element visible, click not actionable (overlay?) | locator tweak / force | -| kabeldeutschland | 3 | `locator('#loginCredintialsId input[type="text"]')` | timeout | **flow break**: goto now redirects to vodafone.de/meinvodafone | re-authoring (Option D, separate) | - -Pilot order: **buhl** (known one-line fix — validates the whole -doctor path against a known answer), then **datev**. - -## Design - -### Placement - -- New module `apps/billcollector/doctor.py` (sync, dev-side; no daemon - import, no protocol change). -- New `doctor` subcommand in `apps/billcollector/cli.py` next to - `serve`/`run` (lazy import, same pattern as `run`). -- New test module `tests/test_bc_doctor.py` (repo test conventions). - -### CLI contract - -``` -billcollector doctor - --pause PATH pause-context JSON file, or a directory - (newest *_pause.json wins) - --from-run RUN_ID read Run.pause_context from the local DB - --llm-url URL OpenAI-compatible base URL (default $BC_LLM_URL) - --llm-model MODEL model name (default $BC_LLM_MODEL) - --llm-api-key KEY optional (default $BC_LLM_API_KEY) - --prompt-file FILE dump the prompt and exit (no LLM call) - --llm-response FILE use a canned LLM reply instead of calling - --out FILE write the proposed YAML to FILE (default: - print to stdout, never touch the recipe) - --apply write the proposed YAML to the recipe file - via save_recipe (the human-approval gate) - --timeout S LLM request timeout seconds (default 120) -``` - -Default is dry-run: print the verdict + unified diff, exit 0. Nothing -on disk changes without `--apply`. - -### LLM endpoint - -OpenAI-compatible `POST {base}/chat/completions` with `requests` -(already in requirements). Base URL may be given with or without the -`/chat/completions` suffix. Text-only input (decision: no vision). -LLM-down / bad-JSON / schema-invalid output → clean error, non-zero -exit, recipe untouched. - -### Prompt (text-only) - -- system: role + method vocabulary (Playwright sync API names as used - by the runner: `get_by_role/label/text/title/placeholder`, - `locator`, `first/nth/last`, `content_frame`, `click/fill/press`, - `goto/close`, `expect_download`) + strict-JSON output contract + - rules (minimal change, keep `{{USERNAME}}/{{PASSWORD}}/{{OTP}}`, - prefer elements present in the snapshot, drop steps whose element is - gone, `press` takes `key` not `value`). -- user: service, failed step number, error record, url/title, the - failing step JSON, the full current recipe YAML, visible elements - (role | name | rect | candidate locator, truncated to ~120 entries), - page text truncated to ~8 KB. - -### Response contract (strict JSON) - -```json -{"explanation": "<= 2 sentences", - "fix_type": "step" | "recipe", - "step_number": 5, - "fixed_step": {"step": 5, "description": "", "methods": [...]}} -``` -or `{"fix_type": "recipe", "fixed_recipe": ""}`. - -Processing: fence-tolerant JSON extraction → for `step`: replace the -top-level step by number in the parsed document, re-serialize -(`yaml.safe_dump(sort_keys=False, allow_unicode=True)`), -`validate_recipe` gate → unified diff → `--out`/`--apply`/stdout. -`save_recipe` already refuses unvalidated text atomically. - -### Exit codes - -0 fix produced (applied if `--apply`); 1 LLM says no in-place fix -(flow break / needs human) with its explanation printed; 2 usage or -LLM/network/parse error; 3 recipe or pause-context load error. - -## Validation - -1. Unit: prompt build, JSON parse (fences, trailing prose), step - replace + re-serialize round-trip against the real schema, canned - `buhl` fix (`key: "Tab"`) passes `validate_recipe` and the diff is - minimal. — DONE (18 tests in tests/test_bc_doctor.py; the test - baseline normalizes the buhl recipe to its pre-fix state so it is - stable against the now-fixed live recipe). -2. Pilot buhl: run the doctor with a canned response (path 1) and, - once the endpoint is provided, for real; `--apply`; re-run the - buhl task locally (display shim) — expect it to get past step 5. — - DONE, see Pilot results. -3. datev: real or canned run; expect a `#login`-scoped or `.first` - variant; apply; re-run. — DONE, see Pilot results. -4. `ruff check apps tests` + pytest for the new module. — DONE (full - suite 426 passed, 1 skipped; ruff clean). - -## Pilot results (2026-10-24) - -All doctor runs used canned responses (`--llm-response`); the real -endpoint is still user-provided at runtime. Fixes were applied via -`--apply` (human gate = the printed diff). - -- **buhl** — two fixes: - 1. step 5 `press(value: "Tab")` → `press(key: "Tab")` (the Stage 0 - TypeError — the known-answer validation). - 2. step 2 (cookie-consent click) → `graceful: true`. The consent is - stored in the persistent profile, so the banner only appears - until first acceptance; a graceful step (single attempt, failure - continues the run) keeps the recipe valid in both states. This - exposed a prompt gap: `graceful` was missing from the doctor's - step vocabulary — added to `build_prompt` plus a - round-trip test. - Re-run (run 16): **success**, `WISO Steuer 2025.pdf` downloaded — - no further stale step surfaced (the WISO label is current). -- **datev** — two fixes: - 1. step 8 `get_by_role(button, "Login")` (strict mode, 2 matches) → - `locator("#login")`. A read-only headed page inspection (the - site rejects plain headless) showed the second match is the - `#SmartLogin` button; `#login` uniquely identifies the form - submit. - 2. step 10 "Check and log in" → "Continue": DATEV renamed the - OTP-confirmation button; surfaced by the post-fix re-run (run 17 - paused there, run 18 after the fix). - Re-run 18: **success**, `Brutto-Netto-Abrechnung 2026 01 - Januar.pdf` downloaded. - -Remaining: a real-LLM run once the endpoint URL/model are provided -(everything else is validated offline via `--prompt-file` / -`--llm-response`); the other five services (1und1, winsim, -freenet_mobilfunk, nuernberger, kabeldeutschland) are covered by -Stage 1b below. - -## Stage 1b: complete the remaining five services (2026-10-24) - -Goal: all 7 services run to the end (PDF downloaded). Remaining: -`1und1`, `winsim`, `freenet_mobilfunk`, `nuernberger` (doctor -iterations) and `kabeldeutschland` (Option D re-authoring). The Stage -0 triage table above stays the classification source; the per-service -forensics below were re-verified against the triage pause contexts -(`/tmp/kilo/triage/*`) on 2026-10-24. - -### First: commit the uncommitted Stage 1 work - -The working tree still carries all of Stage 1 uncommitted -(`doctor.py`, `tests/test_bc_doctor.py`, the `doctor` subcommand in -`cli.py`, the buhl/datev recipe fixes, the plan README + this file, -`.gitignore` `data/`). Suggest the commits via the dev-commits skill -(user confirms each) before starting Stage 1b, so the new recipe -fixes land as separate commits. - -### LLM operation - -- The endpoint is user-provided at runtime (no local engine on this - machine): `BC_LLM_URL` / `BC_LLM_MODEL` / optional `BC_LLM_API_KEY` - env vars, or `--llm-url/--llm-model/--llm-api-key`. OpenAI- - compatible base URL (Ollama/vLLM/LM Studio), text-only. -- No endpoint available: `--prompt-file` (inspect the prompt) and - `--llm-response FILE` (canned reply) still exercise the whole path — - the mechanism is validated, the judgment is not. -- Per-iteration loop, per service: - 1. Fresh pause: re-run the task locally the same way as Stage 0 - (display shim for the WSL2 X-socket quirk); let it pause on the - stale step. Use `--from-run ` — do not reuse the - `/tmp/kilo/triage/*` contexts as doctor input (2026-10-24 - captures; portals may have changed since). They stay forensic - reference only. - 2. `billcollector doctor --from-run ` → read the - explanation + diff (the human gate). - 3. `--apply`, re-run the task. - 4. Repeat until the run succeeds and the PDF lands in Downloads. - Expect ≥ 1 iteration per service: each fix surfaces the next - stale step (datev needed two). - -### Doctor prompt enhancement (small code change) - -`build_prompt` (doctor.py:224) does not mention the `force` argument, -so the LLM can never propose it. Add to the action vocabulary: -`click`/`fill`/`press` accept `{force: true}` in `arguments` — the -runner's kwargs dispatch (runner.py:418-424) passes it through as -`locator.(force=True)`, which bypasses the actionability -checks (schema-valid via the open `arguments` item schema). Guidance -in the prompt: prefer dismissing a covering banner (a `graceful` -dismiss step) when the snapshot shows a dismissable overlay; use -`force: true` only when the target is present, visible, and no -overlay is identifiable — and note that force does not click -*through* an interactive overlay (the event is still dispatched at -the element's point and routed by normal hit-testing). - -Test: extend `tests/test_bc_doctor.py` — the prompt contains the -force guidance; a canned reply adding `{force: true}` to a click's -arguments passes `validate_recipe` and yields a minimal diff. -Validate with `ruff check apps tests` + the full pytest suite. -— DONE (2026-10-24): `test_build_prompt_force_guidance` + -`test_replace_step_force_argument_roundtrip` added; ruff clean, full -suite green (453 passed, 2 skipped). - -### Stage 1b results - -- **nuernberger** — DONE (2026-10-24, run 21 success): re-targeted - step 7 to `get_by_role(link, "Postfach", exact)` (the glyph tile is - gone) and the two invoice rows to the "… PDF" links - (`Leistungsabrechnung PDF` / `Vertragsinformation PDF`); both PDFs - landed in Downloads (`Leistungsabrechnung_1079485_01-10-2026.pdf`, - `Vertragsinformation_1079485_05-10-2026.pdf`). -- **winsim** — DONE (2026-10-25, run 30 success): three doctor - iterations (all canned replies derived from live page probes; the - real endpoint is still user-provided at runtime): - 1. Re-run 26 (original recipe) paused at step 7: the "Bestätigen" - dialog was visible this time (run 14 had surfaced step 8). Fix 1 - (`/tmp/kilo/winsim_fix1.json`): steps 7 + 8 → `graceful: true` - (both dialogs are transient), stale step 9 "Meine Rechnung" (gone - from the start page) → `get_by_role(link, "Rechnungen", exact)` - + `.first`. - 2. Run 27 paused at step 9 (still on /start): the nav "Rechnungen" - link is covered by the userDataSwitch header widget (click lands - on the overlay). Fix 2 (`winsim_fix2.json`): step 9 → - `locator a[title='Rechnungen']` (the quick-entry tile; probe - `winsim_inspect3.py` verified it navigates to - /mytariff/invoice/showAll). - 3. Runs 28/29 paused at the step 11 `expect_download` - (download_failed): the first "Rechnung vom" anchor is itself the - PDF link (`/mytariff/invoice/showPDF/...`) — the plain step 10 - click already fires the download outside the window, and the - inner "Rechnung" link no longer exists (probe - `winsim_inspect4.py` confirmed the click downloads - `B954329320_rech.pdf`). Fix 3 (`winsim_fix3.json`): the - `expect_download` window now wraps the "Rechnung vom" + `first` - click; the stale inner step is dropped. - Run 30: **success** — `B954329320_rech.pdf` in apps/Downloads. - -### Status (2026-10-25) - -4 of 7 services are done (buhl, datev, nuernberger, winsim); the -Stage 1b work so far is committed on dev. The remaining three services -deliberately stay open for a follow-up session: - -- **1und1** — in progress: the expected fixes (drop the stale - "Click to reveal" step, graceful consent) are applied and committed, - but runs 22–25 still pause `step_failed`; next: re-capture a fresh - pause, then the next doctor iteration. -- **freenet_mobilfunk** — not started: the overlay/consent case - (checklist item 4), including the `force: true` decision point. -- **kabeldeutschland** — not started: flow break → Option D - re-authoring against the new vodafone portal (checklist item 5). - -The per-iteration loop (fresh pause → doctor → `--apply` → re-run), -the definition of done, and the wrap-up steps below are unchanged; the -wrap-up (batch commit, plan archival) happens after all seven are -green. - -### Per-service checklist (order: cheap wins first) - -1. **nuernberger** — re-target (not an overlay case). - Forensics: step 7 targets the glyph tile - `get_by_role(link, "Postfach \uf054")`; the pause snapshot shows - the plain nav link `"Postfach"` visible at (460,72) linking to - `/kunde/pages/postfach/postfach.html` — the intended target. - Expected fix: rename step 7 to `get_by_role(link, "Postfach", - exact: true)`; add `.first` if strict mode complains (multiple - visible matches). Next stale step likely in the Postfach table - (steps 8/11: cells "Leistungsabrechnung" / "Vertragsinformation") - — iterate. — DONE (2026-10-24, run 21 success; see Stage 1b - results). -2. **1und1** — drop a stale step. - Forensics: the failing step 4 clicks the Keycloak password- - visibility toggle `get_by_role(button, "Click to reveal", exact)` — - that button is **absent** from the pause snapshot, the step has no - automation purpose, and the recipe contains a duplicated step - number 4 (the first step 4, `get_by_title("Zum Control-Center- - Login")`, works and lands on the Keycloak page). - Expected fix: drop the "Click to reveal" step (clean up the - numbering). Hardening (operator judgment): make step 2 "Ablehnen" - (cookie consent) `graceful: true` so the run survives when the - banner is absent (buhl precedent). Next stale step unknown (e.g. - step 10 `test_id "invoice-link"` in the Control-Center) — iterate. - — IN PROGRESS (2026-10-25): the two expected fixes are applied - in the working tree, but runs 22–25 still pause - `step_failed`; the next doctor iteration is pending (re-capture a - fresh pause first). -3. **winsim** — drop a hidden-dialog step, then re-target. - Forensics: step 8's link "Dialog Schließen" has rect 0×0 in the - pause snapshot (the dialog is no longer shown) → the click times - out waiting for visibility. Step 9's "Meine Rechnung" no longer - exists on the start page; "Rechnungen" is visible in two places - (nav at y≈240, tile at y≈635). - Expected fixes, two iterations: drop step 8 (or make it graceful); - re-target step 9 to `get_by_role(link, "Rechnungen")` + `.first` - (strict mode: two visible matches). — DONE (2026-10-25, run 30 - success; three iterations, see Stage 1b results). -4. **freenet_mobilfunk** — overlay case (the force decision point). - Forensics: step 3 "Jetzt einloggen" is visible on the pre-login - page; the snapshot has no top-frame consent element — the - SP-consent iframe content is invisible to the DOM scan (top-frame - only) — and the recipe carries a commented-out consent step - (iframe `SP Consent Message` → "Alle akzeptieren"): the consent - banner is likely back, covering the CTA. - Plan: re-enable the consent step as `graceful: true` first (the - banner text/iframe may have changed — the re-run shows), re-run. - If the same timeout recurs, drive the paused run once via noVNC / - Copilot to see the actual banner and set the dismiss locator. - `force: true` on the step 3 click is only the fallback if no - covering overlay is found (see the force semantics note above). - Also watch step 4: its `goto` carries a frozen Keycloak `state` - parameter; if login is rejected as an invalid/expired session, - re-capture the login URL from a fresh "Jetzt einloggen" click (a - new state is issued per click) — a one-line re-record. -5. **kabeldeutschland** — flow break → Option D re-authoring (last). - Forensics: step 3's `#loginCredintialsId input[type="text"]` no - longer exists — the portal moved to vodafone.de/meinvodafone - (pause URL `vodafone.de/meinvodafone/account/login`; the snapshot - shows the new login form: "Benutzername*", "Passwort*", - "Zugangsdaten vergessen?"). - The doctor is expected to answer `fix_type: none` here — that is - the correct outcome, do not force a step edit. Re-author per - Option D: codegen against the live portal - (`helpers/BillCollectorCreateRecipe_pw.py`, manual login in the - codegen browser) → translate to YAML (the mechanical `-t` - translator; optionally feed the recorded Python to the LLM for - locator quality, `expect_download` wrapping, `graceful` banner - steps, `{{USERNAME}}/{{PASSWORD}}/{{OTP}}` kept verbatim) → the - usual validate/apply gate → Run-now verification. The whole - post-login invoice flow must be re-recorded inside MeinVodafone; - this is the most labor-intensive service. - -### Definition of done (per service) - -Task run completes, PDF downloaded (verify via the Downloads page / -consumer folder), recipe change landed through the validation gate -with the reviewed diff. - -### Wrap-up (after all seven are green) - -1. Commit the recipe fixes + the doctor prompt enhancement - (dev-commits, user confirms). -2. Archive the delivered plans: move this file and - `1792793307574-recipe-repair-llm-options.md` to - `.kilo/plans/completed/`, update `.kilo/plans/README.md` per its - own rules (the daemon master plan stays active). -3. Stage 2 (Option C in-daemon copilot) stays a separate plan; the - pause context + the proven doctor prompt are its foundation. - -### Risks (Stage 1b) - -- Pause contexts go stale: always re-capture before each doctor - iteration; the triage captures are forensic reference only. -- The element scan is top-frame only: iframe overlays (freenet - consent) are invisible to the doctor → noVNC/Copilot inspection as - the manual fallback. -- No local LLM: the endpoint arrives at runtime; without it the - canned-reply path validates the mechanism, not the judgment. -- A wrong LLM fix is caught by `validate_recipe` + the diff review; - worst case is another pause — the recipe file is only written via - `save_recipe` with `--apply`. -- Portals may change again between now and the runs — per-iteration - re-capture keeps the doctor input fresh. - -## Out of scope - -- Option C (in-daemon copilot at pause time) — Stage 2, separate plan. -- Option D re-authoring for kabeldeutschland (vodafone flow) — per- - service manual work with the codegen helper. -- NAS deployment of the doctor (it is dev-side; the NAS keeps the - current pause/editor repair flow). - -## Risks - -- No local LLM engine on this machine: the endpoint URL/model arrive - at runtime (user-provided). Everything is testable offline via - `--prompt-file` / `--llm-response`. -- YAML re-serialization changes quoting style → noisy diff; - mitigated by `sort_keys=False` and review before `--apply` (the - validation gate is correctness, the diff is the human gate). -- The doctor must never break the daemon: it is a separate - subcommand, imports no daemon state, and writes only through - `save_recipe`. diff --git a/.kilo/plans/1792861675256-issue27-maintainer-commitments.md b/.kilo/plans/1792861675256-issue27-maintainer-commitments.md deleted file mode 100644 index 3082959..0000000 --- a/.kilo/plans/1792861675256-issue27-maintainer-commitments.md +++ /dev/null @@ -1,243 +0,0 @@ -# Issue #27 maintainer commitments (product audit triage) - -Goal: track the maintainer-side work committed in the triage reply to GitHub -issue #27 ("Product audit", closed 2026-10-24, triage decision in comment -6387035799). The audit issue is closed as a triage issue; each accepted item -tracks here (or in its existing lane) until delivered. flowcool's handed-over -items (F07, D08, fixtures, docs, F01, F08-after-contract) are tracked in -their own PRs, not here. - -## Verified context (dev, 2026-10-24) - -All P1 findings re-verified against dev. The audit baseline is public main -`42d43378`; dev has since moved (daemon UI, bounded collection PR #19, -phased login PR #22), but every finding below still reproduces or still holds -on dev. - -- F02: `docker-compose.yml:64-73` mounts Downloads/db/data/profiles but not - recipes; the UI editor saves under `/apps/recipes_playwright` (container - writable layer). Git auto-commit assumes a checkout; the image copies - `apps/` without `.git`. -- F03: `docker-compose.yml:63` publishes `6080-6095` without a loopback host - binding; `config.py:56` `vnc_password` defaults to `""`; `display.py` - runs `x11vnc -localhost -nopw` behind websockify. -- F04: `config.py:54` `ntfy_topic` defaults to `"billcollector"`; - `notify.py:21` `NTFY_HOST` defaults to `https://ntfy.sh`; `notify.py:54-56` - already skips the pause push when the topic is empty (the gap is the - pre-filled default, not the send path). `config.py:53` `ui_base_url` - defaults to loopback `http://127.0.0.1:8000`. The self-hosted ntfy compose - profile (`docker-compose.yml:84-99`) already exists. -- F05: phased-login contract delivered on dev (`doc/SAFE_LOGIN.md`, commit - `9d8ebcf`); none of the 7 provider recipes in `apps/recipes_playwright/` - uses the `login` method; dashboard "available" label does not distinguish - protected vs. legacy login. -- F06: `runner.py:133-148` `run_result` — aborted -> `failure`; completed - with no downloads -> `failure` (explicit legacy contract). -- F08: `recipes.py:28` `validate_recipe` — schema allows any method string; - login-specific validation only. -- F09: `apps/bc_default.ini` ships 7 providers / 11 tasks (winsim x5 users + - kabeldeutschland, datev, buhl, nuernberger, 1und1, freenet_mobilfunk). -- F10: consumer extension slot empty; `track_download` calls consumers - before the Download row insert, catches failures, continues. -- F01: `downloads.py` unchanged since the audit baseline; `track_download` - saves `suggested_filename` before the hash check. - -## Commitments (what I keep) - -### C1 — F04: notification opt-in default (P1, small) - -- Change the `ntfy_topic` default from `"billcollector"` to empty: a fresh - installation sends nothing until the operator sets a topic in Settings. - The send path already no-ops on empty topic (`notify.py:54-56`), so this is - a default + docs change, not a new code path. -- Document public ntfy.sh vs. the bundled self-hosted profile (the compose - header already covers the two-line switch). -- Deep-link base URL: `ui_base_url` must be settable to a reachable - LAN-absolute address for real deployments; keep loopback as the safe - default and surface it in Settings with an explanation. -- Acceptance (from the issue): no notification leaves a fresh installation - before an explicit choice; installation-specific destination; minimal - payload; a user-triggered test notification is a follow-up item (Settings - "Send test push"). -- Status: done 2026-10-24 — `ntfy_topic` now defaults to empty (a fresh - installation sends nothing until the operator sets a topic in Settings; - both push paths no-op on an empty topic, `notify.py:26-28` and - `notify.py:54-57`); Settings surfaces the topic as opt-in with a public - ntfy.sh vs. bundled self-hosted hint, and `ui_base_url` with the - LAN-reachability explanation; the README documents the opt-in flow in the - "Pause notifications (ntfy)" section. The "Send test push" follow-up stays - open. - -### C2 — F09: shipped default INI (P1, small) - -- Remove the personal example accounts from `apps/bc_default.ini` (and - `bc_test.ini` if it ships examples). Fresh import creates no personal - tasks; keep the INI import/migration path working for existing users. -- Acceptance (from the issue): fresh installation starts with no personal - example tasks; demos explicit and synthetic; deliberate INI migration for - existing users preserved; clear first-account action and missing- - configuration visibility before offering scheduled execution (the - first-account wizard itself is U01, stage 2). -- Status: done 2026-10-24 — the shipped `bc_default.ini`/`bc_test.ini` carry - no personal accounts (a fresh import now creates 0 tasks, verified); the - `import_ini` migration path is unchanged and existing users' tasks are - untouched; the README INI example uses synthetic names. First-account - wizard / missing-configuration visibility remains U01 (stage 2). - -### C3 — F08: method contract (P1, unblocks flowcool) - -- Publish the supported method surface: method names, argument names/types, - and the supported variables, as part of the recipe-repair work (see - `.kilo/plans/1792819039884-recipe-doctor-implementation.md` for the - surrounding repair/doctor context). -- The contract must cover the valid legacy collection recipes, the login - recipes (`doc/SAFE_LOGIN.md`), and the bounded collection actions (PR #19) - without rejecting intentionally supported Playwright calls. -- flowcool implements the `validate_recipe` hardening + tests (F08) against - the published contract. -- Acceptance (from the issue): reject unknown methods, invalid argument - names/types and unsupported variables before vault/browser access; report - the service and step; compatibility checks include valid legacy and new - collection/login recipes. -- Status: done 2026-10-24 - the method contract is published in - `doc/RECIPE_METHODS.md` (commit `5745e59`, gate green: ruff + pytest - 451 passed) and posted to issue #27 (comment 6388757576) so flowcool - can start the F08 lane. It covers the legacy collection recipes, the - doc/SAFE_LOGIN.md login recipes and the PR #19 bounded collection - actions; the Playwright surface is defined as the introspectable - public API of the pinned `playwright==1.63.0` sync API (Page, - Locator, Frame, FrameLocator, ElementHandle), and section 5 splits - what the existing gates already enforce from what F08 must add. - -### C4 — F06: result semantics contract (P1, contract-first) - -- Draft the result contract: distinguish proved empty / duplicates-only / - new documents / partial collection / failed collection, so a healthy empty - collection is not a `failure` and an arbitrary no-download run is not - automatically healthy. -- Coordinate with #24 (crash recovery / publication) and #26 (date variables - / collection window) so one coherent semantics covers the collection - window and the publication semantics; no implementation before the - contract is agreed on the issue. -- Then implement: UI, notifications and CLI/exit semantics updated - consistently, with compatibility tests. -- Status: not started; waits for the #24/#26 lanes to settle the shared - semantics. - -### C5 — F02: recipe storage design (P1, design first) - -- Decide: bind-mounted recipes volume (packaged updates vs. local edits), - export/restore, durable recipe history, and the git auto-commit story - (hide or explain it when no usable checkout exists — the image has no - `.git`). -- Acceptance (from the issue): a synthetic UI edit survives container - recreation with identical content; packaged-update interaction defined; - export/restore or durable history provided. -- Blocks U09 (guided repair with revert) — the audit explicitly says resolve - F02 first. -- Status: design not started. - -### C6 — F03: noVNC access path (P1, design first) - -- Specify one supported routing/authentication design: published port - binding (loopback-only host bind + reverse proxy on the NAS) vs. - authentication through the UI session; document reverse-proxy/TLS - requirements. -- Acceptance (from the issue): an unauthenticated client cannot view or - control a live browser through any published path; authenticated - intervention remains usable. -- Changes the reference Compose deployment (`docker-compose.yml:63`); needs - the deployment-process validation (deploy skill) before NAS rollout. -- Status: design not started. - -### C7 — F05: protected-login conversion (P1, slow) - -- UI: distinguish protected vs. legacy login recipes on the dashboard (the - "available" label can be mistaken for a verified login/session state). -- Conversions: convert each of the 7 provider recipes to the phased `login` - method only with a valid final-session proof and appropriate MFA phases — - real-provider validation with my accounts (flowcool may prepare structure - and fixtures, not claim real-provider validation). Preserve existing - session, optional-TOTP and timeout behavior; synthetic tests for no- - resubmit and persistent-budget behavior. -- Overlaps the recipe-doctor stage 1b pilot work (same recipes, same portal - access windows) — schedule the conversions together with the doctor runs - to reuse the access sessions. -- Status: not started; needs portal access windows per provider. - -### C8 — F10: archive confirmation (after #24) - -- Complete #24 first (crash recovery, publication, consumer delivery — - flowcool's implementation lane). Then expose downloaded / pending delivery - / delivered / failed delivery states, using the agreed stable idempotency - key and safe retries; a plain consume-folder integration states what it can - prove when no ingestion callback exists. No competing implementation. -- Status: crash-recovery prerequisite delivered via PR #33 (2026-10-26); - schedulable now. Issue #24 stays open per maintainer decision — the - delivery-state exposure (C8) is its remaining lane. - -## Handed over to flowcool (tracked in their PRs, not here) - -- F07 onboarding toggle closure + multi-control acceptance test. -- D08 README cron example fix (`0 6 1 1,15 * *` at `README.md:378`). -- Acceptance fixtures: synthetic multi-account setup, persistence/restart - checks. -- Docs: D02 tested quick start, D05 troubleshooting, D07 recipe author - reference. -- F01 collision-safe storage, together with the #24 implementation. -- F08 validation + tests, once C3 publishes the method contract. - -## Related lanes - -- #24 crash recovery (flowcool implementation PR #33) — **DELIVERED - 2026-10-26** (Gitea `main` `181a9cc` / GitHub `main` `5116627`, PR closed - without merge, comment 6425517713). F01 (collision-safe storage) is in; the - consumer-delivery half (F10/C8) remains. Issue #24 left OPEN per maintainer - decision 2026-10-26 (no closure comment); C8 is now schedulable. -- #26 date variables / collection window (flowcool PR #34, last in the - continuation order) — overlaps C4; once it lands with #33, C4 becomes - draftable. -- #28 Chromium sandbox (flowcool, contract accepted 2026-10-24) — - **DELIVERED 2026-10-26 via PR #32** (Gitea `main` `bde713c` / GitHub - `main` `93bfcc8`); issue #28 closed as delivered (comment 6429514585). -- #25 publication routine — CLOSED 2026-10-25 (state_reason completed): - the S7 routine was demonstrated across the PR #30 + #31 deliveries, so the - "stays open until a delivery runs the S7 routine" condition is met. Closed - with a comment (issuecomment-6403353716) pointing at the demonstrated - deliveries (#30: Gitea `main` `84a9d8b` / GitHub `41294731df`; #31: Gitea - `main` `5cd1f31` / GitHub `d9362b3`). The GitHub Discussion channel for - per-delivery summaries is still being stood up (the Discussion write API is - not reachable from this tooling) — that refinement is tracked here, not on - #25. - -## Proposed order - -1. C2 (F09) — small, independent, shipped-default fix. -2. C1 (F04) — small, unsafe-default fix. -3. C3 (method contract) — unblocks flowcool's F08 work. -4. C4 (F06 contract) — needs the #24/#26 context; draft once they settle. -5. C5 (F02 design) — unblocks U09. -6. C6 (F03 design) — security; NAS deployment impact. -7. C7 (F05) — slow; bundle with doctor stage 1b portal windows. -8. C8 (F10) — after #24 lands. - -After P1: my top stage-2 picks are U04 (mobile navigation) and U05 -(downloads as document access). Remaining backlog accepted in triage: -U01-U03, U06-U11, D01-D07, D09, O02-O08 (each spins out as its own issue -when picked up). O01 stays deferred per #21. - -## Status - -- Plan created 2026-10-24; triage decision posted on issue #27 - (comment 6387035799). C2 (F09) done 2026-10-24; C1 (F04) done 2026-10-24 - (gate green: ruff + pytest 451 passed); C3 (F08 contract) done - 2026-10-24 (`doc/RECIPE_METHODS.md` commit `5745e59`, issue #27 comment - 6388757576). PR #33 (the #24 implementation lane) DELIVERED 2026-10-26 - (Gitea `main` `181a9cc` / GitHub `5116627`): crash recovery + F01 on main, - C8 (F10) now schedulable, C4 still needs #34 (#26). PR #32 (Chromium - sandbox, #28) DELIVERED 2026-10-26 (Gitea `main` `bde713c` / GitHub - `93bfcc8`). Next per the sequencing plan - (`1792862814701-next-up-sequencing.md`): PR chain #34, then doctor - Stage 1b (needs the user's portal-access windows; C7 conversions ride - the same windows). C4–C8 remain open — this plan stays Active until - they are delivered. diff --git a/.kilo/plans/1792862814701-next-up-sequencing.md b/.kilo/plans/1792862814701-next-up-sequencing.md deleted file mode 100644 index 52f520d..0000000 --- a/.kilo/plans/1792862814701-next-up-sequencing.md +++ /dev/null @@ -1,499 +0,0 @@ -# Next up: sequencing decision (2026-10-24) - -Question: start one of the three active plans now, or wait for a PR generated -from one of the open GitHub issues? - -**Decision: do not wait for a PR. Start plan 3 (issue #27 commitments) with C2 -and C1 now, then C3; then move to plan 2 Stage 1b.** - -Superseded in part (2026-10-25): the PRs arrived — flowcool's #30–#34 are -open. The "PR continuation order" section below is the current primary -next-up, interleaved with the remaining Stage 1b work. - -## Grounds (verified on dev, 2026-10-24) - -- GitHub has **no open PRs** (`s-t-e-f-a-n/BillCollector`, state=open → - empty). Open issues are #24, #25, #26, #28 (all flowcool). "Waiting for a - PR" means idling on a deliverable that does not exist yet, on someone else's - schedule. -- The pending PRs only gate plan 3's **C4** (needs #24/#26 semantics) and - **C8** (needs #24) — items 4 and 8 in the plan's proposed order. They gate - nothing that could start now. -- Plan 1 (`1792793307574-recipe-repair-llm-options.md`) is a pre-step options - analysis with Stage 0 done — a reference document, not an execution target. -- Plan 2's Stage 1 is landed and pushed (doctor subcommand, unit suite, - buhl/datev pilots green, runs 16/18). **Stage 1b** (remaining 5 services) - is open and its plan already documents the no-LLM-endpoint fallback - (canned `--llm-response`, proven in Stage 1). -- Plan 3's C1/C2/C3 have **zero external dependencies** (no portal access, no - LLM endpoint, no flowcool input). C3 (publish the method contract) - actively **unblocks flowcool's F08 lane** — doing it now shortens their - wait, the reverse of blocking on them. -- Plan 3's C7 (protected-login conversion) is explicitly scheduled to bundle - with the Stage 1b portal-access windows to reuse the sessions. - -## Order - -0. **Housekeeping first:** commit the pending plan work (untracked - `1792861675256-issue27-maintainer-commitments.md` + the README line for it) - via the dev-commits skill — user confirms each suggested commit, push to - Gitea origin/dev. -1. **C2 (F09):** remove personal example accounts from - `apps/bc_default.ini` (and `bc_test.ini` if it ships examples); fresh - import creates no personal tasks; INI import/migration path preserved. -2. **C1 (F04):** `ntfy_topic` default `"billcollector"` → empty (send path - already no-ops on empty, `notify.py:54-56`); document public ntfy.sh vs. - the bundled self-hosted compose profile; surface `ui_base_url` in Settings - with the LAN-reachability explanation. -3. **C3 (F08 contract):** publish the supported method surface (method names, - argument names/types, supported variables) covering legacy collection - recipes, `doc/SAFE_LOGIN.md` login recipes, and PR #19 bounded-collection - actions, without rejecting intentionally supported Playwright calls. - Posts to issue #27 so flowcool can start F08. -4. **Stage 1b** (plan 2): nuernberger → 1und1 → winsim (cheap wins) → - freenet_mobilfunk (overlay case) → kabeldeutschland (Option D - re-authoring, last). Per-service loop per the plan's checklist; fresh - pause context per iteration. **C7 conversions ride the same portal - windows.** Start when the user is available for the live portal sessions - (TOTP/OTP at run time). LLM endpoint is runtime-pluggable; without it the - canned-response path carries the loop (mechanism validated, judgment is - manual) — this is the same path Stage 1's green pilots used. -5. **Stays deferred by design (no action now):** C4 (draft once #24/#26 - settle), C5 (F02 design), C6 (F03 design), C8 (after #24 lands). - -## Status (2026-10-24) - -- Step 0 housekeeping: done (plan files committed and pushed). -- Step 1 C2 (F09): done and pushed (`09fb199` + plan update `5324051`). -- Step 2 C1 (F04): done — code, Settings hints, README section, opt-in test; - gate green (ruff + pytest 451 passed); committed and pushed per the - dev-commits cycle. -- Step 3 C3 (F08 contract): done 2026-10-24 — the draft - `doc/RECIPE_METHODS.md` was verified line-by-line against runner/login/ - collection/recipes/schema sources and the 15 shipped recipes; four - accuracy fixes applied (Frame/FrameLocator chain types, the - login-chain locator set, exact-match variable substitution, the - existing-gates vs. F08-additions split); gate green (ruff + pytest - 451 passed); published as commit `5745e59` and posted to issue #27 - (comment 6388757576) so flowcool can start F08. -- Next: Step 4 Stage 1b — start when the user is available for the live - portal sessions; C7 conversions ride the same windows. - -## PR continuation order (2026-10-25) - -Question: in which sequence do we continue the five open community PRs? -(All flowcool, all based on public `main` `b4cf023`, all -`mergeable_state: clean`; none depends on another's code.) - -**Decision: #30 → #31 → #33 → #32 → #34, one pr-review chain at a time, -interleaved with the remaining Stage 1b portal sessions.** - -### Grounds - -- No code dependency between the PRs (same base, disjoint contracts: - #30 docs D08/#27, #31 onboarding F07/#27, #32 sandbox #28, #33 download - recovery #24 + F01/#27, #34 date variables #26). Shared-file overlap is - regional, not hunk-level: `runner.py` (#32 launch flags / #33 startup - recovery / #34 date expansion), `db.py` (#33 `Download` migration / #34 - `Task.date_window_days`), `config.py` (#32, #33), `README.md` + - `CHANGELOG.md` (#30–#34), `tests/test_bc_residual_logs.py` (#33, #34). - Dev has already diverged from `b4cf023`, so every S1 re-runs the - pre-image check against current dev; the skill's divergence handling - covers it. -- **#30 + #31 first** — the cheapest full S1–S7 run. The #25 publication - routine (S7 close comment + related-issue closure + Discussion summary) - has not yet been exercised in a delivery; running it on a one-line docs - PR and a one-line fix PR validates the pipeline end-to-end at minimal - risk before the heavy reviews start. -- **#33 before #32** — the only true downstream unblocks in the batch: - #33 lands #24 → **C8 (F10)** can be scheduled; #33 + #34 land #24/#26 → - **C4 (F06)** can be drafted. And #33's S5 is fully local (unit + - regression + ruff; no Dockerfile/compose/install-script/INI/.env - changes — the new `download_recovery_timeout_seconds` is a Settings row - with a default, so record "not deployment-relevant" with that judgment - in the plan doc) — it cannot stall on an external host. -- **#32 fourth** — the only PR with a hard external gate: S5 fires the - full deployment-validation block (docker build + canary check + smoke - run) on a Docker-capable host (the NAS). Schedule its S5 to the NAS - window. **Swap rule:** if the NAS is free and the user prefers the - deployment-qualified PR before the large #33 review, swap #32 and #33 — - no code dependency makes the swap safe. -- **#34 last** — it extends the shared recipe-contract interface (agreed - in #26, cited in the PR body, so the S1 process check passes) and - changes cron-TZ semantics (no `TZ` → agreed UTC fallback instead of the - host timezone). After it lands, `doc/RECIPE_METHODS.md` (C3), the - doctor-prompt variable vocabulary, and the README each need one - contract-doc update — bundle them into #34's S6 against the final dev - tree so the contract docs are revised once, not per earlier PR. The - kabeldeutschland Option D re-author (the last Stage 1b item) can - optionally use the date variables once #34 is on dev. - -### Per-PR notes (input to each S1) - -| PR | Scope | Weight / gate | Watch in S1 | -|---|---|---|---| -| #30 | README cron example (D08) | trivial; docs-only validation | README migration-section pre-image | -| #31 | onboarding.py closure fix + test (F07) | small | `onboarding.py` pre-image (dev's C1/C2 touched other files) | -| #33 | downloads.py rewrite, db/config, recovery in runner + ui/app, `doc/DOWNLOAD_RECOVERY.md`, 373-line crash tests (#24, F01) | large; S5 fully local | unblocks C8; `runner.py`/`db.py`/`config.py` regions; CHANGELOG + README merges | -| #32 | sandbox flags in daemon + batch launchers, `docker/`, `.gitattributes`, 15 files (#28) | large; S5 needs the NAS Docker host | S5 trigger fires (docker/ + runtime config); `BillCollectorServices_pw.py` legacy path; record host + validated tree hash | -| #34 | date variables, `Task.date_window_days`, Run-now pipe, scheduler TZ (#26) | large; contract + TZ behavior change | interface agreed in #26; `test_bc_residual_logs.py` hunks adapt on top of #33 | - -### Interleave with Stage 1b - -- PR sessions need no portal; Stage 1b iterations need live TOTP/OTP. - Advance the PR stages in the non-portal sessions. -- Stage 1b stands at 4/7 (buhl, datev, nuernberger, winsim green and - committed). Next portal session: **1und1** (fixes committed; runs 22–25 - still pause → fresh pause + doctor iteration per plan 2's loop), then - freenet_mobilfunk, then kabeldeutschland (Option D re-authoring). C7 - conversions ride the same portal windows. -- After #33 lands: C8 can be scheduled. After #33 + #34: C4 can be - drafted. Neither blocks this PR order. - -### Status (2026-10-25) - -- PRs #30–#34 open since 2026-10-24; order decided above. -- PR #30 DELIVERED 2026-10-25 (S1–S7 in one chain): hunk applied + - CHANGELOG entry, Gitea `main` `84a9d8b`, GitHub `main` `41294731df`, - PR closed without merge; the #25 publication routine ran end-to-end - (Discussion summary skipped by user — small fix). -- PR #31 DELIVERED 2026-10-25: S1+S2 verdict approve (byte-identical - pre-images), S4 both files applied + CHANGELOG entry, S5 green (unit 29 - / full 454 / regression PASS / ruff + bandit baseline; deployment skip - recorded + user-confirmed), S7 — commits `7a0ad6b` + `36a8d31` + - `5cd1f31`, Gitea `main` = `5cd1f31`, GitHub `main` = `d9362b3`, PR - closed without merge (comment 6399102074, 14:49:55Z). The Discussion - summary is pending user creation in the web UI (Discussion write API not - usable from this environment; exact text recorded in the review doc, - now in `completed/`). -- PR #33 S1+S2 done 2026-10-25 ("proceed with pr #33" session): - review plan doc `1792965931629-pr33-download-crash-recovery-review.md` - written, verdict approve (head `b675ebe` unchanged), 3 diverged files with - non-overlapping hunks, two nits as S3 decision points, deployment - validation recorded as not deployment-relevant — Status: - AWAITING-DECISION. -- PR #33 DELIVERED 2026-10-26 (S3–S7 across three "continue PR #33" - sessions): S3 M1 + M2(a) applied, S4 all 12 files integrated (3 manual - non-overlapping merges), S5 green (unit 36 / full 487 / regression PASS / - ruff clean / bandit baseline; deployment skip user-confirmed - 2026-10-25T22:44Z), S6 docs confirmed, S7 — commits `78ed5f9` / `20d99e3` - / `181a9cc`, Gitea `main` `181a9cc` (ff-only from `5cd1f31`), GitHub - `main` `5116627`, PR closed without merge (comment 6425517713, - 2026-10-26T19:36:31Z). Issue #24 left OPEN per user decision (no - comment); the Discussion summary is deferred (draft recorded in the - review doc, now in `completed/`). C8 (F10) is now schedulable. -- PR #32 S1+S2 done 2026-10-26 ("proceed with PR #32 per the sequencing" - session): review plan doc - `1793044794581-pr32-chromium-sandbox-review.md` written, verdict approve - (head `52bc1c9` unchanged; 15 files — 11 clean pre-images, 4 diverged with - non-overlapping hunks; seccomp profile byte-identical to the pinned - upstream Playwright source at `54c7115`, SHA256 `cc3e61ca…`; #28 - five-point contract verified point-by-point), two doc-only nits (M1 - fork-era policy wording, M2 host name in the doc) as S3 decision points, - S5 trigger fires (docker-compose + docker/ + launch behaviour) → full - deployment-validation block on the NAS, scheduled to the NAS window — - Status: AWAITING-DECISION. -- PR #32 DELIVERED 2026-10-26 (S3–S7 across four "continue PR #32" - sessions): S3 gate confirmed as-is (11 byte-faithful + 4 manual - non-overlapping merges, M1/M2 applied, 5 commits), S4 all 15 files - integrated, S5 green (41 new unit tests / full 528 / regression PASS / - ruff clean / bandit baseline; NAS deployment-validation block green on - tree `2bc849d` — sandbox proof active on both launch paths, runtime - regression PASS, compose smoke healthy), S6 docs confirmed, S7 — - commits `b01be29` / `61c69d8` / `62c5bb6` / `12a3281` / `2bc849d`, - Gitea `main` `bde713c` (ff-only from `181a9cc`), GitHub `main` - `93bfcc8`, PR closed without merge (comment 6429513980, - 2026-10-26T23:10Z). Issue #28 closed as delivered (comment - 6429514585); the Discussion summary is deferred (draft recorded in - the review doc, now in `completed/`). The NAS still runs the dev tree - from S5; a deploy to `main` is a separate user request. -- PR #34 DELIVERED 2026-10-27 (S3–S7 across five "continue PR #34" - sessions): S1+S2 approve (20 files — 11 clean pre-images, 3 new, - 6 diverged with non-overlapping hunks; the 17-case #26 contract - verified), S3 gate as-is (14 as-is + 6 keep-both + the M1a/M1b/M2 - contract-doc bundle + 4 commits), S4 all 22 files integrated, S5 - green (60 new-suite + 1 new test / full 589 / regression PASS / ruff - clean / bandit no new; NAS deployment-validation block green on - `dc0ef7f` — build, canary 18/5, TZ contract 3/3, compose smoke - healthy, service left running on the dev tree), S6 CHANGELOG `### - Changed` entry confirmed, S7 — delivery commits `cfb7875` / `99c0f2e` - / `9c00d83` / `dc0ef7f`, Gitea `main` `53a5dff` (ff-only from - `bde713c`), GitHub `main` `6bb3348`, PR closed without merge (comment - 6453946786, 2026-10-27T21:25:24Z). Issue #26 closed as delivered - (comment 6453947542, 2026-10-27T21:25:26Z); the Discussion summary is - deferred (draft recorded in the review doc, now in `completed/`). - With #33 + #34 landed, **C4 (F06)** can now be drafted. The NAS still - runs the dev tree from S5; a deploy to `main` is a separate user - request. -- New PRs #35/#39/#40/#41 arrived after the sequencing decision; they do not - change #32's position (no known overlap with its 15 files) — a sequencing - addendum is a separate decision. **Addendum decided 2026-10-27** (plan - `completed/1793124431799-open-issues-plans-20261027.md`): #34 → #35 → #42 → #39 → - #40, with #41 HELD on the #38 archive decision; PR #42 (issue #43) arrived - 2026-10-27. - - PR #35 S1+S2 done 2026-10-27 ("proceed with next pr" session): review plan - doc `1793138983439-pr35-contributor-guides-review.md` written, verdict - approve (head `205e086` unchanged; actual content vs fork base `d9362b3` = - 22 files +602/−17 — 5 new, 16 clean pre-images, 2 diverged non-overlapping; - the 104 deleted private-file entries in the merge-base diff are fork-based - STRIP artifacts, skipped), two doc-currency fixes (M1 TROUBLESHOOTING TZ - row against the post-#34 baseline, M2 QUICKSTART pending list) as S3 - decision points, deployment validation recorded as not deployment-relevant - — Status: AWAITING-DECISION. - - PR #35 DELIVERED 2026-10-28 (S3–S7 across four "continue PR #35" - sessions): S3 gate as-is (M1/M2 confirmed, 3-commit split), S4 all 23 - files integrated (16 clean pre-images + 5 new + README insertion + - authored CHANGELOG), S5 green (PR test 1 passed / full 590 / regression - PASS / ruff clean / bandit zero-new / 15/15 recipes validate; deployment - skip user-confirmed), S6 docs confirmed as-is, S7 — commits `5f45502` / - `5896f40` / `75b885b`, Gitea `main` `75b885b` (ff-only from `53a5dff`), - GitHub `main` `484048d03b86500c9abdb548ef052763050630e4`, PR closed - without merge (comment 6474460727, 2026-10-28T16:21:27Z). Related issue - #27 already closed (2026-10-24) — no closure action; the Discussion - summary is deferred (draft recorded in the review doc, now in - `completed/`). Chain head is now #42. - - PR #42 S1+S2 done 2026-10-28 ("continue pr#42" session): review plan - doc `completed/1793207695728-pr42-daemon-runtime-directories-review.md` written, - verdict approve, no maintainer-side changes (head `0efaec90` stable; - actual content vs fork base `93bfcc85` = 3 files +90/−1 — helper import - line removed, `DatabaseManager` first-open dir creation, new 86-line - pytest module; the 89 deleted private-file entries in the merge-base - diff are STRIP artifacts of the orphaned parent commit `93bfcc85`, - skipped); issue #43 acceptance fully covered by the two shipped tests; - deployment validation recorded as not deployment-relevant — Status: - AWAITING-DECISION. - - PR #42 DELIVERED 2026-10-28 (S3–S7 across four "continue PR #42" - sessions): S3 gate as-is (3 files from head `0efaec90`, no - maintainer-side changes), S4 applied (2 files + new test, +90/−1), - S5 green (PR tests 2 passed / full 592 / regression PASS / ruff + - bandit nothing new; deployment skip user-confirmed), S6 CHANGELOG - `### Fixed` entry confirmed as-is (one re-split incident on the S6 - doc commit, corrected by force-push of `origin/dev`), S7 — commits - `f8e16c7` / `550ef80` / `2a6e462`, Gitea `main` `2a6e462` (ff-only - from `75b885b`), GitHub `main` `c2fb7f8`, PR #42 closed without merge - (comment 6478433499, 2026-10-28T19:22:36Z), issue #43 closed (comment - 6478434230, 2026-10-28T19:22:39Z); Discussion summary deferred to - manual web-UI posting (write API 404). Chain head was then #40 (now - DELIVERED 2026-10-29; the #34 → #35 → #42 → #39 → #40 chain is complete). - - PR #39 S1+S2 done 2026-10-28 ("continue with pr#39" session): review plan - doc `completed/1793217175507-pr39-input-triggered-login-review.md` - written, verdict - approve (head `0053ec6` stable; actual content vs fork base `d9362b3` = - 3 files +192/−5 — the 1-line submit-terminal `fill` validation, - SAFE_LOGIN contract doc, 4 new tests + 1 parametrization; the 87 deleted - private-file entries in the merge-base diff are fork-behind-base - artifacts, skipped); issue #36 contract verified point-by-point against - the unchanged `runner._execute_login` (reserve-before, fill-once, - proof-gated, durable interruption); the PR ships no CHANGELOG entry — - maintainer authors the `### Added` line (S3 decision); deployment - validation recorded as not deployment-relevant — Status: - DELIVERED (2026-10-28: S7 complete; Gitea `main` `b693517` / GitHub - `main` `ad18f7a`; PR #39 closed without merge; issue #36 closed - completed). - - PR #40 S1+S2 done 2026-10-28 ("continue pr#40" session): review plan - doc `completed/1793228064786-pr40-replacement-calendar-review.md` - written, verdict - approve (head `73acc023` stable; branch re-commits the sanitized tip - `d9362b3` as an orphaned public-history commit whose 87 private-file - deletions are comparison artifacts, skipped; actual content = 9 files - +816/−11 — new `calendar.py` bounded traversal, optional - `download_all` deadline/before-download hooks, new daemon action + - legacy-runner refusal, schema + contract doc + 382-line pytest module; - the #37 seven-point contract verified point-by-point; 4 dev-diverged - files (runner.py, BillCollectorServices_pw.py, README, CHANGELOG) all - non-overlapping hunks; M1 doc currency + M2 CHANGELOG wording as S3 - decision points), deployment validation recorded as not - deployment-relevant) — Status: DELIVERED (2026-10-29: S7 complete; S3 - as-is, S4 applied, S5 all green on the exact tree, S6 docs as-is, S7 - commits `df3ec4b`/`d34e396`/`61defd3`, Gitea `main` `61defd3` (ff-only - from `b693517`) / GitHub `main` `f26b2a9`; PR #40 closed without merge - (comment 6509093900, 2026-10-29T21:48:00Z); issue #37 closed (comment - 6509221226, 2026-10-29T21:55:06Z); Discussion summary omitted at user - request). New PR #44 (C3 method-chain validation) arrived after the - addendum — separate sequencing decision. - - PR #41 S1+S2 done 2026-10-29 ("continue PR #41" session): review plan - doc `completed/1793313075441-pr41-notice-identity-review.md` written, - verdict approve with maintainer changes (head `74421dd` stable; branch - re-commits the sanitized tip `d9362b3` as an orphaned - public-history commit, skipped; actual content = 10 files +828/−10; #38 - HOLD revised toward integration 2026-10-29 with F10/C8 as an accepted - risk; M1 contract / M2 schema / M3 #33 re-author as S3 decision - points; deployment-validation skip recorded), S3–S7 across three - "continue PR #41" sessions: S3 as-is, S4 selective integration (6 clean - files + db/recipes manual merge + downloads re-authored on the #33 - journal + 2 test adaptations + CHANGELOG fold), S5 all green on the - exact tree (14/14 unit, 647 + 53 subtests, harness PASS 7/7, 15/15 - recipes, ruff/bandit baseline; deployment skip user-confirmed), S6 docs - as-is (README pointer + NOTICE_IDENTITY header reframe, docs-only), - S7 — commits `14bab50`/`5c8216f`/`74ba171`, Gitea `main` `74ba171` - (ff-only from `61defd3`) / GitHub `main` `e83c8e2`; PR #41 closed - without merge (comment 6531164937, 2026-10-30T19:00:36Z); issue #38 - closed completed (comment 6531165804, 2026-10-30T19:00:39Z). - **The planned PR chain #34 → #35 → #42 → #39 → #40 → #41 is - complete.** Discussion summary text recorded in the review doc for - manual web-UI post (write API unavailable). Flowcool's archive-side - contract proposal on #38 (2026-10-30, comment 6518010065) awaits a - maintainer decision — a dedicated new issue if tracked. - - The 1und1 portal iteration remains independent and can run in a live - session at any time. - -## PR batch #2 (2026-10-30) - -Six open PRs, all flowcool, all based on public `main` `e83c8e2` (the -post-#41 mirror tip): #53 (cron weekday numbering, #46), #54 (persistent -job-store rebuild, #47), #56 (terminal ntfy failures, #49), #55 (exact vault -item identity, #48), #45 (browser cache precedence, no issue), #44 (C3/F08 -recipe method-chain validation, contract agreed in #27). Open issues without -PRs: #50 (incremental download keys + single-PDF ZIP extraction), #51 -(Bourse Direct original-PDF adapter agreement), #52 (archive receipts after -notice identity — the pending #38 archive-side contract), plus the -maintainer lane #24. - -**Decision (user-confirmed 2026-10-30): #53 → #54 → #56 → #55 → #45 → #44, -one pr-review chain at a time.** - -Grounds: - -- #53 and #54 both touch `scheduler.py` + `test_bc_scheduler.py` - (non-overlapping hunks: #53 adds `cron_trigger` + the `_add_job_for` - guard; #54 changes `start()`/`resync()`). Kept adjacent so #54's S1 - pre-image check runs against the settled #53 tree. -- #56 (monitor/orchestrator/notify) and #55 (vault transport) are disjoint - from the scheduler and from each other; both single-commit PRs with S5 - fully local. -- #45 starts from the old sanitized tip `f26b2a9` (103 files / −22928 in the - diff = orphaned-tip STRIP artifact; the real change is ~10 files) and shows - `mergeable_state: dirty` — S1 must untangle the base before any verdict; - scheduled after the clean single-commit PRs. -- #44 (18 files +698/−90) extends the recipe-contract interface (agreed in - #27, cited in the PR body) and is the heaviest S1; it closes the batch. - -- PR #53 S1+S2 done 2026-10-30 ("continue last sessions" session): review - plan doc `1793397084822-pr53-cron-weekdays-review.md` — verdict approve - with maintainer changes (M1: the PR's new `### Fixed` header lands above - `### Added` and duplicates the existing `### Fixed` subsection; the entry - moves into the existing one); head `6b41040b` stable, merge-base - `e83c8e2`, single commit, 4 files — 3 as-is + 1 manual CHANGELOG; issue - #46 before-claim verified on the pinned APScheduler 3.11.3; deployment - validation recorded not deployment-relevant — Status: AWAITING-DECISION. -- PR #53 DELIVERED 2026-10-30 (S3-S7 in follow-up sessions): dev commits - `9ccb1ff`/`44c693a`/`fa1ade5` (fix/test/doc); Gitea `main` `fa1ade5` - (ff-only from `74ba171`); GitHub `main` `a24668a` (mirrored from - `e83c8e2`); PR #53 closed without merge, issue #46 closed completed; - Discussion summary text recorded for manual web-UI post. Plan doc archived - to `completed/`. Next in the batch-2 chain: **#54** (persistent job-store - rebuild, #47) — its S1 pre-image check runs against the settled #53 - `scheduler.py`/`test_bc_scheduler.py` tree. -- PR #54 S1+S2 done 2026-10-30 ("continue pr 54" session): review plan doc - `1793401881422-pr54-persistent-job-resync-review.md` — verdict approve with - maintainer changes (M1: same CHANGELOG `### Fixed`-placement fix as #53; the - PR's header lands above `### Added` and duplicates the existing `### Fixed` - subsection); head `7f6cb02` stable, branch point = orphaned public tip - `e83c8e2` (the 101-file GitHub "Files changed" view is a STRIP artifact, - real content 3 files +42/−4), both code hunks apply cleanly on the post-#53 - dev tree (verified `git apply --check`, no 3way); issue #47 before-claim - verified on the pinned APScheduler 3.11.3 (`Float(25)` next_run_time); no - scheduler overlap with open #44/#45/#55/#56; deployment validation recorded - not deployment-relevant — Status: AWAITING-DECISION, S2 gate passed, S3 - next. -- PR #54 S3-S5 done 2026-10-31 ("continue pr 54" session): S3 proposal - confirmed as-is (2 code files as-is + M1 CHANGELOG Fixed-placement); S4 - integration applied as hunks on dev (`git apply` of - `git diff e83c8e2..refs/pr/54` for scheduler.py + test — clean; M1 by - hand), tree left dirty for S5; S5 validation green on the exact dev tree — - PR module 14 passed / 15 subtests, before/after proof (new test fails on - the pre-integration `scheduler.py`: 2 stale jobs survive `start()`), full - unit 652 / 10 skipped / 68 subtests, regression VERDICT: PASS (7 scenarios - / 8 PDFs), ruff + bandit clean vs baseline; deployment skip recorded (not - deployment-relevant) — Status: VALIDATED, S5 gate passed, S6 next. -- PR #54 S6 done 2026-10-31 ("continue pr 54" session): documentation - confirmed — the CHANGELOG M1 entry stands as the first item of - `[Unreleased]` → `### Fixed` (sanitized, references #47 only); no README - change (line 233 already documents the Task-table-is-authoritative rebuild - behavior the fix makes true); no CONTRIBUTING change (no rule change) — - Status: VALIDATED, S6 gate passed, S7 next (commit + push: Gitea dev → - main promotion → GitHub mirror → PR close + issue #47 closure + Discussion - summary). -- PR #54 S7 complete 2026-10-31 ("continue pr 54" session): DELIVERED — - commits `f68a663`/`f163369`/`a75861f` (fix/test/doc) pushed to `origin/dev`; - Gitea `main` `fa1ade5` → `a75861f` (ff-only, user-gated, no tag); GitHub - `main` `a24668a` → `ebf20bb` (sanitize-by-amend, public gate green: no - private paths at the tip, fix present); PR #54 closed without merge - (comment 6540230969, 2026-10-31T06:45:07Z), issue #47 closed completed - (comment 6540231280, 2026-10-31T06:45:09Z); Discussion summary text - recorded for manual web-UI post (write API unavailable). Plan doc archived - to `completed/`. Next in the batch-2 chain: **#56** (terminal ntfy - failures, #49) — its S1 runs against the settled `a75861f` tree. -- PR #56 S1+S2 done 2026-10-31 ("continue pr 56" session): review plan doc - `1793430417736-pr56-terminal-ntfy-failures-review.md` — verdict approve with - maintainer changes (M1: the PR's top-level `## Terminal run notifications - (#49)` section breaks the Keep a Changelog structure; the entry moves under - `[Unreleased] → ### Added` in house style); head `95b11de` stable, branch - point = orphaned public tip `e83c8e2` (the 103-file GitHub "Files changed" - view is a STRIP artifact, real content 5 files +215/−1), 4 of 5 files - byte-identical pre-images on dev `497dbdb` (clean apply), only `CHANGELOG.md` - manual; no overlap with #53/#54 code, no overlap with open #44/#45/#55 - besides CHANGELOG; deployment validation recorded not deployment-relevant — - Status: AWAITING-DECISION, S2 gate pending (user review of verdict + plan), - S3 next. -- PR #56 S3-S5 done 2026-10-31 (same session): S2 gate passed (verdict - confirmed, doc commit `a96fabe` pushed); S3 proposal confirmed as-is; S4 - applied on dev (4 files byte-identical from head `95b11de` + M1 CHANGELOG - `[Unreleased] → ### Added` first entry, tree left dirty); S5 green on the - exact dev tree — before/after proof (21 new tests fail pre-integration, - 49 OK after), full unit 665 passed / 10 skipped / 83 subtests, regression - VERDICT: PASS (7 scenarios / 8 PDFs, exit 0), ruff clean, bandit 0 new on - app files; deployment recorded skip (not deployment-relevant) — S5 found one - extra maintainer change, M2: `tests/test_bc_login.py` - `test_slow_login_notification_keeps_daemon_loop_responsive` pinned one - `requests.post` per path, but the `finalize`/`orphan` paths now also send - the terminal run notification (the PR's scoped evidence never ran that - module); adapted to path-dependent post counts (2/2/1/1) — S5 gate passed - (same session); Status: VALIDATED, S6 next. -- PR #56 S6 done 2026-10-31 (next session): S6 docs in tree — CHANGELOG M1 - entry stands under `[Unreleased] → ### Added` (sanitized, first item); - README section `### Pause notifications (ntfy)` → `### Notifications (ntfy)` - (TOC anchor, opt-in sentence extended to terminal run outcomes, new - terminal-notices paragraph, both `NTFY_HOST` descriptions); CONTRIBUTING - unchanged — S6 gate passed (user confirmed as-is) — Status: VALIDATED, S7 - next (dev-commits proposal, push `origin/dev`, ff-only promotion, GitHub - mirror, close PR #56 + issue #49). -- PR #56 S7 complete 2026-10-31 (follow-up "continue pr 56" session): - DELIVERED — commits `da01d5b`/`32140f7`/`2bf53d8` (feat/test/doc) pushed to - `origin/dev`; Gitea `main` `a75861f` → `2bf53d8` (ff-only, user-gated, no - tag); GitHub `main` `ebf20bb` → `4265ddb` (sanitize-by-amend, public gate - green: no private paths at the tip, the #49 CHANGELOG entry present); PR - #56 closed without merge (comment 6544710707, 2026-10-31T14:29:39Z), issue - #49 closed completed (comment 6544711134, 2026-10-31T14:29:42Z); Discussion - summary text recorded in the review doc for manual web-UI post (write API - unavailable). Plan doc archived to `completed/` (this session). Next in the - batch-2 chain: **#55** (exact vault item identity, #48) — its S1 runs - against the settled `2bf53d8` tree. -- PR #55 S1+S2 done 2026-10-31 ("continue pr 55" session): review plan doc - `1793461722991-pr55-vault-exact-item-identity-review.md` — verdict approve - with maintainer changes (M1: the PR's `### Fixed` section is inserted - before `### Added`, breaking the Keep a Changelog order; the entry moves - to the top of the existing `[Unreleased] → ### Fixed` section); head - `397b0b6` stable, branch point = orphaned public tip `e83c8e2` (the 104-file - GitHub "Files changed" view is a STRIP artifact, real content 6 files - +177/−11), 4 of 5 existing files byte-identical pre-images on dev - `19c4164` (clean apply) + the new `tests/test_bc_vault_exact.py` a clean - add, only `CHANGELOG.md` manual; overlap with open #44/#45 limited to - `runner.py` (different area) + CHANGELOG; deployment validation recorded - not deployment-relevant — Status: AWAITING-DECISION, S2 gate pending - (user review of verdict + plan), S3 next. - -## Explicitly not doing - -- No idling on the PRs: since 2026-10-25 they are handled through the - pr-review skill in the order above, interleaved with Stage 1b rather - than waiting for either to finish. -- No Option C (in-daemon LLM copilot) — Stage 2, separate plan, per plan 2. - -## Validation - -Each step validates per its own plan file's acceptance criteria (C1/C2/C3 in -`1792861675256-issue27-maintainer-commitments.md`; Stage 1b in -`1792819039884-recipe-doctor-implementation.md`). Repo gate before each push: -`ruff check apps tests` + full pytest suite; commits via dev-commits with -per-item user confirmation. diff --git a/.kilo/plans/1793461722991-pr55-vault-exact-item-identity-review.md b/.kilo/plans/1793461722991-pr55-vault-exact-item-identity-review.md deleted file mode 100644 index 60c08fb..0000000 --- a/.kilo/plans/1793461722991-pr55-vault-exact-item-identity-review.md +++ /dev/null @@ -1,300 +0,0 @@ -# PR #55 review: exact vault item identity before credential reads (issue #48) - -> Status: VALIDATED (2026-10-31) — S5 all green on the exact dev tree: -> before/after proof (8/8 new tests fail pre-integration, pass after), PR -> commands exact (130 passed/30 subtests; 18/14), full unit 675 passed/10 -> skipped/90 subtests, regression VERDICT: PASS exit 0 (7 scenarios/8 PDFs), -> ruff clean, bandit zero-new on production files (test file: B101 asserts + -> B105 synthetic only); deployment skip recorded + user-confirmed at the S5 -> gate; S6 docs confirmed (CHANGELOG M1 stands + vault_transport.md -> correction, no README/CONTRIBUTING change). S7 next. - -## PR metadata - -- PR: s-t-e-f-a-n/BillCollector#55 "fix(vault): resolve exact item identity - before credential reads" (flowcool), head - `397b0b617aaa0eaaf174d0a93897b244e492fc06` (fetched to `refs/pr/55`, - matches `git ls-remote github`). Two commits in the fetched ref (orphaned - public tip `e83c8e2` + the fix commit `397b0b6`); the real content is - **6 files +177/−11** against the branch point `e83c8e2`. The GitHub "Files - changed" view (104 files, +290/−23395) is an orphaned-tip STRIP artifact: - the merge-base `5c8216f` carries 32 `openwiki/` files, the PR head has 0. - Base ref `main` @ `4265ddb` (current GitHub tip); the branch actually - starts at the orphaned public tip `e83c8e2` (GitHub `main` at the PR #41 - delivery; Gitea-side equivalent `74ba171`). -- Issue: #48 (flowcool, open) — `fetch_credentials()` retrieves - `/object/item/` and `/object/totp/`; the CLI's name resolution - can select a case-insensitive or ambiguous match, and the fresh-TOTP - lookup repeats name selection. Required: list/search, filter on the exact - full name, exactly one match, item and TOTP by the validated id; optional - TOTP and existing retries preserved; item naming unchanged; the pinned - local transport must allow the endpoint search query while continuing to - reject queried base URLs, userinfo, fragments, redirects and non-local - targets; errors/logs must not include names, query values, credentials or - raw endpoint bodies; no schema, recipe, control-protocol or exit-code - change. -- Created 2026-10-30T20:11:14Z; last updated 2026-10-31T13:59:37Z (before - this review's fetch). -- Local state at review time: Gitea `dev` = `origin/dev` = `19c4164`, Gitea - `main` = `origin/main` = `2bf53d8`, GitHub `main` = `4265ddb`. Working - tree clean. -- AI disclosure present: "AI-assisted focused port and synthetic - verification under my responsibility. Branch starts directly at public - main `e83c8e2`." - -## Verdict - -**Approve with maintainer changes** — M1 (CHANGELOG placement) only; all PR -code and tests apply as-is from head (byte-identical pre-images on `dev`, -the new test file is a clean add). - -## Verified facts - -1. Head stable: fetched `refs/pr/55` = `397b0b61...`; PR `updated_at` - 2026-10-31T13:59:37Z predates the fetch. S4 preflight re-checks via - `git ls-remote github refs/pull/55/head`. -2. Branch point: the fork starts at the orphaned public tip `e83c8e2` - (GitHub `main` at the PR #41 delivery; Gitea-side equivalent `74ba171`). - `e83c8e2` is **not** an ancestor of the current GitHub `main` `4265ddb` - (each mirror sync re-pushes the Gitea lineage plus one amended sanitized - tip, orphaning the previous public tip — same pattern as #53/#54/#56). - The GitHub "Files changed" view (104 files, +290/−23395; the - merge-base-`5c8216f`-to-`e83c8e2` part is 99 files −23384 of STRIPped - private content, `openwiki/` et al.) is an orphaned-tip STRIP artifact, - not PR content. Integration applies the 6-file delta against `e83c8e2` - only. -3. Issue #48 before-claim plausible and S5-verifiable: on the pre-integration - tree the new `test_case_variants_resolve_exact_id_and_totp_by_id` must - fail (the base `fetch_credentials` resolves by name, so the mocked - search response is misread and `user`/`password` come back None); after - integration the PR's stated command passes: 130 passed, 30 subtests. -4. Pre-image vs Gitea `dev` `19c4164`: `apps/billcollector/vault.py`, - `apps/billcollector/runner.py`, `apps/vault_transport.py` and - `tests/test_vault_transport.py` are **byte-identical** to the PR base → - hunks apply cleanly. `tests/test_bc_vault_exact.py` is a new file (absent - in both base and dev) → clean add. `CHANGELOG.md` diverges (PR #53/#54/ - #56 entries under `[Unreleased]`) → manual merge (M1). Dev files changed - since the Gitea branch point `74ba171`: `CHANGELOG.md`, `README.md`, - `apps/billcollector/notify.py`, `apps/billcollector/orchestrator.py`, - `apps/billcollector/scheduler.py`, `apps/billcollector/ui/tasks.py`, - `tests/test_bc_login.py`, `tests/test_bc_notify.py`, - `tests/test_bc_orchestrator.py`, `tests/test_bc_scheduler.py` — only - `CHANGELOG.md` overlaps the PR. -5. Code review (per file), behavior preserved vs changed: - - `vault.py` `fetch_credentials` (the core fix): search - `/list/object/items?search=` → `json.loads` → require - `success is True` + list `data.data` → **exact case-sensitive** name - match (`item.get("name") == item_name`) → **exactly one** match, else - `VaultAPIError("Expected one exact vault item match")` → `item_id` - validated against `[A-Za-z0-9_-]{1,128}` → item fetched **by id** - `/object/item/` → detail re-validated (`success is True`, - `data.id == item_id`, `data.name == item_name`) → TOTP fetched **by id** - (optional TOTP preserved: `get_totp` still treats 400 as "no TOTP"). - Return dict gains an additive `item_id`; the `user`/`password`/`otp`/ - `url` extraction is unchanged. Error handling: - `ValueError/TypeError/KeyError/AttributeError` from parsing → - `VaultAPIError("Invalid vault item response")` **`from None`** (no - names, queries, credentials or bodies leak); `VaultAPIError` itself - (e.g. a real 4xx from the search) is not in the catch tuple and - propagates with its status intact, as does the "exactly one match" - error. Retry behavior preserved (unchanged `vault_request` policy). - - `runner.py` (2 hunks): `_fresh_otp` now resolves the TOTP target as - `getattr(bcs, "vault_item_id", None) or bcs.vault_item_name` — the - run-start credential re-fetch targets the validated id when present - and falls back to the name (backwards-compatible with any context that - sets only the name). `_RunContext.__init__` stores - `self.bcs.vault_item_id = credentials.get("item_id")` alongside the - existing `vault_item_name`/`otp_fallback` attributes. - - `vault_transport.py` (1 hunk pair): `pinned_api_urls(url, *, - allow_query=True)` — the rejection clause changes from `parsed.query` - to `(parsed.query and not allow_query)`, so request URLs may carry a - query while every other authority check (scheme, hostname, userinfo, - fragment, port, local-only DNS resolution, sorted numeric targets) is - untouched. `pinned_api_url` (the configured-base validator) now calls - `pinned_api_urls(url, allow_query=False)` → **queried base URLs stay - rejected**. `vault_http_request` (the request path) uses the default - `allow_query=True`, preserving path+query in every pinned target - (`parsed._replace(netloc=...)`). Redirect refusal, Host-header ASCII - check and exception sanitization are unchanged. - - Test review: `tests/test_bc_vault_exact.py` (new, 115 lines, pytest — - consistent with the repo's 21 pytest test modules + `pytest.ini`): - case-variant exact selection + item/TOTP by id (asserts the exact - request URLs); missing/ambiguous/invalid-id (`../SECRET_TOKEN`) all - fail before TOTP with no "SECRET_TOKEN"/"Genevois" in the error; - changed item identity rejected; `_fresh_otp` uses the stored id; and a - real loopback `ThreadingHTTPServer` preflight test (status, escaped - name `CC Genevois & #/?`, exact item, optional TOTP 400 — only config - injected, DNS/transport/HTTP stay real). `tests/test_vault_transport.py` - +24: the search query is preserved in every pinned target (IPv4+IPv6), - more unsafe forms rejected for the base validator, and a query never - bypasses the authority/fragment/port/public-IP checks. Synthetic - markers only, no secrets, no network beyond 127.0.0.1. -6. Process check (CONTRIBUTING.md): issue referenced (#48); intended - behavior + verification stated in the PR body; proof shipped - (before/after pytest with expected counts + ruff/bandit/whitespace - results); AI assistance disclosed; no secrets in code/tests/artifacts; no - new network exposure (local pinning preserved; only the search query on - request URLs is new — URL-quoted and local-only); no shared-interface - change (no DB schema, control protocol, recipe contract or exit code — - confirmed in the diff; the `fetch_credentials` return dict gains an - additive `item_id` key read by its only caller). -7. Cross-PR: open PRs are #55 (this), #45 (Playwright browser cache), #44 - (recipe method-chain validation) — all flowcool, all base `main`. - Batch-2 chain `#53 → #54 → #56 → #55 → #45 → #44` (user-confirmed - 2026-10-30); #53/#54/#56 DELIVERED → #55 is next in order. File overlap - with #45/#44 besides `CHANGELOG.md`: `apps/billcollector/runner.py` - (their real content also touches runner.py; #55's two hunks sit in - `_fresh_otp`/`_RunContext`, a different area — S4/S5 pre-image re-checks - cover any movement). #55's `vault.py`, `vault_transport.py` and both - test files are unique to this PR. - -## Risks - -- Low overall: the change narrows credential resolution (name → validated - id) behind the existing transport pinning; new failure modes are - `VaultAPIError`s the runner already records as run failures; no behavior - change for a vault with exactly one correctly-cased item. -- New hard dependency on the `/list/object/items?search=` endpoint in the - operator's bw serve build; if absent, runs fail with a clear 4xx - `VaultAPIError` (final, no retry) instead of silently resolving the wrong - item — a deliberate behavior change per issue #48. -- M1 as-shipped breaks the Keep a Changelog ordering (a new `### Fixed` - section inserted before `### Added`) → manual placement fix required. - -## Integration plan (S4, on Gitea `dev`) - -| File | Action | Note | -|---|---|---| -| `apps/billcollector/vault.py` | apply from head | pre-image byte-identical to dev | -| `apps/billcollector/runner.py` | apply from head | pre-image byte-identical to dev | -| `apps/vault_transport.py` | apply from head | pre-image byte-identical to dev | -| `tests/test_bc_vault_exact.py` | apply from head (new file) | absent in base and dev | -| `tests/test_vault_transport.py` | apply from head | pre-image byte-identical to dev | -| `CHANGELOG.md` | **manual (M1)** | drop the author's misplaced `### Fixed` section (inserted before `### Added`); add the entry at the top of the existing `## [Unreleased] → ### Fixed` section in house style: `- Resolve one exact full vault item name, then fetch credentials and fresh TOTP by its validated id (#48). The pinned local HTTP transport preserves endpoint queries while continuing to refuse queried base URLs and unsafe authorities.` | - -Commits (dev-commits, one concern each, proposed in S7): -1. `fix: resolve the exact vault item id before credential reads (#48)` — - vault.py + runner.py + vault_transport.py -2. `test: cover exact vault item identity and the query-preserving transport - pin (#48)` — test_bc_vault_exact.py + test_vault_transport.py -3. `doc: changelog + vault transport doc — exact vault item identity (#48)` - — CHANGELOG.md (M1 placement) + doc/vault_transport.md (S6 correction: - base/request query distinction, exact-id resolution description) - -## Validation plan (S5, on the exact dev tree, `apps/.venv/bin/python`) - -1. Before/after proof: the new `tests/test_bc_vault_exact.py` fails on the - pre-integration tree (name-based resolution), passes after (PR's stated - evidence: 130 passed, 30 subtests across - `test_vault_transport.py test_bc_vault_exact.py test_bc_login.py - test_bc_runner.py`; plus 18 passed, 14 subtests for - `test_vault_diagnostics.py test_bc_vault_settings.py`). -2. PR unit tests: `apps/.venv/bin/python -m pytest - tests/test_bc_vault_exact.py tests/test_vault_transport.py -v` → all - pass. -3. Full unit suite: `apps/.venv/bin/python -m pytest tests/ -q` → all pass - (watch for pinned behavior assertions in other modules, as in PR #56's - M2). -4. Full regression: `apps/.venv/bin/python tests/run_regression.py --ini - tests/bc_regression.ini` → exit 0, VERDICT: PASS (7 scenarios / 8 PDFs). -5. `ruff` + `bandit` on the changed files → nothing new beyond the known - baseline (PR claim: ruff + whitespace pass, bandit retains the same four - baseline findings). - -## Deployment validation - -**Recorded skip, user-confirmed at the S5 gate (2026-10-31).** Not -deployment-relevant: no `Dockerfile`, `.dockerignore`, `BillCollector.sh`, -`install_docker-image.sh`, `deploy_remote.sh`, INI key or `.env`-handling -change (the change is pure Python inside the existing runtime). Validated tree -basis for S7: dev `53bed6b` + the dirty integration (5 PR files + M1 -CHANGELOG); if the tree moves after S6, re-validate per the S7 prerequisite. - -## Out of scope - -- PRs #45, #44 (later in the batch-2 chain). -- The bw serve search-endpoint availability (operator environment, not - shipped by this PR). -- Any vault-side data change (item naming unchanged per issue #48). - -## Open questions - -- None: S3 confirmed, S4 applied, S5 validated (deployment skip - user-confirmed), S6 docs confirmed. S7 (commit + push, promote, mirror, - close) is the only remaining stage. - -## Execution record - -- S1 (2026-10-31, "continue pr 55" session): fresh S1 review — no plan doc - existed for #55 (plans jumped #54 → #56). Head fetched to `refs/pr/55` - (`397b0b61...`, matches `ls-remote`); PR metadata via the GitHub API - (issue #48, open PRs #55/#45/#44). Merge-base analysis untangled the - orphaned-tip base (`e83c8e2`, not an ancestor of GitHub `main` - `4265ddb`; merge-base `5c8216f`); real content = 6 files +177/−11 against - `e83c8e2` (the 104-file GitHub diff is an orphaned-tip STRIP artifact). - Pre-image vs Gitea `dev` `19c4164`: 4/5 existing files byte-identical - (clean hunks), the new test file a clean add, CHANGELOG diverged (manual - M1). Code review per file (vault.py exact-id resolution + error - sanitization, runner.py fresh-OTP id, vault_transport.py `allow_query` - scoping); process check PASS; cross-PR overlap limited to runner.py + - CHANGELOG. Verdict: approve with M1 (CHANGELOG placement). S1 gate passed - (user "proceed, yes", 2026-10-31). -- S2 (2026-10-31, same session): plan doc created (this file, - AWAITING-DECISION); README + sequencing cross-references updated; doc-only - commit pushed to `origin/dev`. S3 next. -- S3 (2026-10-31, "continue pr 55" session): proposal presented (5 files - as-is from head, M1 CHANGELOG placement, 3 commits); user confirmed as-is. -- S4 (2026-10-31, same session): preflight re-verified (dev = origin/dev = - `53bed6b` doc-only atop the reviewed `19c4164`; PR head still `397b0b61`; - 4 files byte-identical to the PR base, new test file absent). Applied the - 5 files from `refs/pr/55` to the worktree (worktree-only, index untouched) - and verified byte-identity: the 4 modified files share the PR delta's - patch-id `034ddf9b...`, the new file's blob hash `f5995304...` matches the - head. M1 applied: the author's misplaced `### Fixed` block (before - `### Added`) was NOT taken; the identical entry text was inserted at the - top of the existing `[Unreleased] → ### Fixed` section (above the #47 - entry). Working tree left dirty (5 code/test files + CHANGELOG) for S5. - Status: INTEGRATING. -- S5 (2026-10-31, same session): all checks green on the exact dev tree - (`53bed6b` + dirty integration), from repo root with the venv: - - Before/after proof: with the 3 production files stashed to base, the new - `test_bc_vault_exact.py` fails **8/8** (name-based resolution); with the - integration, all pass. - - PR unit tests (2 files): 30 passed, 30 subtests. - - PR-stated 4-file command (`test_vault_transport.py test_bc_vault_exact.py - test_bc_login.py test_bc_runner.py`): **130 passed, 30 subtests** — exact - match to the PR's stated evidence. - - PR-stated 2-file command (`test_vault_diagnostics.py - test_bc_vault_settings.py`): 18 passed, 14 subtests — exact match. - - Full unit suite: 675 passed, 10 skipped, 90 subtests (Δ +10 vs PR #56's - 665 = exactly the 8 new exact-identity tests + 2 new transport tests). - - Full regression: `run_regression.py --ini tests/bc_regression.ini` → - exit 0, VERDICT: PASS (7 scenarios / 8 PDFs). - - ruff (F821, E4/E7/E9) on all 5 changed files: all checks passed. - - bandit per-file vs the pre-image: vault.py / vault_transport.py / - test_vault_transport.py 0 findings (unchanged); runner.py 1×B110 - (pre-existing, identical to base); new test file 11×B101 (asserts) + - 3×B105 (`synthetic-password`) — test-file baseline only, no secrets. - Production files: zero new. - - Deployment validation: **recorded skip** — not deployment-relevant (no - Dockerfile / .dockerignore / BillCollector.sh / install_docker-image.sh / - deploy_remote.sh / INI key / .env change; pure Python inside the existing - runtime). - S5 gate passed: user confirmed the deployment skip and proceeded - (2026-10-31). Status: VALIDATED. -- S6 (2026-10-31, same session): CHANGELOG M1 entry stands as-is in the - `[Unreleased] → Fixed` section (sanitized, public-safe). No README or - CONTRIBUTING change: the operator-visible contract (`BW_API_URL`, item - naming, optional TOTP) is unchanged, only the internal resolution path - narrows. `doc/vault_transport.md` corrected (two edits, in tree, pending - the S6 gate): (a) the "URL credentials, query strings, fragments, HTTPS and - public targets are refused" line now distinguishes the base URL (still - refuses query strings) from request URLs (endpoint query preserved, e.g. the - item search query); (b) the stale "Exact item resolution and stable account - identity are separate work" + "The diagnostics fix in upstream PR #11 is - also needed" sentences (PR #11 already delivered; #48 now landed) replaced - with the exact-id resolution description. S6 gate passed: user confirmed - the docs (2026-10-31). S7 next: dev-commits (3 commits: code / test / - doc — the doc commit now covers CHANGELOG.md + doc/vault_transport.md), - push origin/dev, ff-only promote to main, sync_github.sh mirror, PR + - issue close comments per the publication routine. diff --git a/.kilo/plans/README.md b/.kilo/plans/README.md deleted file mode 100644 index 0cb59e1..0000000 --- a/.kilo/plans/README.md +++ /dev/null @@ -1,22 +0,0 @@ -# Kilo plan files - -Layout (since 2026-10-11): - -- **Root** (this directory): active plans — current execution plans and open - references. -- **`completed/`**: delivered, done, superseded, or consumed plans — retained for - the record, not for execution. - -Rule: when a plan is fully delivered (or superseded/consumed), move it to -`completed/` and update its cross-references. References use `.kilo/plans/...` -paths, so a moved file gains the `completed/` prefix. - -## Active - -| File | Role | -|---|---| -| `1792793307574-recipe-repair-llm-options.md` | Public service recipe recovery — LLM-assisted repair options analysis (pre-step); Stage 0 triage done 2026-10-24, all 7 services classified | -| `1792819039884-recipe-doctor-implementation.md` | Stage 1: offline LLM recipe doctor (`billcollector doctor`) implementation plan + pilot (buhl, datev) — both pilots green 2026-10-24 (runs 16/18, downloads succeeded); Stage 1b in progress (2026-10-25): 4/7 green (buhl, datev, nuernberger, winsim), remaining 1und1 (in progress), freenet_mobilfunk (not started), kabeldeutschland (Option D re-authoring) | -| `1792861675256-issue27-maintainer-commitments.md` | Issue #27 product-audit triage (closed 2026-10-24): maintainer-side commitments C1-C8 (F04, F09, F08 contract, F06 contract, F02, F03, F05, F10) with acceptance criteria, dependencies and proposed order | -| `1792862814701-next-up-sequencing.md` | Next-up sequencing decision (2026-10-24), superseded in part (2026-10-25): C1–C3 done and pushed; current primary next-up is the PR continuation order #30 → #31 → #33 → #32 → #34 (one pr-review chain at a time), interleaved with Stage 1b portal sessions (next: 1und1); PR #30 + #31 DELIVERED 2026-10-25, PR #33 DELIVERED 2026-10-26, PR #32 DELIVERED 2026-10-26 (Gitea `main` `bde713c` / GitHub `93bfcc8`), PR #34 DELIVERED 2026-10-27 (Gitea `main` `53a5dff` / GitHub `6bb3348`), PR #35 DELIVERED 2026-10-28 (Gitea `main` `75b885b` / GitHub `484048d`), PR #42 DELIVERED 2026-10-28 (Gitea `main` `2a6e462` / GitHub `c2fb7f8`), PR #39 DELIVERED 2026-10-28 (Gitea `main` `b693517` / GitHub `ad18f7a`), PR #40 DELIVERED 2026-10-29 (Gitea `main` `61defd3` / GitHub `f26b2a9`), PR #41 DELIVERED 2026-10-30 (Gitea `main` `74ba171` / GitHub `e83c8e2`), issue #38 closed — the planned chain #34 → #35 → #42 → #39 → #40 → #41 is complete; flowcool's archive-side contract proposal on #38 (2026-10-30) awaits a maintainer decision; PR #53 DELIVERED 2026-10-30 (Gitea `main` `fa1ade5` / GitHub `a24668a`), issue #46 closed, PR #54 DELIVERED 2026-10-31 (Gitea `main` `a75861f` / GitHub `ebf20bb`), issue #47 closed; PR #56 DELIVERED 2026-10-31 (Gitea `main` `2bf53d8` / GitHub `4265ddb`), issue #49 closed — batch-2 chain now at #55 (exact vault item identity, #48); PR #55 S1+S2 done 2026-10-31 (approve with M1 CHANGELOG placement, 5/6 files clean from head `397b0b6`, AWAITING-DECISION) | -| `1793461722991-pr55-vault-exact-item-identity-review.md` | PR #55 review (exact vault item identity, issue #48) — S1+S2 done 2026-10-31: verdict approve with M1 CHANGELOG placement, 4/5 existing files byte-identical from head `397b0b6` (orphaned-tip branch point `e83c8e2`) + new test file, Status: AWAITING-DECISION | diff --git a/.kilo/plans/completed/1790289661685-billcollector-daemon-webui-plan.md b/.kilo/plans/completed/1790289661685-billcollector-daemon-webui-plan.md deleted file mode 100644 index 509778a..0000000 --- a/.kilo/plans/completed/1790289661685-billcollector-daemon-webui-plan.md +++ /dev/null @@ -1,331 +0,0 @@ -# BillCollector — Daemon, Web UI & Interactive Sessions - -> Status: SUPERSEDED by 1790364274061-billcollector-nicegui-daemon-plan.md (v2). - -## Goal - -Transform BillCollector from a cron batch job into a 24/7 daemon with a web UI as its -"window": recipe-driven tasks are managed in the UI, the user can interactively step into -scraping sessions (CAPTCHAs, cookie banners, recipe repair after website changes), and the -system becomes easier for other users to adopt (onboarding, packaging, DMS-agnostic -document workflow for systems like Paperless-NGX). - -## Current state (facts from the code) - -- Batch: cron → `BillCollector.sh` → `docker run` → `python3 BillCollector.py [debug]` -- Playwright is hardcoded (`BillCollector.py:255`); the Selenium half is dead code -- Config: `bc_default.ini` / `bc_test.ini` (service + user list), recipes as YAML in - `recipes_playwright/` validated with jsonschema (`recipe-pw-schema.yaml`) -- Credentials: Vaultwarden (Bitwarden API), name-based link (`Service User` → vault item) -- State: SQLite `bc.db` — one table `PageStatus__RunN` per run (unbounded growth) -- The Playwright profile is deleted and cookies cleared on every run - (`BillCollectorServices_pw.py:35` `shutil.rmtree`, `:405` `clear_cookies()`) → - every run is a cold start (cookie banners, CAPTCHAs on every run) -- Recipe authoring: `playwright codegen` + translation to YAML - (`helpers/BillCollectorCreateRecipe_pw.py`) -- Debug: VS Code F5 on `BillCollector.py` → automatically `bc_test.ini` + SPACE pauses -- Recipe steps are method chains (locator methods + `click`/`fill`/`goto`/`expect_download`), - variables `{{USERNAME}}`/`{{PASSWORD}}`/`{{OTP}}` - -## Decisions - -1. **Daemon (24/7)** with an internal scheduler; the UI is the window to the daemon. - Cron is retired in M4. -2. **Stack:** FastAPI (uvicorn) + Jinja2 + htmx. No SPA/Node toolchain. Python stays the - only language. -3. **Auth:** single user, session cookie, UI behind the existing NPM proxy with HTTPS. - The UI password is stored in the DB `settings` table (bcrypt hash), optionally seeded - from `.env` on first start and changeable later via onboarding/settings (no `.env` edit). - No Vaultwarden SSO. -4. **Task** = one `(Service, User)` pair (today: one ini line). Carries: recipe file - reference, enabled flag, cron schedule, last/next run, status, profile directory. -5. **Source of truth:** redesigned SQLite DB. One-time import of the ini files. - **Recipes stay as YAML files on disk** (`recipes_playwright/`), actively versioned in git. - Vaultwarden link stays name-based. -6. **Daemon git:** recipe changes made in the UI are auto-committed (`git add` + `commit`, - structured message, toggleable in settings). -7. **Persistent browser profile per task** (`browser/profiles/_/`): - never deleted automatically, no `clear_cookies()`, resettable via UI. - Goal: minimize cookie banners and CAPTCHA prompts. -8. **Interaction in paused sessions, two levels:** - - **Copilot (primary):** live screenshot + numbered element overlays (reusing the - `PageState.set_interactive_elements` logic); click/fill executed via API by the daemon. - - **noVNC (fallback):** Xvfb + x11vnc + websockify in the image, noVNC embedded in the UI; - full mouse/keyboard control (e.g. drag-and-drop CAPTCHAs). -9. **Pause triggers:** a step fails 3× → status `waiting`; explicit recipe method - `wait_user(prompt)` for known friction points. **No** CAPTCHA heuristics (deliberate). -10. **Waiting session:** held for 30 min (configurable), then aborted as `needs_attention`; - profile is preserved, manual retry possible. -11. **Notification:** UI badge (polling/WebSocket) + ntfy push on pauses. E-mail: out of scope. -12. **Downloads:** existing folder drop stays (DMS-agnostic) + DB tracking - (service, task, timestamp, filename, size, sha256) + deduplication (same hash) - + post-download hook (later Paperless API consumer, without core changes). -13. **Concurrency:** default max 1 parallel task (NAS hardware), configurable. -14. **UI language:** English (public project). -15. **Test fixture:** local mock portal (cookie banner, login, fake CAPTCHA with a displayed - code, drag CAPTCHA, document download) for M1–M3. -16. **Headful always:** every task browser runs under Xvfb (virtual display). noVNC is - therefore available for any session — running or waiting. Copilot screenshots stay - Playwright-native (`page.screenshot()`). - -## Target architecture - -``` -[Scheduler (APScheduler, persistent jobstore)] - │ -[Task queue (global semaphore, per-task locks)] - │ -[TaskRunner (Playwright, pausable)] ←→ [Waiting sessions (in memory)] - │ │ -[Vaultwarden client] [Download tracker] [Copilot API / WebSocket] - │ │ -[SQLite: Task/Run/PageStatus/Download] [noVNC (Xvfb + x11vnc)] - │ -[FastAPI + htmx UI] [ntfy client] [Git committer] -``` - -### Code layout - -- New Python package `apps/billcollector/` (daemon core): - - `cli.py` / `__main__.py` (`serve`, `run `) - - `config.py` (`.env` + settings), `db.py` (SQLite WAL, migrations, Task/Run/PageStatus/Download) - - `scheduler.py` (APScheduler, persistent jobstore) - - `runner.py` (pausable task runner, Playwright, per-task profiles) + - `pagestate.py` (element scan logic extracted from `BillCollectorServices_pw.py`) - - `vault.py` (Vaultwarden client: timeouts, retries, clean TOTP handling) - - `downloads.py` (tracking, dedup, post-download hook), `notify.py` (ntfy), `gitcommit.py` - - `api/` (FastAPI routes), `web/` (Jinja templates + htmx + static), `consumers/` (hook slot, empty) -- The step-execution logic in `BillCollectorServices_pw.py` - (`transform_step_to_json`, `process_step`, `PageState`) is **ported to async** in - `billcollector/runner.py` / `pagestate.py` for the daemon. The legacy `BillCollector.py` - batch path keeps its current sync Playwright until M4 and reuses the shared `vault.py` / - `downloads.py` modules instead of its own copies. -- `tests/` (pytest) + `tests/mock_portal/` (mock portal, from M1) - -### Data model (SQLite, WAL mode, daemon = sole writer) - -- `Task(id, service, user, recipe_file, enabled, cron, last_run, next_run, status, profile_dir, created_at, updated_at)` -- `Run(id, task_id, started_at, ended_at, result, pause_reason, notes)` -- `PageStatus(id, run_id, step_number, locator_action, interactive_elements, error)` — - **one** table, replacing the table-per-run structure -- `Download(id, run_id, task_id, url, filename, size, sha256, saved_path, created_at, duplicate_of_id NULL)` -- `Settings(key TEXT PRIMARY KEY, value TEXT)` — UI password hash, ntfy topic, pause - deadline, concurrency, git-commit toggle, download folder, `UI_BASE_URL` - (all overridable at runtime in the UI) -- Migration: one-time import of `bc_default.ini` / `bc_test.ini` (section `[Playwright]`) → - tasks (default: monthly, disabled until confirmed in the UI), performed on first daemon - start (M1). The legacy `Service` / `PageStatus_*` tables are left as-is (the DB is runtime - status, not an archive). - -### Recipe schema extension - -- New method `wait_user(prompt)`: pauses the task with a user-visible prompt in the UI. -- The method-chain format otherwise stays unchanged; `recipe-pw-schema.yaml` is extended - with the method; `CheckRecipe` continues to validate. -- **Recipes are per-service (shared):** a recipe file is keyed by service - (`recipe-pw__.yaml`) and shared by all tasks of that service (same portal flow, - different credentials). Editing a recipe in the UI affects every task of that service; - the UI must make this explicit. - -### Execution flow - -1. Scheduler (per-task cron) → task enabled? → task lock + global semaphore -2. `launch_persistent_context(user_data_dir=)` under Xvfb — profile reset only via UI -3. Execute recipe steps; each step → `PageStatus` row (reusing the `PageState` logic - from `BillCollectorServices_pw.py`) -4. Step failure: up to 3 attempts (existing per-step timeout) → then pause -5. Pause: DB status `waiting` + session kept in memory, ntfy push, UI badge, 30-min deadline -6. User interacts: Copilot (screenshot, element click, text entry) or noVNC (full control) - → "Resume" -7. Deadline exceeded: close session, status `needs_attention`, profile preserved -8. Download: track (sha256), dedupe (hash already present for the task → skip, - set `duplicate_of_id`), call post-download hook -9. Finalize run (result in DB); recipe changed via UI → write YAML, schema validation, git commit - -### Runner & pause mechanism - -- The **daemon** runner uses **async Playwright** (`async_playwright`); each task run is an - asyncio task. (The M0 batch path keeps today's sync Playwright unchanged.) -- A run is a step loop. Between steps it checks a per-run `asyncio.Event`: - - **Pause** = set status `waiting`, keep the browser context + page alive, then - `await pause_event.wait(timeout=)`. - - **Resume** = the API handler sets `pause_event`; the loop re-executes the failed step - from the beginning. - - **Timeout** = `wait()` returns at the deadline; the run closes the context and sets - `needs_attention`. -- **Copilot actions** (click/fill) are executed directly by the async API on the shared live - context (`await locator.click()` / `await locator.fill()`) — safe because everything runs - on the single asyncio event loop. Screenshots are `await page.screenshot()` per WebSocket tick. -- **noVNC** is independent: it drives the Xvfb display directly (x11vnc), not via Playwright. -- **Xvfb display allocation:** one virtual display per concurrent session (e.g. `:99`, `:100`), - started/stopped with the session. - -### Edge cases & semantics - -- **Daemon restart during a run:** all `running` / `waiting` runs are marked `interrupted` - at startup (manual retry via UI; browser profile preserved). -- **Pause granularity:** pause happens **between** steps. After resume, the current step is - re-executed from the beginning (never mid-step). User interaction while `waiting` does not - advance the step. -- **Copilot view:** running session = read-only live view; interactive (click/fill/resume) - only in status `waiting` — prevents races between step execution and user input. -- **Deduplication:** permanent per task by sha256 (repeat run with the same document → skip, - set `duplicate_of_id`); no time window. -- **Post-download hook:** simple callback interface in the `consumers/` package - (`consume(download, task)`), initially empty — no plugin system. -- **Git commit:** only the changed recipe file is committed; identity from the existing git - config, fallback `BillCollector Daemon `; git failure → warning - in the log, the file write is not rolled back. -- **ntfy:** title `BillCollector: waiting`, body = reason + UI link - (new env var `UI_BASE_URL`). -- **noVNC security:** x11vnc with password (`-rfbauth`) + NPM host with HTTPS/basic auth. -- **Python versions:** image target 3.12 (Ubuntu 24.04); code must also run in the local - dev venv (3.8) → prefer 3.8-compatible version pins (APScheduler 3.x, pydantic v2). -- **Vault item name:** stays `f"{service} {user}".strip()` (current behavior); a task with an - empty user → name = service. - -### API (FastAPI) - -- `/api/auth/login`, `/api/auth/logout`, `/api/health` -- `/api/tasks` (CRUD), `/api/tasks/{id}/run`, `/api/tasks/{id}/reset-profile`, - `/api/tasks/{id}/runs`, `/api/tasks/{id}/downloads` -- `/api/recipes` (list/detail), `/api/recipes/{name}` (GET/PUT with schema validation) -- `/api/sessions` (waiting sessions), `/api/sessions/{run_id}/screenshot`, - `/api/sessions/{run_id}/elements`, `/api/sessions/{run_id}/click`, - `/api/sessions/{run_id}/fill`, `/api/sessions/{run_id}/resume`, - `/api/sessions/{run_id}/cancel` -- `/api/downloads`, `/api/settings`, `/api/config/import-ini` -- WebSocket: session live view (screenshot + elements on change), dashboard badge - -### UI pages - -Login · Dashboard (task table: status, last/next run, waiting badge) · -Task detail (history, per-run step results from `PageStatus`, downloads, actions) · -Session view (Copilot: screenshot + overlays + action bar; noVNC tab) · Recipe editor -(YAML + step tree + validation; on failed step: failure context with candidate elements) · -Downloads (dedup status) · -Settings (password, ntfy topic, pause deadline, concurrency, git-commit toggle, download folder) · -Onboarding - -### Docker / deployment - -- Image `billcollector:latest` becomes a long-running service (uvicorn; plus Xvfb, x11vnc, - websockify, `git` for auto-commits; **non-root user**) -- Volumes: `Downloads`, `db`, `browser/profiles`, git repo (recipes + commits), `.env` -- Ports: 8000 (UI) + noVNC (websockify) — both via NPM with HTTPS -- Config: the daemon reads `apps/.env` (existing `VAULT_HOST`, `BW_API_URL`) plus new vars - (`UI_BASE_URL`, ntfy topic, pause deadline, concurrency, git-commit toggle, optional - UI password seed); runtime-overridable settings live in the `settings` table -- `BillCollector.sh` + cron stay operational until M4 (M0 stabilizes them) - -## Milestones - -### M0 — Stabilize the batch system (cron keeps running during development) - -1. Remove the Selenium half: `BillCollectorServices.py`, `recipes_selenium/`, - `chrome-linux64*` / `chromedriver-linux64*` (repo + Dockerfile), `[Selenium]` sections - from `bc_default.ini` / `bc_test.ini`, `selenium` from `requirements.txt`, `test_selenium.py` -2. Error propagation: a failed step aborts the run (respecting the `graceful` flag), - result recorded in the DB instead of silently continuing -3. Vaultwarden client: timeouts + retry (3×) for all API calls; replace the "No TOTP" - magic string (`BillCollector.py:89`) with proper status handling -4. flock against duplicate runs (lock file `apps/.bc.lock`) -5. `logging` module instead of the `print = logging.debug` hack; `RotatingFileHandler` - for `bc.log`; remove the `os.chdir` side effect in `latest_download_file()` (absolute paths). - **No DB schema change in M0** — the batch path keeps writing to the existing - table-per-run structure unchanged; the single-table `PageStatus` redesign lands in M1's `db.py`. -6. Docker: enable the non-root user (Dockerfile, commented-out block) - -**Exit:** the existing cron run completes unchanged; a deliberately broken step is visible -in log + DB as an abort; a double start is rejected. - -### M1 — Daemon core - -1. FastAPI app skeleton (uvicorn); extend `.env` config (optional UI password seed, ntfy - topic, pause deadline, concurrency, git-commit toggle, `UI_BASE_URL`) -2. DB redesign (Task/Run/PageStatus/Download, WAL) + one-time ini import -3. TaskRunner: Playwright logic refactored from `BillCollectorServices_pw.py`, pausable - (pause/resume API), persistent per-task profile, no profile deletion / cookie clears -4. APScheduler: per-task cron, persistent jobstore, global semaphore (default 1), - per-task locks -5. Pause triggers (3× failure, `wait_user`) + 30-min hold + ntfy push + DB status -6. Download tracking + deduplication + post-download hook -7. CLI: `python -m billcollector serve` / `python -m billcollector run `; - the legacy `BillCollector.py` stays the batch entry point until M4 - -**Exit:** the daemon runs tasks on schedule; on the mock portal: pause → ntfy → resume → -download tracked; profile accumulation verifiable (cookie banner appears only on the 1st run). - -### M2 — UI core - -1. Login + session cookie (bcrypt) -2. Dashboard: task table (status, last/next run, "N waiting" badge via WebSocket/polling) -3. Task management: create (service, user, recipe, schedule), enable/disable, run now, - profile reset, history, downloads -4. Recipe editor: YAML editor + step-tree view + validation via `CheckRecipe`; - save = file write + validation + git commit -5. Downloads page (dedup status), settings page, ini import - -**Exit:** the full task lifecycle is operable from the UI against the mock portal; -recipe edit → save → validation → git commit works. - -### M3 — Interactive sessions - -1. Copilot: WebSocket live view (screenshot + numbered element overlays from the - `PageState` logic), click/fill actions, resume -2. noVNC: Xvfb + x11vnc + websockify in the image, embedded noVNC view on the session page -3. Failure context in the recipe editor: screenshot at failure time, expected locator, - candidate elements → pick/edit → save (→ git commit) -4. Git auto-commit on recipe changes (structured message: `recipe(): `) -5. `wait_user` method in schema + runner - -**Exit (mock portal, E2E):** cookie banner → Copilot click; CAPTCHA code typed via Copilot; -drag CAPTCHA via noVNC; recipe repair after a "changed website" → changed YAML + -commit visible in `git log`. - -### M4 — Adoption for other users - -1. Onboarding wizard (first start: set password, vault config, ini import, create first task) -2. Docker Compose package (BillCollector, optional ntfy) -3. README/docs update: daemon mode, UI usage, recipe authoring; cron→daemon migration guide -4. Mark `BillCollector.sh` / cron as deprecated - -**Exit:** a new user sets up the stack from the docs and collects a document from the mock -portal — without shell access to the daemon. - -## Validation - -- **M0:** production cron run; debug via F5 (`bc_test.ini`); `BillCollectorCheckRecipe.py` - over all remaining recipes -- **M1:** pytest (DB migration/import, scheduler, Vaultwarden client, pause/resume); - integration test against the mock portal (headful under Xvfb) -- **M2:** Playwright UI tests (login, task CRUD, recipe save); schema tests incl. `wait_user` -- **M3:** E2E on the mock portal: Copilot flows (banner, code entry), noVNC flow (drag), - recipe repair + `git log` check -- **Mock portal:** `tests/mock_portal/` — small FastAPI app: cookie banner, login, - fake CAPTCHA (displayed 6-digit code), drag-and-drop CAPTCHA, document download - -## Risks - -- **CAPTCHA detection:** no heuristics by design — only `wait_user` + auto-pause after 3× - failure. No "CAPTCHAs are solved automatically" promise in the docs. -- **SQLite concurrency:** WAL + daemon as sole writer; UI is read-mostly. -- **Chrome/Xvfb + persistent profiles:** pin Playwright/Chromium versions in the image. -- **Async port fidelity:** porting `process_step` to async must preserve current semantics — - `VARIABLE_MAP` placeholder substitution, nested `expect_download` blocks, `first()` - chaining, per-step `PageStatus` recording. Verify against the mock portal that the async - runner behaves identically to the batch path. -- **NAS resources:** concurrency default 1; profiles open only for running tasks; - one Xvfb display per session. -- **Git auto-commit:** repo as bind mount, single user → conflicts unlikely; toggle in settings. -- **Vaultwarden API:** the fragile "No TOTP" text matching is replaced in M0; status checks - stay idempotent. -- **Non-root switch (M0):** bind-mount ownership (Downloads, db, profiles) must be adjusted - for the new user. - -## Out of scope - -- E-mail / SMTP notification (later optional add-on) -- Paperless REST API upload (post-download hook only; later consumer) -- Multi-user / multi-tenancy (single user per instance) -- `playwright codegen` in the UI (advanced; stays a VS Code dev workflow) diff --git a/.kilo/plans/completed/1790364274061-billcollector-nicegui-daemon-plan.md b/.kilo/plans/completed/1790364274061-billcollector-nicegui-daemon-plan.md deleted file mode 100644 index 1e1b648..0000000 --- a/.kilo/plans/completed/1790364274061-billcollector-nicegui-daemon-plan.md +++ /dev/null @@ -1,1463 +0,0 @@ -# BillCollector — Daemon, NiceGUI Web UI & Interactive Sessions (v3) - -Supersedes `.kilo/plans/completed/1790289661685-billcollector-daemon-webui-plan.md`. -Goal (sharpened in v3): turn the cron batch job into a 24/7 daemon whose web UI is its -window — recipe-driven task management, interactive stepping into scraping sessions, -and easier adoption for other users (one instance per operator, DMS-agnostic document -workflow). - -**User-friendliness objective (v3):** the daemon makes BillCollector as easy to operate -as possible: the robot does the routine work, and at every friction point it stops with -a clear "here is what happened — here is what you need to do" screen that a human -resolves from the browser (desktop or phone) — no shell access, no YAML needed for the -common case. - -**First development wave (v3):** the three current complicating circumstances, each -supported by human interaction within the daemon + web UI environment: -1. websites often change their DOM and the path to the documents' download area, -2. sudden ads or cookie-request forms jump into the way of the user / BillCollector, -3. CAPTCHAs are difficult to circumvent for a robot like BillCollector. -The architecture must stay open for future ML / LLM-based AI support to automate tasks — -**human interaction is still required** (human-in-the-loop is a permanent design gate, -see "Intervention architecture"). AI implementation itself is out of Wave 1 scope. - -Refinements vs. v2: -1. **Wave 1 framing** — M1+M2+M3, with the three friction flows as acceptance criteria - (the mock portal's Layer B sites are the acceptance fixture, delivered in v0.4). -2. **Intervention architecture** — every pause carries a machine-readable context and is - resolved through a stable, versioned action protocol. The web UI is the first client; - a future AI assistant is a second client of the same protocol, gated by human approval. -3. **Corrections & additions** — `wait_user` resume semantics (v2's "re-pause is - acceptable" was a trap; now: complete-on-resume + optional `until` verification - locator, the AI-friendliest variant); the 3× step retry is explicitly new runner - behavior (the batch engine has no retry); the element scan gains bounding boxes - (needed for the overlay + the AI context); the protocol gains `context` / `press` / - `drag` actions; ntfy pushes carry a deep link to the session page. - -Carried over from v1/v2 (unchanged): -- **UI stack: NiceGUI** (on FastAPI/uvicorn) instead of Jinja2+htmx — no SPA/Node - toolchain, Python stays the only language, and the hard parts of this UI (live - dashboard, live screenshots, log streaming) are NiceGUI's native patterns. -- **Decoupled processes:** the UI process is a pure orchestrator. Each task run is an - isolated `python -m billcollector run ` **subprocess** running the **existing - sync Playwright** code — no async port, zero crash propagation (a Playwright OOM never - takes down the UI/scheduler), and full OS-level memory reclamation after every run. -- **Dependency baseline (checked 2026-09-25):** Python 3.12 everywhere (image already - on 3.12; dev environment migrates to Ubuntu 24.04 in M0). Direct pins: - `nicegui==3.17.1`, `playwright==1.63.0`, `apscheduler==3.11.3`, `bcrypt==5.0.0`. - -## Current state (facts from the code) - -- Batch: cron → `BillCollector.sh` → `docker run` → `python3 BillCollector.py [debug]` -- Playwright is the only automation path (`BillCollector.py:255`); the Selenium half - was removed in M0 -- `BillCollectorServices_pw.py` is **fully synchronous**: `retrieve_from_service_with_playwright( - service, url, user, pwd, otp, debug)`, `process_step`, `PageState.set_interactive_elements(page)`, - `DatabaseManager` (table-per-run `PageStatus__RunN`) -- Config: `bc_default.ini` / `bc_test.ini` — `[Playwright]` lines `service=user1, user2` - (e.g. `winSIM=Stefan, Auto, Brigitte, Eva, Anna`); recipes as YAML in - `recipes_playwright/` validated by `helpers/BillCollectorCheckRecipe.py` + - `recipe-pw-schema.yaml` -- Credentials: Vaultwarden, name-based link (`f"{service} {user}".strip()` → vault item) -- State: SQLite `bc.db`; Playwright profile deleted + cookies cleared every run - (`BillCollectorServices_pw.py:46`, `:434`) → cold start every run -- Image (`Dockerfile`): ubuntu 24.04, Python 3.12, **xvfb already installed**, - **non-root user (UID/GID 1000) in the Dockerfile**, `CMD ["/bin/bash"]`; the - deployed NAS image was rebuilt from the current requirements on 2026-10-05 - (Playwright 1.63.0) — M0 item 6 done -- Dev environment: Ubuntu 24.04 (WSL2) host, system Python 3.12 (migrated in M0 step 0 - — see M0 status below); `apps/.venv` created by `install_local.sh playwright` from - the **system** `python3` (24.04 apt package names). No CI (`.github` holds only issue - templates). -- `requirements.txt` is a full `pip freeze` regenerated on 3.12 (M0 step 0.4): direct - pins playwright 1.63.0, nicegui 3.17.1, apscheduler 3.11.3, bcrypt 5.0.0, sqlalchemy - plus PyYAML, jsonschema, requests, python-dotenv, nslookup, flatten-json, sshkeyboard - (SPACE-pause helper); `selenium`, stale `dotenv`, unused `websocket-client` removed -- Debug: VS Code F5 → `bc_test.ini` + SPACE pauses -- **Batch engine has no retry:** a failed step aborts the run on the first failure - (`fail_step` in `BillCollectorServices_pw.py:592-599`); the 3×-retry-then-pause is - **new daemon-runner behavior**, not existing -- **Element scan has no coordinates:** `PageState.set_interactive_elements` - (`BillCollectorServices_pw.py:179-380`) returns role/name/candidate-locator per - element but **no bounding boxes** — the Copilot overlay and the machine-readable - pause context both need `getBoundingClientRect()` added -- **Regression environment delivered (v0.4):** `tests/` — mock portal (FastAPI, port - 8787) with Layer A sites (happy/graceful/badlogin/empty/dlfail) and **Layer B - friction sites** (`waituser`: displayed 6-digit code · `captchadrag`: slider · - `autopause`: "I am not a robot" interstitial), `POST /admin//mutate|reset` for - the "changed website" drill; Layer B recipes - (`recipe-pw__testwaituser|testcaptchadrag|testautopause.yaml`) use the `wait_user` - method (free-form schema — already validates); `tests/scenarios.py` = single source - of truth. Layer B execution is deferred to the daemon runner = **Wave 1**. See - `.kilo/plans/completed/1790714158680-local-regression-test-web-service.md` (DELIVERED, all - validation green incl. vault e2e) -- **Fast-track manual run UI delivered:** `apps/bc_ui.py` — minimal NiceGUI subprocess - supervisor (start/stop + live log); absorbed into `billcollector/ui/` in M2 - -## Library state (verified on PyPI, 2026-09-25) - -| Package | Latest | requires_python | Notes | -|---|---|---|---| -| NiceGUI | 3.17.1 | `>=3.10,<4` | UI framework; FastAPI/uvicorn underneath | -| Playwright | 1.63.0 | `>=3.10` | bundles Chromium 153.0.8010.12; requirements pin 1.63.0, deployed NAS image still 1.48.0 | -| APScheduler | 3.11.3 | `>=3.8` | 3.x is the current stable line; SQLAlchemy jobstore | -| FastAPI | 0.141.1 | `>=3.10` | NiceGUI dependency | -| uvicorn | 0.54.0 | `>=3.10` | NiceGUI dependency | -| bcrypt | 5.0.0 | `>=3.8` | UI password hashing | - -Consequences: -- The modern web stack requires **Python >=3.10** — the 3.8 dev environment is a dead - end (3.8 has been EOL since Oct 2024). The image (ubuntu 24.04, Python 3.12) already - matches the target baseline. -- **APScheduler 3.x** (3.11.3) is the current stable line → the "APScheduler 3.x + - SQLAlchemy jobstore" decision stands. -- Playwright 1.48 → 1.63 spans 15 minor releases: the image must be rebuilt with a fresh - `playwright install chromium` (Chromium 153). - -## Wave 1 — objective and acceptance - -**Scope:** M1 (daemon core) + M2 (UI core) + M3 (interactive sessions). M0, M1, M2 -and M3 are done (status below); the Wave 1 exit is complete — the live exit pass -(2026-10-18) and the NAS deploy + daemon cutover (Post-Wave 1 item C, -2026-10-22, `.kilo/plans/completed/1792384312843-wave1-nas-deploy.md`); -M4 (adoption for other users) is done (exit pass passed 2026-10-21). -Post-Wave 1 concessions: B (login-failure lock) delivered via the #20 -contract / PR #22 (DELIVERED 2026-10-23); A (date variables) is followed up -separately in GitHub issue #26 (contract confirmed 2026-10-24; flowcool -implements as a PR against main) and is no longer tracked in this plan. -All plan items are done or transferred — the plan is complete (2026-10-24). -Wave 1's focus is the three current complicating circumstances, each made operable by a -human from the web UI: - -| # | Friction | How Wave 1 handles it (human interaction) | -|---|---|---| -| 1 | Site changed its DOM / path to the download area | The step's locator fails → 3 attempts → pause with **failure context** (screenshot at failure, expected locator, candidate elements). **One-off:** human clicks the real element (Copilot) or navigates (noVNC) → resume → run finishes. **Permanent (repair):** from the pause, the recipe step is fixed — pick a candidate element or edit the step, validate, save, git commit → the next run passes unattended. | -| 2 | Sudden ads / cookie-request forms in the way | **Prevention:** persistent per-task profile → banner only on the 1st run (known banners can additionally be encoded as `graceful` steps, the existing pattern). **Reaction:** the overlay blocks the expected step → same pause flow; the overlay is visible in the screenshot; human dismisses it (Copilot click or noVNC) → resume. No auto-dismiss heuristics. | -| 3 | CAPTCHAs | **No detection heuristics** — three explicit paths: (a) displayed code: recipe `wait_user(prompt)` → pause → human reads the code from the screenshot, types it via Copilot `fill` (or noVNC), presses Verify on the page → resume (optional `until` check verifies). (b) drag/slider: `wait_user(prompt)` → human drags via Copilot `drag` or noVNC → resume. (c) unknown challenge: the 3×-failure **auto-pause** is the generic safety net — the human sees the challenge in the screenshot and acts. | - -**AI-readiness (Wave 1 scope line):** Wave 1 implements the **architecture, not the AI**: -a stable versioned intervention protocol, a machine-readable pause context, and the -human-approval gate (below). No ML/LLM code, no model calls, no API keys. A future -assistant plugs in as a **second client of the same protocol**; its actions are -*proposals* the operator approves in the UI before the runner executes them — human -interaction stays required by construction, not by convention. - -**Acceptance (all on the mock portal, Layer B — the delivered fixture):** -1. `testautopause`: 3× step failure → `waiting` + ntfy (deep link) → human clicks "I am - not a robot" via Copilot → resume → step re-executes → download tracked. -2. `testwaituser`: `wait_user` pause with prompt → human types the displayed 6-digit code - via Copilot `fill` + clicks "Verify" on the page → resume → download. -3. `testcaptchadrag`: `wait_user` pause → human drags the slider via Copilot `drag` or - noVNC → resume → download. -4. Persistent profile: `testhappy` run 1 shows the banner (graceful step succeeds); run 2 - (same task) has the banner absent; login re-runs (session cookie expired). -5. Download dedup: two `testhappy` runs → run 2's downloads deduped by sha256. -6. "Changed website" drill: `POST /admin/happy/mutate` renames "Log in" → run pauses at - the login step with failure context → repair in the recipe UI (candidate picker or - YAML) → validate + save + git commit (visible in `git log`) → re-run succeeds → - `POST /admin/happy/reset`. - -## Decisions - -1. **Daemon** = one long-running process: `python -m billcollector serve` (NiceGUI on - uvicorn) hosting UI + scheduler + orchestrator. Cron retired in M4. -2. **Process isolation:** the orchestrator spawns each task run as a subprocess - (`asyncio.create_subprocess_exec`), monitoring it and streaming its stdout to the UI - (`ui.log`) and to a per-run log file. The UI never imports scraping code. -3. **Sync Playwright stays.** The runner refactors the existing sync logic from - `BillCollectorServices_pw.py` — no async port (v1's "async port fidelity" risk is - eliminated). Job lifecycle is async in the UI layer only. -4. **UI stack:** NiceGUI (FastAPI + uvicorn underneath). No Jinja2, no htmx, no custom - REST API — UI actions run as NiceGUI event handlers. Health = NiceGUI's built-in - `/_alive`. -5. **Auth:** single operator per instance (confirmed). Session cookie login; bcrypt hash - in the DB `settings` table, optionally seeded from `.env` on first start, changeable in - Settings. "User" in this system always means **portal account** (the service+user pair), - never a login identity. No multi-user/multi-tenancy. -6. **Task** = one `(Service, User)` pair (today: one ini line). Carries: recipe file - reference, enabled flag, cron schedule, last/next run, status, profile directory. -7. **Source of truth:** redesigned SQLite DB, one-time ini import. **Recipes stay as YAML - files on disk** (`recipes_playwright/`), git-versioned. Recipes are per-service - (`recipe-pw__.yaml`), shared by all tasks of that service — the UI makes this - explicit. Vaultwarden link stays name-based. -8. **Daemon git:** recipe changes made in the UI are auto-committed (toggleable). -9. **Persistent browser profile per task** (`browser/profiles/_/`): never - auto-deleted, no `clear_cookies()`, resettable via UI. Minimizes cookie banners / - CAPTCHAs. (This is the per-portal-account isolation; it replaces the expert's - per-login-identity directories.) -10. **Interaction in paused sessions, two levels:** - - **Copilot (primary):** live screenshot + numbered element overlays positioned by - bounding boxes (reusing `PageState.set_interactive_elements`, extended — see - "Current state"); click/fill/press/drag executed by the runner on demand; - mobile-first (the phone is a first-class intervention device). - - **noVNC (fallback):** per-run Xvfb + x11vnc + websockify; noVNC web client embedded - in the UI via `ui.html` iframe; full mouse/keyboard for anything Copilot cannot - express. -11. **Pause triggers:** a step fails 3× → status `waiting`; explicit recipe method - `wait_user(prompt[, until])` for known friction points. **No** CAPTCHA heuristics. - The 3× retry is new runner behavior (the batch engine aborts on first failure): - the runner re-executes the step's method chain up to 3 times — `expect_download` - nested steps included — before pausing. -12. **Waiting session:** held 30 min (configurable), then aborted as `needs_attention`; - profile preserved, manual retry possible. -13. **Notification:** UI badge (live element updates / `ui.timer`) + ntfy push on pauses, - with a **deep link** to the session page (`/runs/`) — the - operator can act from a phone without opening the dashboard first. -14. **Downloads:** existing folder drop stays (DMS-agnostic) + DB tracking - (service, task, timestamp, filename, size, sha256) + per-task sha256 dedup + - post-download hook (`consumers/`, initially empty) executed **inside the runner - subprocess**. -15. **Concurrency:** default max 1 parallel task (NAS hardware), configurable. -16. **Headful always:** every run's browser targets a virtual display allocated per run - (file-lock registry, starting at `:99`), started and reaped with the run. noVNC is - therefore available for any session — running or waiting. -17. **UI language:** English. -18. **Python baseline 3.12 everywhere:** image already 3.12 (ubuntu 24.04); dev - environment migrates to Ubuntu 24.04 (WSL2/VM) in M0 and `.venv` is rebuilt from the - system `python3` (3.12). Direct deps pinned, transitive deps regenerated as a full - freeze. -19. **`wait_user` resume semantics (v3 correction, user-confirmed):** on resume the - `wait_user` step **completes** (it is consumed, not re-executed — v2's "re-pause is - acceptable" would trap the operator, since a consumed CAPTCHA code cannot be - re-entered). Optional `until` locator argument: before completing, the runner polls - the page for that locator, bounded by the pause deadline; found → complete, - deadline exceeded → re-pause with the same prompt. Without `until`: complete - immediately; subsequent recipe steps verify the action (failure there pauses again - with fresh context). This is the most AI-friendliest variant: the `until` check is - the objective "did the action work" signal that future AI proposals will be checked - against. -20. **Intervention protocol + pause context are a stable, versioned contract** — the - mechanism that keeps the architecture open for future ML/LLM support (see - "Intervention architecture"). The runner is client-agnostic: it executes explicit - action requests; Wave 1 has exactly one client (the human web UI). No AI code in - Wave 1. -21. **Operator reach:** every pause pushes ntfy with the session deep link (decision 13); - the Copilot session view is mobile-first; the candidate-element recipe repair is the - primary no-YAML fix path. - -## Target architecture - -``` -[serve process — 24/7, NiceGUI on uvicorn] - [NiceGUI pages] [auth middleware] [live logs: ui.log] - [Scheduler (APScheduler, persistent jobstore)] - [Orchestrator: task locks (DB row), global semaphore, - create_subprocess_exec, stdout→log file+ui.log, exit handling] - │ spawns, one subprocess per run -[runner subprocess: python -m billcollector run (sync Playwright)] - [Xvfb :N + x11vnc + websockify — child processes, reaped on exit] - [step loop / pause loop] [control socket (unix, JSON)] [PageState] - [Vaultwarden] [downloads+dedup+hook] [git committer] - │ -[SQLite WAL bc.db: Task / Run / PageStatus / Download / Settings] ← shared state bus -``` - -### Process & IPC design - -- **serve process:** `ui.run(port=8000, on_startup=..., on_shutdown=...)`. Startup: - start APScheduler, kill orphaned runner subprocesses (from `Run.pid`), mark leftover - `running`/`waiting` runs `interrupted`. Shutdown (SIGTERM): terminate child runners, - mark their runs `interrupted`. -- **Orchestration (per scheduled or manual run):** - 1. Task enabled? → DB row lock: `UPDATE Task SET status='running' WHERE id=? AND - status NOT IN ('running','waiting')` (0 rows → skip); global semaphore (default 1). - 2. Allocate display `N` (lockfile registry `/displays/:N.lock`); websockify port - = `6080 + (N - display_start)`. - 3. `INSERT Run (pid, display, novnc_port, log_path, control_sock, ...)`; spawn - `python3 -m billcollector run ` (env inherited: `VAULT_HOST`, `BW_API_URL`, ...). - 4. Stream stdout line-by-line → append to `/logs/run-.log` + push to - registered `ui.log` elements (page registers on connect, unregisters on disconnect). - 5. On exit: finalize `Run` (result from DB status / exit code; non-zero → `error`, - traceback tail into `Run.notes`). -- **Runner control socket** (Copilot channel): unix socket at - `/run-.sock`, JSON-line protocol, served by the runner's **main thread - while in the waiting loop** (between steps, so sync Playwright calls are safe there): - - `screenshot` → base64 PNG · `elements` → numbered interactive elements (existing - `PageState.set_interactive_elements` logic **+ bounding boxes**) · `context` → full - machine-readable pause payload (below) · `click {n}` · `fill {n, text}` · - `press {key}` · `drag {n, dx, dy}` (relative mouse drag from element n — slider - CAPTCHAs on the phone path) · `resume` · `cancel` - - The UI opens a short-lived connection per request; a `ui.timer` polls - `screenshot`/`elements` while the session page is open. -- **Protocol versioning:** the first JSON field of every message is `v` (starts at 1). - The protocol is the AI-integration contract — changes need a version bump and a doc - update (`doc/`), and the `context` payload must stay complete and small enough to ship - over a socket. - -### Intervention architecture (human-in-the-loop, AI-open) - -Every friction point is an **intervention**: the runner gets stuck, persists a -structured pause with machine-readable context, and yields into the waiting loop. The -intervention is resolved by explicit actions and a final `resume`. In Wave 1 the only -action source is the human via the web UI (Copilot / noVNC). The architecture is built -so a future ML/LLM assistant becomes a **second action source without touching the -runner**: - -1. **Stable intervention protocol** — the control-socket JSON protocol above: - versioned, documented, client-agnostic. -2. **Machine-readable pause context** — the `context` response; also persisted in - `Run.pause_context` per pause. This is exactly the input an LLM/ML system would - consume: - - `reason` ∈ `step_failed | wait_user | download_failed` - - `prompt` (human-readable; from `wait_user` or generated from the step description) - - `url`, `page_title`, `error` - - `step` (the failing step's JSON: description, locator chain, actions) - - `screenshot` (path + base64) - - `elements[]` — numbered: `role`, `name`, `rect {x, y, w, h}` (bounding box), - candidate `locator`, `actionTypes`, optional `url` - - `page_text` — visible page text, truncated (≈20 KB) - - `deadline_at` -3. **Human-in-the-loop is a permanent gate.** The runner executes only explicit action - requests from a client; the protocol has no "auto" mode. Wave 1: all actions - originate from the human UI. Future (out of scope): an assistant may *propose* - actions or recipe edits; proposals surface in the UI and execute only after the - operator approves — the runner's execution path is identical either way. The - `wait_user` `until` check (decision 19) is the objective "did it work" signal that - makes such proposals checkable. -4. **No heuristics, no AI in Wave 1** — the runner never detects CAPTCHAs or banners - and never calls a model. Openness is concrete and testable: protocol + context - contract + approval gate, all exercised by the Wave 1 E2E scenarios. - -The element scan (`PageState.set_interactive_elements`) gains **bounding boxes** -(`getBoundingClientRect`) in Wave 1 — required for overlay placement and for the -context's `rect` fields; the scan's existing role/name/locator-candidate logic is -reused unchanged. -- **Running-session live view:** the runner saves a step screenshot to - `/runs//step_.png` after each step (also the M3 failure-context - evidence). The UI shows the latest file, refreshed by `ui.timer` (~3 s). This is - coarser than v1's per-tick WebSocket view — accepted semantic change; full real-time - control is what noVNC provides. -- **noVNC:** the runner spawns `Xvfb :N`, `x11vnc -display :N -rfbport 5900+N - -rfbauth ` (password from settings, per-run file) and `websockify 6080+N - 127.0.0.1:5900+N` as children; all reaped in `finally`. UI iframe: - `ui.html('