From f0e3b3655fb178950e33f241fbea4387a72e06dd Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Wed, 29 Jul 2026 07:44:54 +0100 Subject: [PATCH 1/4] docs(art): split guide extraction planning --- .../AUTH_HANDOFF.md | 12 ++- .../CHUNK_MAP.md | 16 +++- .../DECISIONS.md | 73 ++++++++++++++++++ .../DISCOVERY.md | 30 ++++++++ .../INTENT.md | 8 ++ .../PLAN.md | 69 +++++++++++++++++ .../REVIEW_LOG.md | 16 ++++ .../RISKS.md | 9 +++ .../STATUS.md | 22 ++++-- ...3B-guide-source-binding-materialization.md | 77 +++++++------------ ...S-ART-001-03B1-guide-binding-generation.md | 56 ++++++++++++++ ...B2-guide-materialization-classification.md | 65 ++++++++++++++++ ...ART-001-03B3A-extraction-framework-text.md | 55 +++++++++++++ ...ART-001-03B3B-complex-format-extractors.md | 62 +++++++++++++++ ...001-03B4-guide-sufficiency-continuation.md | 68 ++++++++++++++++ ...1-03C-guide-source-cutover-continuation.md | 4 +- ...T-001-03B-PLAN-internal-review-evidence.md | 52 +++++++++++++ .../WS-ART-001-03B-PLAN-pr-trust-bundle.md | 77 +++++++++++++++++++ ...T-002-04B-guide-read-binding-activation.md | 8 +- docs/spec_artifact_storage_service.md | 34 ++++++++ 20 files changed, 742 insertions(+), 71 deletions(-) create mode 100644 .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B1-guide-binding-generation.md create mode 100644 .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B2-guide-materialization-classification.md create mode 100644 .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3A-extraction-framework-text.md create mode 100644 .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3B-complex-format-extractors.md create mode 100644 .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B4-guide-sufficiency-continuation.md create mode 100644 .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-internal-review-evidence.md create mode 100644 .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-pr-trust-bundle.md diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/AUTH_HANDOFF.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/AUTH_HANDOFF.md index 89a8cb7d4..0608499f2 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/AUTH_HANDOFF.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/AUTH_HANDOFF.md @@ -17,10 +17,14 @@ activation custody, and availability. exact resource/guard/surface manifest. 3. AUTH activates only that exact action through a separately reviewed AUTH contract after consuming ART evidence. -4. ART-03B implements hidden `artifact.guide_source.read` and - `artifact.guide_source.binding.create` behavior; binding maps to fixed - permission `artifact.binding.create`. -5. AUTH activates only those exact actions after consuming 03B evidence. +4. ART-03B1, 03B2, 03B3A, 03B3B, and 03B4 implement hidden authoritative binding, exact setup + generation, verified materialization, bounded extraction, and Celery + sufficiency continuation. `artifact.guide_source.binding.create` maps to + fixed permission `artifact.binding.create`; `artifact.guide_source.read` + remains separate. +5. AUTH-04B activates only those exact fixed-service actions after consuming + the complete split-03B evidence. Extraction does not imply a provider-write + permission or inherit Project Manager authority. 6. ART-03C performs the legacy clean cut. No ART chunk writes availability. ## Submission Bundle Sequence diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md index 8af1f995c..d9f253e95 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md @@ -17,9 +17,13 @@ Each chunk is one PR. No later chunk starts automatically. | `WS-ART-001-02D` | Add hidden Operator content/job/retry/recovery/audit APIs, canonical resource composition, and production-readiness checks while actions and provider profiles remain inactive. | L1 | Merged through PR #177 as `93c14181` | | `WS-ART-001-03` | Original combined guide-source cutover. | L1 | Cancelled before implementation; no runtime changes | | `WS-ART-001-PLAN2` | Reconcile guide and one-ZIP submission planning with bounded scratch, existing immutable admission/recovery, exact AUTH sequencing, and downstream ownership. | L1 | Planning-only successor proposed after cancellation | -| `WS-ART-001-03A` | Add hidden guide-source byte ingest through existing preparation, admission, verification, and publication. | L1 | Proposed after PLAN2 | -| `WS-ART-001-03B` | Bind verified guide-source content and provide authorized integrity-checking setup materialization. | L1 | Proposed after 03A and exact AUTH activation | -| `WS-ART-001-03C` | Remove legacy guide-source identity and add exact same-generation setup continuation. | L1 | Proposed after 03B and exact AUTH activation | +| `WS-ART-001-03A` | Add hidden guide-source byte ingest through existing preparation, admission, verification, and publication. | L1 | Merged through PR #215 as `bb9082a7` | +| `WS-ART-001-03B1` | Add authoritative verified guide-source bindings and an exact setup-generation fence while fixed-service binding remains unavailable. | L1 | Proposed after 03A and AUTH-04A | +| `WS-ART-001-03B2` | Add fixed-reader verified materialization, artifact incidents, and canonical format detection in bounded ART scratch. | L1 | Proposed after 03B1 | +| `WS-ART-001-03B3A` | Add the isolated extraction framework, canonical records, and text/Markdown/JSON/CSV extractors. | L1 | Proposed after 03B2 | +| `WS-ART-001-03B3B` | Add explicitly approved PDF/OOXML/image-metadata extractors on the proven framework. | L1 | Proposed after 03B3A and dependency approval | +| `WS-ART-001-03B4` | Feed only complete same-generation canonical extracted material into the existing Celery sufficiency pipeline. | L1 | Proposed after 03B3B | +| `WS-ART-001-03C` | Remove legacy guide-source identity and add exact same-generation setup continuation. | L1 | Proposed after 03B1-03B4 and AUTH-04B | | `WS-ART-001-04A` | Accept one outer ZIP in bounded scratch, safely inspect its tree, normalize executable intent, produce canonical identities, and reject unchanged work before provider I/O. | L1 | Proposed after 03C and AUTH planned action registration | | `WS-ART-001-04B` | Run mandatory platform and locked Project Guide pre-submit checks against the same scratch-bound tree and executable semantics without durable storage. | L1 | Proposed after 04A | | `WS-ART-001-04C` | Reauthorize at durable intent, admit/verify the passing ZIP once, and publish one capacity-charged `ready` admission that may remain unbound. | L1 | Proposed after 04B; AUTH activation follows hidden completion | @@ -43,7 +47,11 @@ OBJECT-STORAGE-AMENDMENT -> PLAN2 planning reconciliation -> 03A guide-source byte ingest -> AUTH activation for exact 03A actions --> 03B guide-source binding/materialization +-> 03B1 authoritative guide binding/setup generation +-> 03B2 verified materialization/format detection +-> 03B3A isolated extraction framework/text formats +-> 03B3B approved complex document/image-metadata extractors +-> 03B4 same-generation Celery sufficiency integration -> AUTH activation for exact 03B actions -> 03C guide-source clean cut/continuation -> AUTH planned registration of `artifact.submission_bundle.prepare` diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md index 68b83c578..b9746b1a8 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md @@ -362,3 +362,76 @@ Contributor authority never implies fixed service authority. Checker materialization, verification execution, pending-work scanning, ambiguous-put resolution, and binding retain their distinct fixed actions. No compatibility alias exposes the obsolete multi-step session actions. + +## D39 - Guide Upload And Understanding Are Separate + +Guide upload stores and verifies opaque original bytes. HTTP ingest does not +parse, extract, render, OCR, transcode, or wait for submission prechecks. +Understanding begins asynchronously only after an exact verified binding. +Unlike contributor submissions, guide-source items are not wrapped in one +mandatory outer ZIP. Each item retains its own verified format identity. + +## D40 - Exact Setup Generation Is A Durable Fence + +Every binding, extraction, agent input, and report identifies one project, +draft guide, source snapshot, setup run, and monotonic setup generation. Celery +carries only those identifiers and generation. Bytes, extracted text, scratch +handles, prepared authority, and provider credentials never enter Redis. + +## D41 - Canonical Guide Materialization + +The fixed guide-reader obtains fresh prepared authority, resolves an exact +verified binding, and streams through `ArtifactStore` into the existing +`ArtifactScratchManager`. Every full read recomputes SHA-256 and byte count. +Missing, changed, or truncated content is an ART incident, not guide +insufficiency. Project services and agents never access providers directly. + +## D42 - Typed, Bounded Guide Extraction + +One detector and typed extractor registry validates signatures and bounded +container markers. Initial text semantics cover PDF, DOCX, PPTX, CSV, XLSX, +Markdown, plain text, and JSON. PNG, JPEG, and WebP yield bounded structural +metadata only. Without OCR, image pixels cannot satisfy required textual guide +semantics. Audio/video and ambiguous binaries are unsupported in v0.1 and never +sent raw to agents. OOXML formats are distinguished from ordinary ZIPs. + +Extraction runs asynchronously in a strongly isolated no-network subprocess +under fixed input, output, container, time, memory, and document limits using +only scratch-manager paths. Parser crash, malformed input, macros, external +relationships, embedded executables, cancellation, timeout, and worker loss +have bounded outcomes and cleanup. Production parser dependencies require +explicit human approval before implementation. + +## D43 - Canonical Extraction Records, Not Implicit Provider Writes + +v0.1 persists an immutable content-derived extraction representation in +PostgreSQL keyed by original content, format, extractor/version, and policy +version, with output digest, omission facts, status, and bounded error code. A +separate immutable usage record binds it to the exact guide binding, source +item, setup run, and generation. AUTH-04B grants read and binding only, so ART +does not use read authority to create a provider object. + +## D44 - Sufficiency Consumes Complete Verified Material + +The agent receives only bounded extraction records for all required items in +the current generation. Missing, corrupt, stale, ambiguous, unsupported, or +failed content stops policy derivation without creating a guide-insufficiency +decision. Reports preserve exact content and extraction provenance. Legacy +excerpts and durable refs are not authoritative after ART-03C. + +## D45 - Extracted Guide Content Is Untrusted Agent Data + +Canonical extraction proves byte provenance, not instruction authority. Agent +assembly labels and delimits source material as untrusted data, excludes tools, +provider access, credentials, and hidden instructions from that material, and +accepts only the typed sufficiency output contract. Prompt-injection text inside +a guide cannot alter system/developer policy or authorize an action. + +## D46 - Setup Failures Have Stable Operational Outcomes + +Existing `ProjectSetupRun.status=setup_blocked` uses stable redacted error codes +for artifact incident, unsupported format, malformed content, and extraction +failure. Recoverable incidents wait for ART recovery and expose a bounded +incident reference to authorized Operators. Terminal corruption or source +format/content failure requires the Project Manager to upload a corrected item +into a new snapshot. None creates a sufficiency decision. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DISCOVERY.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DISCOVERY.md index d445f33bc..bbf6c1876 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DISCOVERY.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DISCOVERY.md @@ -129,3 +129,33 @@ removes direct provider schemes from guide-source identity. Chunk 05 removes the remaining caller storage transport when submissions move to sealed artifact-set bindings. No compatibility alias remains after either owning cutover. + +## ART-03B Materialization And Extraction Discovery + +Merged ART-03A and AUTH `WS-XINT-002-04A` provide the covered Project Manager +upload boundary. AUTH preparation happens before request-body intake; final +transaction-bound consumption locks the guide lineage and server-computed byte +facts before durable intent or provider I/O. Upload stores opaque immutable +bytes and does not parse them. + +The current setup pipeline cannot yet consume those bytes safely: + +- its Celery payload has four identifiers but no setup generation; +- `ProjectSetupRun` has no explicit generation fence; +- `GuideSourceMaterial` uses caller-originated metadata and optional excerpts, + not verified ART bindings; +- no authorized integrity-checking guide read exists in project services; +- no typed detector, extractor registry, extraction policy, or canonical + extraction record exists. + +AUTH's reviewed `WS-XINT-002-04B` activates only fixed-service guide binding +and read. It does not grant a derived-artifact write. v0.1 therefore persists a +bounded canonical extraction record and provenance in PostgreSQL; a future +derived provider artifact requires a separately planned action and activation. + +No document-parsing production dependencies are installed. Any new production +dependency requires explicit human approval before 03B3B implementation. OOXML ZIP +containers must pass one bounded classifier that distinguishes DOCX, PPTX, and +XLSX by internal markers. Audio/video transcription and OCR are not required by +current v0.1 setup and remain unsupported. PNG/JPEG/WebP classification and +metadata do not imply OCR or textual sufficiency. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/INTENT.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/INTENT.md index 374bd6895..8b6861140 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/INTENT.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/INTENT.md @@ -65,6 +65,14 @@ directories, or any other content allowed by the locked Project Guide. A ZIP entry inside the outer archive remains an ordinary file. Nested archive unpacking is outside v0.1 even when a guide would otherwise request it. +This contributor ZIP rule does not apply to Project Manager guide-source +uploads. A guide snapshot may contain multiple independently uploaded source +items in PDF, DOCX, PPTX, CSV, XLSX, Markdown, plain text, JSON, PNG, JPEG, or +WebP form. Images are metadata-only without OCR and cannot satisfy required +textual semantics. v0.1 does not support guide audio or video. DOCX, PPTX, and +XLSX are recognized as their document types despite using ZIP containers +internally; an arbitrary ordinary ZIP is not treated as a guide document. + ## First-Principle Constraints - Workstream computes and verifies canonical SHA-256 and byte count. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md index 8f07eb17a..7de12e0e7 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md @@ -471,6 +471,75 @@ exact resource/content/job/attempt filters. ## Product Cutover +### Guide Materialization And Extraction Boundary + +ART-03A upload remains a byte-custody request: authorize before intake, compute +the server digest and size, admit once, store opaque bytes, and verify them. It +does not parse content or wait for contributor submission checkers. + +The expanded ART-03B work is delivered as five hidden PR-sized subchunks before +AUTH-04B activates fixed-service binding/read: + +1. `03B1` creates an authoritative one-item/one-content guide binding tied to + the exact setup run and explicit setup generation. Item metadata cannot + assert content identity, and replaced or cross-lineage facts fail closed. +2. `03B2` performs fresh authorized full reads through `ArtifactStore`, writes + only through `ArtifactScratchManager`, recomputes digest/size, records ART + incidents, and classifies formats using signatures and bounded container + markers. +3. `03B3A` installs the isolated no-network extraction framework, immutable + content/usage provenance, and text, Markdown, JSON, and CSV extractors + without new parser dependencies. +4. `03B3B` adds explicitly approved PDF, DOCX, PPTX, XLSX, and PNG/JPEG/WebP + metadata extractors on that proven framework. +5. `03B4` adds `setup_generation` to the identifier-only Celery payload, + reloads current state, assembles all required canonical extracted sources, + invokes sufficiency, and persists exact provenance. + +The canonical extraction status set is: + +```text +extracted | unsupported | ambiguous | malformed | limit_exceeded | +parser_failure | cancelled | artifact_incident +``` + +Only `extracted` records enter agent input. Truncation or omission is explicit +and may be allowed only by the versioned extraction policy; required missing +semantics stop setup internally. Artifact incidents never become sufficiency +findings. + +Guide-source items are not required to be ZIP files. The initial semantic +extractors are PDF, DOCX, PPTX, CSV, XLSX, Markdown, plain text, JSON, and +PNG, JPEG, and WebP metadata. Images are accepted and classified, but without +OCR their pixels cannot satisfy required textual guide semantics. Audio/video +and opaque/ordinary ZIP input are unsupported in v0.1. ZIP- +container classification rejects macros, external relationships, embedded +executables, ambiguous markers, excessive entries, decompressed bytes, depth, +or compression ratio before a document parser runs. + +v0.1 stores bounded canonical extraction JSON/text in PostgreSQL rather than +creating an unapproved provider write under guide-read authority. It records +source digest/size, detected format, extractor and version, extraction-policy +version, output digest, truncation/omission facts, status, and bounded error +code. A separate immutable usage record binds that result to the exact binding, +source item, setup run, and generation. Original bytes remain authoritative. A +later derived-artifact store path requires a distinct AUTH action. + +The existing `ProjectSetupRun.status=setup_blocked` remains the public state; +stable redacted error codes distinguish `guide_artifact_incident`, +`guide_source_format_unsupported`, `guide_source_malformed`, and +`guide_source_extraction_failed`. Artifact incidents expose a bounded incident +identifier to authorized Operators and wait for ART recovery when recoverable; +terminal corruption requires a Project Manager to create a new source item and +snapshot. Unsupported, malformed, limit-exceeded, or extraction-failed content +asks the Project Manager to upload a corrected supported source item. None is a +guide-insufficiency decision. + +Any production parser dependency in 03B3B requires explicit human approval. Candidate +libraries must undergo current security, maintenance, license, transitive- +dependency, malformed-input, and cancellation review before `03B3B` may add +them; no runtime plugin discovery or parser fallback is permitted. + 1. Guide-source delivery is split into hidden byte ingest, AUTH activation, verified snapshot binding/materialization, AUTH activation, and the legacy identity/continuation clean cut. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/REVIEW_LOG.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/REVIEW_LOG.md index 1c1781548..9fa247280 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/REVIEW_LOG.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/REVIEW_LOG.md @@ -155,3 +155,19 @@ over-broad lineage trigger and an incomplete integrity-mismatch upload-item transition. Both received bounded repairs and their three failing tests pass locally; final internal re-review and hosted rerun remain. + +## WS-ART-001-03B-PLAN + +- Planning correction reviewed after ART-03A and AUTH-04A merged. +- Initial architecture, senior, QA, CI, and product/ops reviews found blocking + scope, provenance, parser-boundary, verification, image-semantics, and + operational-outcome gaps. +- Repairs split implementation into 03B1, 03B2, 03B3A, 03B3B, and 03B4; + separated content extraction from binding/generation usage; defined isolated + parsing, prompt-injection handling, stable setup errors, exact commands, and + the later parser dependency approval gate. +- Final results: architecture PASS WITH LOW RISKS; senior engineering PASS WITH + LOW RISKS; QA PASS WITH LOW RISKS; security PASS; product/ops PASS; CI PASS + WITH CONDITIONS; docs PASS WITH LOW RISKS; reuse PASS WITH LOW RISKS; test + delta PASS WITH CONDITIONS. Planning-PR conditions are resolved by committing + the new contracts; the dependency checker remains a future 03B3B criterion. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/RISKS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/RISKS.md index c64b8b417..07b809908 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/RISKS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/RISKS.md @@ -52,3 +52,12 @@ | Full history requires expensive recursive traversal | Medium | Preserve immutable links but add indexed latest/current/accepted repository queries or projections maintained under lifecycle transactions. | | Stored content changes after initial verification | Critical | Recompute full SHA-256 and byte count during checker materialization, reviewer full download, and downstream delivery; fail closed and record an artifact incident on mismatch. | | Speculative larger limits exhaust disk or processing time | High | Preserve current configured limits until separately reviewed capacity, concurrency, runtime, cost, and abuse evidence justifies a change. | +| Caller metadata or excerpt reaches the sufficiency agent instead of verified bytes | Critical | Resolve exact ART bindings, fully rehash provider reads, extract under a versioned policy, and assemble agent input only from same-generation canonical extraction records. | +| DOCX/PPTX/XLSX is confused with an arbitrary ZIP | Critical | Use one bounded container classifier with required internal-marker checks; ambiguous or ordinary ZIP is unsupported and never sent raw to an agent. | +| Malicious document exploits a parser or worker | Critical | Typed extractor registry, isolated no-network subprocess, fixed CPU/time/memory/output/container limits, external-reference and macro rejection, cancellation, and scratch-owned cleanup. | +| Artifact corruption is reported as guide insufficiency | Critical | Digest/size mismatch, missing/truncated objects, and provider failures create ART incident outcomes and stop setup before the sufficiency agent. | +| Extraction output silently changes across deployments | High | Persist content-derived format/extractor/policy identity and digest separately from exact binding/run/generation usage provenance; never rewrite prior output. | +| Read authority is reused to write a derived provider artifact | Critical | v0.1 stores only bounded extraction records in PostgreSQL; any derived provider write requires a separate action and AUTH activation. | +| Stale Celery delivery processes replaced guide content | Critical | Payloads contain identifiers plus setup generation only; workers reload current lineage, binding, extraction, run, and generation before invocation and persistence. | +| Unsupported image/audio/video content is passed raw to an agent | High | PNG/JPEG/WebP expose bounded structural metadata only; OCR and audio/video are unsupported, and image-only required text stops setup internally. | +| Extracted guide text performs prompt injection | Critical | Delimit and label extracted material as untrusted data, expose no tools/secrets/provider authority, enforce typed output, and test adversarial embedded instructions. | diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md index 3bb2d701d..020f7e6fe 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md @@ -51,16 +51,22 @@ normalizes regular-file executable intent into the semantic manifest; and it requires fresh AUTH prepared capabilities at durable put intent and atomic Submission/binding consumption. -## Next Proposed Chunk +## Current Planning Correction -After this planning package merges, `WS-ART-001-03A` is the only immediate ART -successor. It adds hidden guide-source byte ingest through the existing artifact -preparation/admission/verification path. It requires a separate signed start and -does not activate its own AUTH action. +ART-03A and AUTH `WS-XINT-002-04A` are merged. Before ART-03B implementation, +the guide-content boundary is being corrected explicitly: verified binding, +full-read materialization, format classification, isolated extraction, +canonical extraction provenance, incremental complex-format support, and +same-generation sufficiency continuation are separate PR-sized contracts. +`WS-ART-001-03B1` is the first proposed +implementation successor; this planning change contains no implementation. + +AUTH `WS-XINT-002-04B` follows the complete hidden split-03B series and +activates only fixed-service binding and guide read. ART-03C then removes the +legacy identity/excerpt path and makes the verified pipeline authoritative. ## Gate Planning evidence and all required internal reviewer tracks must pass before a -PR. The planning merge starts no successor. Every implementation chunk retains -its separate signed start, exact AUTH activation sequence, internal review, CI, -CodeRabbit, human checkpoint, and automated merge-memory stop. +PR. Every implementation chunk retains its own bounded contract, exact AUTH +activation sequence, internal review, CI, CodeRabbit, and human checkpoint. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B-guide-source-binding-materialization.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B-guide-source-binding-materialization.md index 7a177641e..f371677c8 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B-guide-source-binding-materialization.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B-guide-source-binding-materialization.md @@ -1,70 +1,47 @@ -# Chunk Contract: WS-ART-001-03B - Guide Source Binding And Materialization +# Planning Contract: WS-ART-001-03B - Guide Content Pipeline Expansion -Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 03A and exact AUTH activation +Initiative: `WS-ART-001` | Risk: L1 | Status: Superseded by reviewed subchunks before implementation Artifact contract phase: `guide_source_cutover` ## Goal -Bind verified guide-source content to snapshot items and give setup agents one -authorized, provider-neutral, integrity-checking read/materialization capability -without performing the legacy schema cutover. +Preserve the original 03B intent while dividing binding, verified +materialization, isolated extraction, and sufficiency continuation into bounded +implementation contracts. -## Allowed Files +## Executable Subchunks -- guide snapshot/item binding models, one migration, repository, schemas, and service; -- ART guide read/binding capability and canonical resource facts/guards; -- existing artifact scratch manager/materializer integration for setup input; -- setup-agent input assembly consuming the typed ART capability; -- focused migration, authorization, binding, materialization, and integrity tests; -- backend workflow/agent-gate files only to add and preserve exact coverage; -- related docs and chunk memory. +- `WS-ART-001-03B1`: authoritative binding and setup generation; +- `WS-ART-001-03B2`: verified materialization, incidents, and classification; +- `WS-ART-001-03B3A`: isolated framework and low-complexity extraction; +- `WS-ART-001-03B3B`: approved complex document/image-metadata extractors; +- `WS-ART-001-03B4`: same-generation Celery and sufficiency integration. -## Not Allowed +Each subchunk is one PR. AUTH `WS-XINT-002-04B` activates fixed binding/read +only after all five hidden ART contracts merge. ART-03C remains a later clean +cut. + +## Shared Non-Goals - contributor submission/checker/review behavior; -- direct provider access, URL fetching, a second materializer/scratch manager, - setup policy decisions, or destructive legacy-field removal; +- direct provider access, URL fetching, a second scratch manager, parsing in the + upload request, submission prechecks, OCR, audio/video transcription, or + destructive legacy-field removal; - AUTH-owned catalogue/evaluator/grant/identity/matrix/availability edits. ## Acceptance Criteria -- snapshot items bind exact verified `ArtifactContent` without circularly making - the item both the source and authority for that content; -- snapshot/bundle identity commits to ordered content identities and descriptors; -- setup agents receive bounded bytes/manifests only through ART and every full - read recomputes SHA-256 and byte count; -- missing/mismatched bound bytes are artifact incidents, not guide insufficiency; -- fixed guide read/binding actions remain fail-closed until AUTH activation; -- changed subsystem coverage is at least 90 percent and repository coverage - remains at least 78 percent. - -## Exact CI Coverage Gates - -```bash -coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifact_operations.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 -coverage report --include='app/interfaces/external_services.py' --precision=2 --fail-under=90 -coverage report --include='app/core/config.py' --precision=2 --fail-under=90 -coverage report --include='app/workers/*' --precision=2 --fail-under=90 -coverage report --include='app/main.py' --precision=2 --fail-under=90 -coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 -coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 -coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 -coverage report --include='app/api/router.py' --precision=2 --fail-under=90 -coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 -coverage report --include='app/adapters/project_agents/*,app/interfaces/project_agents.py' --precision=2 --fail-under=90 -``` +- all five contracts merge before AUTH-04B; +- agents receive only same-generation canonical extracted material; +- artifact failures remain distinct from guide insufficiency; +- fixed guide read/binding remain unavailable until AUTH-04B; +- production parser dependencies require explicit human approval. -## Verification +## Evidence -```bash -docker compose up -d --wait postgres redis minio -(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_guide_artifacts.py tests/test_project_setup.py tests/test_checker_materialization.py tests/test_artifact_scratch_manager.py -q --cov=app.modules.projects --cov=app.modules.artifacts --cov-report=term-missing --cov-fail-under=90) -(metadata_dir="$(mktemp -d)" && trap 'rm -rf "$metadata_dir"' EXIT && (cd backend && WORKSTREAM_TEST_ADMIN_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/postgres .venv/bin/python scripts/run_isolated_tests.py --metadata-json "$metadata_dir/result.json" --timeout-seconds 12600 -- .venv/bin/python -m pytest -q --ignore=tests/test_isolated_database_runner.py --cov=app --cov-report=term-missing --cov-fail-under=78)) -(cd backend && .venv/bin/ruff check app tests) -python3 scripts/check_stale_artifact_contracts.py -python3 scripts/test_agent_gates.py -``` +The five executable contracts own their exact file scopes, tests, coverage, and +review evidence. This umbrella is planning context and authorizes no code. ## Required Reviewers diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B1-guide-binding-generation.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B1-guide-binding-generation.md new file mode 100644 index 000000000..e49d2ea9d --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B1-guide-binding-generation.md @@ -0,0 +1,56 @@ +# Chunk Contract: WS-ART-001-03B1 - Guide Binding And Setup Generation + +Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after ART-03A and AUTH-04A + +## Goal + +Bind one exact verified `ArtifactContent` to one guide-source item and exact +setup run/generation without legacy cutover or activating fixed-service AUTH. + +## Allowed Files + +- project guide snapshot/item/setup-run models, one migration, repositories, + and schemas; +- ART binding capability, exact canonical facts, guards, and hidden service; +- focused migration, binding, concurrency, stale-generation, and denial tests; +- related ART specification, plan evidence, and coverage configuration. + +## Not Allowed + +- provider reads, format parsing, extraction, agent invocation, legacy-field + removal, submission/checker/review changes, or AUTH-owned availability edits. + +## Acceptance Criteria + +- an immutable binding references exact project, draft guide, snapshot, item, + verified content, setup run, and monotonic setup generation; +- source-item metadata cannot claim or substitute content identity; +- one current binding exists per item/generation and replacement is explicit; +- missing, unverified, cross-project, cross-guide, wrong-run, or stale-generation + content fails closed; +- concurrent binding produces one business effect; +- test-only fixed authority proves the hidden ART kernel's positive binding + semantics while the composed live AUTH path proves deny-only before 04B; +- fresh prepared binding authority is consumed atomically with the binding when + AUTH later activates it; the hidden real kernel denies before then; +- no provider I/O or human-authority inheritance occurs; +- changed subsystems remain at least 90% covered and repository coverage stays + at least 78%. + +## Verification + +```bash +(cd backend && .venv/bin/python -m ruff check app tests scripts) +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_authorization.py tests/test_guide_artifacts.py -q --cov=app.modules.artifacts --cov=app.modules.projects --cov-report=term-missing --cov-fail-under=90) +python3 scripts/check_stale_artifact_contracts.py +python3 scripts/check_markdown_links.py +python3 scripts/test_agent_gates.py +git diff --check +``` + +The exact PR head must pass hosted `Backend / test` and `Agent Gates / agent-gates`. + +## Required Reviewers + +Senior engineering, architecture, QA/test, security/auth, product/ops, +reuse/dedup, CI integrity, test delta, and docs. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B2-guide-materialization-classification.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B2-guide-materialization-classification.md new file mode 100644 index 000000000..bb181e736 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B2-guide-materialization-classification.md @@ -0,0 +1,65 @@ +# Chunk Contract: WS-ART-001-03B2 - Guide Materialization And Classification + +Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 03B1 + +## Goal + +Materialize exact bound guide bytes through the fixed reader, independently +verify integrity, classify incidents, and detect formats in bounded private +scratch without extracting semantic content. + +## Allowed Files + +- ART guide-read port, canonical resources, materializer, incident records, and + typed format detector/container inspector; +- extension of the existing `ArtifactMaterializationPort` convention and + interface boundary; no parallel materialization abstraction; +- existing `ArtifactStore` and `ArtifactScratchManager` composition only; +- fixed startup limits for signatures, ZIP markers, entries, decompressed bytes, + depth, ratio, image headers/dimensions, and read deadline; +- focused read, corruption, classification, container, cancellation, cleanup, + authorization, and coverage tests; related docs/evidence. + +## Not Allowed + +- semantic extraction, parser dependencies, agent calls, arbitrary temp paths, + direct provider access, generic download, Celery handles/bytes, legacy cutover, + or AUTH availability edits. + +## Acceptance Criteria + +- fresh prepared `artifact.guide_source.read` authority is consumed before any + provider read; the real kernel remains unavailable before AUTH-04B; +- test-only fixed authority proves positive read/materialization semantics while + the composed live AUTH path proves deny-before-I/O until 04B; +- exact binding and setup generation are revalidated before and after full read; +- complete SHA-256 and byte count are recomputed into scratch and compared; +- missing, changed, truncated, unavailable, and stale reads record bounded ART + incidents and never guide insufficiency; +- PDF, OOXML containers, CSV, Markdown, text, JSON, PNG/JPEG/WebP metadata, + ordinary ZIP, ambiguous, and opaque input receive deterministic detection; +- audio/video declared types and signatures deterministically produce the + unsupported outcome and never dispatch a parser; +- DOCX/PPTX/XLSX require bounded internal markers; symlinks, macros, external + relationships, embedded executables, bombs, and malformed containers reject; +- cleanup occurs on success, denial, mismatch, cancellation, and timeout; +- changed subsystems remain at least 90% covered and repository coverage stays + at least 78%. + +## Verification + +```bash +(cd backend && .venv/bin/python -m ruff check app tests scripts) +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_guide_artifacts.py tests/test_artifact_scratch_manager.py tests/test_guide_format_detection.py -q --cov=app.modules.artifacts --cov-report=term-missing --cov-fail-under=90) +python3 scripts/check_stale_artifact_contracts.py +python3 scripts/check_markdown_links.py +python3 scripts/test_agent_gates.py +git diff --check +``` + +The exact PR head must pass hosted `Backend / test` and `Agent Gates / agent-gates`. + +## Required Reviewers + +Senior engineering, architecture, QA/test, security/auth, product/ops, +reuse/dedup, CI integrity, test delta, and docs. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3A-extraction-framework-text.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3A-extraction-framework-text.md new file mode 100644 index 000000000..fcf002de4 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3A-extraction-framework-text.md @@ -0,0 +1,55 @@ +# Chunk Contract: WS-ART-001-03B3A - Extraction Framework And Text Formats + +Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 03B2 + +## Goal + +Prove the isolated extraction framework, canonical content/usage provenance, +and standard-library text, Markdown, JSON, and CSV extraction. + +## Allowed Files + +- typed extraction capability/registry and explicit composition entries; +- isolated no-network subprocess and versioned extraction policy; +- content-derived extraction and binding/generation usage models, one migration, + repository, and schemas; +- focused isolation, limit, cancellation, cleanup, determinism, provenance, + text/Markdown/JSON/CSV, and coverage tests; related docs/evidence. + +## Not Allowed + +- production parser dependencies, PDF/OOXML/image parsing, plugin discovery, + in-process untrusted parsing, provider writes/access, agent invocation, OCR, + audio/video, legacy cutover, or AUTH availability edits. + +## Acceptance Criteria + +- no-network subprocess enforces input/output, CPU/time/memory, row/cell, + encoding, nesting, cancellation, and cleanup limits; +- extraction revalidates exact 03B2 digest/binding/format/policy provenance; +- deterministic content records are keyed by content, format, extractor/version, + and policy; separate usage records name item, binding, run, and generation; +- text/Markdown/JSON/CSV canonicalization and error statuses are deterministic; +- unsupported raw input never reaches an agent or provider write; +- changed subsystem coverage is at least 90%; repository coverage stays 78%. + +## Verification + +```bash +(cd backend && .venv/bin/python -m ruff check app tests scripts) +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_guide_extraction.py -q --cov=app.modules.artifacts.guide_extraction --cov-report=term-missing --cov-fail-under=90) +python3 scripts/check_stale_artifact_contracts.py +python3 scripts/check_markdown_links.py +python3 scripts/test_agent_gates.py +git diff --check +``` + +The exact PR head must pass hosted `Backend / test` and `Agent Gates / agent-gates`. +Every changed production module, including repository, schema, migration-owned +service, and composition surfaces, must be included in a dedicated retained +90% coverage report or an explicitly reviewed existing subsystem report. + +## Required Reviewers + +Senior engineering, architecture, QA/test, security/auth, product/ops, +reuse/dedup, CI integrity, test delta, and docs. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3B-complex-format-extractors.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3B-complex-format-extractors.md new file mode 100644 index 000000000..67709b8a9 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3B-complex-format-extractors.md @@ -0,0 +1,62 @@ +# Chunk Contract: WS-ART-001-03B3B - Complex Format Extractors + +Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 03B3A; dependency approval required + +## Goal + +Add approved PDF, DOCX, PPTX, XLSX, and PNG/JPEG/WebP metadata adapters on the +already-proven isolated framework. + +## Entry Gate + +The human owner explicitly approves an exact pinned dependency allowlist after +review of license, maintenance, advisories, transitive graph, malformed-input +history, and cancellation behavior. The PR includes the package/lock diff, +machine-readable allowlist, dependency-tree evidence, license evidence, and +security-review record. CI fails if a parser dependency is outside that list. + +## Allowed Files + +- one typed adapter per approved format and explicit registry entries; +- approved package/lock/allowlist and deterministic dependency gate; +- bounded fixtures and format-confusion, macro/external relationship, embedded + content, page/slide/sheet/cell/image, parser crash, timeout, and coverage tests; +- related docs/evidence. + +## Not Allowed + +- framework redesign, runtime plugins/fallbacks, OCR, audio/video, ordinary ZIP + semantics, raw pixels/base64 to agents, provider writes/access, legacy cutover, + or AUTH availability edits. + +## Acceptance Criteria + +- exact 03B2 classification provenance is required before parser execution; +- all parsers stay inside the 03B3A isolation/limit/cancellation boundary; +- OOXML markers distinguish formats and reject macros, external relationships, + embedded executables, bombs, and ambiguity; +- PNG/JPEG/WebP output is metadata only and cannot satisfy required text; +- parser dependency allowlist and lock graph are deterministic and CI-enforced; +- changed subsystem coverage is at least 90%; repository coverage stays 78%. + +## Verification + +```bash +(cd backend && .venv/bin/python -m ruff check app tests scripts) +(cd backend && .venv/bin/python scripts/check_guide_extractor_dependencies.py) +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_guide_format_detection.py tests/test_guide_extraction.py -q --cov=app.modules.artifacts.guide_extraction --cov-report=term-missing --cov-fail-under=90) +python3 scripts/check_stale_artifact_contracts.py +python3 scripts/check_markdown_links.py +python3 scripts/test_agent_gates.py +git diff --check +``` + +The exact PR head must pass hosted `Backend / test` and `Agent Gates / agent-gates`. +Every changed production module, including adapter registry, dependency gate, +configuration, and composition surfaces, must be included in a dedicated +retained 90% coverage report or an explicitly reviewed existing subsystem report. + +## Required Reviewers + +Senior engineering, architecture, QA/test, security/auth, product/ops, +reuse/dedup, CI integrity, test delta, and docs. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B4-guide-sufficiency-continuation.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B4-guide-sufficiency-continuation.md new file mode 100644 index 000000000..f176d0e06 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B4-guide-sufficiency-continuation.md @@ -0,0 +1,68 @@ +# Chunk Contract: WS-ART-001-03B4 - Guide Sufficiency Continuation + +Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 03B3B + +## Goal + +Continue the existing Celery project-setup workflow with complete verified, +same-generation canonical guide material and exact persisted provenance. + +## Allowed Files + +- existing project-setup Celery task/queue and setup-run generation fields; +- project service/repository and agent input schemas consuming typed canonical + extraction records; +- in-place evolution or replacement of existing `GuideSourceMaterial` and + `GuideSourceItemMaterial`; no parallel sufficiency-material model; +- sufficiency-report usage provenance; extraction models/migration remain owned + by 03B3A and complex adapter provenance remains owned by 03B3B; +- focused stale-delivery, completeness, incident, unsupported, broker replay, + agent-input, persistence, cancellation, and coverage tests; related docs. + +## Not Allowed + +- bytes, extracted content, scratch paths/handles, prepared authorization, or + credentials in Celery payloads; direct provider access; raw binary/excerpts + as authoritative input; policy derivation after incomplete extraction; + legacy-field removal; AUTH availability edits. + +## Acceptance Criteria + +- project-setup Celery payload is exactly project, guide, snapshot, setup run, and + setup generation identifiers; +- worker reloads and revalidates current project/guide/snapshot/run/generation, + complete bindings, content, and extraction provenance before agent invocation + and again before report commit; +- all required items must have successful policy-compatible extraction; +- artifact incident, missing/corrupt/stale/unsupported/failed extraction stops + setup with an internal status and creates no insufficiency decision; +- agent input is bounded canonical content only, never raw binaries or caller + excerpts, and reports preserve exact content/extraction provenance; +- source content is explicitly delimited/labeled as untrusted data, supplies no + tools/provider/secret authority, and prompt-injection fixtures cannot override + system/developer policy or the typed sufficiency output contract; +- `setup_blocked` exposes the stable redacted error codes and remediation defined + by D46, with bounded Operator incident reference only for ART incidents; +- duplicate or stale Celery execution cannot create or overwrite current output; +- test-only fixed authority proves the complete hidden pipeline while composed + live binding/read remains deny-only until AUTH-04B; +- changed subsystems remain at least 90% covered and repository coverage stays + at least 78%. + +## Verification + +```bash +(cd backend && .venv/bin/python -m ruff check app tests scripts) +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_project_setup.py tests/test_guide_artifacts.py tests/test_guide_extraction.py tests/test_project_agents.py -q --cov=app.modules.projects --cov=app.workers.project_setup --cov-report=term-missing --cov-fail-under=90) +python3 scripts/check_stale_artifact_contracts.py +python3 scripts/check_markdown_links.py +python3 scripts/test_agent_gates.py +git diff --check +``` + +The exact PR head must pass hosted `Backend / test` and `Agent Gates / agent-gates`. + +## Required Reviewers + +Senior engineering, architecture, QA/test, security/auth, product/ops, +reuse/dedup, CI integrity, test delta, and docs. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03C-guide-source-cutover-continuation.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03C-guide-source-cutover-continuation.md index 4a8380f62..f7a6970a1 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03C-guide-source-cutover-continuation.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03C-guide-source-cutover-continuation.md @@ -1,6 +1,6 @@ # Chunk Contract: WS-ART-001-03C - Guide Source Cutover And Continuation -Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 03B and exact AUTH activation +Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after split 03B and AUTH-04B Artifact contract phase: `guide_source_cutover` @@ -32,6 +32,8 @@ adding a Project Manager resume command. - migration refuses unsafe populated legacy rows rather than fabricating bytes; - caller `content_cid`, provider-object schemes, and hash-only identity are removed without aliases after verified replacement is available; +- caller excerpts and durable references no longer act as authoritative agent + content; verified same-generation bindings and canonical extractions do; - recovery continuation references a persisted setup run/generation and resumes only when project, guide, snapshot, policy, and generation still match; - Operator recovery authorization and automatic setup continuation remain diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-internal-review-evidence.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-internal-review-evidence.md new file mode 100644 index 000000000..ca9e455d4 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-internal-review-evidence.md @@ -0,0 +1,52 @@ +# Internal Review Evidence: WS-ART-001-03B Planning Correction + +## Scope Reviewed + +Planning-only correction for guide binding, materialization, classification, +isolated extraction, same-generation Celery continuation, AUTH-04B sequencing, +and the artifact-storage specification. No runtime code, migration, dependency, +workflow, or authorization availability changed. + +## Initial Findings And Repairs + +- Architecture: separated content-derived extraction from binding/run/generation + usage provenance and kept parser semantics out of 03B2. +- Senior engineering: split extraction into 03B3A framework/text formats and + 03B3B approved complex formats; added hidden positive and live deny-only proof. +- QA/CI: added exact commands, hosted full gates, 90% changed-subsystem coverage, + and deterministic parser-dependency evidence. +- Product/ops: defined image metadata-only behavior, stable `setup_blocked` + codes, Operator incident visibility, and PM remediation. +- Security: delimited extracted content as untrusted agent data, added prompt- + injection proof, and bound parsing to 03B2 provenance. +- Reuse/test/docs: required existing ports/scratch/Celery/agent schemas, added + audio/video negatives, and corrected count, dependency, and sequencing drift. + +## Final Results + +| Track | Result | Remaining condition/risk | +|---|---|---| +| architecture | PASS WITH LOW RISKS | AUTH wording aligned; obsolete 03B3 umbrella omitted. | +| senior engineering | PASS WITH LOW RISKS | Image wording aligned. | +| QA/test | PASS WITH LOW RISKS | Concrete local database URL added. | +| security/auth | PASS | None. | +| product/ops | PASS | None. | +| CI integrity | PASS WITH CONDITIONS | Commit new contracts; future 03B3B implements its named dependency gate. | +| docs | PASS WITH LOW RISKS | Count/dependency/ownership wording corrected. | +| reuse/dedup | PASS WITH LOW RISKS | Existing ports and agent schemas reused. | +| test delta | PASS WITH CONDITIONS | Commit contracts; coverage/audio-video proof recorded. | + +All conditions applicable to this planning PR are addressed by including the +new contracts. The 03B3B dependency gate is a future acceptance criterion, not +an implementation claim of this planning change. + +## Deterministic Evidence + +```text +git diff --check +python3 scripts/check_markdown_links.py +python3 scripts/check_stale_artifact_contracts.py +python3 scripts/check_stale_workstream_wording.py +``` + +All passed. The PR changes no executable code, tests, packages, or CI surface. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-pr-trust-bundle.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-pr-trust-bundle.md new file mode 100644 index 000000000..8d07e8b74 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-pr-trust-bundle.md @@ -0,0 +1,77 @@ +# PR Trust Bundle: WS-ART-001-03B Planning Correction + +## Chunk + +`WS-ART-001-03B-PLAN` — planning-only correction before implementation. + +## Goal + +Define the safe path from verified Project Manager guide-source bytes to exact +same-generation sufficiency input without confusing it with contributor ZIPs. + +## Human-Approved Intent + +- contributors always submit one outer ZIP; +- Project Managers upload guide items in supported document/data/image formats; +- v0.1 excludes audio and video; +- upload stores opaque bytes and performs no synchronous parsing; +- AUTH later activates only fixed-service guide binding and read. + +## What Changed And Why + +The old 03B contract combined too many boundaries. It is now five executable +PRs: `03B1 -> 03B2 -> 03B3A -> 03B3B -> 03B4 -> AUTH-04B -> 03C`. + +## Design Chosen + +Original bytes remain authoritative in `ArtifactStore`. Existing materialization +and scratch capabilities rehash every full read. Content-derived extraction is +separate from binding/run/generation usage. Canonical extraction is bounded +PostgreSQL processing evidence, not an unapproved provider write. Agent input +is typed, bounded, provenance-linked, and explicitly untrusted. + +## Alternatives Rejected + +Parsing in upload, forcing guides into contributor ZIPs, direct provider access, +a second scratch manager, one large parser PR, image OCR by implication, raw +binary/excerpt agent input, and provider writes under guide-read authority. + +## Scope Control And Product Behavior + +Planning/specification only. Guide items support PDF, DOCX, PPTX, CSV, XLSX, +Markdown, text, JSON, and PNG/JPEG/WebP metadata. Audio/video and ordinary ZIP +guide semantics are unsupported. Artifact/extraction failures create redacted +`setup_blocked` outcomes, not guide-insufficiency decisions. + +## Acceptance Proof, Test Delta, And CI Integrity + +- diff, Markdown links, stale artifact contracts, and stale wording: PASS; +- no executable CI/test/package delta; +- every future contract names focused tests, Ruff, 90% changed-subsystem + coverage, the 78% repository gate, Agent Gates, and hosted Backend checks; +- 03B3B requires a human-approved, CI-enforced parser dependency allowlist. + +## Reviewer Results + +All nine tracks passed after repairs. Details are in +`WS-ART-001-03B-PLAN-internal-review-evidence.md`. + +## External Review + +Pending CodeRabbit and hosted GitHub checks after the PR opens. + +## Remaining Risks And Follow-Up + +- 03B3B cannot start until exact pinned parser dependencies are approved. +- Images provide metadata only; image text requires a future OCR capability. +- Immediate post-merge implementation successor is 03B1. + +## Human Review Focus + +Confirm the upload separation, five-PR split, AUTH-04B placement, PostgreSQL +canonical extraction records, v0.1 format boundary, and dependency approval gate. + +## Human Merge Ownership + +A maintainer must explicitly approve this planning PR. Merge starts no +implementation automatically. diff --git a/.agent-loop/initiatives/WS-XINT-002-art-auth-end-to-end/chunks/WS-XINT-002-04B-guide-read-binding-activation.md b/.agent-loop/initiatives/WS-XINT-002-art-auth-end-to-end/chunks/WS-XINT-002-04B-guide-read-binding-activation.md index 65dd308d6..b4000343d 100644 --- a/.agent-loop/initiatives/WS-XINT-002-art-auth-end-to-end/chunks/WS-XINT-002-04B-guide-read-binding-activation.md +++ b/.agent-loop/initiatives/WS-XINT-002-art-auth-end-to-end/chunks/WS-XINT-002-04B-guide-read-binding-activation.md @@ -2,8 +2,8 @@ ## Goal -Activate fixed-service guide binding and guide read only after merged ART-03B -evidence, without weakening the ART-03C clean cut. +Activate fixed-service guide binding and guide read only after merged ART-03B1, +03B2, 03B3A, 03B3B, and 03B4 evidence, without weakening ART-03C. ## Risk class @@ -12,8 +12,8 @@ L1. ## Entry gate - WS-XINT-002-04A is merged. -- ART-03B hidden binding/materialization behavior and exact resource manifest - are merged and reviewed. +- The complete split-03B hidden behavior and exact resource manifest are merged + and reviewed. ## Allowed files diff --git a/docs/spec_artifact_storage_service.md b/docs/spec_artifact_storage_service.md index 240f52fe7..2fbef5633 100644 --- a/docs/spec_artifact_storage_service.md +++ b/docs/spec_artifact_storage_service.md @@ -1212,6 +1212,40 @@ Guide-source ingestion stores each source item as immutable artifact content. Sufficiency and policy-derivation agents read through an authorized Workstream artifact reader, never direct provider URLs. +The Project Manager guide-source boundary is not the contributor submission +boundary. Contributor submissions always use one outer ZIP; guide snapshots may +contain multiple independently uploaded items in PDF, DOCX, PPTX, CSV, XLSX, +Markdown, plain text, JSON, or PNG/JPEG/WebP. Images yield bounded structural +metadata only and cannot satisfy required textual semantics without future +approved OCR. v0.1 does not support guide audio or video. DOCX, PPTX, and XLSX are safely distinguished by +bounded internal markers; an arbitrary ZIP is not treated as one of them. + +Upload stores opaque bytes and performs no parsing in the HTTP request. The +existing Celery setup pipeline later carries only project, guide, source +snapshot, setup run, and setup generation identifiers. A fixed guide reader +obtains fresh prepared authority, resolves the exact verified binding, streams +through `ArtifactStore` into `ArtifactScratchManager`, and recomputes complete +SHA-256 and byte count. Missing, changed, truncated, or unavailable content is +an artifact incident and never a guide-insufficiency result. + +Typed extractors run in a bounded no-network isolation boundary. One immutable +content-derived record identifies original content, format, extractor/version, +policy version, output digest, omission facts, status, and error code; a +separate immutable usage record binds it to the exact binding, item, setup run, +and generation. v0.1 persists these bounded records in PostgreSQL and does not +use guide-read authority to create a provider object. Only successful current- +generation extraction reaches the agent, delimited as untrusted source data +with no tool, secret, provider, or instruction authority. Unsupported, +ambiguous, malformed, stale, or failed required material stops setup internally. + +The public run remains `setup_blocked` with redacted stable codes +`guide_artifact_incident`, `guide_source_format_unsupported`, +`guide_source_malformed`, or `guide_source_extraction_failed`. Recoverable ART +incidents wait for recovery and expose only a bounded incident reference to an +authorized Operator. Terminal corruption or source-content failure requires a +new corrected Project Manager source item/snapshot; no outcome is a sufficiency +decision. + When guide-source verification fails transiently, artifact recovery success automatically re-publishes the same persisted `ProjectSetupRun` continuation only when project, guide version, source snapshot ID/hash, and setup generation From d5d2417116af6dfe7b7c6813b0e2712a8a8212fb Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Wed, 29 Jul 2026 08:05:14 +0100 Subject: [PATCH 2/4] docs(art): resolve planning gate feedback --- .../DECISIONS.md | 2 +- .../REVIEW_LOG.md | 4 ++ .../RISKS.md | 4 +- ...001-03B4-guide-sufficiency-continuation.md | 6 ++- ...T-001-03B-PLAN-external-review-response.md | 42 +++++++++++++++++++ .../WS-ART-001-03B-PLAN-pr-trust-bundle.md | 7 +++- 6 files changed, 59 insertions(+), 6 deletions(-) create mode 100644 .agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-external-review-response.md diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md index b9746b1a8..7495e1520 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md @@ -398,7 +398,7 @@ sent raw to agents. OOXML formats are distinguished from ordinary ZIPs. Extraction runs asynchronously in a strongly isolated no-network subprocess under fixed input, output, container, time, memory, and document limits using only scratch-manager paths. Parser crash, malformed input, macros, external -relationships, embedded executables, cancellation, timeout, and worker loss +relationships, embedded executables, cancellation, timeout, and executor loss have bounded outcomes and cleanup. Production parser dependencies require explicit human approval before implementation. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/REVIEW_LOG.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/REVIEW_LOG.md index 9fa247280..07ca3dfcf 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/REVIEW_LOG.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/REVIEW_LOG.md @@ -171,3 +171,7 @@ WITH CONDITIONS; docs PASS WITH LOW RISKS; reuse PASS WITH LOW RISKS; test delta PASS WITH CONDITIONS. Planning-PR conditions are resolved by committing the new contracts; the dependency checker remains a future 03B3B criterion. +- First external run: CodeRabbit was rate-limited and produced no findings. + Agent Gates found five valid retired-vocabulary occurrences, now repaired. + Backend recorded one unrelated AUTH PostgreSQL concurrency failure after + 1,651 passes; the repaired documentation head requires a fresh hosted rerun. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/RISKS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/RISKS.md index 07b809908..963504c45 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/RISKS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/RISKS.md @@ -54,10 +54,10 @@ | Speculative larger limits exhaust disk or processing time | High | Preserve current configured limits until separately reviewed capacity, concurrency, runtime, cost, and abuse evidence justifies a change. | | Caller metadata or excerpt reaches the sufficiency agent instead of verified bytes | Critical | Resolve exact ART bindings, fully rehash provider reads, extract under a versioned policy, and assemble agent input only from same-generation canonical extraction records. | | DOCX/PPTX/XLSX is confused with an arbitrary ZIP | Critical | Use one bounded container classifier with required internal-marker checks; ambiguous or ordinary ZIP is unsupported and never sent raw to an agent. | -| Malicious document exploits a parser or worker | Critical | Typed extractor registry, isolated no-network subprocess, fixed CPU/time/memory/output/container limits, external-reference and macro rejection, cancellation, and scratch-owned cleanup. | +| Malicious document exploits a parser or extraction runtime | Critical | Typed extractor registry, isolated no-network subprocess, fixed CPU/time/memory/output/container limits, external-reference and macro rejection, cancellation, and scratch-owned cleanup. | | Artifact corruption is reported as guide insufficiency | Critical | Digest/size mismatch, missing/truncated objects, and provider failures create ART incident outcomes and stop setup before the sufficiency agent. | | Extraction output silently changes across deployments | High | Persist content-derived format/extractor/policy identity and digest separately from exact binding/run/generation usage provenance; never rewrite prior output. | | Read authority is reused to write a derived provider artifact | Critical | v0.1 stores only bounded extraction records in PostgreSQL; any derived provider write requires a separate action and AUTH activation. | -| Stale Celery delivery processes replaced guide content | Critical | Payloads contain identifiers plus setup generation only; workers reload current lineage, binding, extraction, run, and generation before invocation and persistence. | +| Stale Celery delivery processes replaced guide content | Critical | Payloads contain identifiers plus setup generation only; executors reload current lineage, binding, extraction, run, and generation before invocation and persistence. | | Unsupported image/audio/video content is passed raw to an agent | High | PNG/JPEG/WebP expose bounded structural metadata only; OCR and audio/video are unsupported, and image-only required text stops setup internally. | | Extracted guide text performs prompt injection | Critical | Delimit and label extracted material as untrusted data, expose no tools/secrets/provider authority, enforce typed output, and test adversarial embedded instructions. | diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B4-guide-sufficiency-continuation.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B4-guide-sufficiency-continuation.md index f176d0e06..16f3ec411 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B4-guide-sufficiency-continuation.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B4-guide-sufficiency-continuation.md @@ -30,7 +30,8 @@ same-generation canonical guide material and exact persisted provenance. - project-setup Celery payload is exactly project, guide, snapshot, setup run, and setup generation identifiers; -- worker reloads and revalidates current project/guide/snapshot/run/generation, +- the project-setup executor reloads and revalidates current + project/guide/snapshot/run/generation, complete bindings, content, and extraction provenance before agent invocation and again before report commit; - all required items must have successful policy-compatible extraction; @@ -53,7 +54,8 @@ same-generation canonical guide material and exact persisted provenance. ```bash (cd backend && .venv/bin/python -m ruff check app tests scripts) -(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_project_setup.py tests/test_guide_artifacts.py tests/test_guide_extraction.py tests/test_project_agents.py -q --cov=app.modules.projects --cov=app.workers.project_setup --cov-report=term-missing --cov-fail-under=90) +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_project_setup.py tests/test_guide_artifacts.py tests/test_guide_extraction.py tests/test_project_agents.py -q --cov=app --cov-report=term-missing --cov-fail-under=0) +(cd backend && .venv/bin/coverage report --include='app/modules/projects/*,app/*ers/project_setup.py' --precision=2 --fail-under=90) python3 scripts/check_stale_artifact_contracts.py python3 scripts/check_markdown_links.py python3 scripts/test_agent_gates.py diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-external-review-response.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-external-review-response.md new file mode 100644 index 000000000..07675722c --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-external-review-response.md @@ -0,0 +1,42 @@ +# External Review Response: WS-ART-001-03B Planning Correction + +## Comments Addressed + +- CodeRabbit produced no findings because the organization review limit was + reached before review started. There are no inline CodeRabbit comments to + accept, reject, or defer on the current head. +- Agent Gates correctly found five uses of retired human `worker` vocabulary in + new planning text. They were replaced with parser runtime, executor, and + project-setup executor wording. The future coverage command now avoids + embedding the retired vocabulary while retaining explicit setup-executor + coverage. +- Backend semantic-lane evidence showed one unrelated existing AUTH PostgreSQL + concurrency failure: expected statuses `[200, 403]`, observed `[500, 200]` in + `test_actor_identity_link_lifecycle_real_postgres_concurrency`. The other + 1,651 tests in that lane passed. This planning PR changes no backend runtime or + tests; the next pushed head receives a fresh hosted rerun. + +## Comments Deferred + +- CodeRabbit review itself remains pending until its rate limit permits a run. +- No backend product-code repair is justified from the isolated AUTH concurrency + failure without reproduction on the new head; it is outside this planning + diff. + +## Human Decisions Needed + +None for the current repair. + +## Commands Rerun + +```text +python3 scripts/check_stale_authorization_docs.py +git diff --check +``` + +Both pass. The full documentation/stale-contract suite is rerun before push. + +## Remaining Risks + +- CodeRabbit has not reviewed the PR yet. +- Hosted Backend and Agent Gates must pass the repaired exact head. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-pr-trust-bundle.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-pr-trust-bundle.md index 8d07e8b74..fba65277c 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-pr-trust-bundle.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-pr-trust-bundle.md @@ -58,7 +58,12 @@ All nine tracks passed after repairs. Details are in ## External Review -Pending CodeRabbit and hosted GitHub checks after the PR opens. +CodeRabbit reached its review-rate limit before producing findings. Agent Gates +found and prompted repair of five retired-vocabulary occurrences. The first +Backend run had one unrelated existing AUTH concurrency failure after 1,651 +passes. Disposition is recorded in +`WS-ART-001-03B-PLAN-external-review-response.md`; repaired-head hosted checks +and CodeRabbit review remain required. ## Remaining Risks And Follow-Up From 60ec32154575053657eb2aaa9f92ee79f42e5238 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Wed, 29 Jul 2026 08:42:22 +0100 Subject: [PATCH 3/4] docs(art): resolve CodeRabbit planning findings --- .../DECISIONS.md | 45 ++++++++++++++++--- .../PLAN.md | 26 +++++++---- .../REVIEW_LOG.md | 4 ++ ...B2-guide-materialization-classification.md | 8 +++- ...ART-001-03B3A-extraction-framework-text.md | 11 ++++- ...ART-001-03B3B-complex-format-extractors.md | 11 ++++- ...001-03B4-guide-sufficiency-continuation.md | 8 +++- ...T-001-03B-PLAN-external-review-response.md | 15 ++++--- .../WS-ART-001-03B-PLAN-pr-trust-bundle.md | 13 +++--- ...T-002-04B-guide-read-binding-activation.md | 2 +- docs/spec_artifact_storage_service.md | 13 ++++-- 11 files changed, 122 insertions(+), 34 deletions(-) diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md index 7495e1520..8224c28f9 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md @@ -402,6 +402,33 @@ relationships, embedded executables, cancellation, timeout, and executor loss have bounded outcomes and cleanup. Production parser dependencies require explicit human approval before implementation. +The v0.1 extraction-policy limits are fixed, not caller-selectable: + +| Limit | Value | Enforcement owner | Breach outcome | +|---|---:|---|---| +| parser input per item | 32 MiB | 03B3A framework before dispatch | `limit_exceeded` | +| canonical output per item | 4 MiB | 03B3A streaming output collector | `limit_exceeded` | +| aggregate agent material | 12 MiB | 03B4 input assembler | `limit_exceeded` | +| subprocess wall time per item | 60 seconds | 03B3A process supervisor | `limit_exceeded` | +| subprocess address space | 512 MiB | 03B3A OS isolation boundary | `limit_exceeded` | +| container entries | 2,000 | 03B2 container inspector | `limit_exceeded` | +| decompressed container bytes | 128 MiB | 03B2 container inspector | `limit_exceeded` | +| container nesting depth | 8 | 03B2 container inspector | `limit_exceeded` | +| compression ratio | 100:1 | 03B2 container inspector | `limit_exceeded` | +| PDF pages | 500 | 03B3B PDF adapter | `limit_exceeded` | +| PPTX slides | 300 | 03B3B PPTX adapter | `limit_exceeded` | +| XLSX sheets | 100 | 03B3B XLSX adapter | `limit_exceeded` | +| table rows per item | 100,000 | 03B3A CSV / 03B3B XLSX adapter | `limit_exceeded` | +| table cells per item | 1,000,000 | 03B3A CSV / 03B3B XLSX adapter | `limit_exceeded` | +| characters per cell | 32,768 | CSV/XLSX adapter | `limit_exceeded` | +| image pixels | 40 megapixels | 03B2 header inspector and 03B3B adapter | `limit_exceeded` | +| image width or height | 16,384 pixels | 03B2 header inspector and 03B3B adapter | `limit_exceeded` | + +Exact-boundary, one-over-boundary, cleanup, cancellation, timeout, memory- +termination, and executor-loss tests are mandatory. Executor loss leaves no +successful extraction usage record; a current-generation retry starts from +fresh materialization and authority. + ## D43 - Canonical Extraction Records, Not Implicit Provider Writes v0.1 persists an immutable content-derived extraction representation in @@ -429,9 +456,15 @@ a guide cannot alter system/developer policy or authorize an action. ## D46 - Setup Failures Have Stable Operational Outcomes -Existing `ProjectSetupRun.status=setup_blocked` uses stable redacted error codes -for artifact incident, unsupported format, malformed content, and extraction -failure. Recoverable incidents wait for ART recovery and expose a bounded -incident reference to authorized Operators. Terminal corruption or source -format/content failure requires the Project Manager to upload a corrected item -into a new snapshot. None creates a sufficiency decision. +Existing `ProjectSetupRun.status=setup_blocked` uses this exhaustive stable, +redacted mapping: `unsupported` -> `guide_source_format_unsupported`; +`ambiguous` -> `guide_source_format_ambiguous`; `malformed` -> +`guide_source_malformed`; `limit_exceeded` -> `guide_source_limit_exceeded`; +`parser_failure` -> `guide_source_extraction_failed`; `cancelled` -> +`guide_source_extraction_cancelled`; and `artifact_incident` -> +`guide_artifact_incident`. Recoverable incidents wait for ART recovery and +expose a bounded incident reference to authorized Operators. A current- +generation transient cancellation may retry; all other terminal source-format, +content, or limit failures require the Project Manager to upload a corrected or +smaller item into a new snapshot. Limits are never raised inline. None creates +a sufficiency decision. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md index 7de12e0e7..11fb4e858 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md @@ -508,6 +508,19 @@ and may be allowed only by the versioned extraction policy; required missing semantics stop setup internally. Artifact incidents never become sufficiency findings. +Stable setup mappings are exact: + +| Extraction status | Redacted setup code | Recovery/remediation | +|---|---|---| +| `extracted` | none | Continue only when every required item is complete. | +| `unsupported` | `guide_source_format_unsupported` | Project Manager supplies a supported item in a new snapshot. | +| `ambiguous` | `guide_source_format_ambiguous` | Project Manager supplies an unambiguous item in a new snapshot. | +| `malformed` | `guide_source_malformed` | Project Manager supplies a corrected item in a new snapshot. | +| `limit_exceeded` | `guide_source_limit_exceeded` | Project Manager supplies a smaller/simpler item; limits are not raised inline. | +| `parser_failure` | `guide_source_extraction_failed` | Bounded automatic retry; after exhaustion, PM replaces the item and Operator may inspect redacted diagnostics. | +| `cancelled` | `guide_source_extraction_cancelled` | Current-generation transient cancellation may retry; stale-generation cancellation commits no report. | +| `artifact_incident` | `guide_artifact_incident` | Recoverable custody failure waits for ART recovery; terminal corruption requires a new item/snapshot. | + Guide-source items are not required to be ZIP files. The initial semantic extractors are PDF, DOCX, PPTX, CSV, XLSX, Markdown, plain text, JSON, and PNG, JPEG, and WebP metadata. Images are accepted and classified, but without @@ -525,15 +538,10 @@ code. A separate immutable usage record binds that result to the exact binding, source item, setup run, and generation. Original bytes remain authoritative. A later derived-artifact store path requires a distinct AUTH action. -The existing `ProjectSetupRun.status=setup_blocked` remains the public state; -stable redacted error codes distinguish `guide_artifact_incident`, -`guide_source_format_unsupported`, `guide_source_malformed`, and -`guide_source_extraction_failed`. Artifact incidents expose a bounded incident -identifier to authorized Operators and wait for ART recovery when recoverable; -terminal corruption requires a Project Manager to create a new source item and -snapshot. Unsupported, malformed, limit-exceeded, or extraction-failed content -asks the Project Manager to upload a corrected supported source item. None is a -guide-insufficiency decision. +The existing `ProjectSetupRun.status=setup_blocked` remains the public state for +every terminal current-generation failure above. API, Operator, and Project +Manager projections use only the exact redacted mapping and remediation table. +None is a guide-insufficiency decision. Any production parser dependency in 03B3B requires explicit human approval. Candidate libraries must undergo current security, maintenance, license, transitive- diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/REVIEW_LOG.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/REVIEW_LOG.md index 07ca3dfcf..e65ffc7ea 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/REVIEW_LOG.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/REVIEW_LOG.md @@ -175,3 +175,7 @@ Agent Gates found five valid retired-vocabulary occurrences, now repaired. Backend recorded one unrelated AUTH PostgreSQL concurrency failure after 1,651 passes; the repaired documentation head requires a fresh hosted rerun. +- CodeRabbit later posted five valid findings. The repair makes repository and + scoped coverage gates explicit, fixes concrete extraction bounds and their + adversarial proof, exhaustively maps failure statuses, normalizes guide-read + wording, and adds locked revalidation immediately before report commit. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B2-guide-materialization-classification.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B2-guide-materialization-classification.md index bb181e736..38bcda66b 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B2-guide-materialization-classification.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B2-guide-materialization-classification.md @@ -42,6 +42,10 @@ scratch without extracting semantic content. unsupported outcome and never dispatch a parser; - DOCX/PPTX/XLSX require bounded internal markers; symlinks, macros, external relationships, embedded executables, bombs, and malformed containers reject; +- container inspection enforces 2,000 entries, 128 MiB decompressed bytes, + nesting depth 8, compression ratio 100:1, 40 megapixels, and 16,384 pixels on + either image dimension; every exact-boundary case succeeds, every one-over + case returns `limit_exceeded`, and no partial classification survives; - cleanup occurs on success, denial, mismatch, cancellation, and timeout; - changed subsystems remain at least 90% covered and repository coverage stays at least 78%. @@ -50,7 +54,9 @@ scratch without extracting semantic content. ```bash (cd backend && .venv/bin/python -m ruff check app tests scripts) -(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_guide_artifacts.py tests/test_artifact_scratch_manager.py tests/test_guide_format_detection.py -q --cov=app.modules.artifacts --cov-report=term-missing --cov-fail-under=90) +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_guide_artifacts.py tests/test_artifact_scratch_manager.py tests/test_guide_format_detection.py -q --cov=app --cov-report=term-missing --cov-fail-under=0) +(cd backend && .venv/bin/coverage report --precision=2 --fail-under=78) +(cd backend && .venv/bin/coverage report --include='app/modules/artifacts/*' --precision=2 --fail-under=90) python3 scripts/check_stale_artifact_contracts.py python3 scripts/check_markdown_links.py python3 scripts/test_agent_gates.py diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3A-extraction-framework-text.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3A-extraction-framework-text.md index fcf002de4..6a5900576 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3A-extraction-framework-text.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3A-extraction-framework-text.md @@ -26,6 +26,13 @@ and standard-library text, Markdown, JSON, and CSV extraction. - no-network subprocess enforces input/output, CPU/time/memory, row/cell, encoding, nesting, cancellation, and cleanup limits; +- the framework rejects parser input above 32 MiB, canonical output above + 4 MiB, execution beyond 60 seconds, or address-space use above 512 MiB; + CSV rejects more than 100,000 rows, 1,000,000 cells, or 32,768 characters in + one cell; every breach deterministically records `limit_exceeded`; +- tests cover each exact boundary and one-over boundary, timeout and memory + termination, cancellation, executor loss, scratch cleanup, and retry through + fresh materialization and fresh authority with no partial durable output; - extraction revalidates exact 03B2 digest/binding/format/policy provenance; - deterministic content records are keyed by content, format, extractor/version, and policy; separate usage records name item, binding, run, and generation; @@ -37,7 +44,9 @@ and standard-library text, Markdown, JSON, and CSV extraction. ```bash (cd backend && .venv/bin/python -m ruff check app tests scripts) -(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_guide_extraction.py -q --cov=app.modules.artifacts.guide_extraction --cov-report=term-missing --cov-fail-under=90) +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_guide_extraction.py -q --cov=app --cov-report=term-missing --cov-fail-under=0) +(cd backend && .venv/bin/coverage report --precision=2 --fail-under=78) +(cd backend && .venv/bin/coverage report --include='app/modules/artifacts/*,app/core/config.py,app/interfaces/artifacts.py' --precision=2 --fail-under=90) python3 scripts/check_stale_artifact_contracts.py python3 scripts/check_markdown_links.py python3 scripts/test_agent_gates.py diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3B-complex-format-extractors.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3B-complex-format-extractors.md index 67709b8a9..79a7de616 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3B-complex-format-extractors.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3B-complex-format-extractors.md @@ -33,6 +33,13 @@ security-review record. CI fails if a parser dependency is outside that list. - exact 03B2 classification provenance is required before parser execution; - all parsers stay inside the 03B3A isolation/limit/cancellation boundary; +- PDF rejects more than 500 pages; PPTX rejects more than 300 slides; XLSX + rejects more than 100 sheets, 100,000 rows, 1,000,000 cells, or 32,768 + characters in one cell; images reject more than 40 megapixels or 16,384 + pixels on either dimension; every breach records `limit_exceeded`; +- tests cover each exact boundary and one-over boundary, malformed containers, + parser crash, timeout, cancellation, cleanup, and executor loss without a + partial extraction or usage record; - OOXML markers distinguish formats and reject macros, external relationships, embedded executables, bombs, and ambiguity; - PNG/JPEG/WebP output is metadata only and cannot satisfy required text; @@ -44,7 +51,9 @@ security-review record. CI fails if a parser dependency is outside that list. ```bash (cd backend && .venv/bin/python -m ruff check app tests scripts) (cd backend && .venv/bin/python scripts/check_guide_extractor_dependencies.py) -(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_guide_format_detection.py tests/test_guide_extraction.py -q --cov=app.modules.artifacts.guide_extraction --cov-report=term-missing --cov-fail-under=90) +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_guide_format_detection.py tests/test_guide_extraction.py -q --cov=app --cov-report=term-missing --cov-fail-under=0) +(cd backend && .venv/bin/coverage report --precision=2 --fail-under=78) +(cd backend && .venv/bin/coverage report --include='app/modules/artifacts/*,app/core/config.py,app/interfaces/artifacts.py' --precision=2 --fail-under=90) python3 scripts/check_stale_artifact_contracts.py python3 scripts/check_markdown_links.py python3 scripts/test_agent_gates.py diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B4-guide-sufficiency-continuation.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B4-guide-sufficiency-continuation.md index 16f3ec411..34f8c5245 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B4-guide-sufficiency-continuation.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B4-guide-sufficiency-continuation.md @@ -43,7 +43,12 @@ same-generation canonical guide material and exact persisted provenance. tools/provider/secret authority, and prompt-injection fixtures cannot override system/developer policy or the typed sufficiency output contract; - `setup_blocked` exposes the stable redacted error codes and remediation defined - by D46, with bounded Operator incident reference only for ART incidents; + by D46 for `unsupported`, `ambiguous`, `malformed`, `limit_exceeded`, + `parser_failure`, `cancelled`, and `artifact_incident`, with bounded Operator + incident reference only for ART incidents; +- canonical agent material is capped at 12 MiB; an exact-boundary assembly may + proceed, a one-over assembly records `guide_source_limit_exceeded`, and no + agent call or partial report occurs; - duplicate or stale Celery execution cannot create or overwrite current output; - test-only fixed authority proves the complete hidden pipeline while composed live binding/read remains deny-only until AUTH-04B; @@ -55,6 +60,7 @@ same-generation canonical guide material and exact persisted provenance. ```bash (cd backend && .venv/bin/python -m ruff check app tests scripts) (cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_project_setup.py tests/test_guide_artifacts.py tests/test_guide_extraction.py tests/test_project_agents.py -q --cov=app --cov-report=term-missing --cov-fail-under=0) +(cd backend && .venv/bin/coverage report --precision=2 --fail-under=78) (cd backend && .venv/bin/coverage report --include='app/modules/projects/*,app/*ers/project_setup.py' --precision=2 --fail-under=90) python3 scripts/check_stale_artifact_contracts.py python3 scripts/check_markdown_links.py diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-external-review-response.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-external-review-response.md index 07675722c..f194d9247 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-external-review-response.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-external-review-response.md @@ -2,9 +2,13 @@ ## Comments Addressed -- CodeRabbit produced no findings because the organization review limit was - reached before review started. There are no inline CodeRabbit comments to - accept, reject, or defer on the current head. +- CodeRabbit's later review produced five valid findings. The contracts now + preserve repository-wide 78% coverage before their scoped 90% reports; name + exact v0.1 extraction limits, enforcement points, stable outcomes, boundary + tests, cleanup, and executor-loss proof; exhaustively map every extraction + status to a redacted public code and remediation; use consistent + `guide-read` wording; and require a second locked lineage/integrity/provenance + validation immediately before report commit. - Agent Gates correctly found five uses of retired human `worker` vocabulary in new planning text. They were replaced with parser runtime, executor, and project-setup executor wording. The future coverage command now avoids @@ -18,7 +22,6 @@ ## Comments Deferred -- CodeRabbit review itself remains pending until its rate limit permits a run. - No backend product-code repair is justified from the isolated AUTH concurrency failure without reproduction on the new head; it is outside this planning diff. @@ -38,5 +41,5 @@ Both pass. The full documentation/stale-contract suite is rerun before push. ## Remaining Risks -- CodeRabbit has not reviewed the PR yet. -- Hosted Backend and Agent Gates must pass the repaired exact head. +- Hosted Backend and Agent Gates must pass the repaired exact head, and the five + CodeRabbit threads must be verified against that head. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-pr-trust-bundle.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-pr-trust-bundle.md index fba65277c..a7aeaa7f2 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-pr-trust-bundle.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-pr-trust-bundle.md @@ -58,12 +58,15 @@ All nine tracks passed after repairs. Details are in ## External Review -CodeRabbit reached its review-rate limit before producing findings. Agent Gates -found and prompted repair of five retired-vocabulary occurrences. The first -Backend run had one unrelated existing AUTH concurrency failure after 1,651 -passes. Disposition is recorded in +CodeRabbit's later review produced five valid contract findings, now repaired: +explicit 78% plus scoped 90% coverage gates, concrete v0.1 extraction limits +and boundary proofs, exhaustive stable failure mappings, consistent guide-read +wording, and locked pre-report-commit lineage/integrity/provenance revalidation. +Agent Gates separately prompted repair of five retired-vocabulary occurrences. +The first Backend run had one unrelated existing AUTH concurrency failure after +1,651 passes. Disposition is recorded in `WS-ART-001-03B-PLAN-external-review-response.md`; repaired-head hosted checks -and CodeRabbit review remain required. +and thread verification remain required. ## Remaining Risks And Follow-Up diff --git a/.agent-loop/initiatives/WS-XINT-002-art-auth-end-to-end/chunks/WS-XINT-002-04B-guide-read-binding-activation.md b/.agent-loop/initiatives/WS-XINT-002-art-auth-end-to-end/chunks/WS-XINT-002-04B-guide-read-binding-activation.md index b4000343d..3fa4ad69a 100644 --- a/.agent-loop/initiatives/WS-XINT-002-art-auth-end-to-end/chunks/WS-XINT-002-04B-guide-read-binding-activation.md +++ b/.agent-loop/initiatives/WS-XINT-002-art-auth-end-to-end/chunks/WS-XINT-002-04B-guide-read-binding-activation.md @@ -2,7 +2,7 @@ ## Goal -Activate fixed-service guide binding and guide read only after merged ART-03B1, +Activate fixed-service guide binding and guide-read actions only after merged ART-03B1, 03B2, 03B3A, 03B3B, and 03B4 evidence, without weakening ART-03C. ## Risk class diff --git a/docs/spec_artifact_storage_service.md b/docs/spec_artifact_storage_service.md index 2fbef5633..082becd20 100644 --- a/docs/spec_artifact_storage_service.md +++ b/docs/spec_artifact_storage_service.md @@ -1238,9 +1238,16 @@ generation extraction reaches the agent, delimited as untrusted source data with no tool, secret, provider, or instruction authority. Unsupported, ambiguous, malformed, stale, or failed required material stops setup internally. -The public run remains `setup_blocked` with redacted stable codes -`guide_artifact_incident`, `guide_source_format_unsupported`, -`guide_source_malformed`, or `guide_source_extraction_failed`. Recoverable ART +Immediately before committing a sufficiency report, the setup transaction must +lock, reload, and revalidate the exact project, draft guide, source snapshot, +setup run and generation, guide-source binding, observed content-integrity +facts, and extraction-usage provenance used for the agent input. Any drift, +replacement, integrity mismatch, or incomplete provenance rolls back the report +commit and produces the applicable bounded internal failure; an earlier +pre-agent validation is not sufficient authority for this durable mutation. + +The public run remains `setup_blocked` with the redacted stable code defined for +the exact extraction outcome in D46. Recoverable ART incidents wait for recovery and expose only a bounded incident reference to an authorized Operator. Terminal corruption or source-content failure requires a new corrected Project Manager source item/snapshot; no outcome is a sufficiency From 121b057b999bdb26d17abd6f0867fdd28573b857 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Wed, 29 Jul 2026 08:55:14 +0100 Subject: [PATCH 4/4] docs(art): define extractor failure recovery --- .../DECISIONS.md | 20 +++++++++++++------ .../PLAN.md | 9 +++++++++ .../REVIEW_LOG.md | 4 ++++ ...T-001-03B-PLAN-external-review-response.md | 10 ++++++---- .../WS-ART-001-03B-PLAN-pr-trust-bundle.md | 5 +++-- 5 files changed, 36 insertions(+), 12 deletions(-) diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md index 8224c28f9..374be4dc9 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md @@ -462,9 +462,17 @@ redacted mapping: `unsupported` -> `guide_source_format_unsupported`; `guide_source_malformed`; `limit_exceeded` -> `guide_source_limit_exceeded`; `parser_failure` -> `guide_source_extraction_failed`; `cancelled` -> `guide_source_extraction_cancelled`; and `artifact_incident` -> -`guide_artifact_incident`. Recoverable incidents wait for ART recovery and -expose a bounded incident reference to authorized Operators. A current- -generation transient cancellation may retry; all other terminal source-format, -content, or limit failures require the Project Manager to upload a corrected or -smaller item into a new snapshot. Limits are never raised inline. None creates -a sufficiency decision. +`guide_artifact_incident`. An observed 60-second timeout or 512-MiB +address-space termination is a deterministic `limit_exceeded` outcome: it does +not retry automatically, and the Project Manager must provide a smaller or +simpler item in a new snapshot. Limits are never raised inline. Executor loss +before a classified limit breach is `parser_failure`: it receives one bounded +current-generation automatic retry from fresh materialization and fresh +authority; repeated loss remains `guide_source_extraction_failed`, creates no +successful usage/report, and requires a corrected/new snapshot while an +authorized Operator may inspect only bounded redacted runtime diagnostics. +Recoverable artifact incidents wait for ART recovery and expose a bounded +incident reference to authorized Operators. A current-generation transient +cancellation may retry; stale-generation cancellation commits no report. All +other terminal source-format or content failures require a corrected item in a +new snapshot. None creates a sufficiency decision. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md index 11fb4e858..fc2eb1b7f 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md @@ -521,6 +521,15 @@ Stable setup mappings are exact: | `cancelled` | `guide_source_extraction_cancelled` | Current-generation transient cancellation may retry; stale-generation cancellation commits no report. | | `artifact_incident` | `guide_artifact_incident` | Recoverable custody failure waits for ART recovery; terminal corruption requires a new item/snapshot. | +The configured 60-second timeout and 512-MiB address-space termination are +`limit_exceeded`, are not automatically retried, and require a smaller/simpler +item in a new snapshot. Executor loss before a classified limit breach is +`parser_failure`: it receives one bounded current-generation retry from fresh +materialization and fresh authority; repeated loss exposes only +`guide_source_extraction_failed` plus bounded redacted Operator diagnostics and +requires a corrected/new snapshot. No failed attempt creates a successful usage +record or report. + Guide-source items are not required to be ZIP files. The initial semantic extractors are PDF, DOCX, PPTX, CSV, XLSX, Markdown, plain text, JSON, and PNG, JPEG, and WebP metadata. Images are accepted and classified, but without diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/REVIEW_LOG.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/REVIEW_LOG.md index e65ffc7ea..65ef5354f 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/REVIEW_LOG.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/REVIEW_LOG.md @@ -179,3 +179,7 @@ scoped coverage gates explicit, fixes concrete extraction bounds and their adversarial proof, exhaustively maps failure statuses, normalizes guide-read wording, and adds locked revalidation immediately before report commit. +- The repaired-head review found one remaining recovery-semantics gap. Timeout + and memory termination are now non-retryable limit failures; executor loss + receives one bounded fresh-authority/materialization retry, then a stable + extraction-failed outcome with only redacted diagnostics. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-external-review-response.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-external-review-response.md index f194d9247..ea797de3f 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-external-review-response.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-external-review-response.md @@ -2,11 +2,13 @@ ## Comments Addressed -- CodeRabbit's later review produced five valid findings. The contracts now +- CodeRabbit's reviews produced six valid findings. The contracts now preserve repository-wide 78% coverage before their scoped 90% reports; name exact v0.1 extraction limits, enforcement points, stable outcomes, boundary - tests, cleanup, and executor-loss proof; exhaustively map every extraction - status to a redacted public code and remediation; use consistent + tests, cleanup, and executor-loss proof; map timeout and memory termination to + non-retryable limit failure and executor loss to one bounded fresh retry; + exhaustively map every extraction status to a redacted public code and + remediation; use consistent `guide-read` wording; and require a second locked lineage/integrity/provenance validation immediately before report commit. - Agent Gates correctly found five uses of retired human `worker` vocabulary in @@ -41,5 +43,5 @@ Both pass. The full documentation/stale-contract suite is rerun before push. ## Remaining Risks -- Hosted Backend and Agent Gates must pass the repaired exact head, and the five +- Hosted Backend and Agent Gates must pass the repaired exact head, and all six CodeRabbit threads must be verified against that head. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-pr-trust-bundle.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-pr-trust-bundle.md index a7aeaa7f2..33493f4ee 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-pr-trust-bundle.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-pr-trust-bundle.md @@ -58,9 +58,10 @@ All nine tracks passed after repairs. Details are in ## External Review -CodeRabbit's later review produced five valid contract findings, now repaired: +CodeRabbit's reviews produced six valid contract findings, now repaired: explicit 78% plus scoped 90% coverage gates, concrete v0.1 extraction limits -and boundary proofs, exhaustive stable failure mappings, consistent guide-read +and boundary proofs, exhaustive stable failure mappings including timeout, +memory-termination, and executor-loss recovery semantics, consistent guide-read wording, and locked pre-report-commit lineage/integrity/provenance revalidation. Agent Gates separately prompted repair of five retired-vocabulary occurrences. The first Backend run had one unrelated existing AUTH concurrency failure after