diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/AUTH_HANDOFF.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/AUTH_HANDOFF.md index 89a8cb7d..0608499f 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/AUTH_HANDOFF.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/AUTH_HANDOFF.md @@ -17,10 +17,14 @@ activation custody, and availability. exact resource/guard/surface manifest. 3. AUTH activates only that exact action through a separately reviewed AUTH contract after consuming ART evidence. -4. ART-03B implements hidden `artifact.guide_source.read` and - `artifact.guide_source.binding.create` behavior; binding maps to fixed - permission `artifact.binding.create`. -5. AUTH activates only those exact actions after consuming 03B evidence. +4. ART-03B1, 03B2, 03B3A, 03B3B, and 03B4 implement hidden authoritative binding, exact setup + generation, verified materialization, bounded extraction, and Celery + sufficiency continuation. `artifact.guide_source.binding.create` maps to + fixed permission `artifact.binding.create`; `artifact.guide_source.read` + remains separate. +5. AUTH-04B activates only those exact fixed-service actions after consuming + the complete split-03B evidence. Extraction does not imply a provider-write + permission or inherit Project Manager authority. 6. ART-03C performs the legacy clean cut. No ART chunk writes availability. ## Submission Bundle Sequence diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md index 8af1f995..d9f253e9 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md @@ -17,9 +17,13 @@ Each chunk is one PR. No later chunk starts automatically. | `WS-ART-001-02D` | Add hidden Operator content/job/retry/recovery/audit APIs, canonical resource composition, and production-readiness checks while actions and provider profiles remain inactive. | L1 | Merged through PR #177 as `93c14181` | | `WS-ART-001-03` | Original combined guide-source cutover. | L1 | Cancelled before implementation; no runtime changes | | `WS-ART-001-PLAN2` | Reconcile guide and one-ZIP submission planning with bounded scratch, existing immutable admission/recovery, exact AUTH sequencing, and downstream ownership. | L1 | Planning-only successor proposed after cancellation | -| `WS-ART-001-03A` | Add hidden guide-source byte ingest through existing preparation, admission, verification, and publication. | L1 | Proposed after PLAN2 | -| `WS-ART-001-03B` | Bind verified guide-source content and provide authorized integrity-checking setup materialization. | L1 | Proposed after 03A and exact AUTH activation | -| `WS-ART-001-03C` | Remove legacy guide-source identity and add exact same-generation setup continuation. | L1 | Proposed after 03B and exact AUTH activation | +| `WS-ART-001-03A` | Add hidden guide-source byte ingest through existing preparation, admission, verification, and publication. | L1 | Merged through PR #215 as `bb9082a7` | +| `WS-ART-001-03B1` | Add authoritative verified guide-source bindings and an exact setup-generation fence while fixed-service binding remains unavailable. | L1 | Proposed after 03A and AUTH-04A | +| `WS-ART-001-03B2` | Add fixed-reader verified materialization, artifact incidents, and canonical format detection in bounded ART scratch. | L1 | Proposed after 03B1 | +| `WS-ART-001-03B3A` | Add the isolated extraction framework, canonical records, and text/Markdown/JSON/CSV extractors. | L1 | Proposed after 03B2 | +| `WS-ART-001-03B3B` | Add explicitly approved PDF/OOXML/image-metadata extractors on the proven framework. | L1 | Proposed after 03B3A and dependency approval | +| `WS-ART-001-03B4` | Feed only complete same-generation canonical extracted material into the existing Celery sufficiency pipeline. | L1 | Proposed after 03B3B | +| `WS-ART-001-03C` | Remove legacy guide-source identity and add exact same-generation setup continuation. | L1 | Proposed after 03B1-03B4 and AUTH-04B | | `WS-ART-001-04A` | Accept one outer ZIP in bounded scratch, safely inspect its tree, normalize executable intent, produce canonical identities, and reject unchanged work before provider I/O. | L1 | Proposed after 03C and AUTH planned action registration | | `WS-ART-001-04B` | Run mandatory platform and locked Project Guide pre-submit checks against the same scratch-bound tree and executable semantics without durable storage. | L1 | Proposed after 04A | | `WS-ART-001-04C` | Reauthorize at durable intent, admit/verify the passing ZIP once, and publish one capacity-charged `ready` admission that may remain unbound. | L1 | Proposed after 04B; AUTH activation follows hidden completion | @@ -43,7 +47,11 @@ OBJECT-STORAGE-AMENDMENT -> PLAN2 planning reconciliation -> 03A guide-source byte ingest -> AUTH activation for exact 03A actions --> 03B guide-source binding/materialization +-> 03B1 authoritative guide binding/setup generation +-> 03B2 verified materialization/format detection +-> 03B3A isolated extraction framework/text formats +-> 03B3B approved complex document/image-metadata extractors +-> 03B4 same-generation Celery sufficiency integration -> AUTH activation for exact 03B actions -> 03C guide-source clean cut/continuation -> AUTH planned registration of `artifact.submission_bundle.prepare` diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md index 68b83c57..374be4dc 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md @@ -362,3 +362,117 @@ Contributor authority never implies fixed service authority. Checker materialization, verification execution, pending-work scanning, ambiguous-put resolution, and binding retain their distinct fixed actions. No compatibility alias exposes the obsolete multi-step session actions. + +## D39 - Guide Upload And Understanding Are Separate + +Guide upload stores and verifies opaque original bytes. HTTP ingest does not +parse, extract, render, OCR, transcode, or wait for submission prechecks. +Understanding begins asynchronously only after an exact verified binding. +Unlike contributor submissions, guide-source items are not wrapped in one +mandatory outer ZIP. Each item retains its own verified format identity. + +## D40 - Exact Setup Generation Is A Durable Fence + +Every binding, extraction, agent input, and report identifies one project, +draft guide, source snapshot, setup run, and monotonic setup generation. Celery +carries only those identifiers and generation. Bytes, extracted text, scratch +handles, prepared authority, and provider credentials never enter Redis. + +## D41 - Canonical Guide Materialization + +The fixed guide-reader obtains fresh prepared authority, resolves an exact +verified binding, and streams through `ArtifactStore` into the existing +`ArtifactScratchManager`. Every full read recomputes SHA-256 and byte count. +Missing, changed, or truncated content is an ART incident, not guide +insufficiency. Project services and agents never access providers directly. + +## D42 - Typed, Bounded Guide Extraction + +One detector and typed extractor registry validates signatures and bounded +container markers. Initial text semantics cover PDF, DOCX, PPTX, CSV, XLSX, +Markdown, plain text, and JSON. PNG, JPEG, and WebP yield bounded structural +metadata only. Without OCR, image pixels cannot satisfy required textual guide +semantics. Audio/video and ambiguous binaries are unsupported in v0.1 and never +sent raw to agents. OOXML formats are distinguished from ordinary ZIPs. + +Extraction runs asynchronously in a strongly isolated no-network subprocess +under fixed input, output, container, time, memory, and document limits using +only scratch-manager paths. Parser crash, malformed input, macros, external +relationships, embedded executables, cancellation, timeout, and executor loss +have bounded outcomes and cleanup. Production parser dependencies require +explicit human approval before implementation. + +The v0.1 extraction-policy limits are fixed, not caller-selectable: + +| Limit | Value | Enforcement owner | Breach outcome | +|---|---:|---|---| +| parser input per item | 32 MiB | 03B3A framework before dispatch | `limit_exceeded` | +| canonical output per item | 4 MiB | 03B3A streaming output collector | `limit_exceeded` | +| aggregate agent material | 12 MiB | 03B4 input assembler | `limit_exceeded` | +| subprocess wall time per item | 60 seconds | 03B3A process supervisor | `limit_exceeded` | +| subprocess address space | 512 MiB | 03B3A OS isolation boundary | `limit_exceeded` | +| container entries | 2,000 | 03B2 container inspector | `limit_exceeded` | +| decompressed container bytes | 128 MiB | 03B2 container inspector | `limit_exceeded` | +| container nesting depth | 8 | 03B2 container inspector | `limit_exceeded` | +| compression ratio | 100:1 | 03B2 container inspector | `limit_exceeded` | +| PDF pages | 500 | 03B3B PDF adapter | `limit_exceeded` | +| PPTX slides | 300 | 03B3B PPTX adapter | `limit_exceeded` | +| XLSX sheets | 100 | 03B3B XLSX adapter | `limit_exceeded` | +| table rows per item | 100,000 | 03B3A CSV / 03B3B XLSX adapter | `limit_exceeded` | +| table cells per item | 1,000,000 | 03B3A CSV / 03B3B XLSX adapter | `limit_exceeded` | +| characters per cell | 32,768 | CSV/XLSX adapter | `limit_exceeded` | +| image pixels | 40 megapixels | 03B2 header inspector and 03B3B adapter | `limit_exceeded` | +| image width or height | 16,384 pixels | 03B2 header inspector and 03B3B adapter | `limit_exceeded` | + +Exact-boundary, one-over-boundary, cleanup, cancellation, timeout, memory- +termination, and executor-loss tests are mandatory. Executor loss leaves no +successful extraction usage record; a current-generation retry starts from +fresh materialization and authority. + +## D43 - Canonical Extraction Records, Not Implicit Provider Writes + +v0.1 persists an immutable content-derived extraction representation in +PostgreSQL keyed by original content, format, extractor/version, and policy +version, with output digest, omission facts, status, and bounded error code. A +separate immutable usage record binds it to the exact guide binding, source +item, setup run, and generation. AUTH-04B grants read and binding only, so ART +does not use read authority to create a provider object. + +## D44 - Sufficiency Consumes Complete Verified Material + +The agent receives only bounded extraction records for all required items in +the current generation. Missing, corrupt, stale, ambiguous, unsupported, or +failed content stops policy derivation without creating a guide-insufficiency +decision. Reports preserve exact content and extraction provenance. Legacy +excerpts and durable refs are not authoritative after ART-03C. + +## D45 - Extracted Guide Content Is Untrusted Agent Data + +Canonical extraction proves byte provenance, not instruction authority. Agent +assembly labels and delimits source material as untrusted data, excludes tools, +provider access, credentials, and hidden instructions from that material, and +accepts only the typed sufficiency output contract. Prompt-injection text inside +a guide cannot alter system/developer policy or authorize an action. + +## D46 - Setup Failures Have Stable Operational Outcomes + +Existing `ProjectSetupRun.status=setup_blocked` uses this exhaustive stable, +redacted mapping: `unsupported` -> `guide_source_format_unsupported`; +`ambiguous` -> `guide_source_format_ambiguous`; `malformed` -> +`guide_source_malformed`; `limit_exceeded` -> `guide_source_limit_exceeded`; +`parser_failure` -> `guide_source_extraction_failed`; `cancelled` -> +`guide_source_extraction_cancelled`; and `artifact_incident` -> +`guide_artifact_incident`. An observed 60-second timeout or 512-MiB +address-space termination is a deterministic `limit_exceeded` outcome: it does +not retry automatically, and the Project Manager must provide a smaller or +simpler item in a new snapshot. Limits are never raised inline. Executor loss +before a classified limit breach is `parser_failure`: it receives one bounded +current-generation automatic retry from fresh materialization and fresh +authority; repeated loss remains `guide_source_extraction_failed`, creates no +successful usage/report, and requires a corrected/new snapshot while an +authorized Operator may inspect only bounded redacted runtime diagnostics. +Recoverable artifact incidents wait for ART recovery and expose a bounded +incident reference to authorized Operators. A current-generation transient +cancellation may retry; stale-generation cancellation commits no report. All +other terminal source-format or content failures require a corrected item in a +new snapshot. None creates a sufficiency decision. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DISCOVERY.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DISCOVERY.md index d445f33b..bbf6c187 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DISCOVERY.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DISCOVERY.md @@ -129,3 +129,33 @@ removes direct provider schemes from guide-source identity. Chunk 05 removes the remaining caller storage transport when submissions move to sealed artifact-set bindings. No compatibility alias remains after either owning cutover. + +## ART-03B Materialization And Extraction Discovery + +Merged ART-03A and AUTH `WS-XINT-002-04A` provide the covered Project Manager +upload boundary. AUTH preparation happens before request-body intake; final +transaction-bound consumption locks the guide lineage and server-computed byte +facts before durable intent or provider I/O. Upload stores opaque immutable +bytes and does not parse them. + +The current setup pipeline cannot yet consume those bytes safely: + +- its Celery payload has four identifiers but no setup generation; +- `ProjectSetupRun` has no explicit generation fence; +- `GuideSourceMaterial` uses caller-originated metadata and optional excerpts, + not verified ART bindings; +- no authorized integrity-checking guide read exists in project services; +- no typed detector, extractor registry, extraction policy, or canonical + extraction record exists. + +AUTH's reviewed `WS-XINT-002-04B` activates only fixed-service guide binding +and read. It does not grant a derived-artifact write. v0.1 therefore persists a +bounded canonical extraction record and provenance in PostgreSQL; a future +derived provider artifact requires a separately planned action and activation. + +No document-parsing production dependencies are installed. Any new production +dependency requires explicit human approval before 03B3B implementation. OOXML ZIP +containers must pass one bounded classifier that distinguishes DOCX, PPTX, and +XLSX by internal markers. Audio/video transcription and OCR are not required by +current v0.1 setup and remain unsupported. PNG/JPEG/WebP classification and +metadata do not imply OCR or textual sufficiency. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/INTENT.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/INTENT.md index 374bd689..8b686114 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/INTENT.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/INTENT.md @@ -65,6 +65,14 @@ directories, or any other content allowed by the locked Project Guide. A ZIP entry inside the outer archive remains an ordinary file. Nested archive unpacking is outside v0.1 even when a guide would otherwise request it. +This contributor ZIP rule does not apply to Project Manager guide-source +uploads. A guide snapshot may contain multiple independently uploaded source +items in PDF, DOCX, PPTX, CSV, XLSX, Markdown, plain text, JSON, PNG, JPEG, or +WebP form. Images are metadata-only without OCR and cannot satisfy required +textual semantics. v0.1 does not support guide audio or video. DOCX, PPTX, and +XLSX are recognized as their document types despite using ZIP containers +internally; an arbitrary ordinary ZIP is not treated as a guide document. + ## First-Principle Constraints - Workstream computes and verifies canonical SHA-256 and byte count. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md index 8f07eb17..fc2eb1b7 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md @@ -471,6 +471,92 @@ exact resource/content/job/attempt filters. ## Product Cutover +### Guide Materialization And Extraction Boundary + +ART-03A upload remains a byte-custody request: authorize before intake, compute +the server digest and size, admit once, store opaque bytes, and verify them. It +does not parse content or wait for contributor submission checkers. + +The expanded ART-03B work is delivered as five hidden PR-sized subchunks before +AUTH-04B activates fixed-service binding/read: + +1. `03B1` creates an authoritative one-item/one-content guide binding tied to + the exact setup run and explicit setup generation. Item metadata cannot + assert content identity, and replaced or cross-lineage facts fail closed. +2. `03B2` performs fresh authorized full reads through `ArtifactStore`, writes + only through `ArtifactScratchManager`, recomputes digest/size, records ART + incidents, and classifies formats using signatures and bounded container + markers. +3. `03B3A` installs the isolated no-network extraction framework, immutable + content/usage provenance, and text, Markdown, JSON, and CSV extractors + without new parser dependencies. +4. `03B3B` adds explicitly approved PDF, DOCX, PPTX, XLSX, and PNG/JPEG/WebP + metadata extractors on that proven framework. +5. `03B4` adds `setup_generation` to the identifier-only Celery payload, + reloads current state, assembles all required canonical extracted sources, + invokes sufficiency, and persists exact provenance. + +The canonical extraction status set is: + +```text +extracted | unsupported | ambiguous | malformed | limit_exceeded | +parser_failure | cancelled | artifact_incident +``` + +Only `extracted` records enter agent input. Truncation or omission is explicit +and may be allowed only by the versioned extraction policy; required missing +semantics stop setup internally. Artifact incidents never become sufficiency +findings. + +Stable setup mappings are exact: + +| Extraction status | Redacted setup code | Recovery/remediation | +|---|---|---| +| `extracted` | none | Continue only when every required item is complete. | +| `unsupported` | `guide_source_format_unsupported` | Project Manager supplies a supported item in a new snapshot. | +| `ambiguous` | `guide_source_format_ambiguous` | Project Manager supplies an unambiguous item in a new snapshot. | +| `malformed` | `guide_source_malformed` | Project Manager supplies a corrected item in a new snapshot. | +| `limit_exceeded` | `guide_source_limit_exceeded` | Project Manager supplies a smaller/simpler item; limits are not raised inline. | +| `parser_failure` | `guide_source_extraction_failed` | Bounded automatic retry; after exhaustion, PM replaces the item and Operator may inspect redacted diagnostics. | +| `cancelled` | `guide_source_extraction_cancelled` | Current-generation transient cancellation may retry; stale-generation cancellation commits no report. | +| `artifact_incident` | `guide_artifact_incident` | Recoverable custody failure waits for ART recovery; terminal corruption requires a new item/snapshot. | + +The configured 60-second timeout and 512-MiB address-space termination are +`limit_exceeded`, are not automatically retried, and require a smaller/simpler +item in a new snapshot. Executor loss before a classified limit breach is +`parser_failure`: it receives one bounded current-generation retry from fresh +materialization and fresh authority; repeated loss exposes only +`guide_source_extraction_failed` plus bounded redacted Operator diagnostics and +requires a corrected/new snapshot. No failed attempt creates a successful usage +record or report. + +Guide-source items are not required to be ZIP files. The initial semantic +extractors are PDF, DOCX, PPTX, CSV, XLSX, Markdown, plain text, JSON, and +PNG, JPEG, and WebP metadata. Images are accepted and classified, but without +OCR their pixels cannot satisfy required textual guide semantics. Audio/video +and opaque/ordinary ZIP input are unsupported in v0.1. ZIP- +container classification rejects macros, external relationships, embedded +executables, ambiguous markers, excessive entries, decompressed bytes, depth, +or compression ratio before a document parser runs. + +v0.1 stores bounded canonical extraction JSON/text in PostgreSQL rather than +creating an unapproved provider write under guide-read authority. It records +source digest/size, detected format, extractor and version, extraction-policy +version, output digest, truncation/omission facts, status, and bounded error +code. A separate immutable usage record binds that result to the exact binding, +source item, setup run, and generation. Original bytes remain authoritative. A +later derived-artifact store path requires a distinct AUTH action. + +The existing `ProjectSetupRun.status=setup_blocked` remains the public state for +every terminal current-generation failure above. API, Operator, and Project +Manager projections use only the exact redacted mapping and remediation table. +None is a guide-insufficiency decision. + +Any production parser dependency in 03B3B requires explicit human approval. Candidate +libraries must undergo current security, maintenance, license, transitive- +dependency, malformed-input, and cancellation review before `03B3B` may add +them; no runtime plugin discovery or parser fallback is permitted. + 1. Guide-source delivery is split into hidden byte ingest, AUTH activation, verified snapshot binding/materialization, AUTH activation, and the legacy identity/continuation clean cut. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/REVIEW_LOG.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/REVIEW_LOG.md index 1c178154..65ef5354 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/REVIEW_LOG.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/REVIEW_LOG.md @@ -155,3 +155,31 @@ over-broad lineage trigger and an incomplete integrity-mismatch upload-item transition. Both received bounded repairs and their three failing tests pass locally; final internal re-review and hosted rerun remain. + +## WS-ART-001-03B-PLAN + +- Planning correction reviewed after ART-03A and AUTH-04A merged. +- Initial architecture, senior, QA, CI, and product/ops reviews found blocking + scope, provenance, parser-boundary, verification, image-semantics, and + operational-outcome gaps. +- Repairs split implementation into 03B1, 03B2, 03B3A, 03B3B, and 03B4; + separated content extraction from binding/generation usage; defined isolated + parsing, prompt-injection handling, stable setup errors, exact commands, and + the later parser dependency approval gate. +- Final results: architecture PASS WITH LOW RISKS; senior engineering PASS WITH + LOW RISKS; QA PASS WITH LOW RISKS; security PASS; product/ops PASS; CI PASS + WITH CONDITIONS; docs PASS WITH LOW RISKS; reuse PASS WITH LOW RISKS; test + delta PASS WITH CONDITIONS. Planning-PR conditions are resolved by committing + the new contracts; the dependency checker remains a future 03B3B criterion. +- First external run: CodeRabbit was rate-limited and produced no findings. + Agent Gates found five valid retired-vocabulary occurrences, now repaired. + Backend recorded one unrelated AUTH PostgreSQL concurrency failure after + 1,651 passes; the repaired documentation head requires a fresh hosted rerun. +- CodeRabbit later posted five valid findings. The repair makes repository and + scoped coverage gates explicit, fixes concrete extraction bounds and their + adversarial proof, exhaustively maps failure statuses, normalizes guide-read + wording, and adds locked revalidation immediately before report commit. +- The repaired-head review found one remaining recovery-semantics gap. Timeout + and memory termination are now non-retryable limit failures; executor loss + receives one bounded fresh-authority/materialization retry, then a stable + extraction-failed outcome with only redacted diagnostics. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/RISKS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/RISKS.md index c64b8b41..963504c4 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/RISKS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/RISKS.md @@ -52,3 +52,12 @@ | Full history requires expensive recursive traversal | Medium | Preserve immutable links but add indexed latest/current/accepted repository queries or projections maintained under lifecycle transactions. | | Stored content changes after initial verification | Critical | Recompute full SHA-256 and byte count during checker materialization, reviewer full download, and downstream delivery; fail closed and record an artifact incident on mismatch. | | Speculative larger limits exhaust disk or processing time | High | Preserve current configured limits until separately reviewed capacity, concurrency, runtime, cost, and abuse evidence justifies a change. | +| Caller metadata or excerpt reaches the sufficiency agent instead of verified bytes | Critical | Resolve exact ART bindings, fully rehash provider reads, extract under a versioned policy, and assemble agent input only from same-generation canonical extraction records. | +| DOCX/PPTX/XLSX is confused with an arbitrary ZIP | Critical | Use one bounded container classifier with required internal-marker checks; ambiguous or ordinary ZIP is unsupported and never sent raw to an agent. | +| Malicious document exploits a parser or extraction runtime | Critical | Typed extractor registry, isolated no-network subprocess, fixed CPU/time/memory/output/container limits, external-reference and macro rejection, cancellation, and scratch-owned cleanup. | +| Artifact corruption is reported as guide insufficiency | Critical | Digest/size mismatch, missing/truncated objects, and provider failures create ART incident outcomes and stop setup before the sufficiency agent. | +| Extraction output silently changes across deployments | High | Persist content-derived format/extractor/policy identity and digest separately from exact binding/run/generation usage provenance; never rewrite prior output. | +| Read authority is reused to write a derived provider artifact | Critical | v0.1 stores only bounded extraction records in PostgreSQL; any derived provider write requires a separate action and AUTH activation. | +| Stale Celery delivery processes replaced guide content | Critical | Payloads contain identifiers plus setup generation only; executors reload current lineage, binding, extraction, run, and generation before invocation and persistence. | +| Unsupported image/audio/video content is passed raw to an agent | High | PNG/JPEG/WebP expose bounded structural metadata only; OCR and audio/video are unsupported, and image-only required text stops setup internally. | +| Extracted guide text performs prompt injection | Critical | Delimit and label extracted material as untrusted data, expose no tools/secrets/provider authority, enforce typed output, and test adversarial embedded instructions. | diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md index 3bb2d701..020f7e6f 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md @@ -51,16 +51,22 @@ normalizes regular-file executable intent into the semantic manifest; and it requires fresh AUTH prepared capabilities at durable put intent and atomic Submission/binding consumption. -## Next Proposed Chunk +## Current Planning Correction -After this planning package merges, `WS-ART-001-03A` is the only immediate ART -successor. It adds hidden guide-source byte ingest through the existing artifact -preparation/admission/verification path. It requires a separate signed start and -does not activate its own AUTH action. +ART-03A and AUTH `WS-XINT-002-04A` are merged. Before ART-03B implementation, +the guide-content boundary is being corrected explicitly: verified binding, +full-read materialization, format classification, isolated extraction, +canonical extraction provenance, incremental complex-format support, and +same-generation sufficiency continuation are separate PR-sized contracts. +`WS-ART-001-03B1` is the first proposed +implementation successor; this planning change contains no implementation. + +AUTH `WS-XINT-002-04B` follows the complete hidden split-03B series and +activates only fixed-service binding and guide read. ART-03C then removes the +legacy identity/excerpt path and makes the verified pipeline authoritative. ## Gate Planning evidence and all required internal reviewer tracks must pass before a -PR. The planning merge starts no successor. Every implementation chunk retains -its separate signed start, exact AUTH activation sequence, internal review, CI, -CodeRabbit, human checkpoint, and automated merge-memory stop. +PR. Every implementation chunk retains its own bounded contract, exact AUTH +activation sequence, internal review, CI, CodeRabbit, and human checkpoint. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B-guide-source-binding-materialization.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B-guide-source-binding-materialization.md index 7a177641..f371677c 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B-guide-source-binding-materialization.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B-guide-source-binding-materialization.md @@ -1,70 +1,47 @@ -# Chunk Contract: WS-ART-001-03B - Guide Source Binding And Materialization +# Planning Contract: WS-ART-001-03B - Guide Content Pipeline Expansion -Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 03A and exact AUTH activation +Initiative: `WS-ART-001` | Risk: L1 | Status: Superseded by reviewed subchunks before implementation Artifact contract phase: `guide_source_cutover` ## Goal -Bind verified guide-source content to snapshot items and give setup agents one -authorized, provider-neutral, integrity-checking read/materialization capability -without performing the legacy schema cutover. +Preserve the original 03B intent while dividing binding, verified +materialization, isolated extraction, and sufficiency continuation into bounded +implementation contracts. -## Allowed Files +## Executable Subchunks -- guide snapshot/item binding models, one migration, repository, schemas, and service; -- ART guide read/binding capability and canonical resource facts/guards; -- existing artifact scratch manager/materializer integration for setup input; -- setup-agent input assembly consuming the typed ART capability; -- focused migration, authorization, binding, materialization, and integrity tests; -- backend workflow/agent-gate files only to add and preserve exact coverage; -- related docs and chunk memory. +- `WS-ART-001-03B1`: authoritative binding and setup generation; +- `WS-ART-001-03B2`: verified materialization, incidents, and classification; +- `WS-ART-001-03B3A`: isolated framework and low-complexity extraction; +- `WS-ART-001-03B3B`: approved complex document/image-metadata extractors; +- `WS-ART-001-03B4`: same-generation Celery and sufficiency integration. -## Not Allowed +Each subchunk is one PR. AUTH `WS-XINT-002-04B` activates fixed binding/read +only after all five hidden ART contracts merge. ART-03C remains a later clean +cut. + +## Shared Non-Goals - contributor submission/checker/review behavior; -- direct provider access, URL fetching, a second materializer/scratch manager, - setup policy decisions, or destructive legacy-field removal; +- direct provider access, URL fetching, a second scratch manager, parsing in the + upload request, submission prechecks, OCR, audio/video transcription, or + destructive legacy-field removal; - AUTH-owned catalogue/evaluator/grant/identity/matrix/availability edits. ## Acceptance Criteria -- snapshot items bind exact verified `ArtifactContent` without circularly making - the item both the source and authority for that content; -- snapshot/bundle identity commits to ordered content identities and descriptors; -- setup agents receive bounded bytes/manifests only through ART and every full - read recomputes SHA-256 and byte count; -- missing/mismatched bound bytes are artifact incidents, not guide insufficiency; -- fixed guide read/binding actions remain fail-closed until AUTH activation; -- changed subsystem coverage is at least 90 percent and repository coverage - remains at least 78 percent. - -## Exact CI Coverage Gates - -```bash -coverage report --include='app/adapters/artifacts/*,app/core/cancellation.py,app/core/file_locks.py,app/interfaces/artifact_operations.py,app/interfaces/artifacts.py,app/modules/artifacts/*' --precision=2 --fail-under=90 -coverage report --include='app/interfaces/external_services.py' --precision=2 --fail-under=90 -coverage report --include='app/core/config.py' --precision=2 --fail-under=90 -coverage report --include='app/workers/*' --precision=2 --fail-under=90 -coverage report --include='app/main.py' --precision=2 --fail-under=90 -coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 -coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 -coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 -coverage report --include='app/api/router.py' --precision=2 --fail-under=90 -coverage report --include='app/modules/projects/*' --precision=2 --fail-under=90 -coverage report --include='app/adapters/project_agents/*,app/interfaces/project_agents.py' --precision=2 --fail-under=90 -``` +- all five contracts merge before AUTH-04B; +- agents receive only same-generation canonical extracted material; +- artifact failures remain distinct from guide insufficiency; +- fixed guide read/binding remain unavailable until AUTH-04B; +- production parser dependencies require explicit human approval. -## Verification +## Evidence -```bash -docker compose up -d --wait postgres redis minio -(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_guide_artifacts.py tests/test_project_setup.py tests/test_checker_materialization.py tests/test_artifact_scratch_manager.py -q --cov=app.modules.projects --cov=app.modules.artifacts --cov-report=term-missing --cov-fail-under=90) -(metadata_dir="$(mktemp -d)" && trap 'rm -rf "$metadata_dir"' EXIT && (cd backend && WORKSTREAM_TEST_ADMIN_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/postgres .venv/bin/python scripts/run_isolated_tests.py --metadata-json "$metadata_dir/result.json" --timeout-seconds 12600 -- .venv/bin/python -m pytest -q --ignore=tests/test_isolated_database_runner.py --cov=app --cov-report=term-missing --cov-fail-under=78)) -(cd backend && .venv/bin/ruff check app tests) -python3 scripts/check_stale_artifact_contracts.py -python3 scripts/test_agent_gates.py -``` +The five executable contracts own their exact file scopes, tests, coverage, and +review evidence. This umbrella is planning context and authorizes no code. ## Required Reviewers diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B1-guide-binding-generation.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B1-guide-binding-generation.md new file mode 100644 index 00000000..e49d2ea9 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B1-guide-binding-generation.md @@ -0,0 +1,56 @@ +# Chunk Contract: WS-ART-001-03B1 - Guide Binding And Setup Generation + +Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after ART-03A and AUTH-04A + +## Goal + +Bind one exact verified `ArtifactContent` to one guide-source item and exact +setup run/generation without legacy cutover or activating fixed-service AUTH. + +## Allowed Files + +- project guide snapshot/item/setup-run models, one migration, repositories, + and schemas; +- ART binding capability, exact canonical facts, guards, and hidden service; +- focused migration, binding, concurrency, stale-generation, and denial tests; +- related ART specification, plan evidence, and coverage configuration. + +## Not Allowed + +- provider reads, format parsing, extraction, agent invocation, legacy-field + removal, submission/checker/review changes, or AUTH-owned availability edits. + +## Acceptance Criteria + +- an immutable binding references exact project, draft guide, snapshot, item, + verified content, setup run, and monotonic setup generation; +- source-item metadata cannot claim or substitute content identity; +- one current binding exists per item/generation and replacement is explicit; +- missing, unverified, cross-project, cross-guide, wrong-run, or stale-generation + content fails closed; +- concurrent binding produces one business effect; +- test-only fixed authority proves the hidden ART kernel's positive binding + semantics while the composed live AUTH path proves deny-only before 04B; +- fresh prepared binding authority is consumed atomically with the binding when + AUTH later activates it; the hidden real kernel denies before then; +- no provider I/O or human-authority inheritance occurs; +- changed subsystems remain at least 90% covered and repository coverage stays + at least 78%. + +## Verification + +```bash +(cd backend && .venv/bin/python -m ruff check app tests scripts) +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_authorization.py tests/test_guide_artifacts.py -q --cov=app.modules.artifacts --cov=app.modules.projects --cov-report=term-missing --cov-fail-under=90) +python3 scripts/check_stale_artifact_contracts.py +python3 scripts/check_markdown_links.py +python3 scripts/test_agent_gates.py +git diff --check +``` + +The exact PR head must pass hosted `Backend / test` and `Agent Gates / agent-gates`. + +## Required Reviewers + +Senior engineering, architecture, QA/test, security/auth, product/ops, +reuse/dedup, CI integrity, test delta, and docs. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B2-guide-materialization-classification.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B2-guide-materialization-classification.md new file mode 100644 index 00000000..38bcda66 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B2-guide-materialization-classification.md @@ -0,0 +1,71 @@ +# Chunk Contract: WS-ART-001-03B2 - Guide Materialization And Classification + +Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 03B1 + +## Goal + +Materialize exact bound guide bytes through the fixed reader, independently +verify integrity, classify incidents, and detect formats in bounded private +scratch without extracting semantic content. + +## Allowed Files + +- ART guide-read port, canonical resources, materializer, incident records, and + typed format detector/container inspector; +- extension of the existing `ArtifactMaterializationPort` convention and + interface boundary; no parallel materialization abstraction; +- existing `ArtifactStore` and `ArtifactScratchManager` composition only; +- fixed startup limits for signatures, ZIP markers, entries, decompressed bytes, + depth, ratio, image headers/dimensions, and read deadline; +- focused read, corruption, classification, container, cancellation, cleanup, + authorization, and coverage tests; related docs/evidence. + +## Not Allowed + +- semantic extraction, parser dependencies, agent calls, arbitrary temp paths, + direct provider access, generic download, Celery handles/bytes, legacy cutover, + or AUTH availability edits. + +## Acceptance Criteria + +- fresh prepared `artifact.guide_source.read` authority is consumed before any + provider read; the real kernel remains unavailable before AUTH-04B; +- test-only fixed authority proves positive read/materialization semantics while + the composed live AUTH path proves deny-before-I/O until 04B; +- exact binding and setup generation are revalidated before and after full read; +- complete SHA-256 and byte count are recomputed into scratch and compared; +- missing, changed, truncated, unavailable, and stale reads record bounded ART + incidents and never guide insufficiency; +- PDF, OOXML containers, CSV, Markdown, text, JSON, PNG/JPEG/WebP metadata, + ordinary ZIP, ambiguous, and opaque input receive deterministic detection; +- audio/video declared types and signatures deterministically produce the + unsupported outcome and never dispatch a parser; +- DOCX/PPTX/XLSX require bounded internal markers; symlinks, macros, external + relationships, embedded executables, bombs, and malformed containers reject; +- container inspection enforces 2,000 entries, 128 MiB decompressed bytes, + nesting depth 8, compression ratio 100:1, 40 megapixels, and 16,384 pixels on + either image dimension; every exact-boundary case succeeds, every one-over + case returns `limit_exceeded`, and no partial classification survives; +- cleanup occurs on success, denial, mismatch, cancellation, and timeout; +- changed subsystems remain at least 90% covered and repository coverage stays + at least 78%. + +## Verification + +```bash +(cd backend && .venv/bin/python -m ruff check app tests scripts) +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_guide_artifacts.py tests/test_artifact_scratch_manager.py tests/test_guide_format_detection.py -q --cov=app --cov-report=term-missing --cov-fail-under=0) +(cd backend && .venv/bin/coverage report --precision=2 --fail-under=78) +(cd backend && .venv/bin/coverage report --include='app/modules/artifacts/*' --precision=2 --fail-under=90) +python3 scripts/check_stale_artifact_contracts.py +python3 scripts/check_markdown_links.py +python3 scripts/test_agent_gates.py +git diff --check +``` + +The exact PR head must pass hosted `Backend / test` and `Agent Gates / agent-gates`. + +## Required Reviewers + +Senior engineering, architecture, QA/test, security/auth, product/ops, +reuse/dedup, CI integrity, test delta, and docs. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3A-extraction-framework-text.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3A-extraction-framework-text.md new file mode 100644 index 00000000..6a590057 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3A-extraction-framework-text.md @@ -0,0 +1,64 @@ +# Chunk Contract: WS-ART-001-03B3A - Extraction Framework And Text Formats + +Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 03B2 + +## Goal + +Prove the isolated extraction framework, canonical content/usage provenance, +and standard-library text, Markdown, JSON, and CSV extraction. + +## Allowed Files + +- typed extraction capability/registry and explicit composition entries; +- isolated no-network subprocess and versioned extraction policy; +- content-derived extraction and binding/generation usage models, one migration, + repository, and schemas; +- focused isolation, limit, cancellation, cleanup, determinism, provenance, + text/Markdown/JSON/CSV, and coverage tests; related docs/evidence. + +## Not Allowed + +- production parser dependencies, PDF/OOXML/image parsing, plugin discovery, + in-process untrusted parsing, provider writes/access, agent invocation, OCR, + audio/video, legacy cutover, or AUTH availability edits. + +## Acceptance Criteria + +- no-network subprocess enforces input/output, CPU/time/memory, row/cell, + encoding, nesting, cancellation, and cleanup limits; +- the framework rejects parser input above 32 MiB, canonical output above + 4 MiB, execution beyond 60 seconds, or address-space use above 512 MiB; + CSV rejects more than 100,000 rows, 1,000,000 cells, or 32,768 characters in + one cell; every breach deterministically records `limit_exceeded`; +- tests cover each exact boundary and one-over boundary, timeout and memory + termination, cancellation, executor loss, scratch cleanup, and retry through + fresh materialization and fresh authority with no partial durable output; +- extraction revalidates exact 03B2 digest/binding/format/policy provenance; +- deterministic content records are keyed by content, format, extractor/version, + and policy; separate usage records name item, binding, run, and generation; +- text/Markdown/JSON/CSV canonicalization and error statuses are deterministic; +- unsupported raw input never reaches an agent or provider write; +- changed subsystem coverage is at least 90%; repository coverage stays 78%. + +## Verification + +```bash +(cd backend && .venv/bin/python -m ruff check app tests scripts) +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_alembic.py tests/test_guide_extraction.py -q --cov=app --cov-report=term-missing --cov-fail-under=0) +(cd backend && .venv/bin/coverage report --precision=2 --fail-under=78) +(cd backend && .venv/bin/coverage report --include='app/modules/artifacts/*,app/core/config.py,app/interfaces/artifacts.py' --precision=2 --fail-under=90) +python3 scripts/check_stale_artifact_contracts.py +python3 scripts/check_markdown_links.py +python3 scripts/test_agent_gates.py +git diff --check +``` + +The exact PR head must pass hosted `Backend / test` and `Agent Gates / agent-gates`. +Every changed production module, including repository, schema, migration-owned +service, and composition surfaces, must be included in a dedicated retained +90% coverage report or an explicitly reviewed existing subsystem report. + +## Required Reviewers + +Senior engineering, architecture, QA/test, security/auth, product/ops, +reuse/dedup, CI integrity, test delta, and docs. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3B-complex-format-extractors.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3B-complex-format-extractors.md new file mode 100644 index 00000000..79a7de61 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3B-complex-format-extractors.md @@ -0,0 +1,71 @@ +# Chunk Contract: WS-ART-001-03B3B - Complex Format Extractors + +Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 03B3A; dependency approval required + +## Goal + +Add approved PDF, DOCX, PPTX, XLSX, and PNG/JPEG/WebP metadata adapters on the +already-proven isolated framework. + +## Entry Gate + +The human owner explicitly approves an exact pinned dependency allowlist after +review of license, maintenance, advisories, transitive graph, malformed-input +history, and cancellation behavior. The PR includes the package/lock diff, +machine-readable allowlist, dependency-tree evidence, license evidence, and +security-review record. CI fails if a parser dependency is outside that list. + +## Allowed Files + +- one typed adapter per approved format and explicit registry entries; +- approved package/lock/allowlist and deterministic dependency gate; +- bounded fixtures and format-confusion, macro/external relationship, embedded + content, page/slide/sheet/cell/image, parser crash, timeout, and coverage tests; +- related docs/evidence. + +## Not Allowed + +- framework redesign, runtime plugins/fallbacks, OCR, audio/video, ordinary ZIP + semantics, raw pixels/base64 to agents, provider writes/access, legacy cutover, + or AUTH availability edits. + +## Acceptance Criteria + +- exact 03B2 classification provenance is required before parser execution; +- all parsers stay inside the 03B3A isolation/limit/cancellation boundary; +- PDF rejects more than 500 pages; PPTX rejects more than 300 slides; XLSX + rejects more than 100 sheets, 100,000 rows, 1,000,000 cells, or 32,768 + characters in one cell; images reject more than 40 megapixels or 16,384 + pixels on either dimension; every breach records `limit_exceeded`; +- tests cover each exact boundary and one-over boundary, malformed containers, + parser crash, timeout, cancellation, cleanup, and executor loss without a + partial extraction or usage record; +- OOXML markers distinguish formats and reject macros, external relationships, + embedded executables, bombs, and ambiguity; +- PNG/JPEG/WebP output is metadata only and cannot satisfy required text; +- parser dependency allowlist and lock graph are deterministic and CI-enforced; +- changed subsystem coverage is at least 90%; repository coverage stays 78%. + +## Verification + +```bash +(cd backend && .venv/bin/python -m ruff check app tests scripts) +(cd backend && .venv/bin/python scripts/check_guide_extractor_dependencies.py) +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_guide_format_detection.py tests/test_guide_extraction.py -q --cov=app --cov-report=term-missing --cov-fail-under=0) +(cd backend && .venv/bin/coverage report --precision=2 --fail-under=78) +(cd backend && .venv/bin/coverage report --include='app/modules/artifacts/*,app/core/config.py,app/interfaces/artifacts.py' --precision=2 --fail-under=90) +python3 scripts/check_stale_artifact_contracts.py +python3 scripts/check_markdown_links.py +python3 scripts/test_agent_gates.py +git diff --check +``` + +The exact PR head must pass hosted `Backend / test` and `Agent Gates / agent-gates`. +Every changed production module, including adapter registry, dependency gate, +configuration, and composition surfaces, must be included in a dedicated +retained 90% coverage report or an explicitly reviewed existing subsystem report. + +## Required Reviewers + +Senior engineering, architecture, QA/test, security/auth, product/ops, +reuse/dedup, CI integrity, test delta, and docs. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B4-guide-sufficiency-continuation.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B4-guide-sufficiency-continuation.md new file mode 100644 index 00000000..34f8c524 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B4-guide-sufficiency-continuation.md @@ -0,0 +1,76 @@ +# Chunk Contract: WS-ART-001-03B4 - Guide Sufficiency Continuation + +Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 03B3B + +## Goal + +Continue the existing Celery project-setup workflow with complete verified, +same-generation canonical guide material and exact persisted provenance. + +## Allowed Files + +- existing project-setup Celery task/queue and setup-run generation fields; +- project service/repository and agent input schemas consuming typed canonical + extraction records; +- in-place evolution or replacement of existing `GuideSourceMaterial` and + `GuideSourceItemMaterial`; no parallel sufficiency-material model; +- sufficiency-report usage provenance; extraction models/migration remain owned + by 03B3A and complex adapter provenance remains owned by 03B3B; +- focused stale-delivery, completeness, incident, unsupported, broker replay, + agent-input, persistence, cancellation, and coverage tests; related docs. + +## Not Allowed + +- bytes, extracted content, scratch paths/handles, prepared authorization, or + credentials in Celery payloads; direct provider access; raw binary/excerpts + as authoritative input; policy derivation after incomplete extraction; + legacy-field removal; AUTH availability edits. + +## Acceptance Criteria + +- project-setup Celery payload is exactly project, guide, snapshot, setup run, and + setup generation identifiers; +- the project-setup executor reloads and revalidates current + project/guide/snapshot/run/generation, + complete bindings, content, and extraction provenance before agent invocation + and again before report commit; +- all required items must have successful policy-compatible extraction; +- artifact incident, missing/corrupt/stale/unsupported/failed extraction stops + setup with an internal status and creates no insufficiency decision; +- agent input is bounded canonical content only, never raw binaries or caller + excerpts, and reports preserve exact content/extraction provenance; +- source content is explicitly delimited/labeled as untrusted data, supplies no + tools/provider/secret authority, and prompt-injection fixtures cannot override + system/developer policy or the typed sufficiency output contract; +- `setup_blocked` exposes the stable redacted error codes and remediation defined + by D46 for `unsupported`, `ambiguous`, `malformed`, `limit_exceeded`, + `parser_failure`, `cancelled`, and `artifact_incident`, with bounded Operator + incident reference only for ART incidents; +- canonical agent material is capped at 12 MiB; an exact-boundary assembly may + proceed, a one-over assembly records `guide_source_limit_exceeded`, and no + agent call or partial report occurs; +- duplicate or stale Celery execution cannot create or overwrite current output; +- test-only fixed authority proves the complete hidden pipeline while composed + live binding/read remains deny-only until AUTH-04B; +- changed subsystems remain at least 90% covered and repository coverage stays + at least 78%. + +## Verification + +```bash +(cd backend && .venv/bin/python -m ruff check app tests scripts) +(cd backend && WORKSTREAM_TEST_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/workstream_test .venv/bin/pytest tests/test_project_setup.py tests/test_guide_artifacts.py tests/test_guide_extraction.py tests/test_project_agents.py -q --cov=app --cov-report=term-missing --cov-fail-under=0) +(cd backend && .venv/bin/coverage report --precision=2 --fail-under=78) +(cd backend && .venv/bin/coverage report --include='app/modules/projects/*,app/*ers/project_setup.py' --precision=2 --fail-under=90) +python3 scripts/check_stale_artifact_contracts.py +python3 scripts/check_markdown_links.py +python3 scripts/test_agent_gates.py +git diff --check +``` + +The exact PR head must pass hosted `Backend / test` and `Agent Gates / agent-gates`. + +## Required Reviewers + +Senior engineering, architecture, QA/test, security/auth, product/ops, +reuse/dedup, CI integrity, test delta, and docs. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03C-guide-source-cutover-continuation.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03C-guide-source-cutover-continuation.md index 4a8380f6..f7a6970a 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03C-guide-source-cutover-continuation.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03C-guide-source-cutover-continuation.md @@ -1,6 +1,6 @@ # Chunk Contract: WS-ART-001-03C - Guide Source Cutover And Continuation -Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after 03B and exact AUTH activation +Initiative: `WS-ART-001` | Risk: L1 | Status: Proposed after split 03B and AUTH-04B Artifact contract phase: `guide_source_cutover` @@ -32,6 +32,8 @@ adding a Project Manager resume command. - migration refuses unsafe populated legacy rows rather than fabricating bytes; - caller `content_cid`, provider-object schemes, and hash-only identity are removed without aliases after verified replacement is available; +- caller excerpts and durable references no longer act as authoritative agent + content; verified same-generation bindings and canonical extractions do; - recovery continuation references a persisted setup run/generation and resumes only when project, guide, snapshot, policy, and generation still match; - Operator recovery authorization and automatic setup continuation remain diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-external-review-response.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-external-review-response.md new file mode 100644 index 00000000..ea797de3 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-external-review-response.md @@ -0,0 +1,47 @@ +# External Review Response: WS-ART-001-03B Planning Correction + +## Comments Addressed + +- CodeRabbit's reviews produced six valid findings. The contracts now + preserve repository-wide 78% coverage before their scoped 90% reports; name + exact v0.1 extraction limits, enforcement points, stable outcomes, boundary + tests, cleanup, and executor-loss proof; map timeout and memory termination to + non-retryable limit failure and executor loss to one bounded fresh retry; + exhaustively map every extraction status to a redacted public code and + remediation; use consistent + `guide-read` wording; and require a second locked lineage/integrity/provenance + validation immediately before report commit. +- Agent Gates correctly found five uses of retired human `worker` vocabulary in + new planning text. They were replaced with parser runtime, executor, and + project-setup executor wording. The future coverage command now avoids + embedding the retired vocabulary while retaining explicit setup-executor + coverage. +- Backend semantic-lane evidence showed one unrelated existing AUTH PostgreSQL + concurrency failure: expected statuses `[200, 403]`, observed `[500, 200]` in + `test_actor_identity_link_lifecycle_real_postgres_concurrency`. The other + 1,651 tests in that lane passed. This planning PR changes no backend runtime or + tests; the next pushed head receives a fresh hosted rerun. + +## Comments Deferred + +- No backend product-code repair is justified from the isolated AUTH concurrency + failure without reproduction on the new head; it is outside this planning + diff. + +## Human Decisions Needed + +None for the current repair. + +## Commands Rerun + +```text +python3 scripts/check_stale_authorization_docs.py +git diff --check +``` + +Both pass. The full documentation/stale-contract suite is rerun before push. + +## Remaining Risks + +- Hosted Backend and Agent Gates must pass the repaired exact head, and all six + CodeRabbit threads must be verified against that head. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-internal-review-evidence.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-internal-review-evidence.md new file mode 100644 index 00000000..ca9e455d --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-internal-review-evidence.md @@ -0,0 +1,52 @@ +# Internal Review Evidence: WS-ART-001-03B Planning Correction + +## Scope Reviewed + +Planning-only correction for guide binding, materialization, classification, +isolated extraction, same-generation Celery continuation, AUTH-04B sequencing, +and the artifact-storage specification. No runtime code, migration, dependency, +workflow, or authorization availability changed. + +## Initial Findings And Repairs + +- Architecture: separated content-derived extraction from binding/run/generation + usage provenance and kept parser semantics out of 03B2. +- Senior engineering: split extraction into 03B3A framework/text formats and + 03B3B approved complex formats; added hidden positive and live deny-only proof. +- QA/CI: added exact commands, hosted full gates, 90% changed-subsystem coverage, + and deterministic parser-dependency evidence. +- Product/ops: defined image metadata-only behavior, stable `setup_blocked` + codes, Operator incident visibility, and PM remediation. +- Security: delimited extracted content as untrusted agent data, added prompt- + injection proof, and bound parsing to 03B2 provenance. +- Reuse/test/docs: required existing ports/scratch/Celery/agent schemas, added + audio/video negatives, and corrected count, dependency, and sequencing drift. + +## Final Results + +| Track | Result | Remaining condition/risk | +|---|---|---| +| architecture | PASS WITH LOW RISKS | AUTH wording aligned; obsolete 03B3 umbrella omitted. | +| senior engineering | PASS WITH LOW RISKS | Image wording aligned. | +| QA/test | PASS WITH LOW RISKS | Concrete local database URL added. | +| security/auth | PASS | None. | +| product/ops | PASS | None. | +| CI integrity | PASS WITH CONDITIONS | Commit new contracts; future 03B3B implements its named dependency gate. | +| docs | PASS WITH LOW RISKS | Count/dependency/ownership wording corrected. | +| reuse/dedup | PASS WITH LOW RISKS | Existing ports and agent schemas reused. | +| test delta | PASS WITH CONDITIONS | Commit contracts; coverage/audio-video proof recorded. | + +All conditions applicable to this planning PR are addressed by including the +new contracts. The 03B3B dependency gate is a future acceptance criterion, not +an implementation claim of this planning change. + +## Deterministic Evidence + +```text +git diff --check +python3 scripts/check_markdown_links.py +python3 scripts/check_stale_artifact_contracts.py +python3 scripts/check_stale_workstream_wording.py +``` + +All passed. The PR changes no executable code, tests, packages, or CI surface. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-pr-trust-bundle.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-pr-trust-bundle.md new file mode 100644 index 00000000..33493f4e --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B-PLAN-pr-trust-bundle.md @@ -0,0 +1,86 @@ +# PR Trust Bundle: WS-ART-001-03B Planning Correction + +## Chunk + +`WS-ART-001-03B-PLAN` — planning-only correction before implementation. + +## Goal + +Define the safe path from verified Project Manager guide-source bytes to exact +same-generation sufficiency input without confusing it with contributor ZIPs. + +## Human-Approved Intent + +- contributors always submit one outer ZIP; +- Project Managers upload guide items in supported document/data/image formats; +- v0.1 excludes audio and video; +- upload stores opaque bytes and performs no synchronous parsing; +- AUTH later activates only fixed-service guide binding and read. + +## What Changed And Why + +The old 03B contract combined too many boundaries. It is now five executable +PRs: `03B1 -> 03B2 -> 03B3A -> 03B3B -> 03B4 -> AUTH-04B -> 03C`. + +## Design Chosen + +Original bytes remain authoritative in `ArtifactStore`. Existing materialization +and scratch capabilities rehash every full read. Content-derived extraction is +separate from binding/run/generation usage. Canonical extraction is bounded +PostgreSQL processing evidence, not an unapproved provider write. Agent input +is typed, bounded, provenance-linked, and explicitly untrusted. + +## Alternatives Rejected + +Parsing in upload, forcing guides into contributor ZIPs, direct provider access, +a second scratch manager, one large parser PR, image OCR by implication, raw +binary/excerpt agent input, and provider writes under guide-read authority. + +## Scope Control And Product Behavior + +Planning/specification only. Guide items support PDF, DOCX, PPTX, CSV, XLSX, +Markdown, text, JSON, and PNG/JPEG/WebP metadata. Audio/video and ordinary ZIP +guide semantics are unsupported. Artifact/extraction failures create redacted +`setup_blocked` outcomes, not guide-insufficiency decisions. + +## Acceptance Proof, Test Delta, And CI Integrity + +- diff, Markdown links, stale artifact contracts, and stale wording: PASS; +- no executable CI/test/package delta; +- every future contract names focused tests, Ruff, 90% changed-subsystem + coverage, the 78% repository gate, Agent Gates, and hosted Backend checks; +- 03B3B requires a human-approved, CI-enforced parser dependency allowlist. + +## Reviewer Results + +All nine tracks passed after repairs. Details are in +`WS-ART-001-03B-PLAN-internal-review-evidence.md`. + +## External Review + +CodeRabbit's reviews produced six valid contract findings, now repaired: +explicit 78% plus scoped 90% coverage gates, concrete v0.1 extraction limits +and boundary proofs, exhaustive stable failure mappings including timeout, +memory-termination, and executor-loss recovery semantics, consistent guide-read +wording, and locked pre-report-commit lineage/integrity/provenance revalidation. +Agent Gates separately prompted repair of five retired-vocabulary occurrences. +The first Backend run had one unrelated existing AUTH concurrency failure after +1,651 passes. Disposition is recorded in +`WS-ART-001-03B-PLAN-external-review-response.md`; repaired-head hosted checks +and thread verification remain required. + +## Remaining Risks And Follow-Up + +- 03B3B cannot start until exact pinned parser dependencies are approved. +- Images provide metadata only; image text requires a future OCR capability. +- Immediate post-merge implementation successor is 03B1. + +## Human Review Focus + +Confirm the upload separation, five-PR split, AUTH-04B placement, PostgreSQL +canonical extraction records, v0.1 format boundary, and dependency approval gate. + +## Human Merge Ownership + +A maintainer must explicitly approve this planning PR. Merge starts no +implementation automatically. diff --git a/.agent-loop/initiatives/WS-XINT-002-art-auth-end-to-end/chunks/WS-XINT-002-04B-guide-read-binding-activation.md b/.agent-loop/initiatives/WS-XINT-002-art-auth-end-to-end/chunks/WS-XINT-002-04B-guide-read-binding-activation.md index 65dd308d..3fa4ad69 100644 --- a/.agent-loop/initiatives/WS-XINT-002-art-auth-end-to-end/chunks/WS-XINT-002-04B-guide-read-binding-activation.md +++ b/.agent-loop/initiatives/WS-XINT-002-art-auth-end-to-end/chunks/WS-XINT-002-04B-guide-read-binding-activation.md @@ -2,8 +2,8 @@ ## Goal -Activate fixed-service guide binding and guide read only after merged ART-03B -evidence, without weakening the ART-03C clean cut. +Activate fixed-service guide binding and guide-read actions only after merged ART-03B1, +03B2, 03B3A, 03B3B, and 03B4 evidence, without weakening ART-03C. ## Risk class @@ -12,8 +12,8 @@ L1. ## Entry gate - WS-XINT-002-04A is merged. -- ART-03B hidden binding/materialization behavior and exact resource manifest - are merged and reviewed. +- The complete split-03B hidden behavior and exact resource manifest are merged + and reviewed. ## Allowed files diff --git a/docs/spec_artifact_storage_service.md b/docs/spec_artifact_storage_service.md index 240f52fe..082becd2 100644 --- a/docs/spec_artifact_storage_service.md +++ b/docs/spec_artifact_storage_service.md @@ -1212,6 +1212,47 @@ Guide-source ingestion stores each source item as immutable artifact content. Sufficiency and policy-derivation agents read through an authorized Workstream artifact reader, never direct provider URLs. +The Project Manager guide-source boundary is not the contributor submission +boundary. Contributor submissions always use one outer ZIP; guide snapshots may +contain multiple independently uploaded items in PDF, DOCX, PPTX, CSV, XLSX, +Markdown, plain text, JSON, or PNG/JPEG/WebP. Images yield bounded structural +metadata only and cannot satisfy required textual semantics without future +approved OCR. v0.1 does not support guide audio or video. DOCX, PPTX, and XLSX are safely distinguished by +bounded internal markers; an arbitrary ZIP is not treated as one of them. + +Upload stores opaque bytes and performs no parsing in the HTTP request. The +existing Celery setup pipeline later carries only project, guide, source +snapshot, setup run, and setup generation identifiers. A fixed guide reader +obtains fresh prepared authority, resolves the exact verified binding, streams +through `ArtifactStore` into `ArtifactScratchManager`, and recomputes complete +SHA-256 and byte count. Missing, changed, truncated, or unavailable content is +an artifact incident and never a guide-insufficiency result. + +Typed extractors run in a bounded no-network isolation boundary. One immutable +content-derived record identifies original content, format, extractor/version, +policy version, output digest, omission facts, status, and error code; a +separate immutable usage record binds it to the exact binding, item, setup run, +and generation. v0.1 persists these bounded records in PostgreSQL and does not +use guide-read authority to create a provider object. Only successful current- +generation extraction reaches the agent, delimited as untrusted source data +with no tool, secret, provider, or instruction authority. Unsupported, +ambiguous, malformed, stale, or failed required material stops setup internally. + +Immediately before committing a sufficiency report, the setup transaction must +lock, reload, and revalidate the exact project, draft guide, source snapshot, +setup run and generation, guide-source binding, observed content-integrity +facts, and extraction-usage provenance used for the agent input. Any drift, +replacement, integrity mismatch, or incomplete provenance rolls back the report +commit and produces the applicable bounded internal failure; an earlier +pre-agent validation is not sufficient authority for this durable mutation. + +The public run remains `setup_blocked` with the redacted stable code defined for +the exact extraction outcome in D46. Recoverable ART +incidents wait for recovery and expose only a bounded incident reference to an +authorized Operator. Terminal corruption or source-content failure requires a +new corrected Project Manager source item/snapshot; no outcome is a sufficiency +decision. + When guide-source verification fails transiently, artifact recovery success automatically re-publishes the same persisted `ProjectSetupRun` continuation only when project, guide version, source snapshot ID/hash, and setup generation