WS-ART-001: correct guide materialization and extraction plan

.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/AUTH_HANDOFF.md

@@ -17,10 +17,14 @@ activation custody, and availability.
    exact resource/guard/surface manifest.
 3. AUTH activates only that exact action through a separately reviewed AUTH
    contract after consuming ART evidence.
-4. ART-03B implements hidden `artifact.guide_source.read` and
-   `artifact.guide_source.binding.create` behavior; binding maps to fixed
-   permission `artifact.binding.create`.
-5. AUTH activates only those exact actions after consuming 03B evidence.
+4. ART-03B1, 03B2, 03B3A, 03B3B, and 03B4 implement hidden authoritative binding, exact setup
+   generation, verified materialization, bounded extraction, and Celery
+   sufficiency continuation. `artifact.guide_source.binding.create` maps to
+   fixed permission `artifact.binding.create`; `artifact.guide_source.read`
+   remains separate.
+5. AUTH-04B activates only those exact fixed-service actions after consuming
+   the complete split-03B evidence. Extraction does not imply a provider-write
+   permission or inherit Project Manager authority.
 6. ART-03C performs the legacy clean cut. No ART chunk writes availability.
 
 ## Submission Bundle Sequence

.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/CHUNK_MAP.md

@@ -17,9 +17,13 @@ Each chunk is one PR. No later chunk starts automatically.
 | `WS-ART-001-02D` | Add hidden Operator content/job/retry/recovery/audit APIs, canonical resource composition, and production-readiness checks while actions and provider profiles remain inactive. | L1 | Merged through PR #177 as `93c14181` |
 | `WS-ART-001-03` | Original combined guide-source cutover. | L1 | Cancelled before implementation; no runtime changes |
 | `WS-ART-001-PLAN2` | Reconcile guide and one-ZIP submission planning with bounded scratch, existing immutable admission/recovery, exact AUTH sequencing, and downstream ownership. | L1 | Planning-only successor proposed after cancellation |
-| `WS-ART-001-03A` | Add hidden guide-source byte ingest through existing preparation, admission, verification, and publication. | L1 | Proposed after PLAN2 |
-| `WS-ART-001-03B` | Bind verified guide-source content and provide authorized integrity-checking setup materialization. | L1 | Proposed after 03A and exact AUTH activation |
-| `WS-ART-001-03C` | Remove legacy guide-source identity and add exact same-generation setup continuation. | L1 | Proposed after 03B and exact AUTH activation |
+| `WS-ART-001-03A` | Add hidden guide-source byte ingest through existing preparation, admission, verification, and publication. | L1 | Merged through PR #215 as `bb9082a7` |
+| `WS-ART-001-03B1` | Add authoritative verified guide-source bindings and an exact setup-generation fence while fixed-service binding remains unavailable. | L1 | Proposed after 03A and AUTH-04A |
+| `WS-ART-001-03B2` | Add fixed-reader verified materialization, artifact incidents, and canonical format detection in bounded ART scratch. | L1 | Proposed after 03B1 |
+| `WS-ART-001-03B3A` | Add the isolated extraction framework, canonical records, and text/Markdown/JSON/CSV extractors. | L1 | Proposed after 03B2 |
+| `WS-ART-001-03B3B` | Add explicitly approved PDF/OOXML/image-metadata extractors on the proven framework. | L1 | Proposed after 03B3A and dependency approval |
+| `WS-ART-001-03B4` | Feed only complete same-generation canonical extracted material into the existing Celery sufficiency pipeline. | L1 | Proposed after 03B3B |
+| `WS-ART-001-03C` | Remove legacy guide-source identity and add exact same-generation setup continuation. | L1 | Proposed after 03B1-03B4 and AUTH-04B |
 | `WS-ART-001-04A` | Accept one outer ZIP in bounded scratch, safely inspect its tree, normalize executable intent, produce canonical identities, and reject unchanged work before provider I/O. | L1 | Proposed after 03C and AUTH planned action registration |
 | `WS-ART-001-04B` | Run mandatory platform and locked Project Guide pre-submit checks against the same scratch-bound tree and executable semantics without durable storage. | L1 | Proposed after 04A |
 | `WS-ART-001-04C` | Reauthorize at durable intent, admit/verify the passing ZIP once, and publish one capacity-charged `ready` admission that may remain unbound. | L1 | Proposed after 04B; AUTH activation follows hidden completion |
@@ -43,7 +47,11 @@ OBJECT-STORAGE-AMENDMENT
 -> PLAN2 planning reconciliation
 -> 03A guide-source byte ingest
 -> AUTH activation for exact 03A actions
--> 03B guide-source binding/materialization
+-> 03B1 authoritative guide binding/setup generation
+-> 03B2 verified materialization/format detection
+-> 03B3A isolated extraction framework/text formats
+-> 03B3B approved complex document/image-metadata extractors
+-> 03B4 same-generation Celery sufficiency integration
 -> AUTH activation for exact 03B actions
 -> 03C guide-source clean cut/continuation
 -> AUTH planned registration of `artifact.submission_bundle.prepare`

.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DECISIONS.md

@@ -362,3 +362,117 @@ Contributor authority never implies fixed service authority. Checker
 materialization, verification execution, pending-work scanning, ambiguous-put
 resolution, and binding retain their distinct fixed actions. No compatibility
 alias exposes the obsolete multi-step session actions.
+
+## D39 - Guide Upload And Understanding Are Separate
+
+Guide upload stores and verifies opaque original bytes. HTTP ingest does not
+parse, extract, render, OCR, transcode, or wait for submission prechecks.
+Understanding begins asynchronously only after an exact verified binding.
+Unlike contributor submissions, guide-source items are not wrapped in one
+mandatory outer ZIP. Each item retains its own verified format identity.
+
+## D40 - Exact Setup Generation Is A Durable Fence
+
+Every binding, extraction, agent input, and report identifies one project,
+draft guide, source snapshot, setup run, and monotonic setup generation. Celery
+carries only those identifiers and generation. Bytes, extracted text, scratch
+handles, prepared authority, and provider credentials never enter Redis.
+
+## D41 - Canonical Guide Materialization
+
+The fixed guide-reader obtains fresh prepared authority, resolves an exact
+verified binding, and streams through `ArtifactStore` into the existing
+`ArtifactScratchManager`. Every full read recomputes SHA-256 and byte count.
+Missing, changed, or truncated content is an ART incident, not guide
+insufficiency. Project services and agents never access providers directly.
+
+## D42 - Typed, Bounded Guide Extraction
+
+One detector and typed extractor registry validates signatures and bounded
+container markers. Initial text semantics cover PDF, DOCX, PPTX, CSV, XLSX,
+Markdown, plain text, and JSON. PNG, JPEG, and WebP yield bounded structural
+metadata only. Without OCR, image pixels cannot satisfy required textual guide
+semantics. Audio/video and ambiguous binaries are unsupported in v0.1 and never
+sent raw to agents. OOXML formats are distinguished from ordinary ZIPs.
+
+Extraction runs asynchronously in a strongly isolated no-network subprocess
+under fixed input, output, container, time, memory, and document limits using
+only scratch-manager paths. Parser crash, malformed input, macros, external
+relationships, embedded executables, cancellation, timeout, and executor loss
+have bounded outcomes and cleanup. Production parser dependencies require
+explicit human approval before implementation.
+
+The v0.1 extraction-policy limits are fixed, not caller-selectable:
+
+| Limit | Value | Enforcement owner | Breach outcome |
+|---|---:|---|---|
+| parser input per item | 32 MiB | 03B3A framework before dispatch | `limit_exceeded` |
+| canonical output per item | 4 MiB | 03B3A streaming output collector | `limit_exceeded` |
+| aggregate agent material | 12 MiB | 03B4 input assembler | `limit_exceeded` |
+| subprocess wall time per item | 60 seconds | 03B3A process supervisor | `limit_exceeded` |
+| subprocess address space | 512 MiB | 03B3A OS isolation boundary | `limit_exceeded` |
+| container entries | 2,000 | 03B2 container inspector | `limit_exceeded` |
+| decompressed container bytes | 128 MiB | 03B2 container inspector | `limit_exceeded` |
+| container nesting depth | 8 | 03B2 container inspector | `limit_exceeded` |
+| compression ratio | 100:1 | 03B2 container inspector | `limit_exceeded` |
+| PDF pages | 500 | 03B3B PDF adapter | `limit_exceeded` |
+| PPTX slides | 300 | 03B3B PPTX adapter | `limit_exceeded` |
+| XLSX sheets | 100 | 03B3B XLSX adapter | `limit_exceeded` |
+| table rows per item | 100,000 | 03B3A CSV / 03B3B XLSX adapter | `limit_exceeded` |
+| table cells per item | 1,000,000 | 03B3A CSV / 03B3B XLSX adapter | `limit_exceeded` |
+| characters per cell | 32,768 | CSV/XLSX adapter | `limit_exceeded` |
+| image pixels | 40 megapixels | 03B2 header inspector and 03B3B adapter | `limit_exceeded` |
+| image width or height | 16,384 pixels | 03B2 header inspector and 03B3B adapter | `limit_exceeded` |
+
+Exact-boundary, one-over-boundary, cleanup, cancellation, timeout, memory-
+termination, and executor-loss tests are mandatory. Executor loss leaves no
+successful extraction usage record; a current-generation retry starts from
+fresh materialization and authority.
+
+## D43 - Canonical Extraction Records, Not Implicit Provider Writes
+
+v0.1 persists an immutable content-derived extraction representation in
+PostgreSQL keyed by original content, format, extractor/version, and policy
+version, with output digest, omission facts, status, and bounded error code. A
+separate immutable usage record binds it to the exact guide binding, source
+item, setup run, and generation. AUTH-04B grants read and binding only, so ART
+does not use read authority to create a provider object.
+
+## D44 - Sufficiency Consumes Complete Verified Material
+
+The agent receives only bounded extraction records for all required items in
+the current generation. Missing, corrupt, stale, ambiguous, unsupported, or
+failed content stops policy derivation without creating a guide-insufficiency
+decision. Reports preserve exact content and extraction provenance. Legacy
+excerpts and durable refs are not authoritative after ART-03C.
+
+## D45 - Extracted Guide Content Is Untrusted Agent Data
+
+Canonical extraction proves byte provenance, not instruction authority. Agent
+assembly labels and delimits source material as untrusted data, excludes tools,
+provider access, credentials, and hidden instructions from that material, and
+accepts only the typed sufficiency output contract. Prompt-injection text inside
+a guide cannot alter system/developer policy or authorize an action.
+
+## D46 - Setup Failures Have Stable Operational Outcomes
+
+Existing `ProjectSetupRun.status=setup_blocked` uses this exhaustive stable,
+redacted mapping: `unsupported` -> `guide_source_format_unsupported`;
+`ambiguous` -> `guide_source_format_ambiguous`; `malformed` ->
+`guide_source_malformed`; `limit_exceeded` -> `guide_source_limit_exceeded`;
+`parser_failure` -> `guide_source_extraction_failed`; `cancelled` ->
+`guide_source_extraction_cancelled`; and `artifact_incident` ->
+`guide_artifact_incident`. An observed 60-second timeout or 512-MiB
+address-space termination is a deterministic `limit_exceeded` outcome: it does
+not retry automatically, and the Project Manager must provide a smaller or
+simpler item in a new snapshot. Limits are never raised inline. Executor loss
+before a classified limit breach is `parser_failure`: it receives one bounded
+current-generation automatic retry from fresh materialization and fresh
+authority; repeated loss remains `guide_source_extraction_failed`, creates no
+successful usage/report, and requires a corrected/new snapshot while an
+authorized Operator may inspect only bounded redacted runtime diagnostics.
+Recoverable artifact incidents wait for ART recovery and expose a bounded
+incident reference to authorized Operators. A current-generation transient
+cancellation may retry; stale-generation cancellation commits no report. All
+other terminal source-format or content failures require a corrected item in a
+new snapshot. None creates a sufficiency decision.

.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/DISCOVERY.md

@@ -129,3 +129,33 @@ removes direct provider schemes from guide-source identity. Chunk 05 removes
 the remaining caller storage transport when submissions move to sealed
 artifact-set bindings. No compatibility alias remains after either owning
 cutover.
+
+## ART-03B Materialization And Extraction Discovery
+
+Merged ART-03A and AUTH `WS-XINT-002-04A` provide the covered Project Manager
+upload boundary. AUTH preparation happens before request-body intake; final
+transaction-bound consumption locks the guide lineage and server-computed byte
+facts before durable intent or provider I/O. Upload stores opaque immutable
+bytes and does not parse them.
+
+The current setup pipeline cannot yet consume those bytes safely:
+
+- its Celery payload has four identifiers but no setup generation;
+- `ProjectSetupRun` has no explicit generation fence;
+- `GuideSourceMaterial` uses caller-originated metadata and optional excerpts,
+  not verified ART bindings;
+- no authorized integrity-checking guide read exists in project services;
+- no typed detector, extractor registry, extraction policy, or canonical
+  extraction record exists.
+
+AUTH's reviewed `WS-XINT-002-04B` activates only fixed-service guide binding
+and read. It does not grant a derived-artifact write. v0.1 therefore persists a
+bounded canonical extraction record and provenance in PostgreSQL; a future
+derived provider artifact requires a separately planned action and activation.
+
+No document-parsing production dependencies are installed. Any new production
+dependency requires explicit human approval before 03B3B implementation. OOXML ZIP
+containers must pass one bounded classifier that distinguishes DOCX, PPTX, and
+XLSX by internal markers. Audio/video transcription and OCR are not required by
+current v0.1 setup and remain unsupported. PNG/JPEG/WebP classification and
+metadata do not imply OCR or textual sufficiency.

.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/INTENT.md

@@ -65,6 +65,14 @@ directories, or any other content allowed by the locked Project Guide. A ZIP
 entry inside the outer archive remains an ordinary file. Nested archive
 unpacking is outside v0.1 even when a guide would otherwise request it.
 
+This contributor ZIP rule does not apply to Project Manager guide-source
+uploads. A guide snapshot may contain multiple independently uploaded source
+items in PDF, DOCX, PPTX, CSV, XLSX, Markdown, plain text, JSON, PNG, JPEG, or
+WebP form. Images are metadata-only without OCR and cannot satisfy required
+textual semantics. v0.1 does not support guide audio or video. DOCX, PPTX, and
+XLSX are recognized as their document types despite using ZIP containers
+internally; an arbitrary ordinary ZIP is not treated as a guide document.
+
 ## First-Principle Constraints
 
 - Workstream computes and verifies canonical SHA-256 and byte count.

.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/PLAN.md

@@ -471,6 +471,92 @@ exact resource/content/job/attempt filters.
 
 ## Product Cutover
 
+### Guide Materialization And Extraction Boundary
+
+ART-03A upload remains a byte-custody request: authorize before intake, compute
+the server digest and size, admit once, store opaque bytes, and verify them. It
+does not parse content or wait for contributor submission checkers.
+
+The expanded ART-03B work is delivered as five hidden PR-sized subchunks before
+AUTH-04B activates fixed-service binding/read:
+
+1. `03B1` creates an authoritative one-item/one-content guide binding tied to
+   the exact setup run and explicit setup generation. Item metadata cannot
+   assert content identity, and replaced or cross-lineage facts fail closed.
+2. `03B2` performs fresh authorized full reads through `ArtifactStore`, writes
+   only through `ArtifactScratchManager`, recomputes digest/size, records ART
+   incidents, and classifies formats using signatures and bounded container
+   markers.
+3. `03B3A` installs the isolated no-network extraction framework, immutable
+   content/usage provenance, and text, Markdown, JSON, and CSV extractors
+   without new parser dependencies.
+4. `03B3B` adds explicitly approved PDF, DOCX, PPTX, XLSX, and PNG/JPEG/WebP
+   metadata extractors on that proven framework.
+5. `03B4` adds `setup_generation` to the identifier-only Celery payload,
+   reloads current state, assembles all required canonical extracted sources,
+   invokes sufficiency, and persists exact provenance.
+
+The canonical extraction status set is:
+
+```text
+extracted | unsupported | ambiguous | malformed | limit_exceeded |
+parser_failure | cancelled | artifact_incident
+```
+
+Only `extracted` records enter agent input. Truncation or omission is explicit
+and may be allowed only by the versioned extraction policy; required missing
+semantics stop setup internally. Artifact incidents never become sufficiency
+findings.
+
+Stable setup mappings are exact:
+
+| Extraction status | Redacted setup code | Recovery/remediation |
+|---|---|---|
+| `extracted` | none | Continue only when every required item is complete. |
+| `unsupported` | `guide_source_format_unsupported` | Project Manager supplies a supported item in a new snapshot. |
+| `ambiguous` | `guide_source_format_ambiguous` | Project Manager supplies an unambiguous item in a new snapshot. |
+| `malformed` | `guide_source_malformed` | Project Manager supplies a corrected item in a new snapshot. |
+| `limit_exceeded` | `guide_source_limit_exceeded` | Project Manager supplies a smaller/simpler item; limits are not raised inline. |
+| `parser_failure` | `guide_source_extraction_failed` | Bounded automatic retry; after exhaustion, PM replaces the item and Operator may inspect redacted diagnostics. |
+| `cancelled` | `guide_source_extraction_cancelled` | Current-generation transient cancellation may retry; stale-generation cancellation commits no report. |
+| `artifact_incident` | `guide_artifact_incident` | Recoverable custody failure waits for ART recovery; terminal corruption requires a new item/snapshot. |
+
+The configured 60-second timeout and 512-MiB address-space termination are
+`limit_exceeded`, are not automatically retried, and require a smaller/simpler
+item in a new snapshot. Executor loss before a classified limit breach is
+`parser_failure`: it receives one bounded current-generation retry from fresh
+materialization and fresh authority; repeated loss exposes only
+`guide_source_extraction_failed` plus bounded redacted Operator diagnostics and
+requires a corrected/new snapshot. No failed attempt creates a successful usage
+record or report.
+
+Guide-source items are not required to be ZIP files. The initial semantic
+extractors are PDF, DOCX, PPTX, CSV, XLSX, Markdown, plain text, JSON, and
+PNG, JPEG, and WebP metadata. Images are accepted and classified, but without
+OCR their pixels cannot satisfy required textual guide semantics. Audio/video
+and opaque/ordinary ZIP input are unsupported in v0.1. ZIP-
+container classification rejects macros, external relationships, embedded
+executables, ambiguous markers, excessive entries, decompressed bytes, depth,
+or compression ratio before a document parser runs.
+
+v0.1 stores bounded canonical extraction JSON/text in PostgreSQL rather than
+creating an unapproved provider write under guide-read authority. It records
+source digest/size, detected format, extractor and version, extraction-policy
+version, output digest, truncation/omission facts, status, and bounded error
+code. A separate immutable usage record binds that result to the exact binding,
+source item, setup run, and generation. Original bytes remain authoritative. A
+later derived-artifact store path requires a distinct AUTH action.
+
+The existing `ProjectSetupRun.status=setup_blocked` remains the public state for
+every terminal current-generation failure above. API, Operator, and Project
+Manager projections use only the exact redacted mapping and remediation table.
+None is a guide-insufficiency decision.
+
+Any production parser dependency in 03B3B requires explicit human approval. Candidate
+libraries must undergo current security, maintenance, license, transitive-
+dependency, malformed-input, and cancellation review before `03B3B` may add
+them; no runtime plugin discovery or parser fallback is permitted.
+
 1. Guide-source delivery is split into hidden byte ingest, AUTH activation,
    verified snapshot binding/materialization, AUTH activation, and the legacy
    identity/continuation clean cut.