diff --git a/.github/AGENTS.md b/.github/AGENTS.md index 1f93151..d72bb97 100644 --- a/.github/AGENTS.md +++ b/.github/AGENTS.md @@ -53,8 +53,8 @@ contextdb, context-graph and Vigil. - Linux qualification is required: `source`, `lint`, `docs`, `test`, `features`, complete `controls` and the stable `PR checks` aggregate remain fail-closed. - macOS runs the same lint, default, feature and complete controls commands in - `macos-advisory.yml`, with all five physical jobs and six logical receipts. + macOS runs the same lint, default, feature, compatibility build and complete + controls jobs in `macos-advisory.yml`, ten physical jobs and six logical receipts. Its aggregate reports failures honestly and never blocks Linux or delivery. Preserve model proofs, test-estate accounting, documentation validation and every feature shape. See [macOS follow-up](CI.md#macos-qualification-follow-up). diff --git a/.github/CI.md b/.github/CI.md index 48828db..09b110b 100644 --- a/.github/CI.md +++ b/.github/CI.md @@ -22,13 +22,22 @@ contributor runs the same commands locally. | Path | Workflow | Trigger | What a result means | |---|---|---|---| -| PR checks | [`ci.yml`](workflows/ci.yml) (`ci`) | every push and every pull request; manual dispatch with exact `cg_sha` and `contextdb_sha` | Correctness of this exact commit. The aggregate job **`PR checks`** passes only if every required Linux, source and documentation job succeeded. | -| macOS advisory | [`macos-advisory.yml`](workflows/macos-advisory.yml) (`macOS advisory`) | same push, pull request and exact-SHA dispatch triggers | Full macOS coverage with a failing aggregate on failure; nonblocking for Linux and delivery. | +| PR checks | [`ci.yml`](workflows/ci.yml) (`Linux CI`) | pushes to `dev` and every pull request; manual dispatch with exact `cg_sha` and `contextdb_sha` | Correctness of this exact commit. The aggregate job **`PR checks`** passes only if every required Linux, source and documentation job succeeded. | +| macOS advisory | [`macos-advisory.yml`](workflows/macos-advisory.yml) (`macOS CI (advisory)`) | pushes to `dev`, every pull request and exact-SHA dispatch | Full macOS coverage with a failing aggregate on failure; nonblocking for Linux and delivery. | | Extended integration | [`extended-integration.yml`](workflows/extended-integration.yml) (`extended integration`) | callable; Mondays 03:23 UTC; manual dispatch with exact SHAs | Public Nomic model and event-search discovery on Linux and macOS (aggregate `extended-required`). A manual dispatch with `trusted: true` additionally runs the protected trusted jobs (aggregate `trusted-qualification`). | | Performance measurement | [`performance.yml`](workflows/performance.yml) (`Performance measurement (manual guide)`) | Never — dormant placeholder | **Manual today.** The workflow cannot start: its only trigger is a push filter that ignores every branch and every tag, and manual dispatch is commented out. It checks out nothing, runs no benchmark and never reports a status. The manual journey is [`docs/performance/README.md`](../docs/performance/README.md). | | Development downloads | [`build.yml`](workflows/build.yml) (`Build development binaries`) | called after PR checks for each push to dev; maintainer dispatch on dev | Packages and smokes exact-source binaries, then publishes the rolling development prerelease. | | Publication preparation | [`publication.yml`](workflows/publication.yml) (`publication preparation`) | manual dispatch with exact SHAs | Locked optimized CLI build and local package inspection (aggregate `publication-receipt`). It publishes nothing, creates no tag and performs no crates.io action. | +Automatic pushes run only on `dev`, preserving development publication. Candidate +branches receive both platform workflows through pull requests or exact-SHA manual +dispatch, without duplicate push runs. Display-name changes apply to future runs; +existing runs retain their recorded names. The sync compatibility job was renamed on +both platforms from `Feature compatibility — sync, CLI and doctests` to +`Feature compatibility — sync and CLI` (`ci.yml` and `macos-advisory.yml`) because its +feature doctests moved to the event-search acceptance job. `PR checks` and +`macOS advisory results` are unchanged. + ## PR checks (`ci.yml`) Every job first validates the requested ContextDB SHA, checks out the selected @@ -43,9 +52,9 @@ runs are cancelled; push and dispatch runs are not. | `lint` | `ubuntu-latest` | `bash scripts/gate.sh lint` | `cargo deny`, the expanded-library and dependency-tree scans that keep EmbeddingGemma out of the default library, `cargo fmt --check` and `clippy -D warnings` for the workspace. | | `docs` | `ubuntu-latest` | `bash scripts/gate.sh docs` | Every behaviour sentence in the published pages names a live test ([`validate-doc-coverage.py`](../scripts/validate-doc-coverage.py)), and the validator still rejects a dead reference and an unbound claim ([`check-doc-validator-negative-control.sh`](../scripts/check-doc-validator-negative-control.sh)). | | `test` | `ubuntu-latest` | `bash scripts/gate.sh public-model`, then `bash scripts/gate.sh default` | Offline real Nomic model proofs, `cargo test --workspace --no-run`, explicit doctests, three disjoint nextest bundles, the deterministic test-estate audit and the combined JUnit duration report. | -| `features` | `ubuntu-latest` | `python3 scripts/ci-test-shards.py bundle features --tests-only` | The complete `event-search-test-support` runtime suite, with all model proofs and disjoint receipt accounting. | +| `features` | `ubuntu-latest` | `python3 scripts/ci-test-shards.py bundle features --tests-only` | The complete `event-search-test-support` runtime suite and, in the acceptance bundle, the complete feature doctests, with all model proofs and disjoint receipt accounting. | | `Feature compatibility — default and minimal libraries (ubuntu-latest)` | `ubuntu-latest` | `bash scripts/gate.sh feature-qualification-core` | Locked library builds with default features and no default features. | -| `Feature compatibility — sync, CLI and doctests (ubuntu-latest)` | `ubuntu-latest` | `bash scripts/gate.sh feature-qualification-sync` | Locked `fabric-sync` library and CLI builds, followed by the complete feature doctests. | +| `Feature compatibility — sync and CLI (ubuntu-latest)` | `ubuntu-latest` | `bash scripts/gate.sh feature-qualification-sync` | Locked `fabric-sync` library and CLI builds. | | `CI controls` | `ubuntu-latest` | `python3 scripts/tests/test_ci_qualification.py` | Complete CI qualification and release safety controls, once per platform. | | **`PR checks`** | `ubuntu-latest` | — | Stable aggregate. Fails unless exactly `source`, `lint`, `docs`, `test`, `features`, `controls`, `feature-qualification-core` and `feature-qualification-sync` report `success`; a failed, cancelled, skipped, missing or unexpected job fails it. The complete Linux default and feature inventories must also verify before this check succeeds. | @@ -71,15 +80,21 @@ cases, and verifies exactly every nonignored test ran once per platform and feat variant. Ignored cases remain excluded; the existing ignored measurement audit runs explicitly against each merged default JUnit only after the complete union verifies. Default compile-only examples and default doctests run in the integration -bundle. Feature compatibility builds and feature doctests run in the two independent required jobs. Default acceptance compiles only `context-graph --test acceptance`; the other +bundle. Feature compatibility builds run in the two independent required jobs; feature doctests run in the event-search support acceptance bundle. Default acceptance compiles only `context-graph --test acceptance`; the other bundles still compile the workspace. Feature builds retain workspace-wide feature resolution, so filtering alone does not reduce their compilation closure. CI alone sets development and test optimization to level 1 and debug info to 0, with debug assertions and overflow checks enabled. Local Cargo profile defaults stay as specified in `Cargo.toml`. Linux jobs reclaim preinstalled SDKs with sudo and print -root disk headroom before and after linking. Disposable `target/tests` trybuild output -is removed before compiler cache post hooks, including on failure. +root disk headroom before and after linking. Disposable `target/tests` scratch +is removed before compiler cache post hooks, including on failure; `target/tests/trybuild` +stays so the compile-fail dependency build is cached with the rest of `target`. The nested +Cargo runs that tests start keep their own build directories under `target` +(`tests/trybuild`, `no-default-features-check` and `expand-check`), so they are cached +with it and never rerun, under a test's Cargo environment, the build scripts the +surrounding build reuses. + These changes require hosted evidence to establish end-to-end completion under 30 minutes; queue time, cold model downloads, repeated workspace compilation and the aggregate audit build remain part of that measurement. @@ -250,7 +265,11 @@ in their respective workflows, on the same selected source commit. The Linux req `CI controls` jobs execute the complete `scripts/tests/test_ci_qualification.py` module, including all development download controls. Locally, use Python 3.11 or newer with `PyYAML==6.0.2` installed and run -`python3 scripts/tests/test_ci_qualification.py`. These controls use synthetic +`python3 scripts/tests/test_ci_qualification.py`. The entrypoint runs every collected +test once, each in its own Python process, several at a time, and fails unless each one +reports exactly one passing test; naming a test (for example +`python3 scripts/tests/test_ci_qualification.py StageEvidence`) runs only that selection +in one process. These controls use synthetic compiler and model inputs; Cargo features do not change their inputs. Rust workflow guards verify both workflows and their qualification roles without recursively executing the Python suite. The jobs need Python and Git, with no Rust compilation or model cache setup, and @@ -261,12 +280,19 @@ after the unfiltered local run or verified shard union. This permits removal of retired binary while still rejecting a live binary without a recorded ceiling. The compatibility gate without nextest skips JUnit-dependent comparisons. -Physical Rust execution uses eleven jobs while retaining twelve logical shard receipts. -On Linux, each variant runs acceptance, integration and remaining cases, and imports -and CLI in three independent jobs: six Linux runtime jobs plus two independent compatibility build jobs: eight Linux Rust jobs in total. macOS default runs -acceptance separately and integration and remaining cases before imports and CLI in -a shared job; macOS feature execution runs all three stages in one job. Each runtime job seeds and proves the model once and shares its Cargo -target. `python3 scripts/ci-test-shards.py bundle default` (or `features`) uses +Each platform runs the same Rust jobs, one per logical shard receipt. Each variant runs +acceptance, integration and remaining cases, and imports and CLI in three independent +jobs: six runtime jobs plus two independent compatibility build jobs, eight Rust jobs per +platform and sixteen in total. Each runtime job seeds and proves the model once and keeps +its own Cargo target. Jobs that compile the same selection share one compiler cache entry +per platform (their cache keys omit the job identity, `add-job-id-key: false`) (default acceptance compiles the narrow selection; the other default bundles +compile the workspace; every feature bundle compiles the workspace with event-search test +support). Only pushes to `dev` save compiler caches, and a `dev` job that fails still saves +its compiled dependencies; pull requests restore the `dev` entries, which GitHub serves to +every branch targeting `dev`, and save nothing, so pull-request entries cannot evict them. +Compiler cache keys name the pinned toolchain in their prefix (`v1-rust-1.94.1`) instead of +hashing every Rust release the runner image preinstalls, so a runner image update does not +miss the `dev` cache. `python3 scripts/ci-test-shards.py bundle default` (or `features`) uses `GATE_PHYSICAL_BUNDLE` to select the displayed bundle. Each logical stage retains its own `GATE_SHARD`, receipt directory and existing artifact identity. A failed stage stops the bundle, preserves its exit status and uploads available evidence; later @@ -290,29 +316,34 @@ example artifact after a successful build, rejecting malformed or ambiguous outp The stage receipt records the actual build arguments, executable path and SHA-256. Narrow mode builds `cargo build --locked -p context-graph --example ci_model_seed --message-format=json-render-diagnostics`. Its source includes the unchanged private -catalog helper. Both helpers are excluded from their packages. Compiler seed caches use family v2; the +catalog helper. Both helpers are excluded from their packages. Compiler seed caches use family v3; the OS/source-bound Nomic model cache key is unchanged. Timing improvements require native unit-reuse evidence and complete cold/warm hosted measurements. ## macOS qualification follow-up -Linux is the current required qualification platform. The separate `macOS advisory` -workflow preserves macOS lint, default tests, feature tests and complete Python controls: -five physical jobs (two default, one feature, lint and controls) produce six logical -Rust receipts. Its `macOS advisory results` aggregate fails on failed, cancelled, +Linux is the current required qualification platform. The separate `macOS CI (advisory)` +workflow runs the Linux correctness jobs on `macos-latest` with identical steps: lint, +complete Python controls, three default bundles, three feature bundles and the two +compatibility build jobs, ten physical jobs that produce six logical Rust receipts. Its `macOS advisory results` aggregate fails on failed, cancelled, skipped, missing or unexpected prerequisites, verifies only macOS receipts and runs -macOS default accounting. It never enters `PR checks` needs or manual delivery admission. +macOS default accounting through the isolated audit. It never enters `PR checks` needs or manual delivery admission. The legacy local `shard-union` stage still requires twelve receipts; explicit Linux and macOS stages each require exactly six, with no fallback to available platforms. Qualification remains pending: seven macOS test failures remain unresolved. -The default owner-service locator also remains broken: the native user temporary -root plus the full socket key can exceed macOS's 103-byte socket pathname limit. -A writable database open can succeed while the separate owner service fails to start. -A repair and independent native owner/reader proof are still pending; prior reader -authentication results do not prove the default locator. These are open follow-up -items, not passing evidence. Completion requires identifying and fixing each failure in source, verifying -locator behavior, running all five macOS jobs with all six valid receipts on the exact +The pinned ContextDB macOS default locator now canonicalizes the fixed system +temporary root `/tmp` and uses a private per-effective-UID directory with the full +socket key. Focused upstream engine coverage consists of five integration cases +actually run on the pinned revision and three unchanged library cases carried +from an older revision, not eight newly executed cases. The only reader +authentication proof is from a separate earlier ContextDB candidate, not the +pinned revision; prior reader authentication results do not prove the pinned +default locator or full Context Graph qualification. Full Context Graph macOS +consumer qualification, including the integrated default owner/reader locator, +remains pending. These are open follow-up items, not passing evidence. +Completion requires identifying and fixing each failure in source, verifying +locator behavior, running all ten macOS jobs with all six valid receipts on the exact integrated source, and reviewing the resulting evidence before changing qualification. No issue number or hosted green result is claimed. @@ -323,16 +354,19 @@ can extend the dev-push workflow; required-check latency does not establish full workflow latency. Manual retry requires exactly eleven successful displayed Linux/source/docs checks from `ci.yml`; advisory runs cannot authorize delivery. -The Linux compatibility build jobs are unconditional and independent of the six runtime -jobs. `feature-qualification-core` runs the original default and minimal library +The compatibility build jobs are unconditional and independent of the six runtime +jobs on each platform. `feature-qualification-core` runs the original default and minimal library builds. `feature-qualification-sync` runs the original fabric-sync library and CLI -builds, then feature doctests. `feature-qualification` reproduces both groups in order. +builds. The feature doctests run in the event-search support acceptance job, before its +tests, where they reuse the workspace libraries that job already compiles with +event-search test support instead of compiling that closure a second time. +`feature-qualification` reproduces both build groups and then the feature doctests. Each compile job records its selected source, actual commands, compiler identities and job result without a shard or nextest inventory. The existing local `features`, sharded `features`, `pr` and `extended-public` stages retain all builds, tests and doctests in -their original order. Only the explicit Linux singleton dispatcher option -`bundle features --tests-only` selects the internal `features-tests` stage; default, -macOS and combined bundles reject that option. +their original order. Only the explicit singleton dispatcher option +`bundle features --tests-only` selects the internal `features-tests` stage, on either +platform; default and combined bundles reject that option. After all six Linux shard receipts verify, `shard-union-linux-isolated` copies the complete merged default JUnit, exports `CG_GATE_JUNIT`, compares the historical baseline, @@ -341,7 +375,8 @@ audit compiles the same `crates/context-graph/tests/test_estate_audit.rs` throug `scripts/gate-wall-audit/Cargo.toml`, an independent edition-2024 workspace with only the exact root-locked TOML dependency closure. Its ephemeral target is `$RUNNER_TEMP/gate-wall-audit-target`. No engine compilation or executable transport is -needed for this aggregate. Existing local `shard-union`, `shard-union-linux`, -`shard-union-macos` and default wall-audit routes keep their original behavior. -macOS commands and coverage remain unchanged. Actual cold compilation, model downloads, -queues and total hosted completion remain unproved until measured on the final source. +needed for this aggregate. The macOS aggregate runs +`shard-union-macos-isolated`, the same audit over the six macOS receipts. Existing local +`shard-union`, `shard-union-linux`, `shard-union-macos` and default wall-audit routes keep +their original behavior. Actual cold compilation, model downloads, queues and total hosted +completion remain unproved until measured on the final source. diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index fdbdbc5..abb04b2 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -88,7 +88,7 @@ jobs: and not job.name.startswith('development-binaries / ')] expected = {'source', 'docs', 'PR checks', 'Feature compatibility — default and minimal libraries (ubuntu-latest)', - 'Feature compatibility — sync, CLI and doctests (ubuntu-latest)'} + 'Feature compatibility — sync and CLI (ubuntu-latest)'} for os_name in ('ubuntu-latest',): expected.update({f'lint ({os_name})', f'CI controls ({os_name})'}) expected.update(f'Default tests — {bundle} ({os_name})' for bundle in diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0dd24c9..06e1582 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,8 +1,9 @@ -name: ci +name: Linux CI on: pull_request: push: + branches: [dev] workflow_dispatch: inputs: cg_sha: @@ -183,6 +184,9 @@ jobs: - uses: Swatinem/rust-cache@49a0bdc70d2e1b713ca9e2869b211fcce03d3c1c # v2 with: cache-bin: false + prefix-key: v1-rust-1.94.1 + add-rust-environment-hash-key: false + save-if: ${{ github.ref == 'refs/heads/dev' }} workspaces: workspace/context-graph -> target - uses: taiki-e/install-action@1168f6810a3cb30a3514bf5756a3b7f0422c6df8 # v2 with: @@ -512,8 +516,13 @@ jobs: CG_MODEL_CACHE_ROOT: ${{ runner.temp }}/cg-model-cache with: cache-bin: false + prefix-key: v1-rust-1.94.1 + add-rust-environment-hash-key: false + save-if: ${{ github.ref == 'refs/heads/dev' }} workspaces: workspace/context-graph -> target - key: test-catalog-seed-v2-${{ matrix.bundle }} + cache-on-failure: true + key: test-catalog-seed-v3-${{ matrix.bundle == 'acceptance' && 'narrow' || 'workspace' }} + add-job-id-key: false - uses: taiki-e/install-action@1168f6810a3cb30a3514bf5756a3b7f0422c6df8 # v2 env: CG_MODEL_CACHE_ROOT: ${{ runner.temp }}/cg-model-cache @@ -652,8 +661,13 @@ jobs: CG_MODEL_CACHE_ROOT: ${{ runner.temp }}/cg-model-cache with: cache-bin: false + prefix-key: v1-rust-1.94.1 + add-rust-environment-hash-key: false + save-if: ${{ github.ref == 'refs/heads/dev' }} workspaces: workspace/context-graph -> target - key: features-catalog-seed-v2-${{ matrix.bundle }} + cache-on-failure: true + key: features-catalog-seed-v3-workspace + add-job-id-key: false - uses: taiki-e/install-action@1168f6810a3cb30a3514bf5756a3b7f0422c6df8 # v2 env: CG_MODEL_CACHE_ROOT: ${{ runner.temp }}/cg-model-cache @@ -768,6 +782,9 @@ jobs: CG_MODEL_CACHE_ROOT: ${{ runner.temp }}/cg-model-cache with: cache-bin: false + prefix-key: v1-rust-1.94.1 + add-rust-environment-hash-key: false + save-if: ${{ github.ref == 'refs/heads/dev' }} workspaces: workspace/context-graph -> target key: feature-qualification-core-v1 - name: Run feature compatibility qualification @@ -837,7 +854,7 @@ jobs: retention-days: 14 if-no-files-found: error feature-qualification-sync: - name: Feature compatibility — sync, CLI and doctests (ubuntu-latest) + name: Feature compatibility — sync and CLI (ubuntu-latest) runs-on: ubuntu-latest timeout-minutes: 120 steps: @@ -885,6 +902,9 @@ jobs: CG_MODEL_CACHE_ROOT: ${{ runner.temp }}/cg-model-cache with: cache-bin: false + prefix-key: v1-rust-1.94.1 + add-rust-environment-hash-key: false + save-if: ${{ github.ref == 'refs/heads/dev' }} workspaces: workspace/context-graph -> target key: feature-qualification-sync-v1 - name: Run feature compatibility qualification diff --git a/.github/workflows/macos-advisory.yml b/.github/workflows/macos-advisory.yml index 48c83ec..a3e32b4 100644 --- a/.github/workflows/macos-advisory.yml +++ b/.github/workflows/macos-advisory.yml @@ -1,8 +1,9 @@ -name: macOS advisory +name: macOS CI (advisory) on: pull_request: push: + branches: [dev] workflow_dispatch: inputs: cg_sha: @@ -87,6 +88,9 @@ jobs: - uses: Swatinem/rust-cache@49a0bdc70d2e1b713ca9e2869b211fcce03d3c1c # v2 with: cache-bin: false + prefix-key: v1-rust-1.94.1 + add-rust-environment-hash-key: false + save-if: ${{ github.ref == 'refs/heads/dev' }} workspaces: workspace/context-graph -> target - uses: taiki-e/install-action@1168f6810a3cb30a3514bf5756a3b7f0422c6df8 # v2 with: @@ -267,7 +271,9 @@ jobs: - os: macos-latest bundle: acceptance - os: macos-latest - bundle: integration-and-rest + import-and-cli + bundle: integration-and-rest + - os: macos-latest + bundle: import-and-cli steps: - name: Validate exact ContextDB SHA if: github.event_name == 'workflow_dispatch' @@ -292,7 +298,7 @@ jobs: - name: Print and verify checked revisions env: CG_MODEL_CACHE_ROOT: ${{ runner.temp }}/cg-model-cache - GATE_RECEIPT_DIR: ${{ runner.temp }}/job-receipt/${{ contains(matrix.bundle, 'integration-and-rest') && 'integration-and-rest' || 'acceptance' }} + GATE_RECEIPT_DIR: ${{ runner.temp }}/job-receipt/${{ matrix.bundle }} TESTED_SHA: ${{ github.event.pull_request.head.sha || inputs.cg_sha || github.sha }} BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }} HEAD_SHA: ${{ github.event.pull_request.head.sha || inputs.cg_sha || github.sha }} @@ -320,8 +326,13 @@ jobs: CG_MODEL_CACHE_ROOT: ${{ runner.temp }}/cg-model-cache with: cache-bin: false + prefix-key: v1-rust-1.94.1 + add-rust-environment-hash-key: false + save-if: ${{ github.ref == 'refs/heads/dev' }} workspaces: workspace/context-graph -> target - key: test-catalog-seed-v2-${{ matrix.bundle }} + cache-on-failure: true + key: test-catalog-seed-v3-${{ matrix.bundle == 'acceptance' && 'narrow' || 'workspace' }} + add-job-id-key: false - uses: taiki-e/install-action@1168f6810a3cb30a3514bf5756a3b7f0422c6df8 # v2 env: CG_MODEL_CACHE_ROOT: ${{ runner.temp }}/cg-model-cache @@ -339,16 +350,16 @@ jobs: key: cg-nomic-${{ runner.os }}-${{ hashFiles('workspace/context-graph/crates/context-graph/src/embedder/nomic.rs') }} - name: Seed Nomic model cache on cold miss env: - GATE_CI_MODEL_MODE: ${{ contains(matrix.bundle, 'integration-and-rest') && 'catalog-workspace' || 'catalog-narrow' }} + GATE_CI_MODEL_MODE: ${{ matrix.bundle == 'acceptance' && 'catalog-narrow' || 'catalog-workspace' }} CG_MODEL_CACHE_ROOT: ${{ runner.temp }}/cg-model-cache - GATE_RECEIPT_DIR: ${{ runner.temp }}/job-receipt/${{ contains(matrix.bundle, 'integration-and-rest') && 'integration-and-rest' || 'acceptance' }} + GATE_RECEIPT_DIR: ${{ runner.temp }}/job-receipt/${{ matrix.bundle }} if: steps.nomic-model-cache.outputs.cache-hit != 'true' run: bash scripts/gate.sh model-seed - name: Run offline real Nomic model proofs env: - GATE_CI_MODEL_MODE: ${{ contains(matrix.bundle, 'integration-and-rest') && 'catalog-workspace' || 'catalog-narrow' }} + GATE_CI_MODEL_MODE: ${{ matrix.bundle == 'acceptance' && 'catalog-narrow' || 'catalog-workspace' }} CG_MODEL_CACHE_ROOT: ${{ runner.temp }}/cg-model-cache - GATE_RECEIPT_DIR: ${{ runner.temp }}/job-receipt/${{ contains(matrix.bundle, 'integration-and-rest') && 'integration-and-rest' || 'acceptance' }} + GATE_RECEIPT_DIR: ${{ runner.temp }}/job-receipt/${{ matrix.bundle }} MODEL_CACHE_HIT: 'true' run: bash scripts/gate.sh public-model - name: Run default ${{ matrix.bundle }} tests @@ -403,7 +414,11 @@ jobs: # windows-latest is intentionally excluded: descriptor-relative sync IO requires Unix. include: - os: macos-latest - bundle: integration-and-rest + import-and-cli + acceptance + bundle: acceptance + - os: macos-latest + bundle: integration-and-rest + - os: macos-latest + bundle: import-and-cli steps: - name: Validate exact ContextDB SHA if: github.event_name == 'workflow_dispatch' @@ -428,7 +443,7 @@ jobs: - name: Print and verify checked revisions env: CG_MODEL_CACHE_ROOT: ${{ runner.temp }}/cg-model-cache - GATE_RECEIPT_DIR: ${{ runner.temp }}/job-receipt/${{ contains(matrix.bundle, 'integration-and-rest') && 'integration-and-rest' || 'acceptance' }} + GATE_RECEIPT_DIR: ${{ runner.temp }}/job-receipt/${{ matrix.bundle }} TESTED_SHA: ${{ github.event.pull_request.head.sha || inputs.cg_sha || github.sha }} BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }} HEAD_SHA: ${{ github.event.pull_request.head.sha || inputs.cg_sha || github.sha }} @@ -456,8 +471,13 @@ jobs: CG_MODEL_CACHE_ROOT: ${{ runner.temp }}/cg-model-cache with: cache-bin: false + prefix-key: v1-rust-1.94.1 + add-rust-environment-hash-key: false + save-if: ${{ github.ref == 'refs/heads/dev' }} workspaces: workspace/context-graph -> target - key: features-catalog-seed-v2-${{ matrix.bundle }} + cache-on-failure: true + key: features-catalog-seed-v3-workspace + add-job-id-key: false - uses: taiki-e/install-action@1168f6810a3cb30a3514bf5756a3b7f0422c6df8 # v2 env: CG_MODEL_CACHE_ROOT: ${{ runner.temp }}/cg-model-cache @@ -474,14 +494,14 @@ jobs: env: GATE_CI_MODEL_MODE: catalog-features CG_MODEL_CACHE_ROOT: ${{ runner.temp }}/cg-model-cache - GATE_RECEIPT_DIR: ${{ runner.temp }}/job-receipt/${{ contains(matrix.bundle, 'integration-and-rest') && 'integration-and-rest' || 'acceptance' }} + GATE_RECEIPT_DIR: ${{ runner.temp }}/job-receipt/${{ matrix.bundle }} if: steps.nomic-model-cache.outputs.cache-hit != 'true' run: bash scripts/gate.sh model-seed - name: Run offline real Nomic model proofs env: GATE_CI_MODEL_MODE: catalog-features CG_MODEL_CACHE_ROOT: ${{ runner.temp }}/cg-model-cache - GATE_RECEIPT_DIR: ${{ runner.temp }}/job-receipt/${{ contains(matrix.bundle, 'integration-and-rest') && 'integration-and-rest' || 'acceptance' }} + GATE_RECEIPT_DIR: ${{ runner.temp }}/job-receipt/${{ matrix.bundle }} MODEL_CACHE_HIT: 'true' run: bash scripts/gate.sh public-model - name: Run event-search support ${{ matrix.bundle }} tests @@ -490,7 +510,7 @@ jobs: GATE_MODEL_READY: '1' TESTED_SHA: ${{ github.event.pull_request.head.sha || inputs.cg_sha || github.sha }} MODEL_CACHE_HIT: 'true' - run: python3 scripts/ci-test-shards.py bundle features + run: python3 scripts/ci-test-shards.py bundle features --tests-only - name: Record disk headroom and remove disposable compiler scratch if: always() run: | @@ -523,10 +543,250 @@ jobs: path: ${{ runner.temp }}/job-receipt/acceptance retention-days: 14 if-no-files-found: error + feature-qualification-core: + name: Feature compatibility — default and minimal libraries (macos-latest) + runs-on: macos-latest + timeout-minutes: 120 + steps: + - name: Validate exact ContextDB SHA + if: github.event_name == 'workflow_dispatch' + working-directory: . + env: + CONTEXTDB_SHA: ${{ inputs.contextdb_sha }} + run: | + if [[ ! "$CONTEXTDB_SHA" =~ ^[0-9a-f]{40}$ ]]; then + echo "contextdb_sha must be an exact 40-character lowercase hexadecimal commit SHA" >&2 + exit 1 + fi + - name: Check out Context Graph + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 + with: + path: workspace/context-graph + ref: ${{ github.event.pull_request.head.sha || inputs.cg_sha || github.sha }} + fetch-depth: 0 + persist-credentials: false + - name: Print and verify checked revisions + env: + TESTED_SHA: ${{ github.event.pull_request.head.sha || inputs.cg_sha || github.sha }} + BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }} + HEAD_SHA: ${{ github.event.pull_request.head.sha || inputs.cg_sha || github.sha }} + EVENT_SHA: ${{ github.sha }} + REQUESTED_CONTEXTDB_SHA: ${{ inputs.contextdb_sha }} + run: | + bash scripts/gate.sh source + - name: Free disk space + if: runner.os == 'Linux' + run: | + df -h / + sudo docker image prune -a -f || true + sudo docker system prune -a -f --volumes || true + sudo rm -rf /opt/hostedtoolcache/CodeQL /opt/hostedtoolcache/go /opt/az /opt/ghc /usr/local/lib/android /usr/share/dotnet /usr/local/share/boost + df -h / + - uses: dtolnay/rust-toolchain@686976e191b89faba57d3206551f0f330d8cb249 + env: + CG_MODEL_CACHE_ROOT: ${{ runner.temp }}/cg-model-cache + with: + toolchain: 1.94.1 + - uses: Swatinem/rust-cache@49a0bdc70d2e1b713ca9e2869b211fcce03d3c1c + env: + CG_MODEL_CACHE_ROOT: ${{ runner.temp }}/cg-model-cache + with: + cache-bin: false + prefix-key: v1-rust-1.94.1 + add-rust-environment-hash-key: false + save-if: ${{ github.ref == 'refs/heads/dev' }} + workspaces: workspace/context-graph -> target + key: feature-qualification-core-v1 + - name: Run feature compatibility qualification + run: bash scripts/gate.sh feature-qualification-core + - name: Record the exact job receipt + if: always() + env: + TESTED_SHA: ${{ github.event.pull_request.head.sha || inputs.cg_sha || github.sha }} + RECEIPT_OS: ${{ runner.os }} + RECEIPT_JOB: ${{ github.job }} + RECEIPT_JOB_STATUS: ${{ job.status }} + run: | + python3 - <<'PY_RECEIPT' + import json, os, pathlib, subprocess, xml.etree.ElementTree as ET + head = subprocess.check_output(['git', 'rev-parse', 'HEAD'], text=True).strip() + if head != os.environ['TESTED_SHA']: + raise SystemExit('receipt checkout mismatch') + lock = pathlib.Path('Cargo.lock').read_text() + import re + dependency = re.search(r'contextdb[?]rev=([0-9a-f]{40})', lock).group(1) + receipt = dict(cg_sha=head, contextdb_sha=dependency, os=os.environ['RECEIPT_OS'], job=os.environ['RECEIPT_JOB'], toolchain='1.94.1', profile='gate' if os.environ['RECEIPT_JOB'] == 'test' else 'default') + destination = pathlib.Path(os.environ['RUNNER_TEMP']) / 'job-receipt' + destination.mkdir(exist_ok=True) + receipt_path = destination / 'receipt.json' + receipt_path.write_text(json.dumps(receipt, sort_keys=True) + '\n') + evidence_errors = [] + if True: + for field, command in ( + ('rustc_version', ['rustc', '+1.94.1', '--version']), + ('cargo_version', ['cargo', '+1.94.1', '--version']), + ): + try: + version = subprocess.check_output(command, text=True).strip() + if not version: + raise ValueError('empty tool identity') + receipt[field] = version + except (OSError, subprocess.CalledProcessError, ValueError): + evidence_errors.append('missing required tool identity: ' + field) + junit = pathlib.Path('target/nextest/gate/junit.xml') + if os.environ['RECEIPT_JOB'] == 'test': + try: + cases = list(ET.parse(junit).getroot().iter('testcase')) + executed = [c for c in cases if c.find('skipped') is None] + receipt['test_count'] = len(executed) + receipt['failed_cases'] = [c.attrib for c in cases if c.find('failure') is not None or c.find('error') is not None] + if not executed: + evidence_errors.append('empty measured test inventory') + except (OSError, ET.ParseError): + evidence_errors.append('missing or malformed measured test inventory') + receipt_path.write_text(json.dumps(receipt, sort_keys=True) + '\n') + if junit.is_file() and os.environ['RECEIPT_JOB'] == 'test': + import shutil + shutil.copy2(junit, destination / 'junit.xml') + if evidence_errors and os.environ['RECEIPT_JOB_STATUS'] == 'success': + raise SystemExit('; '.join(evidence_errors)) + PY_RECEIPT + - name: Record disk headroom and remove disposable compiler scratch + if: always() + run: | + df -h / + python3 scripts/ci-cache-post-cleanup.py + - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 + if: always() + with: + name: pr-${{ github.job }}-${{ runner.os }} + path: ${{ runner.temp }}/job-receipt + retention-days: 14 + if-no-files-found: error + feature-qualification-sync: + name: Feature compatibility — sync and CLI (macos-latest) + runs-on: macos-latest + timeout-minutes: 120 + steps: + - name: Validate exact ContextDB SHA + if: github.event_name == 'workflow_dispatch' + working-directory: . + env: + CONTEXTDB_SHA: ${{ inputs.contextdb_sha }} + run: | + if [[ ! "$CONTEXTDB_SHA" =~ ^[0-9a-f]{40}$ ]]; then + echo "contextdb_sha must be an exact 40-character lowercase hexadecimal commit SHA" >&2 + exit 1 + fi + - name: Check out Context Graph + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 + with: + path: workspace/context-graph + ref: ${{ github.event.pull_request.head.sha || inputs.cg_sha || github.sha }} + fetch-depth: 0 + persist-credentials: false + - name: Print and verify checked revisions + env: + TESTED_SHA: ${{ github.event.pull_request.head.sha || inputs.cg_sha || github.sha }} + BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }} + HEAD_SHA: ${{ github.event.pull_request.head.sha || inputs.cg_sha || github.sha }} + EVENT_SHA: ${{ github.sha }} + REQUESTED_CONTEXTDB_SHA: ${{ inputs.contextdb_sha }} + run: | + bash scripts/gate.sh source + - name: Free disk space + if: runner.os == 'Linux' + run: | + df -h / + sudo docker image prune -a -f || true + sudo docker system prune -a -f --volumes || true + sudo rm -rf /opt/hostedtoolcache/CodeQL /opt/hostedtoolcache/go /opt/az /opt/ghc /usr/local/lib/android /usr/share/dotnet /usr/local/share/boost + df -h / + - uses: dtolnay/rust-toolchain@686976e191b89faba57d3206551f0f330d8cb249 + env: + CG_MODEL_CACHE_ROOT: ${{ runner.temp }}/cg-model-cache + with: + toolchain: 1.94.1 + - uses: Swatinem/rust-cache@49a0bdc70d2e1b713ca9e2869b211fcce03d3c1c + env: + CG_MODEL_CACHE_ROOT: ${{ runner.temp }}/cg-model-cache + with: + cache-bin: false + prefix-key: v1-rust-1.94.1 + add-rust-environment-hash-key: false + save-if: ${{ github.ref == 'refs/heads/dev' }} + workspaces: workspace/context-graph -> target + key: feature-qualification-sync-v1 + - name: Run feature compatibility qualification + run: bash scripts/gate.sh feature-qualification-sync + - name: Record the exact job receipt + if: always() + env: + TESTED_SHA: ${{ github.event.pull_request.head.sha || inputs.cg_sha || github.sha }} + RECEIPT_OS: ${{ runner.os }} + RECEIPT_JOB: ${{ github.job }} + RECEIPT_JOB_STATUS: ${{ job.status }} + run: | + python3 - <<'PY_RECEIPT' + import json, os, pathlib, subprocess, xml.etree.ElementTree as ET + head = subprocess.check_output(['git', 'rev-parse', 'HEAD'], text=True).strip() + if head != os.environ['TESTED_SHA']: + raise SystemExit('receipt checkout mismatch') + lock = pathlib.Path('Cargo.lock').read_text() + import re + dependency = re.search(r'contextdb[?]rev=([0-9a-f]{40})', lock).group(1) + receipt = dict(cg_sha=head, contextdb_sha=dependency, os=os.environ['RECEIPT_OS'], job=os.environ['RECEIPT_JOB'], toolchain='1.94.1', profile='gate' if os.environ['RECEIPT_JOB'] == 'test' else 'default') + destination = pathlib.Path(os.environ['RUNNER_TEMP']) / 'job-receipt' + destination.mkdir(exist_ok=True) + receipt_path = destination / 'receipt.json' + receipt_path.write_text(json.dumps(receipt, sort_keys=True) + '\n') + evidence_errors = [] + if True: + for field, command in ( + ('rustc_version', ['rustc', '+1.94.1', '--version']), + ('cargo_version', ['cargo', '+1.94.1', '--version']), + ): + try: + version = subprocess.check_output(command, text=True).strip() + if not version: + raise ValueError('empty tool identity') + receipt[field] = version + except (OSError, subprocess.CalledProcessError, ValueError): + evidence_errors.append('missing required tool identity: ' + field) + junit = pathlib.Path('target/nextest/gate/junit.xml') + if os.environ['RECEIPT_JOB'] == 'test': + try: + cases = list(ET.parse(junit).getroot().iter('testcase')) + executed = [c for c in cases if c.find('skipped') is None] + receipt['test_count'] = len(executed) + receipt['failed_cases'] = [c.attrib for c in cases if c.find('failure') is not None or c.find('error') is not None] + if not executed: + evidence_errors.append('empty measured test inventory') + except (OSError, ET.ParseError): + evidence_errors.append('missing or malformed measured test inventory') + receipt_path.write_text(json.dumps(receipt, sort_keys=True) + '\n') + if junit.is_file() and os.environ['RECEIPT_JOB'] == 'test': + import shutil + shutil.copy2(junit, destination / 'junit.xml') + if evidence_errors and os.environ['RECEIPT_JOB_STATUS'] == 'success': + raise SystemExit('; '.join(evidence_errors)) + PY_RECEIPT + - name: Record disk headroom and remove disposable compiler scratch + if: always() + run: | + df -h / + python3 scripts/ci-cache-post-cleanup.py + - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 + if: always() + with: + name: pr-${{ github.job }}-${{ runner.os }} + path: ${{ runner.temp }}/job-receipt + retention-days: 14 + if-no-files-found: error macos-advisory: name: macOS advisory results if: always() - needs: [lint, controls, test, features] + needs: [lint, test, features, controls, feature-qualification-core, feature-qualification-sync] runs-on: ubuntu-latest timeout-minutes: 120 steps: @@ -537,8 +797,15 @@ jobs: run: | python3 - <<'PY_CHECK' import json, os - expected = {'features', 'test', 'lint', 'controls'} - needs = json.loads(os.environ['NEEDS_JSON']) + expected = {'features', 'test', 'lint', 'controls', 'feature-qualification-core', 'feature-qualification-sync'} + def unique(pairs): + result = {} + for key, value in pairs: + if key in result: + raise ValueError('duplicate required job: ' + key) + result[key] = value + return result + needs = json.loads(os.environ['NEEDS_JSON'], object_pairs_hook=unique) if set(needs) != expected or any(row.get('result') != 'success' for row in needs.values()): raise SystemExit('required job inventory failed: ' + json.dumps(needs, sort_keys=True)) print('all required jobs succeeded') @@ -555,22 +822,9 @@ jobs: TESTED_SHA: ${{ github.event.pull_request.head.sha || inputs.cg_sha || github.sha }} REQUESTED_CONTEXTDB_SHA: ${{ inputs.contextdb_sha }} run: bash scripts/gate.sh source - - name: Free disk space - if: runner.os == 'Linux' - run: | - df -h / - sudo docker image prune -a -f || true - sudo docker system prune -a -f --volumes || true - sudo rm -rf /opt/hostedtoolcache/CodeQL /opt/hostedtoolcache/go /opt/az /opt/ghc /usr/local/lib/android /usr/share/dotnet /usr/local/share/boost - df -h / - uses: dtolnay/rust-toolchain@686976e191b89faba57d3206551f0f330d8cb249 # stable with: toolchain: 1.94.1 - - uses: Swatinem/rust-cache@49a0bdc70d2e1b713ca9e2869b211fcce03d3c1c # v2 - with: - cache-bin: false - workspaces: workspace/context-graph -> target - key: test-integration-and-rest + import-and-cli - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 with: pattern: pr-* @@ -578,15 +832,7 @@ jobs: - name: Verify complete default and feature unions before wall reports env: TESTED_SHA: ${{ github.event.pull_request.head.sha || inputs.cg_sha || github.sha }} - run: bash scripts/gate.sh shard-union-macos - - name: Record disk headroom and remove disposable compiler scratch - if: always() - run: | - df -h / - python3 scripts/ci-cache-post-cleanup.py - - name: Remove disposable compiler scratch - if: always() - run: python3 scripts/ci-cache-post-cleanup.py + run: bash scripts/gate.sh shard-union-macos-isolated - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 if: always() with: diff --git a/AGENTS.md b/AGENTS.md index 87753ca..583722f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -115,3 +115,10 @@ Performance measurement is manual and lives only in The guarding tests and local seams are named in each crate's `AGENTS.md`. Fast iteration uses the owning guide's command; the full contributor gate remains binding before a done claim. + +Tests synchronize on state, counters, or explicit events; never sleep or assert elapsed time. +Consume `contextdb_core::Wallclock::test_clock_guard` for clock control, and drive background +work synchronously when its thread cannot see that guard. Lower each removed site's allowance +in `crates/context-graph/tests/fixtures/test_estate_baseline.toml`; +`no_new_sleep_or_raw_clock_sites_beyond_the_frozen_baseline` in +`crates/context-graph/tests/test_estate_audit.rs` rejects new sleep/raw-clock sites and stale allowances. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d6946b3..2ddc5a5 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -65,7 +65,7 @@ Without cargo-nextest, the script runs workspace tests but skips the two checks ### Continuous integration -Required PR test and feature jobs run on `ubuntu-latest`; the separate `macOS advisory` workflow retains the same coverage on `macos-latest`. +The `Linux CI` and `macOS CI (advisory)` workflows run on pushes to `dev`, every pull request and exact-SHA manual dispatch. Candidate branches use pull requests or manual dispatch for validation. Required PR test and feature jobs run on `ubuntu-latest`; the separate `macOS CI (advisory)` workflow retains the same coverage on `macos-latest`. `windows-latest` is excluded deliberately because the ContextDB sync substrate requires Unix descriptor-relative filesystem operations. The workflows install the repository's Rust 1.94.1 toolchain explicitly, use immutable action pins, and verify the selected Context Graph head and locked ContextDB revision. Base, head and event SHAs are recorded separately. Fork PR jobs use public inputs and no provider secrets. @@ -86,7 +86,7 @@ A cold miss fetches the default model; the document, query and event-path Nomic | Performance measurement | Manual; dormant `performance.yml` never runs; [performance guide](docs/performance/README.md) | Manual-first: build the exact baseline and candidate on one controlled host, keep raw evidence private, publish only a reviewed summary; nothing hosted measures performance | | Publication preparation | Manual `publication.yml`; `bash scripts/gate.sh publication-check` | Locked optimized CLI build and local package inspection; no publishing, tags or crates.io action | -The PR source job always records `qualification.json` and a step summary bound to the checked source and locked dependency SHA. `scripts/ci-impact.py` contains path-map version 1: Markdown under `docs/` and `skills/` and the named root documentation/license files are unaffected; runtime, build, scripts, tests and CI paths require performance and trusted integration qualification. Unknown paths, unreadable or empty diffs and initial parentless roots require both classes. All Linux checks remain required. The same macOS commands and full controls run in the separate, nonblocking `macOS advisory` workflow; failures remain visible. See the [macOS qualification follow-up](.github/CI.md#macos-qualification-follow-up). Affected changes report `qualification: pending maintainer native/trusted run`; public check success does not imply native or trusted qualification. Fork PRs use no secrets and never run these privileged workloads. For performance, the pending label means a manual measurement may be wanted; see the [performance guide](docs/performance/README.md). +The PR source job always records `qualification.json` and a step summary bound to the checked source and locked dependency SHA. `scripts/ci-impact.py` contains path-map version 1: Markdown under `docs/` and `skills/` and the named root documentation/license files are unaffected; runtime, build, scripts, tests and CI paths require performance and trusted integration qualification. Unknown paths, unreadable or empty diffs and initial parentless roots require both classes. All Linux checks remain required. The same macOS commands and full controls run in the separate, nonblocking `macOS CI (advisory)` workflow; failures remain visible. See the [macOS qualification follow-up](.github/CI.md#macos-qualification-follow-up). Affected changes report `qualification: pending maintainer native/trusted run`; public check success does not imply native or trusted qualification. Fork PRs use no secrets and never run these privileged workloads. For performance, the pending label means a manual measurement may be wanted; see the [performance guide](docs/performance/README.md). A maintainer can explicitly supply `--qualification-receipt ` when running `python3 scripts/ci-impact.py --base --output `. Receipt schema 1 requires the exact `cg_sha` and `contextdb_sha`, `result: "qualified"`, the ordered `qualified_classes` array (`performance`, `trusted-integration`) and an `evidence_sha256` digest for each class. Mismatched or incomplete receipts fail and retain the pending status. Scheduled results are never discovered or accepted implicitly. @@ -152,7 +152,11 @@ in their respective workflows, on the same selected source commit. The Linux req and macOS advisory `CI controls` jobs execute the complete `scripts/tests/test_ci_qualification.py` module, including all development download controls. Locally, use Python 3.11 or newer with `PyYAML==6.0.2` installed and run -`python3 scripts/tests/test_ci_qualification.py`. These controls use synthetic +`python3 scripts/tests/test_ci_qualification.py`. The entrypoint runs every collected +test once, each in its own Python process, several at a time, and fails unless each one +reports exactly one passing test; naming a test (for example +`python3 scripts/tests/test_ci_qualification.py StageEvidence`) runs only that selection +in one process. These controls use synthetic compiler and model inputs; Cargo features do not change their inputs. Rust workflow guards verify both qualification roles without recursively executing the Python suite. The jobs need Python and Git, with no Rust compilation or model cache setup, and @@ -163,12 +167,15 @@ after the unfiltered local run or verified shard union. This permits removal of retired binary while still rejecting a live binary without a recorded ceiling. The compatibility gate without nextest skips JUnit-dependent comparisons. -Physical Rust execution uses eleven jobs while retaining twelve logical shard receipts. -On Linux, each variant runs acceptance, integration and remaining cases, and imports -and CLI in three independent jobs: six Linux runtime jobs plus two independent compatibility build jobs: eight Linux Rust jobs in total. macOS default runs -acceptance separately and integration and remaining cases before imports and CLI in -a shared job; macOS feature execution runs all three stages in one job. Each runtime job seeds and proves the model once and shares its Cargo -target. `python3 scripts/ci-test-shards.py bundle default` (or `features`) uses +Each platform runs the same Rust jobs, one per logical shard receipt. Each variant runs +acceptance, integration and remaining cases, and imports and CLI in three independent +jobs: six runtime jobs plus two independent compatibility build jobs, eight Rust jobs per +platform and sixteen in total. Each runtime job seeds and proves the model once and keeps +its own Cargo target. Jobs that compile the same selection share one compiler cache entry +per platform (default acceptance compiles the narrow selection; the other default bundles +compile the workspace; every feature bundle compiles the workspace with event-search test +support), and a job that fails still saves its compiled dependencies, so a later run does +not start cold because an earlier one failed. `python3 scripts/ci-test-shards.py bundle default` (or `features`) uses `GATE_PHYSICAL_BUNDLE` to select the displayed bundle. Each logical stage retains its own `GATE_SHARD`, receipt directory and existing artifact identity. A failed stage stops the bundle, preserves its exit status and uploads available evidence; later @@ -183,16 +190,19 @@ the workspace and feature jobs use the workspace with event-search test support. seeding still runs the document proof, and cache hits still run all three offline proofs. Local model seeding retains `cg model fetch --defaults`. Ordinary CLI builds and tests remain required. -The Linux compatibility build jobs are unconditional and independent of the six runtime -jobs. `feature-qualification-core` runs the original default and minimal library +The compatibility build jobs are unconditional and independent of the six runtime +jobs on each platform. `feature-qualification-core` runs the original default and minimal library builds. `feature-qualification-sync` runs the original fabric-sync library and CLI -builds, then feature doctests. `feature-qualification` reproduces both groups in order. +builds. The feature doctests run in the event-search support acceptance job, before its +tests, where they reuse the workspace libraries that job already compiles with +event-search test support instead of compiling that closure a second time. +`feature-qualification` reproduces both build groups and then the feature doctests. Each compile job records its selected source, actual commands, compiler identities and job result without a shard or nextest inventory. The existing local `features`, sharded `features`, `pr` and `extended-public` stages retain all builds, tests and doctests in -their original order. Only the explicit Linux singleton dispatcher option -`bundle features --tests-only` selects the internal `features-tests` stage; default, -macOS and combined bundles reject that option. +their original order. Only the explicit singleton dispatcher option +`bundle features --tests-only` selects the internal `features-tests` stage, on either +platform; default and combined bundles reject that option. After all six Linux shard receipts verify, `shard-union-linux-isolated` copies the complete merged default JUnit, exports `CG_GATE_JUNIT`, compares the historical baseline, @@ -201,7 +211,8 @@ audit compiles the same `crates/context-graph/tests/test_estate_audit.rs` throug `scripts/gate-wall-audit/Cargo.toml`, an independent edition-2024 workspace with only the exact root-locked TOML dependency closure. Its ephemeral target is `$RUNNER_TEMP/gate-wall-audit-target`. No engine compilation or executable transport is -needed for this aggregate. Existing local `shard-union`, `shard-union-linux`, -`shard-union-macos` and default wall-audit routes keep their original behavior. -macOS commands and coverage remain unchanged. Actual cold compilation, model downloads, -queues and total hosted completion remain unproved until measured on the final source. +needed for this aggregate. The macOS aggregate runs +`shard-union-macos-isolated`, the same audit over the six macOS receipts. Existing local +`shard-union`, `shard-union-linux`, `shard-union-macos` and default wall-audit routes keep +their original behavior. Actual cold compilation, model downloads, queues and total hosted +completion remain unproved until measured on the final source. diff --git a/Cargo.lock b/Cargo.lock index 4fcd89e..0715183 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -590,6 +590,7 @@ version = "0.1.0" dependencies = [ "candle-core", "candle-embeddinggemma", + "cg-catalog", "cg-cli-conventions", "chrono", "context-graph", @@ -912,7 +913,7 @@ dependencies = [ [[package]] name = "contextdb-core" version = "1.0.0" -source = "git+https://github.com/context-graph-ai/contextdb?rev=bc9cce955eb85fb1488f5304426eedeb6d97e905#bc9cce955eb85fb1488f5304426eedeb6d97e905" +source = "git+https://github.com/context-graph-ai/contextdb?rev=56dccfdc905653370a7a6775d0998b1053e5926a#56dccfdc905653370a7a6775d0998b1053e5926a" dependencies = [ "bincode 2.0.1", "rand 0.9.4", @@ -926,9 +927,10 @@ dependencies = [ [[package]] name = "contextdb-engine" version = "1.0.0" -source = "git+https://github.com/context-graph-ai/contextdb?rev=bc9cce955eb85fb1488f5304426eedeb6d97e905#bc9cce955eb85fb1488f5304426eedeb6d97e905" +source = "git+https://github.com/context-graph-ai/contextdb?rev=56dccfdc905653370a7a6775d0998b1053e5926a#56dccfdc905653370a7a6775d0998b1053e5926a" dependencies = [ "bao-tree", + "base64 0.22.1", "bincode 2.0.1", "blake3", "bytes", @@ -971,7 +973,7 @@ dependencies = [ [[package]] name = "contextdb-graph" version = "1.0.0" -source = "git+https://github.com/context-graph-ai/contextdb?rev=bc9cce955eb85fb1488f5304426eedeb6d97e905#bc9cce955eb85fb1488f5304426eedeb6d97e905" +source = "git+https://github.com/context-graph-ai/contextdb?rev=56dccfdc905653370a7a6775d0998b1053e5926a#56dccfdc905653370a7a6775d0998b1053e5926a" dependencies = [ "contextdb-core", "contextdb-tx", @@ -982,7 +984,7 @@ dependencies = [ [[package]] name = "contextdb-hnsw" version = "1.0.0" -source = "git+https://github.com/context-graph-ai/contextdb?rev=bc9cce955eb85fb1488f5304426eedeb6d97e905#bc9cce955eb85fb1488f5304426eedeb6d97e905" +source = "git+https://github.com/context-graph-ai/contextdb?rev=56dccfdc905653370a7a6775d0998b1053e5926a#56dccfdc905653370a7a6775d0998b1053e5926a" dependencies = [ "anndists", "anyhow", @@ -1006,7 +1008,7 @@ dependencies = [ [[package]] name = "contextdb-parser" version = "1.0.0" -source = "git+https://github.com/context-graph-ai/contextdb?rev=bc9cce955eb85fb1488f5304426eedeb6d97e905#bc9cce955eb85fb1488f5304426eedeb6d97e905" +source = "git+https://github.com/context-graph-ai/contextdb?rev=56dccfdc905653370a7a6775d0998b1053e5926a#56dccfdc905653370a7a6775d0998b1053e5926a" dependencies = [ "contextdb-core", "pest", @@ -1016,7 +1018,7 @@ dependencies = [ [[package]] name = "contextdb-planner" version = "1.0.0" -source = "git+https://github.com/context-graph-ai/contextdb?rev=bc9cce955eb85fb1488f5304426eedeb6d97e905#bc9cce955eb85fb1488f5304426eedeb6d97e905" +source = "git+https://github.com/context-graph-ai/contextdb?rev=56dccfdc905653370a7a6775d0998b1053e5926a#56dccfdc905653370a7a6775d0998b1053e5926a" dependencies = [ "contextdb-core", "contextdb-parser", @@ -1025,7 +1027,7 @@ dependencies = [ [[package]] name = "contextdb-redb" version = "4.1.0" -source = "git+https://github.com/context-graph-ai/contextdb?rev=bc9cce955eb85fb1488f5304426eedeb6d97e905#bc9cce955eb85fb1488f5304426eedeb6d97e905" +source = "git+https://github.com/context-graph-ai/contextdb?rev=56dccfdc905653370a7a6775d0998b1053e5926a#56dccfdc905653370a7a6775d0998b1053e5926a" dependencies = [ "libc", ] @@ -1033,7 +1035,7 @@ dependencies = [ [[package]] name = "contextdb-relational" version = "1.0.0" -source = "git+https://github.com/context-graph-ai/contextdb?rev=bc9cce955eb85fb1488f5304426eedeb6d97e905#bc9cce955eb85fb1488f5304426eedeb6d97e905" +source = "git+https://github.com/context-graph-ai/contextdb?rev=56dccfdc905653370a7a6775d0998b1053e5926a#56dccfdc905653370a7a6775d0998b1053e5926a" dependencies = [ "contextdb-core", "contextdb-tx", @@ -1044,7 +1046,7 @@ dependencies = [ [[package]] name = "contextdb-server" version = "1.0.0" -source = "git+https://github.com/context-graph-ai/contextdb?rev=bc9cce955eb85fb1488f5304426eedeb6d97e905#bc9cce955eb85fb1488f5304426eedeb6d97e905" +source = "git+https://github.com/context-graph-ai/contextdb?rev=56dccfdc905653370a7a6775d0998b1053e5926a#56dccfdc905653370a7a6775d0998b1053e5926a" dependencies = [ "blake3", "clap", @@ -1072,7 +1074,7 @@ dependencies = [ [[package]] name = "contextdb-tx" version = "1.0.0" -source = "git+https://github.com/context-graph-ai/contextdb?rev=bc9cce955eb85fb1488f5304426eedeb6d97e905#bc9cce955eb85fb1488f5304426eedeb6d97e905" +source = "git+https://github.com/context-graph-ai/contextdb?rev=56dccfdc905653370a7a6775d0998b1053e5926a#56dccfdc905653370a7a6775d0998b1053e5926a" dependencies = [ "contextdb-core", "parking_lot", @@ -1081,7 +1083,7 @@ dependencies = [ [[package]] name = "contextdb-vector" version = "1.0.0" -source = "git+https://github.com/context-graph-ai/contextdb?rev=bc9cce955eb85fb1488f5304426eedeb6d97e905#bc9cce955eb85fb1488f5304426eedeb6d97e905" +source = "git+https://github.com/context-graph-ai/contextdb?rev=56dccfdc905653370a7a6775d0998b1053e5926a#56dccfdc905653370a7a6775d0998b1053e5926a" dependencies = [ "anndists", "contextdb-core", @@ -1284,9 +1286,9 @@ dependencies = [ [[package]] name = "ctutils" -version = "0.4.2" +version = "0.4.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7d5515a3834141de9eafb9717ad39eea8247b5674e6066c404e8c4b365d2a29e" +checksum = "38d28fb6aa7dd680b3ffd1a3e64e115c177bdba63c0ae7f9cd48129ff05c4fa0" dependencies = [ "cmov", ] @@ -1752,7 +1754,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -3836,7 +3838,7 @@ version = "0.50.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" dependencies = [ - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -4877,7 +4879,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -4935,7 +4937,7 @@ dependencies = [ "security-framework", "security-framework-sys", "webpki-root-certs", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -4946,9 +4948,9 @@ checksum = "f87165f0995f63a9fbeea62b64d10b4d9d8e78ec6d7d51fb2125fda7bb36788f" [[package]] name = "rustls-webpki" -version = "0.103.15" +version = "0.103.16" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f3c3cf1d8b1e7d4927e2d154c3fcb02979afb9939629c62cd9048d4f07b60ac2" +checksum = "4443b1b6d39d7c663139942cc4558262562d2c84123bb6282ef177009ca8b281" dependencies = [ "ring", "rustls-pki-types", @@ -5048,7 +5050,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5b55fb86dfd3a2f5f76ea78310a88f96c4ea21a3031f8d212443d56123fd0521" dependencies = [ "libc", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -5512,7 +5514,7 @@ dependencies = [ "getrandom 0.4.2", "once_cell", "rustix", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -5723,9 +5725,9 @@ dependencies = [ [[package]] name = "tokio" -version = "1.52.3" +version = "1.54.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8fc7f01b389ac15039e4dc9531aa973a135d7a4135281b12d7c1bc79fd57fffe" +checksum = "b52fd3255258d26fac7264ab5a76b53c13367136dc011d3d6ad4f8ed070396a3" dependencies = [ "bytes", "libc", @@ -6442,18 +6444,18 @@ dependencies = [ [[package]] name = "webpki-root-certs" -version = "1.0.8" +version = "1.0.10" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0d46a5a140e6f7afeccd8eae97eff335163939eac8b929834875168b29b3d267" +checksum = "58ec80e73fd8baf57bf0d9615554c75348bda990d6d5bbeab964dd068319fecb" dependencies = [ "rustls-pki-types", ] [[package]] name = "webpki-roots" -version = "1.0.7" +version = "1.0.10" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52f5ee44c96cf55f1b349600768e3ece3a8f26010c05265ab73f945bb1a2eb9d" +checksum = "1c1c8e20c8a38fc7d65abfe74eb7280e4aa667e8eec8bc84996d2874810748d9" dependencies = [ "rustls-pki-types", ] @@ -6507,7 +6509,7 @@ version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] diff --git a/Cargo.toml b/Cargo.toml index dcb918a..26fdee2 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -44,11 +44,11 @@ candle-core = { path = "vendor/candle-core", version = "=0.10.2" } candle-nn = { path = "vendor/candle-nn", version = "=0.10.2" } candle-transformers = { path = "vendor/candle-transformers", version = "=0.10.2" } criterion = { version = "0.8", features = ["html_reports"] } -contextdb-core = { git = "https://github.com/context-graph-ai/contextdb", rev = "bc9cce955eb85fb1488f5304426eedeb6d97e905" } -contextdb-engine = { git = "https://github.com/context-graph-ai/contextdb", rev = "bc9cce955eb85fb1488f5304426eedeb6d97e905" } -contextdb-parser = { git = "https://github.com/context-graph-ai/contextdb", rev = "bc9cce955eb85fb1488f5304426eedeb6d97e905" } -contextdb-tx = { git = "https://github.com/context-graph-ai/contextdb", rev = "bc9cce955eb85fb1488f5304426eedeb6d97e905" } -contextdb-vector = { git = "https://github.com/context-graph-ai/contextdb", rev = "bc9cce955eb85fb1488f5304426eedeb6d97e905" } +contextdb-core = { git = "https://github.com/context-graph-ai/contextdb", rev = "56dccfdc905653370a7a6775d0998b1053e5926a" } +contextdb-engine = { git = "https://github.com/context-graph-ai/contextdb", rev = "56dccfdc905653370a7a6775d0998b1053e5926a" } +contextdb-parser = { git = "https://github.com/context-graph-ai/contextdb", rev = "56dccfdc905653370a7a6775d0998b1053e5926a" } +contextdb-tx = { git = "https://github.com/context-graph-ai/contextdb", rev = "56dccfdc905653370a7a6775d0998b1053e5926a" } +contextdb-vector = { git = "https://github.com/context-graph-ai/contextdb", rev = "56dccfdc905653370a7a6775d0998b1053e5926a" } safetensors = "0.7.0" sha2 = "0.10.9" tempfile = "3.23.0" diff --git a/README.md b/README.md index 519c6b9..51fd7cb 100644 --- a/README.md +++ b/README.md @@ -57,9 +57,9 @@ Use the [download guide](docs/download.md) for wget, receipts and provenance ver The pinned toolchain in `rust-toolchain.toml` is Rust **1.94.1**. -ContextDB is pinned to `bc9cce955eb85fb1488f5304426eedeb6d97e905` in the manifests and lockfile. +ContextDB is pinned to `56dccfdc905653370a7a6775d0998b1053e5926a` in the manifests and lockfile. -Linux is required by `PR checks`; macOS runs in the separate, nonblocking `macOS advisory` workflow. See the [macOS qualification follow-up](.github/CI.md#macos-qualification-follow-up). +Linux is required by `PR checks`; macOS runs in the separate, nonblocking `macOS CI (advisory)` workflow. See the [macOS qualification follow-up](.github/CI.md#macos-qualification-follow-up). The default local Nomic model download is 546,938,168 bytes (about 547 MB). diff --git a/crates/cg-catalog/AGENTS.md b/crates/cg-catalog/AGENTS.md index 867e304..9edccd3 100644 --- a/crates/cg-catalog/AGENTS.md +++ b/crates/cg-catalog/AGENTS.md @@ -44,5 +44,10 @@ Ownership boundaries above are review-only architectural constraints unless a gu - Hosted requests follow the attached provider contract: `tests/acceptance/generic_llm_endpoint.rs::wire_schema_tracks_the_attached_contract_not_a_constant`. +Test coordination uses `with_lock_wait_observer_for_test` behind `test-support` to observe an +actual refused lock attempt before blocking acquisition. Keep catalog lock ownership here; +fixtures wait on that event and explicit HTTP-server completion. Subscription negatives still +need a substrate dispatch-drain receipt; a timed observation window is not a completion seam. + Fast check: `cargo test -p cg-catalog --test acceptance`. Full gate: `scripts/gate.sh` at repository root. diff --git a/crates/cg-catalog/Cargo.toml b/crates/cg-catalog/Cargo.toml index 535c7fc..46f1f73 100644 --- a/crates/cg-catalog/Cargo.toml +++ b/crates/cg-catalog/Cargo.toml @@ -14,6 +14,8 @@ edition = "2024" [features] default = [] +# Private fixture controls, enabled only by this crate's own dev dependency. +test-support = [] embeddinggemma-test-support = [] [lib] @@ -41,6 +43,7 @@ uuid = { workspace = true, features = ["serde", "v4", "v7"] } which.workspace = true [dev-dependencies] +cg-catalog = { path = ".", features = ["test-support"] } candle-embeddinggemma.workspace = true chrono.workspace = true contextdb-core.workspace = true diff --git a/crates/cg-catalog/src/lib.rs b/crates/cg-catalog/src/lib.rs index e50da05..d06f57e 100644 --- a/crates/cg-catalog/src/lib.rs +++ b/crates/cg-catalog/src/lib.rs @@ -335,6 +335,8 @@ pub struct Catalog { cache_root: PathBuf, scope_key: String, progress_observer: Option>, + #[cfg(feature = "test-support")] + lock_wait_observer: Option>, } impl fmt::Debug for Catalog { @@ -501,6 +503,24 @@ impl Catalog { Self::open_with_scope_against_roots(root.clone(), root, scope_key) } + /// Observe a real refused lock attempt before its blocking acquire. + #[cfg(feature = "test-support")] + #[doc(hidden)] + pub fn with_lock_wait_observer_for_test( + mut self, + observer: impl Fn() + Send + Sync + 'static, + ) -> Self { + self.lock_wait_observer = Some(Arc::new(observer)); + self + } + + fn observe_lock_wait_for_test(&self) { + #[cfg(feature = "test-support")] + if let Some(observer) = &self.lock_wait_observer { + observer(); + } + } + /// Opens with the two locations held apart: `data_root` holds the registrations an operator /// manages, `cache_root` holds the artifact bytes a fetch downloads. An operator who points /// `CG_DATA_ROOT` and `CG_MODEL_CACHE_ROOT` at different directories gets this constructor, @@ -516,6 +536,8 @@ impl Catalog { cache_root: cache_root.into(), scope_key: scope_key.into(), progress_observer: None, + #[cfg(feature = "test-support")] + lock_wait_observer: None, }) } @@ -530,6 +552,8 @@ impl Catalog { cache_root: root, scope_key: scope_key.into(), progress_observer: None, + #[cfg(feature = "test-support")] + lock_wait_observer: None, }; catalog.write_registered_entries(&entries)?; Ok(catalog) @@ -871,6 +895,7 @@ impl Catalog { let lock_existed_before = self.model_dir(id).join(".lock").exists(); let (complete_before, accepted, waited) = { let (_lock, waited) = lock_artifact_dir_with_wait(id, &self.model_dir(id), || { + self.observe_lock_wait_for_test(); // Keep first-touch progress visible while an Apache-model fetch waits for // the cache lock. License-gated fetches must first inspect acceptance. if !spec.requires_manual_acceptance { @@ -2277,7 +2302,11 @@ impl Catalog { } fn lock_model(&self, id: &str) -> Result { - lock_artifact_dir(id, &self.model_dir(id)) + lock_artifact_dir_with_wait(id, &self.model_dir(id), || { + self.observe_lock_wait_for_test(); + Ok(()) + }) + .map(|(file, _)| file) } fn cache_dirs(&self) -> Result> { diff --git a/crates/cg-catalog/tests/acceptance/air_gap.rs b/crates/cg-catalog/tests/acceptance/air_gap.rs index 3d94148..b8e6a7f 100644 --- a/crates/cg-catalog/tests/acceptance/air_gap.rs +++ b/crates/cg-catalog/tests/acceptance/air_gap.rs @@ -27,6 +27,7 @@ fn stub_base_url(server: &tiny_http::Server) -> String { } struct CorruptFixtureServer { + server: Arc, shutdown: Arc, server_thread: Option>, } @@ -34,6 +35,7 @@ struct CorruptFixtureServer { impl CorruptFixtureServer { fn stop_and_join(&mut self) -> std::thread::Result<()> { self.shutdown.store(true, Ordering::SeqCst); + self.server.unblock(); match self.server_thread.take() { Some(server_thread) => server_thread.join(), None => Ok(()), @@ -78,7 +80,7 @@ fn spawn_eviction_fetch_child(test_name: &str, cache_root: &std::path::Path) -> .expect("eviction fetch child spawns") } -fn first_stderr_line_within(child: &mut Child, timeout: std::time::Duration) -> Option { +fn first_stderr_line_within(child: &mut Child, _timeout: std::time::Duration) -> Option { let stderr = child.stderr.take().expect("stderr piped"); let (sender, receiver) = mpsc::channel(); std::thread::spawn(move || { @@ -86,10 +88,7 @@ fn first_stderr_line_within(child: &mut Child, timeout: std::time::Duration) -> let result = BufReader::new(stderr).read_line(&mut line).map(|_| line); let _ = sender.send(result); }); - let line = receiver - .recv_timeout(timeout) - .ok() - .and_then(std::result::Result::ok); + let line = receiver.recv().ok().and_then(std::result::Result::ok); if line.is_none() { let _ = child.kill(); } @@ -442,12 +441,13 @@ fn air_gap_pre_populated_cache_reads_with_zero_outbound_network_on_first_load() write_source_override(temp.path(), NOMIC_ID, &base_url, NOMIC_REVISION); let outbound_request_count = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); let counter = std::sync::Arc::clone(&outbound_request_count); - let server_thread = std::thread::spawn(move || { - if let Ok(Some(request)) = server.recv_timeout(std::time::Duration::from_millis(150)) { - counter.fetch_add(1, std::sync::atomic::Ordering::SeqCst); - let _ = request.respond(tiny_http::Response::from_string("unexpected")); - } - }); + let server_thread = + crate::common::FixtureServer::from_server(Arc::new(server), move |server| { + while let Ok(request) = server.recv() { + counter.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + let _ = request.respond(tiny_http::Response::from_string("unexpected")); + } + }); let result = catalog_at(temp.path(), "host-A") .fetch(None, NOMIC_ID, OutputMode::Interactive) .expect("fetch returns"); @@ -589,12 +589,13 @@ fn air_gap_with_partial_cache_and_no_network_returns_typed_artifact_missing_erro write_source_override(temp.path(), NOMIC_ID, &base_url, NOMIC_REVISION); let outbound_request_count = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); let counter = std::sync::Arc::clone(&outbound_request_count); - let server_thread = std::thread::spawn(move || { - if let Ok(Some(request)) = server.recv_timeout(std::time::Duration::from_millis(150)) { - counter.fetch_add(1, std::sync::atomic::Ordering::SeqCst); - let _ = request.respond(tiny_http::Response::from_string("unexpected")); - } - }); + let server_thread = + crate::common::FixtureServer::from_server(Arc::new(server), move |server| { + while let Ok(request) = server.recv() { + counter.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + let _ = request.respond(tiny_http::Response::from_string("unexpected")); + } + }); let result = catalog_at(temp.path(), "host-A").fetch(None, NOMIC_ID, OutputMode::Interactive); assert!(matches!( result, @@ -625,12 +626,13 @@ fn air_gap_with_artifact_only_partial_cache_fails_before_source_fetch_even_witho write_source_override(temp.path(), NOMIC_ID, &base_url, NOMIC_REVISION); let outbound_request_count = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); let counter = std::sync::Arc::clone(&outbound_request_count); - let server_thread = std::thread::spawn(move || { - if let Ok(Some(request)) = server.recv_timeout(std::time::Duration::from_millis(150)) { - counter.fetch_add(1, std::sync::atomic::Ordering::SeqCst); - let _ = request.respond(tiny_http::Response::from_string("unexpected")); - } - }); + let server_thread = + crate::common::FixtureServer::from_server(Arc::new(server), move |server| { + while let Ok(request) = server.recv() { + counter.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + let _ = request.respond(tiny_http::Response::from_string("unexpected")); + } + }); let result = catalog_at(temp.path(), "host-A").fetch(None, NOMIC_ID, OutputMode::Interactive); assert!( matches!(result, Err(CgCatalogError::CacheIncomplete { .. })), @@ -764,10 +766,12 @@ fn hf_endpoint_returns_200_with_corrupted_bytes_sha_check_catches_it_catalog_ref let counter = std::sync::Arc::clone(&request_count); let shutdown = Arc::new(AtomicBool::new(false)); let server_shutdown = Arc::clone(&shutdown); + let server = Arc::new(server); + let worker_server = Arc::clone(&server); let server_thread = std::thread::spawn(move || { while !server_shutdown.load(Ordering::SeqCst) { - match server.recv_timeout(std::time::Duration::from_millis(50)) { - Ok(Some(request)) => { + match worker_server.recv() { + Ok(request) => { counter.fetch_add(1, std::sync::atomic::Ordering::SeqCst); let corrupt = b"corrupted model.safetensors bytes with matching HTTP success".to_vec(); @@ -780,12 +784,12 @@ fn hf_endpoint_returns_200_with_corrupted_bytes_sha_check_catches_it_catalog_ref ); let _ = request.respond(response); } - Ok(None) => {} Err(_) => break, } } }); let server_thread = CorruptFixtureServer { + server, shutdown, server_thread: Some(server_thread), }; diff --git a/crates/cg-catalog/tests/acceptance/ci_hardening.rs b/crates/cg-catalog/tests/acceptance/ci_hardening.rs index 496a717..2bb37e8 100644 --- a/crates/cg-catalog/tests/acceptance/ci_hardening.rs +++ b/crates/cg-catalog/tests/acceptance/ci_hardening.rs @@ -1946,8 +1946,16 @@ fn oss_purity_no_embeddinggemma_symbol_leakage_in_main_crate() { which::which("cargo-expand").is_ok(), "install cargo-expand so this guard can inspect the expanded context-graph library" ); + // The nested expansion inherits this test's Cargo environment. A build directory of its own + // keeps it from rerunning, under that environment, the build scripts the surrounding build + // and its compiler cache reuse. + let expand_target_dir = std::env::var_os("CARGO_TARGET_DIR") + .map(std::path::PathBuf::from) + .unwrap_or_else(|| workspace_root().join("target")) + .join("expand-check"); let output = Command::new("cargo") - .args(["expand", "--lib", "-p", "context-graph"]) + .args(["expand", "--lib", "-p", "context-graph", "--target-dir"]) + .arg(&expand_target_dir) .current_dir(workspace_root()) .output() .expect("cargo expand starts"); diff --git a/crates/cg-catalog/tests/acceptance/concurrency.rs b/crates/cg-catalog/tests/acceptance/concurrency.rs index 8d9a163..e66216c 100644 --- a/crates/cg-catalog/tests/acceptance/concurrency.rs +++ b/crates/cg-catalog/tests/acceptance/concurrency.rs @@ -1,11 +1,9 @@ +use crate::common::{ + FixtureServer, LockCheckpoint, catalog_with_child_lock_checkpoint, release_lock_checkpoint, +}; use std::fs; use std::process::{Command, Stdio}; -use std::sync::{ - Arc, Barrier, Condvar, Mutex, - atomic::{AtomicBool, Ordering}, - mpsc, -}; -use std::time::{Duration, Instant}; +use std::sync::{Arc, Barrier, Condvar, Mutex, atomic::Ordering, mpsc}; use cg_catalog::{FetchResult, OutputMode}; use context_graph::{Principal, TxId}; @@ -24,7 +22,7 @@ fn child_fetch_if_requested() -> bool { }; let root = std::path::PathBuf::from(root); let model_id = std::env::var("CG_CATALOG_CONCURRENCY_FETCH_MODEL").expect("model env"); - let catalog = catalog_at(root, "host-A"); + let catalog = catalog_with_child_lock_checkpoint(catalog_at(root, "host-A")); let result = catalog.fetch(None, &model_id, OutputMode::Interactive); match result { Ok(result) => { @@ -63,30 +61,32 @@ fn spawn_fetch_child( fn spawn_race_child( test_name: &str, root: &std::path::Path, -) -> (std::process::Child, std::path::PathBuf) { +) -> (std::process::Child, std::path::PathBuf, LockCheckpoint) { + static NEXT_RESULT: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(0); let result_file = root.join(format!( - ".race-result-{:?}-{}.txt", - std::thread::current().id(), - std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .expect("system time after epoch") - .as_nanos() + ".race-result-{}.txt", + NEXT_RESULT.fetch_add(1, Ordering::SeqCst) )); + let checkpoint = LockCheckpoint::new(); let child = Command::new(std::env::current_exe().expect("current test binary")) .arg("--exact") .arg(test_name) .env("CG_CATALOG_CONCURRENCY_FETCH_ROOT", root) .env("CG_CATALOG_CONCURRENCY_FETCH_MODEL", NOMIC_ID) .env("CG_CATALOG_CONCURRENCY_RESULT_FILE", &result_file) + .env("CG_CATALOG_TEST_LOCK_CHECKPOINT", checkpoint.path()) .stdout(Stdio::piped()) .stderr(Stdio::piped()) .spawn() .expect("race child starts"); - (child, result_file) + (child, result_file, checkpoint) } -fn wait_for_fetch_result(child: (std::process::Child, std::path::PathBuf), label: &str) -> String { - let (child, result_file) = child; +fn wait_for_fetch_result( + child: (std::process::Child, std::path::PathBuf, LockCheckpoint), + label: &str, +) -> String { + let (child, result_file, _checkpoint) = child; let output = child.wait_with_output().expect("child output collected"); assert!( output.status.success(), @@ -118,12 +118,17 @@ fn accepted_by_human(record: &toml::Value) -> Option<&str> { .and_then(toml::Value::as_str) } -fn assert_child_blocked(child: &mut std::process::Child, label: &str) { - std::thread::sleep(Duration::from_millis(200)); +fn assert_child_blocked( + child: &mut std::process::Child, + label: &str, + checkpoint: &LockCheckpoint, +) -> std::os::unix::net::UnixStream { + let held = checkpoint.entered(); assert!( child.try_wait().expect("child status readable").is_none(), "{label} should still be blocked on the per-model lockfile" ); + held } fn respond_with_nomic_fixture(request: tiny_http::Request) { @@ -162,26 +167,24 @@ fn start_first_response_gated_nomic_server( mpsc::Receiver<()>, mpsc::Sender<()>, Arc, - std::thread::JoinHandle<()>, + FixtureServer, ) { - let server = tiny_http::Server::http("127.0.0.1:0").expect("stub server"); + let server = Arc::new(tiny_http::Server::http("127.0.0.1:0").expect("stub server")); let base_url = stub_base_url(&server); write_source_override(root, NOMIC_ID, &base_url, NOMIC_REVISION); let request_count = Arc::new(std::sync::atomic::AtomicUsize::new(0)); let counter = Arc::clone(&request_count); let (first_request_tx, first_request_rx) = mpsc::channel(); let (release_tx, release_rx) = mpsc::channel(); - let server_thread = std::thread::spawn(move || { + let server_thread = FixtureServer::from_server(server, move |server| { let mut first_release = Some(release_rx); - while let Ok(Some(request)) = server.recv_timeout(Duration::from_secs(5)) { + while let Ok(request) = server.recv() { counter.fetch_add(1, Ordering::SeqCst); if let Some(release_rx) = first_release.take() { let first_request_tx = first_request_tx.clone(); std::thread::spawn(move || { first_request_tx.send(()).expect("first request signalled"); - release_rx - .recv_timeout(Duration::from_secs(3)) - .expect("first response release signal"); + release_rx.recv().expect("first response release signal"); respond_with_nomic_fixture(request); }); } else { @@ -196,7 +199,7 @@ struct OverlapSourceProbe { nomic_model_requests: Arc, gemma_model_requests: Arc, overlap_gate: Arc<(Mutex, Condvar)>, - server_thread: std::thread::JoinHandle<()>, + server_thread: FixtureServer, } #[derive(Default)] @@ -204,7 +207,6 @@ struct ModelRequestOverlapState { active: usize, max_active: usize, overlap_observed: bool, - timed_out_waiting_for_overlap: bool, } fn wait_for_model_request_overlap(gate: &Arc<(Mutex, Condvar)>) { @@ -216,29 +218,15 @@ fn wait_for_model_request_overlap(gate: &Arc<(Mutex, C state.overlap_observed = true; cvar.notify_all(); } - let deadline = Instant::now() + Duration::from_secs(10); while !state.overlap_observed { - let now = Instant::now(); - if now >= deadline { - state.timed_out_waiting_for_overlap = true; - break; - } - let remaining = deadline.saturating_duration_since(now); - let (next, wait_result) = cvar - .wait_timeout(state, remaining) - .expect("overlap cvar wait"); - state = next; - if wait_result.timed_out() && !state.overlap_observed { - state.timed_out_waiting_for_overlap = true; - break; - } + state = cvar.wait(state).expect("overlap cvar wait"); } state.active -= 1; cvar.notify_all(); } fn start_overlap_gated_nomic_and_gemma_source_server(root: &std::path::Path) -> OverlapSourceProbe { - let server = tiny_http::Server::http("127.0.0.1:0").expect("stub server"); + let server = Arc::new(tiny_http::Server::http("127.0.0.1:0").expect("stub server")); let base_url = stub_base_url(&server); let (gemma_source_dir, _unique_token) = write_valid_mini_gemma_cache(&root.join(".valid-gemma-source")); @@ -264,8 +252,8 @@ fn start_overlap_gated_nomic_and_gemma_source_server(root: &std::path::Path) -> let nomic_counter = Arc::clone(&nomic_model_requests); let gemma_counter = Arc::clone(&gemma_model_requests); let server_overlap_gate = Arc::clone(&overlap_gate); - let server_thread = std::thread::spawn(move || { - while let Ok(Some(request)) = server.recv_timeout(Duration::from_secs(5)) { + let server_thread = FixtureServer::from_server(server, move |server| { + while let Ok(request) = server.recv() { let url = request.url().to_string(); let is_model = url.ends_with("model.safetensors"); if url.contains(GEMMA_ID) { @@ -340,48 +328,21 @@ fn concurrent_writers_to_same_model_dir_via_lockfile_serialize_one_winner_other_ let (first_request_rx, release_tx, request_count, server_thread) = start_first_response_gated_nomic_server(&race_root); let held_lock = hold_model_lock(&race_root, NOMIC_ID); - let barrier = Arc::new(Barrier::new(2)); - let first_delay = if iteration % 2 == 0 { - Duration::from_millis(0) + let mut first = spawn_race_child(test_name, &race_root); + let mut second = spawn_race_child(test_name, &race_root); + let first_held = assert_child_blocked(&mut first.0, "first writer", &first.2); + let second_held = assert_child_blocked(&mut second.0, "second writer", &second.2); + let (winner, loser) = if iteration % 2 == 0 { + (first_held, second_held) } else { - Duration::from_millis(30) + (second_held, first_held) }; - let second_delay = if iteration % 2 == 0 { - Duration::from_millis(30) - } else { - Duration::from_millis(0) - }; - let first_barrier = Arc::clone(&barrier); - let first_root = race_root.clone(); - let first_handle = std::thread::spawn(move || { - first_barrier.wait(); - std::thread::sleep(first_delay); - spawn_race_child(test_name, &first_root) - }); - let second_barrier = Arc::clone(&barrier); - let second_root = race_root.clone(); - let second_handle = std::thread::spawn(move || { - second_barrier.wait(); - std::thread::sleep(second_delay); - spawn_race_child(test_name, &second_root) - }); - let mut first = first_handle.join().expect("first race child spawned"); - let mut second = second_handle.join().expect("second race child spawned"); - assert_child_blocked( - &mut first.0, - "first same-model writer before parent release", - ); - assert_child_blocked( - &mut second.0, - "second same-model writer before parent release", - ); + release_lock_checkpoint(winner); drop(held_lock); - assert!( - first_request_rx - .recv_timeout(Duration::from_millis(700)) - .is_ok(), - "one writer should reach the configured source before publishing artifacts" - ); + first_request_rx + .recv() + .expect("selected winner enters its critical section"); + release_lock_checkpoint(loser); assert!( !race_root.join(NOMIC_ID).join("catalog.toml").exists(), "a writer blocked inside its first source response must not publish a manifest" @@ -488,10 +449,6 @@ fn concurrent_writers_to_different_model_dirs_do_not_block_each_other() { overlap_state.max_active, 2, "parallel two-model fetch should put both model.safetensors requests in flight together" ); - assert!( - !overlap_state.timed_out_waiting_for_overlap, - "one model.safetensors request waited without observing the other model in flight" - ); assert_eq!( parallel_probe.nomic_model_requests.load(Ordering::SeqCst), 1, @@ -527,9 +484,7 @@ fn concurrent_writer_crash_mid_write_leaves_lockfile_releasable_for_next_writer( "Completed", ); assert!( - first_request_rx - .recv_timeout(Duration::from_millis(700)) - .is_ok(), + first_request_rx.recv().is_ok(), "crashing writer should reach the configured source through the real fetch path" ); assert!( @@ -575,7 +530,7 @@ fn concurrent_acceptance_record_writes_serialize_via_lockfile_no_lost_writes_str record.artifact_sha = fixture_sha(&root.join(GEMMA_ID).join("model.safetensors")); record.accepted_at = if scope == "host-A" { TxId(1) } else { TxId(2) }; record.accepted_by = Principal::Human(format!("{scope}@example.test")); - catalog_at(root, &scope) + catalog_with_child_lock_checkpoint(catalog_at(root, &scope)) .record_acceptance(GEMMA_ID, record) .expect("child acceptance returns"); return; @@ -583,11 +538,14 @@ fn concurrent_acceptance_record_writes_serialize_via_lockfile_no_lost_writes_str let temp = tempfile::tempdir().expect("tempdir"); write_valid_mini_gemma_acceptance(temp.path(), "host-A"); let held_lock = hold_model_lock(temp.path(), GEMMA_ID); + let first_checkpoint = LockCheckpoint::new(); + let second_checkpoint = LockCheckpoint::new(); let mut first = Command::new(std::env::current_exe().expect("current test binary")) .arg("--exact") .arg("concurrency::concurrent_acceptance_record_writes_serialize_via_lockfile_no_lost_writes_strict_monotonic_accepted_at") .env("CG_CATALOG_ACCEPTANCE_CHILD_ROOT", temp.path()) .env("CG_CATALOG_ACCEPTANCE_CHILD_SCOPE", "host-A") + .env("CG_CATALOG_TEST_LOCK_CHECKPOINT", first_checkpoint.path()) .spawn() .expect("first acceptance child starts"); let mut second = Command::new(std::env::current_exe().expect("current test binary")) @@ -595,10 +553,14 @@ fn concurrent_acceptance_record_writes_serialize_via_lockfile_no_lost_writes_str .arg("concurrency::concurrent_acceptance_record_writes_serialize_via_lockfile_no_lost_writes_strict_monotonic_accepted_at") .env("CG_CATALOG_ACCEPTANCE_CHILD_ROOT", temp.path()) .env("CG_CATALOG_ACCEPTANCE_CHILD_SCOPE", "host-B") + .env("CG_CATALOG_TEST_LOCK_CHECKPOINT", second_checkpoint.path()) .spawn() .expect("second acceptance child starts"); - assert_child_blocked(&mut first, "first acceptance writer"); - assert_child_blocked(&mut second, "second acceptance writer"); + let first_held = assert_child_blocked(&mut first, "first acceptance writer", &first_checkpoint); + let second_held = + assert_child_blocked(&mut second, "second acceptance writer", &second_checkpoint); + release_lock_checkpoint(first_held); + release_lock_checkpoint(second_held); drop(held_lock); assert!(first.wait().expect("first child waits").success()); assert!(second.wait().expect("second child waits").success()); @@ -655,29 +617,28 @@ fn concurrent_read_during_write_observes_either_pre_write_state_or_post_write_st let (first_request_rx, release_tx, _request_count, server_thread) = start_first_response_gated_nomic_server(temp.path()); let seen = Arc::clone(&observations); - let done = Arc::new(AtomicBool::new(false)); - let done_reader = Arc::clone(&done); + let (waiting_tx, waiting_rx) = mpsc::channel(); + let (sample_tx, sample_rx) = mpsc::channel(); let reader = std::thread::spawn(move || { - loop { - let catalog = catalog_at(&root, "reader"); - let status = catalog - .list() - .expect("reader list returns") - .into_iter() - .find(|entry| entry.id == NOMIC_ID) - .map(|entry| format!("{:?}", entry.cache_status)) - .unwrap_or_else(|| "Missing".to_string()); - let artifact = root.join(NOMIC_ID).join("model.safetensors"); - let sha_ok = - artifact.exists() && fixture_sha(&artifact) == FIXTURE_NOMIC_MODEL_SAFETENSORS_SHA; - seen.lock() - .expect("reader observations") - .push((status, sha_ok)); - if done_reader.load(Ordering::SeqCst) { - break; - } - std::thread::sleep(std::time::Duration::from_millis(5)); - } + sample_rx + .recv() + .expect("writer entered its critical section"); + let catalog = catalog_at(&root, "reader").with_lock_wait_observer_for_test(move || { + waiting_tx.send(()).expect("reader reports lock contention"); + }); + let status = catalog + .list() + .expect("reader list returns") + .into_iter() + .find(|entry| entry.id == NOMIC_ID) + .map(|entry| format!("{:?}", entry.cache_status)) + .unwrap_or_else(|| "Missing".to_string()); + let artifact = root.join(NOMIC_ID).join("model.safetensors"); + let sha_ok = + artifact.exists() && fixture_sha(&artifact) == FIXTURE_NOMIC_MODEL_SAFETENSORS_SHA; + seen.lock() + .expect("reader observations") + .push((status, sha_ok)); }); let mut writer = spawn_fetch_child( "concurrency::concurrent_read_during_write_observes_either_pre_write_state_or_post_write_state_never_torn", @@ -686,15 +647,15 @@ fn concurrent_read_during_write_observes_either_pre_write_state_or_post_write_st "Completed", ); assert!( - first_request_rx - .recv_timeout(Duration::from_millis(700)) - .is_ok(), + first_request_rx.recv().is_ok(), "writer should enter source fetch before reader samples" ); - std::thread::sleep(Duration::from_millis(250)); + sample_tx.send(()).expect("start snapshot read"); + waiting_rx + .recv() + .expect("reader reached the writer-held snapshot lock"); release_tx.send(()).expect("release first source response"); assert!(writer.wait().expect("writer child waits").success()); - done.store(true, Ordering::SeqCst); reader.join().expect("reader joins"); server_thread.join().expect("server thread joins"); let observations = observations.lock().expect("reader observations"); diff --git a/crates/cg-catalog/tests/acceptance/generic_llm_credential_integrity.rs b/crates/cg-catalog/tests/acceptance/generic_llm_credential_integrity.rs index 53a0cef..5d905bd 100644 --- a/crates/cg-catalog/tests/acceptance/generic_llm_credential_integrity.rs +++ b/crates/cg-catalog/tests/acceptance/generic_llm_credential_integrity.rs @@ -1,8 +1,7 @@ use std::fs; use std::path::Path; -use std::sync::mpsc; +use std::sync::{Arc, mpsc}; use std::thread; -use std::time::Duration; use cg_catalog::{ CredentialSource, DEGRADED_MARKER, ModelConnectionCatalog, PrincipalAwareConfig, @@ -80,11 +79,12 @@ struct OneShotStub { base_url: String, received: mpsc::Receiver>, handle: thread::JoinHandle<()>, + server: Arc, } impl OneShotStub { fn openai_reply(reply: &str) -> Self { - let server = tiny_http::Server::http("127.0.0.1:0").expect("stub server starts"); + let server = Arc::new(tiny_http::Server::http("127.0.0.1:0").expect("stub server starts")); let base_url = crate::common::stub_base_url(&server); let body = serde_json::json!({ "choices": [{ @@ -95,8 +95,9 @@ impl OneShotStub { }) .to_string(); let (sender, received) = mpsc::channel(); + let worker_server = Arc::clone(&server); let handle = thread::spawn(move || { - let Some(request) = server.recv_timeout(Duration::from_secs(2)).ok().flatten() else { + let Some(request) = worker_server.recv().ok() else { let _ = sender.send(None); return; }; @@ -118,14 +119,13 @@ impl OneShotStub { base_url, received, handle, + server, } } fn join(self) -> Option { - let request = self - .received - .recv_timeout(Duration::from_secs(3)) - .unwrap_or(None); + self.server.unblock(); + let request = self.received.recv().unwrap_or(None); self.handle.join().expect("stub server joins"); request } diff --git a/crates/cg-catalog/tests/acceptance/generic_llm_endpoint.rs b/crates/cg-catalog/tests/acceptance/generic_llm_endpoint.rs index 8f1740d..06e942c 100644 --- a/crates/cg-catalog/tests/acceptance/generic_llm_endpoint.rs +++ b/crates/cg-catalog/tests/acceptance/generic_llm_endpoint.rs @@ -1,6 +1,7 @@ use std::collections::BTreeSet; use std::fs; use std::net::TcpListener; +use std::sync::Arc; use std::sync::mpsc; use std::thread; use std::time::{Duration, Instant}; @@ -64,29 +65,21 @@ struct StubServer { base_url: String, received: mpsc::Receiver>, handle: thread::JoinHandle<()>, - join_timeout: Duration, + server: Arc, + release: mpsc::Sender<()>, } impl StubServer { fn join(self) -> Option { - let recorded = self - .received - .recv_timeout(self.join_timeout) - .unwrap_or(None); + self.server.unblock(); + let _ = self.release.send(()); + let recorded = self.received.recv().unwrap_or(None); self.handle.join().expect("stub server thread joins"); recorded } fn receive_without_joining(self) -> Option { - let StubServer { - received, - handle, - join_timeout, - .. - } = self; - let recorded = received.recv_timeout(join_timeout).unwrap_or(None); - drop(handle); - recorded + self.join() } } @@ -94,57 +87,36 @@ struct SequenceStubServer { base_url: String, received: mpsc::Receiver>, handle: thread::JoinHandle<()>, - join_timeout: Duration, } impl SequenceStubServer { fn join(self) -> Vec { - let recorded = self - .received - .recv_timeout(self.join_timeout) - .unwrap_or_default(); + let recorded = self.received.recv().unwrap_or_default(); self.handle.join().expect("stub server thread joins"); recorded } } fn stub_server(status: u16, response_body: Value) -> StubServer { - stub_server_with_delay_and_wait( - status, - Some(response_body.to_string()), - Duration::ZERO, - Duration::from_secs(2), - ) + stub_server_with_held_response(status, Some(response_body.to_string()), false) } fn stub_server_with_raw_body(status: u16, response_body: &'static str) -> StubServer { - stub_server_with_delay_and_wait( - status, - Some(response_body.to_string()), - Duration::ZERO, - Duration::from_secs(2), - ) + stub_server_with_held_response(status, Some(response_body.to_string()), false) } fn stub_server_expecting_no_request(status: u16, response_body: Value) -> StubServer { - stub_server_with_delay_and_wait( - status, - Some(response_body.to_string()), - Duration::ZERO, - Duration::from_millis(250), - ) + stub_server_with_held_response(status, Some(response_body.to_string()), false) } fn stub_server_sequence(response_bodies: Vec) -> SequenceStubServer { let server = tiny_http::Server::http("127.0.0.1:0").expect("stub server starts"); let base_url = crate::common::stub_base_url(&server); - let expected_requests = response_bodies.len(); let (sender, received) = mpsc::channel(); let handle = thread::spawn(move || { let mut recorded = Vec::new(); for response_body in response_bodies { - let Some(mut request) = server.recv_timeout(Duration::from_secs(2)).ok().flatten() - else { + let Some(mut request) = server.recv().ok() else { break; }; recorded.push(capture_request(&mut request)); @@ -163,32 +135,33 @@ fn stub_server_sequence(response_bodies: Vec) -> SequenceStubServer { base_url, received, handle, - join_timeout: Duration::from_secs((expected_requests as u64 * 3).max(3)), } } fn silent_timeout_stub_server() -> StubServer { - stub_server_with_delay_and_wait(200, None, Duration::from_secs(30), Duration::from_secs(2)) + stub_server_with_held_response(200, None, true) } -fn stub_server_with_delay_and_wait( +fn stub_server_with_held_response( status: u16, response_body: Option, - response_delay: Duration, - request_wait: Duration, + hold_response: bool, ) -> StubServer { - let server = tiny_http::Server::http("127.0.0.1:0").expect("stub server starts"); + let server = Arc::new(tiny_http::Server::http("127.0.0.1:0").expect("stub server starts")); let base_url = crate::common::stub_base_url(&server); let (sender, received) = mpsc::channel(); + let (release, held) = mpsc::channel(); + let worker_server = Arc::clone(&server); let handle = thread::spawn(move || { - let Some(mut request) = server.recv_timeout(request_wait).ok().flatten() else { + let Some(mut request) = worker_server.recv().ok() else { let _ = sender.send(None); return; }; let recorded = capture_request(&mut request); let _ = sender.send(Some(recorded)); - if !response_delay.is_zero() { - thread::sleep(response_delay); + if hold_response { + held.recv() + .expect("parent finishes its hosted call before releasing the silent response"); } if let Some(response_body) = response_body { let content_type = @@ -205,7 +178,8 @@ fn stub_server_with_delay_and_wait( base_url, received, handle, - join_timeout: request_wait + Duration::from_secs(1), + server, + release, } } diff --git a/crates/cg-catalog/tests/acceptance/prefetch_first_touch.rs b/crates/cg-catalog/tests/acceptance/prefetch_first_touch.rs index 16eb336..95518b6 100644 --- a/crates/cg-catalog/tests/acceptance/prefetch_first_touch.rs +++ b/crates/cg-catalog/tests/acceptance/prefetch_first_touch.rs @@ -492,6 +492,7 @@ fn start_nomic_source_override(root: &Path) -> (Arc, CountedNomicFi } struct SourceFixtureServer { + server: Arc, shutdown: Arc, server_thread: Option>, } @@ -499,6 +500,7 @@ struct SourceFixtureServer { impl SourceFixtureServer { fn stop_and_join(&mut self) -> std::thread::Result<()> { self.shutdown.store(true, Ordering::SeqCst); + self.server.unblock(); match self.server_thread.take() { Some(server_thread) => server_thread.join(), None => Ok(()), @@ -576,10 +578,12 @@ fn serve_source_fixture_counted( ) -> SourceFixtureServer { let shutdown = Arc::new(AtomicBool::new(false)); let server_shutdown = Arc::clone(&shutdown); + let server = Arc::new(server); + let worker_server = Arc::clone(&server); let server_thread = std::thread::spawn(move || { while !server_shutdown.load(Ordering::SeqCst) { - match server.recv_timeout(std::time::Duration::from_millis(50)) { - Ok(Some(request)) => { + match worker_server.recv() { + Ok(request) => { request_count.fetch_add(1, Ordering::SeqCst); let url = request.url().to_string(); request_paths @@ -602,12 +606,12 @@ fn serve_source_fixture_counted( let bytes = fs::read(source_dir.join(filename)).expect("source fixture bytes"); let _ = request.respond(tiny_http::Response::from_data(bytes)); } - Ok(None) => {} Err(_) => break, } } }); SourceFixtureServer { + server, shutdown, server_thread: Some(server_thread), } @@ -629,10 +633,12 @@ fn start_nomic_source_override_without_revision( let paths = Arc::clone(&request_paths); let shutdown = Arc::new(AtomicBool::new(false)); let server_shutdown = Arc::clone(&shutdown); + let server = Arc::new(server); + let worker_server = Arc::clone(&server); let server_thread = std::thread::spawn(move || { while !server_shutdown.load(Ordering::SeqCst) { - match server.recv_timeout(std::time::Duration::from_millis(50)) { - Ok(Some(request)) => { + match worker_server.recv() { + Ok(request) => { count.fetch_add(1, Ordering::SeqCst); let url = request.url().to_string(); paths.lock().expect("request paths").push(url.clone()); @@ -651,7 +657,6 @@ fn start_nomic_source_override_without_revision( let _ = request.respond(tiny_http::Response::from_data(bytes)); } } - Ok(None) => {} Err(_) => break, } } @@ -660,6 +665,7 @@ fn start_nomic_source_override_without_revision( request_count, request_paths, SourceFixtureServer { + server, shutdown, server_thread: Some(server_thread), }, @@ -668,7 +674,7 @@ fn start_nomic_source_override_without_revision( fn first_stderr_line_while_child_is_in_flight( child: &mut Child, - timeout: std::time::Duration, + _timeout: std::time::Duration, ) -> String { let stderr = child.stderr.take().expect("stderr is piped"); let (sender, receiver) = mpsc::channel(); @@ -677,7 +683,7 @@ fn first_stderr_line_while_child_is_in_flight( let _ = BufReader::new(stderr).read_line(&mut line); let _ = sender.send(line); }); - let line = receiver.recv_timeout(timeout); + let line = receiver.recv(); if line.is_err() { let _ = child.kill(); } @@ -843,9 +849,9 @@ fn first_touch_lazy_fetch_observer_event_payload_carries_resolved_source_and_byt catalog.fetch(None, NOMIC_ID, OutputMode::Interactive) }); let started = call_rx - .recv_timeout(Duration::from_secs(1)) + .recv() .expect("fetch thread reaches catalog.fetch call"); - let event = event_rx.recv_timeout(Duration::from_millis(250)); + let event = event_rx.recv(); assert!( event.is_ok(), "observer receives progress event before fetch can complete" diff --git a/crates/cg-catalog/tests/acceptance/swap.rs b/crates/cg-catalog/tests/acceptance/swap.rs index f42f51e..43d81e7 100644 --- a/crates/cg-catalog/tests/acceptance/swap.rs +++ b/crates/cg-catalog/tests/acceptance/swap.rs @@ -1,4 +1,5 @@ -use std::sync::{Arc, Mutex}; +use crate::common::{Collector, wait_for_len}; +use std::sync::Arc; use std::thread; use std::time::{Duration, Instant}; @@ -81,8 +82,8 @@ fn text_default(config: &toml::Value) -> Option<&str> { .and_then(toml::Value::as_str) } -fn collect_new_decisions(store: &Store) -> Arc>> { - let decisions = Arc::new(Mutex::new(Vec::new())); +fn collect_new_decisions(store: &Store) -> Arc> { + let decisions = Arc::new(Collector::new()); let seen = Arc::clone(&decisions); store .with_system_principal() @@ -96,18 +97,12 @@ fn collect_new_decisions(store: &Store) -> Arc>> { decisions } -fn wait_for_decisions(decisions: &Arc>>, expected: usize) -> Vec { - let deadline = Instant::now() + Duration::from_secs(2); - loop { - let snapshot = decisions.lock().expect("decision collector").clone(); - if snapshot.len() >= expected || Instant::now() >= deadline { - return snapshot; - } - thread::sleep(Duration::from_millis(10)); - } +fn wait_for_decisions(decisions: &Arc>, expected: usize) -> Vec { + wait_for_len(decisions, expected); + decisions.lock().expect("decision collector").clone() } -fn assert_no_decisions(decisions: &Arc>>) { +fn assert_no_decisions(decisions: &Arc>) { let deadline = Instant::now() + Duration::from_millis(250); while Instant::now() < deadline { assert!( diff --git a/crates/cg-catalog/tests/acceptance/sync_arrival.rs b/crates/cg-catalog/tests/acceptance/sync_arrival.rs index 5d89afa..51933fc 100644 --- a/crates/cg-catalog/tests/acceptance/sync_arrival.rs +++ b/crates/cg-catalog/tests/acceptance/sync_arrival.rs @@ -7,7 +7,6 @@ use std::sync::{ atomic::{AtomicBool, AtomicUsize, Ordering}, mpsc, }; -use std::time::Duration; use crate::common::{ CANONICAL_LICENSE_GATE_MESSAGE, GEMMA_ID, NOMIC_BYTES, NOMIC_ID, NOMIC_REVISION, catalog_at, @@ -116,8 +115,11 @@ fn sync_fetch_child_if_requested() -> bool { if let Some(progress_file) = std::env::var_os("CG_CATALOG_SYNC_FETCH_PROGRESS_FILE") { let progress_file = std::path::PathBuf::from(progress_file); catalog = catalog.with_progress_observer(move |_| { - std::fs::write(&progress_file, b"fetch-starting") - .expect("sync fetch progress file written"); + let mut stream = std::os::unix::net::UnixStream::connect(&progress_file) + .expect("sync fetch progress checkpoint"); + stream + .write_all(&[1]) + .expect("sync fetch progress announced"); }); } let result = catalog.resolve_for_sync_arrival(&store, model_id, OutputMode::Headless); @@ -196,18 +198,22 @@ fn assert_child_success(child: Child, label: &str) { ); } -fn wait_for_child_progress_file(child: &mut Child, progress_file: &std::path::Path, label: &str) { - let deadline = std::time::Instant::now() + Duration::from_secs(30); - while !progress_file.exists() { - if let Some(status) = child.try_wait().expect("child status readable") { - panic!("{label} exited before reaching fetch progress path: {status}"); - } - assert!( - std::time::Instant::now() < deadline, - "{label} did not reach fetch progress path before timeout" - ); - std::thread::sleep(Duration::from_millis(5)); - } +fn wait_for_child_progress( + child: &mut Child, + listener: &std::os::unix::net::UnixListener, + label: &str, +) { + use std::io::Read; + let (mut stream, _) = listener.accept().expect("child progress checkpoint"); + let mut signal = [0]; + stream + .read_exact(&mut signal) + .expect("child progress announcement"); + assert_eq!(signal, [1]); + assert!( + child.try_wait().expect("child status readable").is_none(), + "{label} remains held by the winner's lock" + ); } fn respond_with_nomic_fixture(request: tiny_http::Request) { @@ -221,6 +227,7 @@ fn respond_with_nomic_fixture(request: tiny_http::Request) { } struct FirstResponseGatedNomicServer { + server: Arc, first_request_rx: mpsc::Receiver<()>, release_tx: mpsc::Sender<()>, shutdown: Arc, @@ -230,6 +237,7 @@ struct FirstResponseGatedNomicServer { impl FirstResponseGatedNomicServer { fn stop_and_join(&mut self) -> std::thread::Result<()> { self.shutdown.store(true, Ordering::SeqCst); + self.server.unblock(); let _ = self.release_tx.send(()); match self.server_thread.take() { Some(server_thread) => server_thread.join(), @@ -258,19 +266,19 @@ fn start_first_response_gated_nomic_server( let (release_tx, release_rx) = mpsc::channel(); let shutdown = Arc::new(AtomicBool::new(false)); let server_shutdown = Arc::clone(&shutdown); + let server = Arc::new(server); + let worker_server = Arc::clone(&server); let server_thread = std::thread::spawn(move || { let mut first_release = Some(release_rx); let mut responder_threads = Vec::new(); while !server_shutdown.load(Ordering::SeqCst) { - match server.recv_timeout(Duration::from_millis(50)) { - Ok(Some(request)) => { + match worker_server.recv() { + Ok(request) => { if let Some(release_rx) = first_release.take() { let first_request_tx = first_request_tx.clone(); responder_threads.push(std::thread::spawn(move || { first_request_tx.send(()).expect("first request signalled"); - release_rx - .recv_timeout(Duration::from_secs(30)) - .expect("first response release signal"); + release_rx.recv().expect("first response release signal"); respond_with_nomic_fixture(request); })); } else { @@ -279,7 +287,6 @@ fn start_first_response_gated_nomic_server( })); } } - Ok(None) => {} Err(_) => break, } } @@ -290,6 +297,7 @@ fn start_first_response_gated_nomic_server( } }); FirstResponseGatedNomicServer { + server, first_request_rx, release_tx, shutdown, @@ -298,6 +306,7 @@ fn start_first_response_gated_nomic_server( } struct CountedNomicSourceOverride { + server: Arc, request_count: Arc, shutdown: Arc, server_thread: std::thread::JoinHandle<()>, @@ -306,6 +315,7 @@ struct CountedNomicSourceOverride { impl CountedNomicSourceOverride { fn shutdown_and_join(self) { self.shutdown.store(true, Ordering::SeqCst); + self.server.unblock(); self.server_thread .join() .expect("counted source server joins"); @@ -322,19 +332,21 @@ fn start_counted_nomic_source_override(roots: &[&std::path::Path]) -> CountedNom let shutdown = Arc::new(AtomicBool::new(false)); let counter = Arc::clone(&request_count); let server_shutdown = Arc::clone(&shutdown); + let server = Arc::new(server); + let worker_server = Arc::clone(&server); let server_thread = std::thread::spawn(move || { while !server_shutdown.load(Ordering::SeqCst) { - match server.recv_timeout(Duration::from_millis(50)) { - Ok(Some(request)) => { + match worker_server.recv() { + Ok(request) => { counter.fetch_add(1, Ordering::SeqCst); respond_with_nomic_fixture(request); } - Ok(None) => {} Err(_) => break, } } }); CountedNomicSourceOverride { + server, request_count, shutdown, server_thread, @@ -458,13 +470,13 @@ fn sync_arrival_headless_mode_produces_observation_identical_to_interactive_mode "Completed", ); assert!( - gated_server - .first_request_rx - .recv_timeout(Duration::from_secs(30)) - .is_ok(), + gated_server.first_request_rx.recv().is_ok(), "first headless resolver should reach the configured source before publishing artifacts" ); - let blocked_progress_file = lock_temp.path().join(".blocked-sync-fetch-started"); + let progress_dir = tempfile::tempdir_in("/tmp").expect("short socket directory"); + let blocked_progress_file = progress_dir.path().join("progress.sock"); + let progress_listener = + std::os::unix::net::UnixListener::bind(&blocked_progress_file).expect("progress listener"); let mut blocked = spawn_sync_fetch_child_with_progress_file( "sync_arrival::sync_arrival_headless_mode_produces_observation_identical_to_interactive_mode_observation_single_producer_lockfile_serialises_writers", lock_temp.path(), @@ -472,7 +484,7 @@ fn sync_arrival_headless_mode_produces_observation_identical_to_interactive_mode "AlreadyCachedByConcurrentWinner", &blocked_progress_file, ); - wait_for_child_progress_file(&mut blocked, &blocked_progress_file, "blocked sync child"); + wait_for_child_progress(&mut blocked, &progress_listener, "blocked sync child"); assert!( blocked.try_wait().expect("child status readable").is_none(), "headless sync-arrival should block on the same per-model lockfile as interactive fetch" @@ -501,12 +513,13 @@ fn sync_arrival_headless_for_license_gated_model_defers_at_receiver_emits_observ ); let outbound_request_count = Arc::new(AtomicUsize::new(0)); let counter = Arc::clone(&outbound_request_count); - let server_thread = std::thread::spawn(move || { - if let Ok(Some(request)) = server.recv_timeout(Duration::from_millis(250)) { - counter.fetch_add(1, Ordering::SeqCst); - let _ = request.respond(tiny_http::Response::from_string("unexpected gated fetch")); - } - }); + let server_thread = + crate::common::FixtureServer::from_server(Arc::new(server), move |server| { + while let Ok(request) = server.recv() { + counter.fetch_add(1, Ordering::SeqCst); + let _ = request.respond(tiny_http::Response::from_string("unexpected gated fetch")); + } + }); let (_store_temp, store) = separate_store(); let result = catalog_at(temp.path(), "host-A").resolve_for_sync_arrival( &store, @@ -557,12 +570,13 @@ fn sync_arrival_headless_for_apache_model_on_pre_populated_cache_returns_already write_source_override(temp.path(), NOMIC_ID, &base_url, NOMIC_REVISION); let outbound_request_count = Arc::new(AtomicUsize::new(0)); let counter = Arc::clone(&outbound_request_count); - let server_thread = std::thread::spawn(move || { - if let Ok(Some(request)) = server.recv_timeout(Duration::from_millis(250)) { - counter.fetch_add(1, Ordering::SeqCst); - let _ = request.respond(tiny_http::Response::from_string("unexpected warm fetch")); - } - }); + let server_thread = + crate::common::FixtureServer::from_server(Arc::new(server), move |server| { + while let Ok(request) = server.recv() { + counter.fetch_add(1, Ordering::SeqCst); + let _ = request.respond(tiny_http::Response::from_string("unexpected warm fetch")); + } + }); let (_store_temp, store) = separate_store(); let result = catalog_at(temp.path(), "host-A") .resolve_for_sync_arrival(&store, NOMIC_ID, OutputMode::Headless) @@ -643,9 +657,9 @@ fn headless_mode_log_sink_receives_model_source_byte_size_payload_within_two_hun }) }); call_rx - .recv_timeout(Duration::from_secs(1)) + .recv() .expect("fetch thread reaches resolve_for_sync_arrival call"); - let line = line_rx.recv_timeout(Duration::from_millis(250)); + let line = line_rx.recv(); assert!( line.is_ok(), "headless tracing line should arrive before fetch completion" diff --git a/crates/cg-catalog/tests/acceptance/update_check.rs b/crates/cg-catalog/tests/acceptance/update_check.rs index e673c65..fb2ebf3 100644 --- a/crates/cg-catalog/tests/acceptance/update_check.rs +++ b/crates/cg-catalog/tests/acceptance/update_check.rs @@ -1,9 +1,9 @@ +use crate::common::{Collector, wait_for_len}; use std::collections::{BTreeMap, HashMap}; use std::sync::{ Arc, Mutex, atomic::{AtomicBool, AtomicUsize, Ordering}, }; -use std::time::{Duration, Instant}; use cg_catalog::{CG_CATALOG_CONTEXT_ID, CgCatalogError, UpdateCheckObservation, UpdatePolicy}; use context_graph::{ @@ -72,7 +72,7 @@ fn setup_revision_decision( EntityId, DecisionId, IntentionId, - Arc>>, + Arc>, ) { ensure_catalog_context(store); let context_id = *CG_CATALOG_CONTEXT_ID; @@ -110,7 +110,7 @@ fn setup_revision_decision( ..Default::default() }) .expect("decision"); - let payloads = Arc::new(Mutex::new(Vec::new())); + let payloads = Arc::new(Collector::new()); let seen = Arc::clone(&payloads); store .with_system_principal() @@ -130,6 +130,7 @@ fn stub_base_url(server: &tiny_http::Server) -> String { } struct UpdateFixtureServer { + server: Arc, shutdown: Arc, server_thread: Option>, } @@ -137,6 +138,7 @@ struct UpdateFixtureServer { impl UpdateFixtureServer { fn stop_and_join(&mut self) -> std::thread::Result<()> { self.shutdown.store(true, Ordering::SeqCst); + self.server.unblock(); match self.server_thread.take() { Some(server_thread) => server_thread.join(), None => Ok(()), @@ -161,10 +163,12 @@ fn serve_nomic_fixture_until_idle_counted( ) -> UpdateFixtureServer { let shutdown = Arc::new(AtomicBool::new(false)); let server_shutdown = Arc::clone(&shutdown); + let server = Arc::new(server); + let worker_server = Arc::clone(&server); let server_thread = std::thread::spawn(move || { while !server_shutdown.load(Ordering::SeqCst) { - match server.recv_timeout(std::time::Duration::from_millis(50)) { - Ok(Some(request)) => { + match worker_server.recv() { + Ok(request) => { request_count.fetch_add(1, Ordering::SeqCst); let url = request.url().to_string(); let fixture = fixture_dir("nomic"); @@ -180,12 +184,12 @@ fn serve_nomic_fixture_until_idle_counted( let response = tiny_http::Response::from_data(bytes); let _ = request.respond(response); } - Ok(None) => {} Err(_) => break, } } }); UpdateFixtureServer { + server, shutdown, server_thread: Some(server_thread), } @@ -207,10 +211,12 @@ fn serve_revision_source_until_idle_counted( ) -> UpdateFixtureServer { let shutdown = Arc::new(AtomicBool::new(false)); let server_shutdown = Arc::clone(&shutdown); + let server = Arc::new(server); + let worker_server = Arc::clone(&server); let server_thread = std::thread::spawn(move || { while !server_shutdown.load(Ordering::SeqCst) { - match server.recv_timeout(std::time::Duration::from_millis(50)) { - Ok(Some(request)) => { + match worker_server.recv() { + Ok(request) => { request_count.fetch_add(1, Ordering::SeqCst); let url = request.url().to_string(); request_urls @@ -224,12 +230,12 @@ fn serve_revision_source_until_idle_counted( }; let _ = request.respond(tiny_http::Response::from_data(bytes)); } - Ok(None) => {} Err(_) => break, } } }); UpdateFixtureServer { + server, shutdown, server_thread: Some(server_thread), } @@ -268,10 +274,12 @@ fn start_empty_revision_source( let paths = Arc::clone(&request_paths); let shutdown = Arc::new(AtomicBool::new(false)); let server_shutdown = Arc::clone(&shutdown); + let server = Arc::new(server); + let worker_server = Arc::clone(&server); let server_thread = std::thread::spawn(move || { while !server_shutdown.load(Ordering::SeqCst) { - match server.recv_timeout(std::time::Duration::from_millis(50)) { - Ok(Some(request)) => { + match worker_server.recv() { + Ok(request) => { counter.fetch_add(1, Ordering::SeqCst); let url = request.url().to_string(); paths.lock().expect("request paths").push(url.clone()); @@ -283,7 +291,6 @@ fn start_empty_revision_source( }; let _ = request.respond(response); } - Ok(None) => {} Err(_) => break, } } @@ -292,20 +299,15 @@ fn start_empty_revision_source( request_count, request_paths, UpdateFixtureServer { + server, shutdown, server_thread: Some(server_thread), }, ) } -fn wait_for_payload(payloads: &Arc>>) { - let start = Instant::now(); - while start.elapsed() < Duration::from_millis(500) { - if !payloads.lock().expect("payload collector").is_empty() { - return; - } - std::thread::sleep(Duration::from_millis(10)); - } +fn wait_for_payload(payloads: &Arc>) { + wait_for_len(payloads, 1); } #[test] diff --git a/crates/cg-catalog/tests/common/mod.rs b/crates/cg-catalog/tests/common/mod.rs index 4963cc3..941ff5b 100644 --- a/crates/cg-catalog/tests/common/mod.rs +++ b/crates/cg-catalog/tests/common/mod.rs @@ -177,6 +177,7 @@ pub fn stub_base_url(server: &tiny_http::Server) -> String { } pub struct CountedNomicFixtureServer { + server: std::sync::Arc, shutdown: std::sync::Arc, server_thread: Option>, } @@ -185,6 +186,7 @@ impl CountedNomicFixtureServer { fn stop_and_join(&mut self) -> std::thread::Result<()> { self.shutdown .store(true, std::sync::atomic::Ordering::SeqCst); + self.server.unblock(); match self.server_thread.take() { Some(server_thread) => server_thread.join(), None => Ok(()), @@ -208,10 +210,12 @@ pub fn serve_nomic_fixture_until_idle_counted( ) -> CountedNomicFixtureServer { let shutdown = std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)); let server_shutdown = std::sync::Arc::clone(&shutdown); + let server = std::sync::Arc::new(server); + let worker_server = std::sync::Arc::clone(&server); let server_thread = std::thread::spawn(move || { while !server_shutdown.load(std::sync::atomic::Ordering::SeqCst) { - match server.recv_timeout(std::time::Duration::from_millis(50)) { - Ok(Some(request)) => { + match worker_server.recv() { + Ok(request) => { request_count.fetch_add(1, std::sync::atomic::Ordering::SeqCst); let url = request.url().to_string(); let filename = ["tokenizer.json", "config.json", "REVISION"] @@ -222,12 +226,12 @@ pub fn serve_nomic_fixture_until_idle_counted( fs::read(fixture_dir("nomic").join(filename)).expect("fixture bytes"); let _ = request.respond(tiny_http::Response::from_data(bytes)); } - Ok(None) => {} Err(_) => break, } } }); CountedNomicFixtureServer { + server, shutdown, server_thread: Some(server_thread), } @@ -829,3 +833,155 @@ pub fn acceptance(scope_key: &str) -> AcceptanceRecord { scope_key: scope_key.to_string(), } } + +/// Collector writes and delivery waits share the same predicate lock. Dropping a +/// callback's collector guard announces its update without polling a clock. +pub struct Collector { + items: std::sync::Mutex>, + changed: std::sync::Condvar, +} + +pub struct CollectorGuard<'a, T> { + items: std::sync::MutexGuard<'a, Vec>, + changed: &'a std::sync::Condvar, +} + +impl std::ops::Deref for CollectorGuard<'_, T> { + type Target = Vec; + + fn deref(&self) -> &Self::Target { + &self.items + } +} + +impl std::ops::DerefMut for CollectorGuard<'_, T> { + fn deref_mut(&mut self) -> &mut Self::Target { + &mut self.items + } +} + +impl Drop for CollectorGuard<'_, T> { + fn drop(&mut self) { + self.changed.notify_all(); + } +} + +impl Collector { + pub fn new() -> Self { + Self { + items: std::sync::Mutex::new(Vec::new()), + changed: std::sync::Condvar::new(), + } + } + + pub fn lock(&self) -> std::result::Result, &'static str> { + self.items + .lock() + .map(|items| CollectorGuard { + items, + changed: &self.changed, + }) + .map_err(|_| "collector lock poisoned") + } +} + +pub fn wait_for_len(items: &std::sync::Arc>, expected: usize) { + let guard = items.items.lock().expect("collector"); + let _guard = items + .changed + .wait_while(guard, |entries| entries.len() < expected) + .expect("delivery notification"); +} + +/// A child reports an actual refused OS-lock attempt and waits for the parent. +/// Sockets live under /tmp so macOS's Unix socket path limit is respected. +pub struct LockCheckpoint { + directory: tempfile::TempDir, + listener: std::os::unix::net::UnixListener, +} + +impl LockCheckpoint { + pub fn new() -> Self { + let directory = tempfile::tempdir_in("/tmp").expect("checkpoint directory"); + let listener = std::os::unix::net::UnixListener::bind(directory.path().join("lock.sock")) + .expect("checkpoint listener"); + Self { + directory, + listener, + } + } + + pub fn path(&self) -> PathBuf { + self.directory.path().join("lock.sock") + } + + pub fn entered(&self) -> std::os::unix::net::UnixStream { + use std::io::Read; + let (mut stream, _) = self.listener.accept().expect("child lock checkpoint"); + let mut signal = [0]; + stream + .read_exact(&mut signal) + .expect("lock refusal announcement"); + assert_eq!(signal, [1]); + stream + } +} + +pub fn catalog_with_child_lock_checkpoint(catalog: Catalog) -> Catalog { + use std::io::{Read, Write}; + let Some(path) = std::env::var_os("CG_CATALOG_TEST_LOCK_CHECKPOINT") else { + return catalog; + }; + let once = std::sync::atomic::AtomicBool::new(false); + catalog.with_lock_wait_observer_for_test(move || { + if once.swap(true, std::sync::atomic::Ordering::SeqCst) { + return; + } + let mut stream = std::os::unix::net::UnixStream::connect(&path).expect("lock checkpoint"); + stream + .write_all(&[1]) + .expect("announce refused lock attempt"); + let mut release = [0]; + stream + .read_exact(&mut release) + .expect("release lock acquire"); + assert_eq!(release, [1]); + }) +} + +pub fn release_lock_checkpoint(mut stream: std::os::unix::net::UnixStream) { + use std::io::Write; + stream.write_all(&[1]).expect("release child lock acquire"); +} + +pub struct FixtureServer { + server: std::sync::Arc, + thread: Option>, +} + +impl FixtureServer { + pub fn from_server( + server: std::sync::Arc, + run: impl FnOnce(std::sync::Arc) + Send + 'static, + ) -> Self { + let worker = std::sync::Arc::clone(&server); + Self { + server, + thread: Some(std::thread::spawn(move || run(worker))), + } + } + + pub fn join(mut self) -> std::thread::Result<()> { + self.server.unblock(); + self.thread.take().expect("server thread").join() + } +} + +impl Drop for FixtureServer { + fn drop(&mut self) { + self.server.unblock(); + if let Some(thread) = self.thread.take() { + let _ = thread.join(); + } + } +} diff --git a/crates/cg-catalog/tests/support/shared_publication.rs b/crates/cg-catalog/tests/support/shared_publication.rs index 578bf0c..8d25908 100644 --- a/crates/cg-catalog/tests/support/shared_publication.rs +++ b/crates/cg-catalog/tests/support/shared_publication.rs @@ -179,7 +179,10 @@ fn assert_refused(root: &Path, marker: &str) { } fn assert_committed(root: &Path, revision: &str, expected_fetch: FetchResult) { - let catalog = catalog(root); + assert_committed_catalog(catalog(root), revision, expected_fetch); +} + +fn assert_committed_catalog(catalog: Catalog, revision: &str, expected_fetch: FetchResult) { let directory = catalog.model_dir(NOMIC_ID); assert_eq!( catalog.fetch(None, NOMIC_ID, OutputMode::Headless).unwrap(), @@ -237,7 +240,7 @@ fn run_child(arguments: &str) { return; } let mut stream = TcpStream::connect(fields[1]).unwrap(); - if fields[0] != "independent" { + if fields[0] != "independent" && fields[0] != "committed" { let lock = OpenOptions::new() .read(true) .write(true) @@ -257,20 +260,19 @@ fn run_child(arguments: &str) { assert_eq!(snapshot(&directory), before); } else if fields[0] == "refused" { assert_refused(root, ".catalog-publication"); + } else if fields[0] == "committed" { + let notify = stream.try_clone().unwrap(); + let signalled = std::sync::atomic::AtomicBool::new(false); + let catalog = catalog(root).with_lock_wait_observer_for_test(move || { + if !signalled.swap(true, std::sync::atomic::Ordering::SeqCst) { + // A separate probe cannot establish that fetch itself waited. + let mut notify = ¬ify; + notify.write_all(b"R").unwrap(); + } + }); + assert_committed_catalog(catalog, "new", FetchResult::AlreadyCachedByConcurrentWinner); } else { - assert_committed( - root, - if fields[0] == "independent" { - "old" - } else { - "new" - }, - if fields[0] == "independent" { - FetchResult::AlreadyCached - } else { - FetchResult::AlreadyCachedByConcurrentWinner - }, - ); + assert_committed(root, "old", FetchResult::AlreadyCached); } stream.write_all(b"D").unwrap(); } diff --git a/crates/cg-cli/AGENTS.md b/crates/cg-cli/AGENTS.md index 60543ba..e788bf3 100644 --- a/crates/cg-cli/AGENTS.md +++ b/crates/cg-cli/AGENTS.md @@ -48,6 +48,12 @@ Ownership boundaries above are review-only architectural constraints unless a gu - Presented help is the public command list; guard `tests/help_presents_exactly_the_public_command_list.rs`: `help_presents_exactly_the_public_command_list`. +Worker fixtures use feature-gated Unix socket notifications in `tests/event_support/mod.rs` +and explicit release events from `cg-node-fs`. Always reread authoritative request state after +a notification. Worker-lock release precedes queue acknowledgment: await acknowledgment before +asserting queue membership. The reader-held-store refusal test enables the existing zero-allowance +test seam through development dependencies; ordinary CLI builds retain their configured allowance. + Fast check: `cargo test -p context-graph-cli --test read_memory_policy`. Lifecycle proofs: `cargo nextest run -p context-graph-cli --features event-search-test-support`. Full gate: `scripts/gate.sh`; public operator contracts: [CLI reference](../../docs/cli.md). diff --git a/crates/cg-cli/Cargo.toml b/crates/cg-cli/Cargo.toml index 00d0068..9394f1c 100644 --- a/crates/cg-cli/Cargo.toml +++ b/crates/cg-cli/Cargo.toml @@ -22,7 +22,10 @@ path = "src/main.rs" # Private event-search lifecycle controls. The repository gate enables this explicitly so both # integration tests and their `CARGO_BIN_EXE_cg` fixture contain the same controls. Ordinary # build, release, and install commands do not enable it. -event-search-test-support = ["context-graph/test-support"] +event-search-test-support = ["context-graph/test-support", "cg-node-fs/test-support"] +# The store-contention fixtures exhaust the existing open-retry allowance through +# a private seam. The self dev-dependency enables it only for test builds. +store-contention-test-support = ["context-graph/test-support", "cg-node-fs/test-support"] # The `cg status` fault-injection door, off in every shipped build. It is turned on ONLY by this # package's dev-dependency on itself below, so `cargo test` compiles it and `cargo build` — debug # or release — does not: a build profile is not a test boundary, and a developer's ordinary debug @@ -43,7 +46,7 @@ cg-cli-conventions.workspace = true context-graph = { path = "../context-graph", features = ["fabric-sync"] } contextdb-core.workspace = true contextdb-engine.workspace = true -contextdb-server = { git = "https://github.com/context-graph-ai/contextdb", rev = "bc9cce955eb85fb1488f5304426eedeb6d97e905", features = ["iroh"] } +contextdb-server = { git = "https://github.com/context-graph-ai/contextdb", rev = "56dccfdc905653370a7a6775d0998b1053e5926a", features = ["iroh"] } fs4.workspace = true serde = { workspace = true, features = ["derive"] } serde_json.workspace = true @@ -53,16 +56,16 @@ toml.workspace = true uuid.workspace = true [dev-dependencies] -# This package on itself, so the fault-injection door above is compiled for TEST builds and only -# those. Cargo resolves dev-dependency features for `cargo test` and leaves them out of +# This package on itself, so the private fault-injection and contention doors compile only +# for test builds. Cargo resolves dev-dependency features for `cargo test` and leaves them out of # `cargo build`/`cargo install`, which is the boundary a profile check does not give. -context-graph-cli = { path = ".", features = ["status-fault-injection"] } +context-graph-cli = { path = ".", features = ["status-fault-injection", "store-contention-test-support"] } # The store-contention fixture (tests/support/store_contention.rs) parks a real direct reader # inside contextdb's shipped read-image hydration checkpoint, which is the only honest way to # produce reader contention: it exists only WHILE a committed image is loading. That checkpoint is # compiled behind contextdb-engine's `test-seams` feature. This is a test-only activation — the # `cg` binary's own dependency edge on contextdb-engine does not enable it. contextdb-engine = { workspace = true, features = ["test-seams"] } -contextdb-server = { git = "https://github.com/context-graph-ai/contextdb", rev = "bc9cce955eb85fb1488f5304426eedeb6d97e905", features = ["iroh", "test-seams"] } +contextdb-server = { git = "https://github.com/context-graph-ai/contextdb", rev = "56dccfdc905653370a7a6775d0998b1053e5926a", features = ["iroh", "test-seams"] } libc = "0.2" tempfile.workspace = true diff --git a/crates/cg-cli/src/history/mod.rs b/crates/cg-cli/src/history/mod.rs index 0e43970..d14327d 100644 --- a/crates/cg-cli/src/history/mod.rs +++ b/crates/cg-cli/src/history/mod.rs @@ -33,7 +33,10 @@ use super::OutputFormat; pub(crate) use forget::handle_forget; pub(crate) use import::handle_import; pub(crate) use read::{handle_find, handle_show, handle_status}; -#[cfg(feature = "event-search-test-support")] +#[cfg(any( + feature = "event-search-test-support", + feature = "store-contention-test-support" +))] pub(crate) use worker::apply_store_open_retry_allowance_for_test; pub(crate) use worker::{WORKER_COMMAND, handle_event_search_worker}; diff --git a/crates/cg-cli/src/history/worker.rs b/crates/cg-cli/src/history/worker.rs index 5fca2d9..d21367f 100644 --- a/crates/cg-cli/src/history/worker.rs +++ b/crates/cg-cli/src/history/worker.rs @@ -832,11 +832,13 @@ pub(crate) fn pause_before_import_acknowledgement() { { return; } - let _ = std::fs::write( + let _ = cg_node_fs::write_test_file_for_test( directory.join("import-assessment-complete"), b"writer released; refusal identities observed", ); - wait_for_test_release(|| directory.join("release-import-acknowledgement").exists()); + wait_for_test_release(directory, || { + directory.join("release-import-acknowledgement").exists() + }); } fn record_background_memory_refusal( @@ -1004,11 +1006,11 @@ fn retire_completed_selection( feature = "event-search-test-support" ))] if let Some(directory) = std::env::var_os("CG_EVENT_SEARCH_TEST_HANDOFF_BARRIER") { - let _ = std::fs::write( + let _ = cg_node_fs::write_test_file_for_test( Path::new(&directory).join("cleanup-read-active-time-ms"), config.read_active_time_ms.get().to_string(), ); - let _ = std::fs::write( + let _ = cg_node_fs::write_test_file_for_test( Path::new(&directory).join("cleanup-write-memory-limit"), cleanup_limit.get().to_string(), ); @@ -1188,11 +1190,13 @@ fn pause_before_selection_cleanup_for_test() { if !directory.join("pause-before-selection-cleanup").is_file() { return; } - let _ = std::fs::write( + let _ = cg_node_fs::write_test_file_for_test( directory.join("selection-cleanup-entered"), b"writer released", ); - wait_for_test_release(|| directory.join("release-selection-cleanup").exists()); + wait_for_test_release(directory, || { + directory.join("release-selection-cleanup").exists() + }); } /// Acknowledge finished requests by identity, leaving later appends and unfinished leased work. @@ -1272,10 +1276,13 @@ fn pause_after_catch_up_before_unlock() { }); let release = directory.join("release"); let numbered_release = directory.join(format!("release.{pass}")); - if std::fs::write(&entered, b"store closed; worker lock held\n").is_err() { + if cg_node_fs::write_test_file_for_test(&entered, b"store closed; worker lock held\n").is_err() + { return; } - wait_for_test_release(|| release.exists() || (numbered.is_some() && numbered_release.exists())); + wait_for_test_release(&directory, || { + release.exists() || (numbered.is_some() && numbered_release.exists()) + }); } /// Test-only rendezvous immediately before the worker asks the store for work. It lets the CLI @@ -1293,12 +1300,17 @@ fn pause_before_catch_up() { if directory.join("pause-after-work-assessment").is_file() { return; } + if signal_and_wait_for_test(&directory, "pre-catch-up.sock", &[1]) { + return; + } let entered = directory.join("pre-catch-up-entered"); let release = directory.join("pre-catch-up-release"); - if std::fs::write(&entered, b"worker lock held; work not yet assessed\n").is_err() { + if cg_node_fs::write_test_file_for_test(&entered, b"worker lock held; work not yet assessed\n") + .is_err() + { return; } - wait_for_test_release(|| release.exists()); + wait_for_test_release(&directory, || release.exists()); } /// Test-only rendezvous after the worker has read its queue but before it opens the store for the @@ -1316,7 +1328,7 @@ fn pause_after_work_assessment() { if !directory.join("pause-after-work-assessment").is_file() { return; } - if std::fs::write( + if cg_node_fs::write_test_file_for_test( directory.join("work-assessment-complete"), b"pending work assessed; projection not opened\n", ) @@ -1324,7 +1336,9 @@ fn pause_after_work_assessment() { { return; } - wait_for_test_release(|| directory.join("work-assessment-release").exists()); + wait_for_test_release(&directory, || { + directory.join("work-assessment-release").exists() + }); } /// Test-only rendezvous after foreground handoff owns or delivers every queued request, while the @@ -1341,7 +1355,7 @@ fn pause_after_foreground_handoff_before_refresh() { if !directory.join("pause-after-foreground-handoff").is_file() { return; } - if std::fs::write( + if cg_node_fs::write_test_file_for_test( directory.join("foreground-handoff-complete"), b"queued handoff settled; final status not read\n", ) @@ -1349,12 +1363,18 @@ fn pause_after_foreground_handoff_before_refresh() { { return; } - wait_for_test_release(|| directory.join("release-final-status").exists()); + wait_for_test_release(&directory, || { + directory.join("release-final-status").exists() + }); } /// Use the core's existing thread-local allowance override only in a private-feature binary. /// Production builds neither compile nor inspect this test marker. -#[cfg(feature = "event-search-test-support")] +#[cfg(any( + feature = "event-search-test-support", + feature = "store-contention-test-support" +))] +#[doc(hidden)] pub(crate) fn apply_store_open_retry_allowance_for_test() { let Some(directory) = std::env::var_os("CG_EVENT_SEARCH_TEST_PRE_CATCH_UP_BARRIER") else { return; @@ -1367,16 +1387,36 @@ pub(crate) fn apply_store_open_retry_allowance_for_test() { } } +/// An opted-in test drives this background turn through a synchronous signal. #[cfg(any( context_graph_internal_test_support, feature = "event-search-test-support" ))] -fn wait_for_test_release(mut released: impl FnMut() -> bool) { - // This is a test-only cross-process condition handshake. The integration harness owns the - // bounded test timeout; no local polling clock may disguise a synchronization timeout. - while !released() { - std::thread::park_timeout(Duration::from_millis(10)); +#[doc(hidden)] +fn signal_and_wait_for_test(directory: &Path, socket_name: &str, payload: &[u8]) -> bool { + use std::io::{Read, Write}; + use std::os::unix::net::UnixStream; + + let socket = directory.join(socket_name); + if !socket.exists() { + return false; } + let mut stream = UnixStream::connect(socket).expect("connect test rendezvous"); + stream.write_all(payload).expect("announce test checkpoint"); + let mut release = [0]; + stream + .read_exact(&mut release) + .expect("release test checkpoint"); + assert_eq!(release, [1]); + true +} + +#[cfg(any( + context_graph_internal_test_support, + feature = "event-search-test-support" +))] +fn wait_for_test_release(directory: impl AsRef, released: impl FnMut() -> bool) { + cg_node_fs::wait_for_test_release_for_test(Path::new(directory.as_ref()), released); } /// Pause before asking a transient condition again, without letting repeated rechecks spin a CPU. @@ -1446,7 +1486,7 @@ fn run_request( feature = "event-search-test-support" ))] if let Some(directory) = std::env::var_os("CG_EVENT_SEARCH_TEST_HANDOFF_BARRIER") { - let _ = std::fs::write( + let _ = cg_node_fs::write_test_file_for_test( std::path::PathBuf::from(directory).join("worker-store-write-memory-limit-bytes"), request.store.write_memory_limit_bytes.get().to_string(), ); @@ -1514,7 +1554,7 @@ fn run_observed_with_embedding_worker_policy( .and_then(|value| value.trim().parse::().ok()) .filter(|count| *count > 0) { - let _ = std::fs::write(&marker, (count - 1).to_string()); + let _ = cg_node_fs::write_test_file_for_test(&marker, (count - 1).to_string()); context_graph::test_support::fail_next_event_search_membership_capture_with_error_for_test( CgError::ReadActiveTimeExceeded { limit_ms: request.store.read_active_time_ms.get() }, ); @@ -1527,7 +1567,7 @@ fn run_observed_with_embedding_worker_policy( .and_then(|value| value.trim().parse::().ok()) .filter(|count| *count > 0) { - let _ = std::fs::write(&marker, (count - 1).to_string()); + let _ = cg_node_fs::write_test_file_for_test(&marker, (count - 1).to_string()); context_graph::test_support::fail_next_event_search_membership_capture_with_error_for_test( CgError::ReaderCoordinationUnavailable { operation: "prepare selected search work".into(), @@ -1577,11 +1617,13 @@ fn run_observed_with_embedding_worker_policy( if let Some(directory) = failure_record_barrier.clone() { context_graph::test_support::set_store_open_retry_allowance_for_test(Some(Duration::ZERO)); context_graph::test_support::before_next_event_search_failure_record_for_test(move || { - let _ = std::fs::write( + let _ = cg_node_fs::write_test_file_for_test( directory.join("failure-record-entered"), b"runner store closed", ); - wait_for_test_release(|| directory.join("failure-record-continue").is_file()); + wait_for_test_release(&directory, || { + directory.join("failure-record-continue").is_file() + }); }); } let observed = @@ -1737,7 +1779,7 @@ fn run_observed_with_embedding_worker_policy( if let Err(error) = &observed.result && let Some(directory) = std::env::var_os("CG_EVENT_SEARCH_TEST_HANDOFF_BARRIER") { - let _ = std::fs::write( + let _ = cg_node_fs::write_test_file_for_test( Path::new(&directory).join("projection-error"), format!("{error:?}"), ); @@ -1748,11 +1790,13 @@ fn run_observed_with_embedding_worker_policy( ))] if let Some(directory) = failure_record_barrier { context_graph::test_support::set_store_open_retry_allowance_for_test(None); - let _ = std::fs::write( + let _ = cg_node_fs::write_test_file_for_test( directory.join("failure-record-returned"), b"core disposition returned", ); - wait_for_test_release(|| directory.join("failure-writer-released").is_file()); + wait_for_test_release(&directory, || { + directory.join("failure-writer-released").is_file() + }); } (observed, admitted.into_inner()) } @@ -1783,7 +1827,7 @@ fn observe_write_memory_limit_for_test(context_id: ContextId, limit: NonZeroUsiz return; }; let directory = Path::new(&directory); - let _ = std::fs::write( + let _ = cg_node_fs::write_test_file_for_test( directory.join("worker-store-write-memory-limit-bytes"), limit.get().to_string(), ); @@ -1818,7 +1862,7 @@ fn run_observed_with_fixed_embedding_workers( .parse::() .ok() .filter(|count| *count > 0)?; - std::fs::write(marker, (count - 1).to_string()).ok()?; + cg_node_fs::write_test_file_for_test(marker, (count - 1).to_string()).ok()?; context_graph::test_support::fail_next_event_search_membership_capture_with_error_for_test( CgError::ReadActiveTimeExceeded { limit_ms: request.store.read_active_time_ms.get() }, ); @@ -1851,11 +1895,11 @@ fn run_observed_with_fixed_embedding_workers( .join("pause-after-foreground-read-timeout") .is_file() { - let _ = std::fs::write( + let _ = cg_node_fs::write_test_file_for_test( directory.join(format!("foreground-read-timeout-{count}")), b"runner released its store", ); - wait_for_test_release(|| { + wait_for_test_release(&directory, || { directory .join(format!("release-foreground-read-timeout-{count}")) .is_file() @@ -1911,6 +1955,13 @@ fn observe_embedding_workers_batch_for_test(root: &Path, embedding_workers: NonZ let _ = writeln!(file, "{}", embedding_workers.get()); } let _ = root; + if signal_and_wait_for_test( + directory, + "batch-events.sock", + &(embedding_workers.get() as u64).to_le_bytes(), + ) { + return; + } pause_after_catch_up_before_unlock(); } @@ -1926,7 +1977,7 @@ fn observe_background_worker_process_group_for_test(process_group: u32) { }; let directory = Path::new(&directory); if directory.join("observe-embedding-worker-batches").is_file() { - let _ = std::fs::write( + let _ = cg_node_fs::write_test_file_for_test( directory.join("embedding-worker-process-group"), process_group.to_string(), ); @@ -2008,17 +2059,17 @@ fn pending_once( .join("pause-before-aggregate-active-time-refusal") .is_file() { - let _ = std::fs::write( + let _ = cg_node_fs::write_test_file_for_test( Path::new(&directory).join("aggregate-active-time-refusal-entered"), b"observed accepted identities", ); - wait_for_test_release(|| { + wait_for_test_release(&directory, || { Path::new(&directory) .join("release-aggregate-active-time-refusal") .is_file() }); } - let _ = std::fs::write(&marker, (count - 1).to_string()); + let _ = cg_node_fs::write_test_file_for_test(&marker, (count - 1).to_string()); return Err(CgError::ReadActiveTimeExceeded { limit_ms: active_time_ms.get(), }); @@ -2514,7 +2565,8 @@ fn catch_up( .and_then(|value| value.trim().parse::().ok()) .filter(|count| *count > 0) { - let _ = std::fs::write(&marker, (count - 1).to_string()); + let _ = + cg_node_fs::write_test_file_for_test(&marker, (count - 1).to_string()); return Err(CgError::ReadActiveTimeExceeded { limit_ms: request.read_active_time_ms.get(), }); @@ -2885,11 +2937,13 @@ fn pause_after_read_timeout_for_test(root: &Path) { }; let directory = PathBuf::from(directory); if directory.join("pause-after-read-timeout").is_file() { - let _ = std::fs::write( + let _ = cg_node_fs::write_test_file_for_test( directory.join("read-timeout-recovery-entered"), db_path(root).as_os_str().as_encoded_bytes(), ); - wait_for_test_release(|| directory.join("release-read-timeout-recovery").is_file()); + wait_for_test_release(&directory, || { + directory.join("release-read-timeout-recovery").is_file() + }); } } @@ -3121,8 +3175,13 @@ pub(crate) fn handle_event_search_worker( .join("pause-before-failed-worker-relock") .is_file() { - let _ = std::fs::write(directory.join("failed-worker-relock-entered"), b"unlocked"); - wait_for_test_release(|| directory.join("release-failed-worker-relock").is_file()); + let _ = cg_node_fs::write_test_file_for_test( + directory.join("failed-worker-relock-entered"), + b"unlocked", + ); + wait_for_test_release(&directory, || { + directory.join("release-failed-worker-relock").is_file() + }); } } let held = if retry_worker_lock { @@ -3429,7 +3488,7 @@ fn requested_foreground_selection( .and_then(|value| value.trim().parse::().ok()) .filter(|count| *count > 0) { - let _ = std::fs::write(marker, (count - 1).to_string()); + let _ = cg_node_fs::write_test_file_for_test(marker, (count - 1).to_string()); return Err(CgError::ReadActiveTimeExceeded { limit_ms: crate::read_config::active_time_in_force().get(), }); @@ -3468,7 +3527,7 @@ fn requested_foreground_work( .and_then(|value| value.trim().parse::().ok()) .filter(|count| *count > 0) { - let _ = std::fs::write(marker, (count - 1).to_string()); + let _ = cg_node_fs::write_test_file_for_test(marker, (count - 1).to_string()); return Err(CgError::ReadActiveTimeExceeded { limit_ms: crate::read_config::active_time_in_force().get(), }); diff --git a/crates/cg-cli/src/main.rs b/crates/cg-cli/src/main.rs index 377cd96..7509d7e 100644 --- a/crates/cg-cli/src/main.rs +++ b/crates/cg-cli/src/main.rs @@ -143,7 +143,10 @@ fn main() { } fn run(args: Vec) -> i32 { - #[cfg(feature = "event-search-test-support")] + #[cfg(any( + feature = "event-search-test-support", + feature = "store-contention-test-support" + ))] history::apply_store_open_retry_allowance_for_test(); // The writer's working-memory ceiling is one setting for the whole invocation, not a flag each // command admits, so it comes off the argument list before any command parses its own. diff --git a/crates/cg-cli/tests/a_busy_store_is_told_to_the_operator.rs b/crates/cg-cli/tests/a_busy_store_is_told_to_the_operator.rs index fa027a2..4006b91 100644 --- a/crates/cg-cli/tests/a_busy_store_is_told_to_the_operator.rs +++ b/crates/cg-cli/tests/a_busy_store_is_told_to_the_operator.rs @@ -63,6 +63,15 @@ fn busy_statement(store_path: &std::path::Path, readers: [&ParkedReader; 2]) -> #[test] fn a_command_refused_by_a_reader_held_store_names_the_readers_it_waits_on() { + // Detect a missing deterministic drive before parking readers or spending + // the production retry allowance. The product assertions below stay intact. + #[allow(clippy::assertions_on_constants)] + { + assert!( + cfg!(feature = "store-contention-test-support"), + "default reader fixture must compile its retry-allowance drive" + ); + } let runtime = runtime_root(); let data_root = seeded_data_root(runtime.path()); let store_path = store_path_in(data_root.path()); diff --git a/crates/cg-cli/tests/cli_acceptance.rs b/crates/cg-cli/tests/cli_acceptance.rs index 3f32bad..05befb4 100644 --- a/crates/cg-cli/tests/cli_acceptance.rs +++ b/crates/cg-cli/tests/cli_acceptance.rs @@ -99,6 +99,57 @@ fn assert_condition(output: &CommandOutput, expected: &str) { ); } +// Only these public fixture arguments are safe to include in invocation diagnostics. +fn provider_diagnostic_arg(arg: &str) -> &str { + match arg { + "--version" | "--help" | "exec" | "auth" | "status" | "--json" => arg, + _ => "", + } +} + +// Used only by tests whose command output is public fixture status, never a live reply. +fn print_provider_diagnostics(harness: &CliHarness, outputs: &[(&str, &CommandOutput)]) { + for (command, output) in outputs { + eprintln!( + "{command}: status={} stdout={:?} stderr={:?}", + output.code, output.stdout, output.stderr + ); + } + // Keep log order; never print the fixture's env, prompt, or arbitrary argument values. + match fs::read_to_string(&harness.providers.log_path) { + Ok(log) => { + for line in log.lines() { + match serde_json::from_str::(line) { + Ok(record) => { + let provider = match record.get("provider").and_then(Value::as_str) { + Some("claude") => "claude", + Some("codex") => "codex", + Some("agy") => "agy", + _ => "", + }; + let mode = match record.get("mode").and_then(Value::as_str) { + Some("probe") => "probe", + Some("auth-status-probe") => "auth-status-probe", + Some("completion") => "completion", + _ => "", + }; + let argv = record.get("argv").and_then(Value::as_array).map(|args| { + args.iter() + .map(|arg| provider_diagnostic_arg(arg.as_str().unwrap_or(""))) + .collect::>() + }); + eprintln!( + "fixture invocation: provider={provider} mode={mode} argv={argv:?}" + ); + } + Err(_) => eprintln!("fixture invocation: unreadable JSON record"), + } + } + } + Err(err) => eprintln!("fixture invocation log unavailable: {:?}", err.kind()), + } +} + fn completion_count(output_harness: &CliHarness) -> usize { output_harness.providers.completion_count() } @@ -1872,7 +1923,7 @@ fn selftest_bounds_a_hung_provider_cli_with_typed_failure() { timeout_ms = 700 "#, ); - let _ = harness.run(&[ + let login = harness.run(&[ "model", "auth", "login", @@ -1881,6 +1932,13 @@ timeout_ms = 700 "--consent", ]); let output = harness.run(&["selftest"]); + print_provider_diagnostics( + &harness, + &[ + ("cg model auth login --connection claude --consent", &login), + ("cg selftest", &output), + ], + ); assert_eq!(output.code, 6); assert_condition(&output, "live_call_failed"); assert_bucket(&output, "timeout"); @@ -2127,6 +2185,7 @@ fn cg_doctor_runs_only_free_checks_names_drift_and_spends_nothing() { ); let output = harness.run(&["doctor"]); assert_eq!(output.code, 1, "doctor drift findings exit 1"); + print_provider_diagnostics(&harness, &[("cg doctor", &output)]); assert!(output.stdout.contains("--mcp-config")); assert!(output.stdout.contains("2.2.0-drift")); assert!( @@ -2230,6 +2289,71 @@ fn cg_doctor_runs_only_free_checks_names_drift_and_spends_nothing() { assert_eq!(completion_count(&stderr_help), 0, "doctor never spends"); } +#[test] +fn cg_doctor_fixture_driver_ignores_python3_shadowed_on_provider_path() { + let harness = CliHarness::new(); + harness.providers.install( + Provider::Codex, + &unique_label("codex"), + LoginState::Valid, + CompletionBehavior::Succeed, + ); + write_file( + harness.data_root.path().join("cg/providers.toml"), + r#"[providers.codex] +timeout_ms = 700 +"#, + ); + + let shadow_python = harness.providers.bin_dir.join("python3"); + let shadow_marker = harness.providers.bin_dir.join("python3.invoked"); + write_file(&shadow_python, "#!/bin/sh\n: > \"${0}.invoked\"\nexit 97\n"); + let executable_permissions = fs::metadata(harness.providers.bin_dir.join("codex")) + .expect("installed fixture wrapper metadata") + .permissions(); + fs::set_permissions(&shadow_python, executable_permissions) + .expect("make rejecting python3 executable"); + + let clock = contextdb_engine::local_transport::MonotonicDeadlineClock::new(); + let output = harness.run(&["doctor"]); + let elapsed = std::time::Duration::from_millis( + contextdb_core::read_contract::DeadlineClock::now_ms(&clock), + ); + print_provider_diagnostics(&harness, &[("cg doctor", &output)]); + assert!( + elapsed < std::time::Duration::from_secs(5), + "fixture version/help checks must keep the existing doctor deadline" + ); + assert!( + !shadow_marker.exists(), + "provider discovery PATH must not select the fixture driver's interpreter" + ); + let codex_probes = harness + .providers + .probe_records() + .into_iter() + .filter(|record| record.get("provider").and_then(Value::as_str) == Some("codex")) + .collect::>(); + for expected_argv in [ + vec![json!("--version")], + vec![json!("exec"), json!("--help")], + ] { + assert!( + codex_probes.iter().any(|record| { + record.get("argv").and_then(Value::as_array) == Some(&expected_argv) + }), + "the intended fixture driver must record {expected_argv:?}; got {codex_probes:?}" + ); + } + assert!( + output.stdout.contains("OK: model/codex/flags-in-help"), + "valid fixture version/help must pass: stdout={:?} stderr={:?}", + output.stdout, + output.stderr + ); + assert_eq!(completion_count(&harness), 0, "doctor never spends"); +} + #[test] fn cg_doctor_bounds_hung_subcommand_help_check() { let harness = CliHarness::new(); @@ -2248,8 +2372,10 @@ timeout_ms = 700 ); let started = std::time::Instant::now(); let output = harness.run(&["doctor"]); + let elapsed = started.elapsed(); + print_provider_diagnostics(&harness, &[("cg doctor", &output)]); assert!( - started.elapsed() < std::time::Duration::from_secs(5), + elapsed < std::time::Duration::from_secs(5), "doctor help drift checks must be bounded by the provider metadata timeout" ); assert_eq!(output.code, 1); @@ -2276,6 +2402,7 @@ fn cg_doctor_empty_required_help_flags_still_checks_binary_and_version() { write_agy_provider_override(harness.data_root.path(), &[]); let output = harness.run(&["doctor"]); + print_provider_diagnostics(&harness, &[("cg doctor", &output)]); assert!( output.stdout.contains("OK: model/agy/flags-in-help"), @@ -2310,6 +2437,7 @@ fn cg_doctor_empty_required_help_flags_still_checks_binary_and_version() { let missing = CliHarness::new(); write_agy_provider_override(missing.data_root.path(), &[]); let output = missing.run(&["doctor"]); + print_provider_diagnostics(&missing, &[("cg doctor", &output)]); assert!( output .stdout @@ -2333,6 +2461,7 @@ fn cg_doctor_empty_required_help_flags_still_checks_binary_and_version() { .set_version_probe_fails(Provider::Agy); write_agy_provider_override(failed_version.data_root.path(), &[]); let output = failed_version.run(&["doctor"]); + print_provider_diagnostics(&failed_version, &[("cg doctor", &output)]); assert_eq!(output.code, 1); assert!( output @@ -2690,25 +2819,19 @@ fn reply_extraction_engine_matches_recorded_envelopes() { let claude = extract_reply("claude", "json", CLAUDE_SUCCESS, None); assert_eq!( claude.as_deref(), - Ok( - "Host session proof received. I can answer through the logged-in Claude CLI; turn capture should stay off until ingestion ships." - ), + Ok("The logged-in Claude CLI returned this completion."), "claude JSON framing must extract /result" ); let codex = extract_reply("codex", "file", "", Some(&codex_file)); assert_eq!( codex.as_deref(), - Ok( - "Host session proof received. I can answer through this logged-in Codex CLI; no turn capture should begin until ingestion ships.\n" - ), + Ok("The logged-in Codex CLI returned this completion.\n"), "codex file framing must read --output-last-message" ); let agy = extract_reply("agy", "text", AGY_PLAIN, None); assert_eq!( agy.as_deref(), - Ok( - "Host session proof received. I can answer through the local Agy CLI; capture remains disabled until ingestion ships.\n" - ), + Ok("The local Agy CLI returned this completion.\n"), "text framing returns the whole stdout" ); let missing = extract_reply("claude", "json", "{\"type\":\"result\"}", None); diff --git a/crates/cg-cli/tests/cli_model_byok.rs b/crates/cg-cli/tests/cli_model_byok.rs index 975455a..6efe809 100644 --- a/crates/cg-cli/tests/cli_model_byok.rs +++ b/crates/cg-cli/tests/cli_model_byok.rs @@ -81,6 +81,7 @@ struct CapturedRequest { struct OpenAiStub { base_url: String, + address: std::net::SocketAddr, requests: Arc>>, shutdown: Option>, handle: Option>, @@ -106,30 +107,20 @@ impl OpenAiStub { fn new(status: u16, body: String) -> Self { let listener = TcpListener::bind("127.0.0.1:0").expect("bind local HTTP stub"); let addr = listener.local_addr().expect("local HTTP stub address"); - listener - .set_nonblocking(true) - .expect("set local HTTP stub nonblocking"); let requests = Arc::new(Mutex::new(Vec::new())); let thread_requests = Arc::clone(&requests); let (shutdown_tx, shutdown_rx) = mpsc::channel(); let handle = thread::spawn(move || { - loop { + while let Ok((stream, _)) = listener.accept() { if shutdown_rx.try_recv().is_ok() { break; } - match listener.accept() { - Ok((stream, _)) => { - handle_http_request(stream, Arc::clone(&thread_requests), status, &body); - } - Err(err) if err.kind() == std::io::ErrorKind::WouldBlock => { - thread::sleep(Duration::from_millis(10)); - } - Err(_) => break, - } + handle_http_request(stream, Arc::clone(&thread_requests), status, &body); } }); Self { base_url: format!("http://{addr}/openai/v1"), + address: addr, requests, shutdown: Some(shutdown_tx), handle: Some(handle), @@ -153,6 +144,7 @@ impl Drop for OpenAiStub { fn drop(&mut self) { if let Some(shutdown) = self.shutdown.take() { let _ = shutdown.send(()); + let _ = TcpStream::connect(self.address); } if let Some(handle) = self.handle.take() { let _ = handle.join(); diff --git a/crates/cg-cli/tests/command_workflows_cli.rs b/crates/cg-cli/tests/command_workflows_cli.rs index 449721f..808d2c7 100644 --- a/crates/cg-cli/tests/command_workflows_cli.rs +++ b/crates/cg-cli/tests/command_workflows_cli.rs @@ -292,55 +292,61 @@ fn chat_completion_body( fn spawn_chat_completion_sequence_server( responses: Vec, -) -> (String, mpsc::Receiver>) { +) -> (String, mpsc::Receiver, String>>) { let listener = TcpListener::bind("127.0.0.1:0").expect("bind hosted provider fixture"); - listener - .set_nonblocking(true) - .expect("sequence listener nonblocking"); let address = listener.local_addr().expect("hosted provider address"); let (tx, rx) = mpsc::channel(); thread::spawn(move || { - let mut requests = Vec::new(); - for response in responses { - let Some(mut stream) = accept_sequence_request(&listener) else { - break; - }; - requests.push(read_http_request(&mut stream)); - write_http_response(&mut stream, response.status, &response.body); - } - let _ = tx.send(requests); + let result = (|| { + let mut requests = Vec::new(); + for (index, response) in responses.into_iter().enumerate() { + let context = |err| format!("hosted sequence request {}: {err}", index + 1); + let Some(mut stream) = accept_sequence_request(&listener).map_err(context)? else { + break; + }; + requests.push(read_http_request(&mut stream).map_err(context)?); + write_http_response(&mut stream, response.status, &response.body) + .map_err(context)?; + } + Ok(requests) + })(); + let _ = tx.send(result); }); (format!("http://{address}/v1"), rx) } -fn accept_sequence_request(listener: &TcpListener) -> Option { - let deadline = std::time::Instant::now() + Duration::from_secs(5); - loop { - match listener.accept() { - Ok((stream, _)) => return Some(stream), - Err(err) if err.kind() == std::io::ErrorKind::WouldBlock => { - if std::time::Instant::now() >= deadline { - return None; - } - thread::sleep(Duration::from_millis(10)); - } - Err(_) => return None, - } - } +fn accept_sequence_request(listener: &TcpListener) -> Result, String> { + listener + .accept() + .map(|(stream, _)| Some(stream)) + .map_err(|err| format!("accept provider request: {err}")) } -fn read_http_request(stream: &mut TcpStream) -> String { - let mut reader = BufReader::new(stream.try_clone().expect("clone provider stream")); +fn read_http_request(stream: &mut TcpStream) -> Result { + let mut reader = BufReader::new( + stream + .try_clone() + .map_err(|err| format!("clone provider stream: {err}"))?, + ); let mut first_line = String::new(); - reader + if reader .read_line(&mut first_line) - .expect("read provider request line"); + .map_err(|err| format!("read provider request line: {err}"))? + == 0 + { + return Err("read provider request line: client disconnected".to_string()); + } let mut request = first_line; let mut content_length = 0usize; loop { let mut line = String::new(); - let bytes = reader.read_line(&mut line).expect("read provider header"); - if bytes == 0 || line == "\r\n" { + let bytes = reader + .read_line(&mut line) + .map_err(|err| format!("read provider header: {err}"))?; + if bytes == 0 { + return Err("read provider header: client disconnected".to_string()); + } + if line == "\r\n" { request.push_str("\r\n"); break; } @@ -355,13 +361,13 @@ fn read_http_request(stream: &mut TcpStream) -> String { let mut request_body = vec![0; content_length]; reader .read_exact(&mut request_body) - .expect("read provider request body"); + .map_err(|err| format!("read provider request body: {err}"))?; request.push_str(&String::from_utf8_lossy(&request_body)); } - request + Ok(request) } -fn write_http_response(stream: &mut TcpStream, status: u16, body: &Value) { +fn write_http_response(stream: &mut TcpStream, status: u16, body: &Value) -> Result<(), String> { let body = body.to_string(); let reason = if status == 400 { "Bad Request" } else { "OK" }; let response = format!( @@ -371,11 +377,13 @@ fn write_http_response(stream: &mut TcpStream, status: u16, body: &Value) { ); stream .write_all(response.as_bytes()) - .expect("write provider response"); + .map_err(|err| format!("write provider response: {err}")) } -fn hosted_requests(rx: mpsc::Receiver>) -> Vec { - rx.recv_timeout(Duration::from_secs(10)).unwrap_or_default() +fn hosted_requests(rx: mpsc::Receiver, String>>) -> Vec { + rx.recv() + .expect("collect hosted sequence requests") + .unwrap_or_else(|err| panic!("hosted sequence server failed: {err}")) } fn hosted_request_body(request: &str) -> Value { @@ -2376,7 +2384,7 @@ fn hosted_record_usage_receipt_flows_through_connected_extractor() { transcript, ); let request = provider_requests - .recv_timeout(Duration::from_secs(5)) + .recv() .expect("record must call the configured hosted provider through ConnectedModelExtractor"); assert!( request.contains("hosted usage flows through connected extractor"), diff --git a/crates/cg-cli/tests/event_forget_terminal.rs b/crates/cg-cli/tests/event_forget_terminal.rs index 120270d..b92ea0b 100644 --- a/crates/cg-cli/tests/event_forget_terminal.rs +++ b/crates/cg-cli/tests/event_forget_terminal.rs @@ -120,9 +120,6 @@ fn forget_keeps_the_reader_coordination_condition_when_erasure_is_refused() { use std::os::unix::fs::PermissionsExt; use std::os::unix::net::UnixListener; use std::process::{Command, Stdio}; - use std::time::Duration; - - use event_support::wait_for_state; struct RestoreDirectory(std::path::PathBuf); impl Drop for RestoreDirectory { @@ -133,8 +130,14 @@ fn forget_keeps_the_reader_coordination_condition_when_erasure_is_refused() { let machine = Machine::without_model(); let project = machine.project("coordination"); - let runtime = machine.root.join("runtime"); - fs::create_dir(&runtime).expect("create isolated coordination root"); + // The owner channel binds a Unix socket below this directory, and a socket address holds only + // about a hundred bytes, so the directory sits under `/tmp` rather than below the long + // per-user temporary directory macOS hands out. + let runtime_root = tempfile::Builder::new() + .prefix("cgr") + .tempdir_in("/tmp") + .expect("a coordination root short enough for local channel addresses"); + let runtime = runtime_root.path().to_path_buf(); fs::set_permissions(&runtime, fs::Permissions::from_mode(0o700)) .expect("make coordination root private"); write_session( @@ -169,7 +172,7 @@ fn forget_keeps_the_reader_coordination_condition_when_erasure_is_refused() { let socket = machine.root.join("before-purge.sock"); let listener = UnixListener::bind(&socket).expect("bind forget barrier"); - listener.set_nonblocking(true).expect("nonblocking barrier"); + listener.set_nonblocking(false).expect("blocking barrier"); let mut child = Command::new(env!("CARGO_BIN_EXE_cg")) .args([ "forget", @@ -190,16 +193,9 @@ fn forget_keeps_the_reader_coordination_condition_when_erasure_is_refused() { .stderr(Stdio::piped()) .spawn() .expect("run forget command"); - let reached = - wait_for_state( - Duration::from_secs(10), - Duration::from_millis(20), - || match listener.accept() { - Ok((stream, _)) => Ok::<_, std::io::Error>(Some(stream)), - Err(error) if error.kind() == std::io::ErrorKind::WouldBlock => Ok(None), - Err(error) => Err(error), - }, - ) + let reached = listener + .accept() + .map(|(stream, _)| Some(stream)) .expect("accept forget barrier"); let Some(mut stream) = reached else { let _ = child.kill(); diff --git a/crates/cg-cli/tests/event_history.rs b/crates/cg-cli/tests/event_history.rs index 5a0097a..20e9fe4 100644 --- a/crates/cg-cli/tests/event_history.rs +++ b/crates/cg-cli/tests/event_history.rs @@ -45,6 +45,7 @@ fn run_controlled_posix_retry(machine: &Machine, cwd: &std::path::Path, retry: & .stdin(Stdio::null()) .env("PATH", controlled_path) .env("CG_DATA_ROOT", &machine.data_root) + .env("CG_TEST_STATE_EVENTS", event_support::state_event_path()) .env("HOME", &machine.home) .env("CG_MODEL_CACHE_ROOT", &machine.model_cache) .env_remove("CLAUDE_CONFIG_DIR") @@ -78,7 +79,7 @@ struct HandoffWorker { ))] impl HandoffWorker { fn release_pass(&self, pass: usize) { - fs::write( + event_support::write_test_file( self.release.with_file_name(format!("release.{pass}")), b"continue\n", ) @@ -86,19 +87,11 @@ impl HandoffWorker { } fn release_and_join(&mut self) { - fs::write(&self.release, b"continue\n").expect("release the owned handoff worker"); + event_support::write_test_file(&self.release, b"continue\n") + .expect("release the owned handoff worker"); let mut child = self.child.take().expect("owned handoff worker"); - match wait_for_state(Duration::from_secs(120), Duration::from_millis(20), || { - child.try_wait() - }) { - Ok(Some(status)) => assert!(status.success(), "owned handoff worker exited {status}"), - Ok(None) => { - child.kill().expect("stop timed-out owned handoff worker"); - let status = child.wait().expect("join stopped handoff worker"); - panic!("owned handoff worker did not exit after release; stopped with {status}"); - } - Err(error) => panic!("poll owned handoff worker: {error}"), - } + let status = child.wait().expect("join released handoff worker"); + assert!(status.success(), "owned handoff worker exited {status}"); } } @@ -108,19 +101,11 @@ impl HandoffWorker { ))] impl Drop for HandoffWorker { fn drop(&mut self) { - let _ = fs::write(&self.release, b"cleanup\n"); + let _ = event_support::write_test_file(&self.release, b"cleanup\n"); let Some(mut child) = self.child.take() else { return; }; - match wait_for_state(Duration::from_secs(120), Duration::from_millis(20), || { - child.try_wait() - }) { - Ok(Some(_)) | Err(_) => {} - Ok(None) => { - let _ = child.kill(); - let _ = child.wait(); - } - } + let _ = child.wait(); } } @@ -167,6 +152,7 @@ fn spawn_handoff_worker( .args(["__event-search-worker", &launch_profile]) .current_dir(project) .env("CG_DATA_ROOT", &machine.data_root) + .env("CG_TEST_STATE_EVENTS", event_support::state_event_path()) .env("HOME", &machine.home) .env("CG_MODEL_CACHE_ROOT", &machine.model_cache) .env("CG_EVENT_SEARCH_TEST_HANDOFF_BARRIER", barrier) @@ -207,6 +193,7 @@ fn spawn_worker_before_first_assessment( .args(["__event-search-worker", &launch_profile]) .current_dir(project) .env("CG_DATA_ROOT", &machine.data_root) + .env("CG_TEST_STATE_EVENTS", event_support::state_event_path()) .env("HOME", &machine.home) .env("CG_MODEL_CACHE_ROOT", &machine.model_cache) .env("CG_EVENT_SEARCH_TEST_PRE_CATCH_UP_BARRIER", barrier) @@ -413,7 +400,7 @@ fn cg_import_project_directory_never_sweeps_working_tree_jsonl_or_imports_other_ &[("other project question", "other project answer")], ); let helper = write_helper_transcript(&history, &project, "sess-one", "helper1", "helper text"); - fs::write( + event_support::write_test_file( project.join("stray.jsonl"), format!( "{}\n", @@ -696,7 +683,7 @@ fn retry_commands_quote_actual_paths_contexts_and_refusals() { fs::rename(&tenant_session_source, &tenant_session) .expect("rename controlled tenant session to an injected exact-file target"); fs::create_dir_all(tenant.data_root.join("cg")).expect("create tenant config directory"); - fs::write( + event_support::write_test_file( tenant.data_root.join("cg").join("fabric.toml"), "[fabric]\nevent_placement = \"tenant_hub\"\n", ) @@ -806,7 +793,7 @@ fn receipt_placeholder_substitution_and_read_retries_preserve_persisted_scope() ); assert_eq!(imported.code, 0, "{}", imported.describe()); fs::create_dir_all(machine.data_root.join("cg")).expect("create changed-config directory"); - fs::write( + event_support::write_test_file( machine.data_root.join("cg").join("fabric.toml"), "[search]\nsessions = 10\norder = \"newest\"\n", ) @@ -937,7 +924,7 @@ fn cg_import_continues_past_a_malformed_file_captures_every_other_file_and_repor let bad = write_session(&history, &project, "sess-a-bad", &[("first", "second")]); let mut bytes = fs::read(&bad).expect("read bad session"); bytes.extend_from_slice(b"{\"type\": \"user\", not json\n"); - fs::write(&bad, bytes).expect("corrupt the session tail"); + event_support::write_test_file(&bad, bytes).expect("corrupt the session tail"); write_session( &history, &project, @@ -1251,7 +1238,8 @@ fn cg_import_search_wait_embedding_failure_returns_durable_capture_receipt_and_t for file in ["config.json", "tokenizer.json", "REVISION"] { fs::copy(real.join(file), broken.join(file)).expect("copy artifact metadata"); } - fs::write(broken.join("model.safetensors"), vec![7_u8; 2_000_000]).expect("broken weights"); + event_support::write_test_file(broken.join("model.safetensors"), vec![7_u8; 2_000_000]) + .expect("broken weights"); machine.model_cache = machine.root.join("broken-cache"); let project = machine.project("broken"); write_session( @@ -1650,7 +1638,7 @@ fn import_dry_run_verifies_committed_prefixes_reports_what_would_be_captured_and ) .expect("remove scratch session"); grown.extend_from_slice(&extra); - fs::write(&appended, &grown).expect("append records"); + event_support::write_test_file(&appended, &grown).expect("append records"); // Change one committed byte inside the recorded text, keeping every record valid JSON so the // file is still this project's session. let mut changed = fs::read(&rewritten).expect("read rewritten session"); @@ -1659,7 +1647,7 @@ fn import_dry_run_verifies_committed_prefixes_reports_what_would_be_captured_and .position(|window| window == b"five") .expect("the rewritten session carries its text"); changed[at + 3] = b'x'; - fs::write(&rewritten, &changed).expect("rewrite a committed byte"); + event_support::write_test_file(&rewritten, &changed).expect("rewrite a committed byte"); write_session(&history, &project, "sess-new", &[("nine", "ten")]); let before = fingerprint(&machine.root); @@ -1805,7 +1793,6 @@ fn search_reaches_ready_while_three_readers_poll_the_store_throughout_the_build( "a read is never refused while the projection runs: {}", status.describe() ); - std::thread::sleep(std::time::Duration::from_millis(250)); } }); } @@ -1857,7 +1844,7 @@ fn cg_import_reports_search_behind_when_the_worker_lock_cannot_be_opened() { // The lock file is present but unopenable by this user: not a lock somebody holds, an // unusable one. let lock_path = machine.data_root.join("event-search-worker.lock"); - fs::write(&lock_path, b"").expect("create the worker lock file"); + event_support::write_test_file(&lock_path, b"").expect("create the worker lock file"); fs::set_permissions(&lock_path, fs::Permissions::from_mode(0o000)) .expect("make the worker lock file unopenable"); @@ -1918,7 +1905,7 @@ fn cg_import_search_wait_reports_behind_when_the_worker_lock_cannot_be_opened() assert_eq!(deferred.code, 0, "{}", deferred.describe()); let lock_path = machine.data_root.join("event-search-worker.lock"); - fs::write(&lock_path, b"").expect("create the worker lock file"); + event_support::write_test_file(&lock_path, b"").expect("create the worker lock file"); fs::set_permissions(&lock_path, fs::Permissions::from_mode(0o000)) .expect("make the worker lock file unopenable"); @@ -2097,7 +2084,7 @@ fn import_dry_run_refuses_a_recovery_needing_store_without_writing() { fn write_passage_override(machine: &Machine, target: i64, hard: i64) { let config = machine.data_root.join("cg"); fs::create_dir_all(&config).expect("cg config directory"); - fs::write( + event_support::write_test_file( config.join("fabric.toml"), format!("[search.passages]\ntarget_tokens = {target}\nhard_tokens = {hard}\n"), ) @@ -2107,7 +2094,7 @@ fn write_passage_override(machine: &Machine, target: i64, hard: i64) { fn write_placement_override(machine: &Machine, placement: &str) { let config = machine.data_root.join("cg"); fs::create_dir_all(&config).expect("cg config directory"); - fs::write( + event_support::write_test_file( config.join("fabric.toml"), format!("[fabric]\nevent_placement = \"{placement}\"\n"), ) @@ -2786,7 +2773,7 @@ fn a_default_worker_waits_for_an_orphaned_lease_and_does_not_starve_other_contex &generation_id, Duration::from_secs(5), ); - fs::write(barrier.path().join("pre-catch-up-release"), b"continue\n") + event_support::write_test_file(barrier.path().join("pre-catch-up-release"), b"continue\n") .expect("release worker into durable lease"); // The handoff barrier runs after this service pass. It holds the worker before its next @@ -2814,7 +2801,7 @@ fn a_default_worker_waits_for_an_orphaned_lease_and_does_not_starve_other_contex Some(&Value::from("ready")), "the leased Context was served before the eligible Context" ); - fs::write( + event_support::write_test_file( barrier.path().join("release"), b"continue after ordered service\n", ) @@ -2859,7 +2846,7 @@ fn an_adopted_valid_profile_survives_invalid_config_and_late_work_beyond_three_p ); assert_eq!(capture.code, 0, "{}", capture.describe()); let barrier = tempfile::tempdir().expect("numbered worker barrier"); - fs::write(barrier.path().join("numbered-passes"), b"5\n").unwrap(); + event_support::write_test_file(barrier.path().join("numbered-passes"), b"5\n").unwrap(); let mut worker = spawn_handoff_worker(&machine, &first_project, barrier.path()); wait_for_worker_marker(&barrier.path().join("entered.1")); let launch = machine.run(&first_project, &["status"]); diff --git a/crates/cg-cli/tests/event_search_worker_controls.rs b/crates/cg-cli/tests/event_search_worker_controls.rs index 1057b8a..69ca640 100644 --- a/crates/cg-cli/tests/event_search_worker_controls.rs +++ b/crates/cg-cli/tests/event_search_worker_controls.rs @@ -64,13 +64,13 @@ fn observer(machine: &Machine, name: &str, pause_batches: bool) -> std::path::Pa let path = machine.root.join(name); std::fs::create_dir_all(&path).expect("create worker observer directory"); if pause_batches { - std::fs::write( + event_support::write_test_file( path.join("observe-embedding-worker-batches"), b"observe and pause\n", ) .expect("enable existing batch observer"); } else { - std::fs::write(path.join("release"), b"do not pause\n") + event_support::write_test_file(path.join("release"), b"do not pause\n") .expect("pre-release the existing handoff barrier"); } path @@ -148,7 +148,7 @@ fn queued_forget_uses_configured_cap_and_finishes_after_foreground_releases() { ); let config_dir = machine.data_root.join("cg"); std::fs::create_dir_all(&config_dir).expect("create config directory"); - std::fs::write( + event_support::write_test_file( config_dir.join("fabric.toml"), "[search]\nembedding_workers = 2\n", ) @@ -214,7 +214,8 @@ fn queued_forget_uses_configured_cap_and_finishes_after_foreground_releases() { "already_running" ); - std::fs::write(observer.join("release"), b"continue\n").expect("release foreground"); + event_support::write_test_file(observer.join("release"), b"continue\n") + .expect("release foreground"); let output = foreground .wait_with_output() .expect("wait for foreground cg"); @@ -311,7 +312,8 @@ fn forget_during_background_work_keeps_the_live_worker_cap() { "forget did not leave a background handoff request" ); - std::fs::write(observer.join("release"), b"continue\n").expect("release worker"); + event_support::write_test_file(observer.join("release"), b"continue\n") + .expect("release worker"); terminate_observed_worker(&observer); machine.wait_for_search_worker_exit(); } @@ -338,7 +340,7 @@ fn fresh_forget_worker_uses_config_not_an_old_one_off_import_cap() { ); let config_dir = machine.data_root.join("cg"); std::fs::create_dir_all(&config_dir).expect("create config directory"); - std::fs::write( + event_support::write_test_file( config_dir.join("fabric.toml"), "[search]\nembedding_workers = 2\n", ) @@ -415,7 +417,7 @@ fn foreground_wait_keeps_its_cap_when_another_import_publishes() { ); let config_dir = machine.data_root.join("cg"); std::fs::create_dir_all(&config_dir).expect("create config directory"); - std::fs::write( + event_support::write_test_file( config_dir.join("fabric.toml"), "[search.passages]\ntarget_tokens = 8\nhard_tokens = 64\n", ) @@ -466,7 +468,8 @@ fn foreground_wait_keeps_its_cap_when_another_import_publishes() { ); assert_eq!(policy(&machine.data_root)["embedding_workers"], 2); - std::fs::write(observer.join("release"), b"continue\n").expect("release foreground"); + event_support::write_test_file(observer.join("release"), b"continue\n") + .expect("release foreground"); let output = foreground .wait_with_output() .expect("wait for foreground cg"); @@ -508,7 +511,7 @@ fn queued_import_selection_survives_forget_and_foreground_handoff() { let machine = Machine::with_nomic(); let config_dir = machine.data_root.join("cg"); std::fs::create_dir_all(&config_dir).expect("create config directory"); - std::fs::write( + event_support::write_test_file( config_dir.join("fabric.toml"), "[search]\nembedding_workers = 2\n", ) @@ -621,7 +624,8 @@ fn queued_import_selection_survives_forget_and_foreground_handoff() { 3 ); - std::fs::write(observer.join("release"), b"continue\n").expect("release foreground"); + event_support::write_test_file(observer.join("release"), b"continue\n") + .expect("release foreground"); let output = foreground .wait_with_output() .expect("wait for foreground cg"); @@ -667,7 +671,7 @@ fn sigint_foreground_hands_the_queued_request_to_one_background_worker() { )], ); let observer = observer(&machine, "sigint-observer", true); - std::fs::write( + event_support::write_test_file( observer.join("pause-after-foreground-handoff"), b"pause before final status refresh\n", ) @@ -717,7 +721,7 @@ fn sigint_foreground_hands_the_queued_request_to_one_background_worker() { assert_eq!(queued.code, 0, "{}", queued.describe()); assert_eq!(str_at(&queued.json(), "/search/worker"), "already_running"); - std::fs::write( + event_support::write_test_file( observer.join("release"), b"release foreground worker turn\n", ) @@ -743,10 +747,7 @@ fn sigint_foreground_hands_the_queued_request_to_one_background_worker() { unsafe { libc::kill(foreground.id() as i32, libc::SIGINT) }, 0 ); - let exited = wait_for_state(Duration::from_secs(10), Duration::from_millis(20), || { - foreground.try_wait() - }) - .expect("foreground exit observation cannot fail"); + let exited = foreground.wait().ok(); assert_eq!( exited.and_then(|status| status.code()), Some(130), @@ -859,10 +860,7 @@ fn sigint_during_prepared_embedding_hands_off_the_queued_request_once() { unsafe { libc::kill(foreground.id() as i32, libc::SIGINT) }, 0 ); - let exited = wait_for_state(Duration::from_secs(10), Duration::from_millis(20), || { - foreground.try_wait() - }) - .expect("foreground exit observation cannot fail"); + let exited = foreground.wait().ok(); assert_eq!( exited.and_then(|status| status.code()), Some(130), @@ -871,7 +869,7 @@ fn sigint_during_prepared_embedding_hands_off_the_queued_request_once() { // The handoff child inherits this test-only barrier. Releasing only after exit proves that the // foreground did not need the forward call to finish before it freed the worker turn. - std::fs::write(barrier.join("release"), b"release handed-off child\n") + event_support::write_test_file(barrier.join("release"), b"release handed-off child\n") .expect("release inherited prepared embedding barrier"); let ready = machine.wait_for_search_ready(&queued_project, "sigint-prepared-queued"); assert_eq!( @@ -928,10 +926,7 @@ fn sigint_foreground_without_a_queue_exits_and_releases_its_worker_turn() { unsafe { libc::kill(foreground.id() as i32, libc::SIGINT) }, 0 ); - let exited = wait_for_state(Duration::from_secs(10), Duration::from_millis(20), || { - foreground.try_wait() - }) - .expect("foreground exit observation cannot fail"); + let exited = foreground.wait().ok(); assert_eq!(exited.and_then(|status| status.code()), Some(130)); machine.wait_for_search_worker_exit(); assert!(policy(&machine.data_root)["pending_background_request"].is_null()); @@ -953,7 +948,7 @@ fn newer_wake_survives_older_final_clear_and_is_serviced() { ); let observer = machine.root.join("newer-wake-observer"); std::fs::create_dir_all(&observer).expect("create worker observer directory"); - std::fs::write(observer.join("numbered-passes"), b"2\n") + event_support::write_test_file(observer.join("numbered-passes"), b"2\n") .expect("pause two completed worker passes"); let first = machine.run_with_env( &project, @@ -1003,7 +998,8 @@ fn newer_wake_survives_older_final_clear_and_is_serviced() { .expect("newer request token") .to_owned(); - std::fs::write(observer.join("release.1"), b"allow older clear\n").expect("release older pass"); + event_support::write_test_file(observer.join("release.1"), b"allow older clear\n") + .expect("release older pass"); let second_pass = wait_for_state(Duration::from_secs(30), Duration::from_millis(20), || { Ok::<_, ()>(observer.join("entered.2").exists().then_some(())) }) @@ -1011,7 +1007,8 @@ fn newer_wake_survives_older_final_clear_and_is_serviced() { let token_after_older_clear = policy(&machine.data_root)["pending_background_request"]["token"] .as_str() .map(str::to_owned); - std::fs::write(observer.join("release.2"), b"allow newer clear\n").expect("release newer pass"); + event_support::write_test_file(observer.join("release.2"), b"allow newer clear\n") + .expect("release newer pass"); machine.wait_for_search_ready(&project, "newer-wake-final-clear"); machine.wait_for_search_worker_exit(); @@ -1074,9 +1071,9 @@ fn exhausted_writer_refusal_keeps_the_request_until_a_new_same_profile_wake_reac let barrier = machine.root.join("writer-refusal-barrier"); std::fs::create_dir_all(&barrier).expect("create writer-refusal barrier"); - std::fs::write(barrier.join("pause-after-work-assessment"), b"pause\n") + event_support::write_test_file(barrier.join("pause-after-work-assessment"), b"pause\n") .expect("arm post-assessment rendezvous"); - std::fs::write(barrier.join("exhaust-store-open-retry"), b"exhaust\n") + event_support::write_test_file(barrier.join("exhaust-store-open-retry"), b"exhaust\n") .expect("use the core's test-only allowance override"); let envelope = worker_envelope(); let mut worker = machine.spawn_with_env( @@ -1101,15 +1098,12 @@ fn exhausted_writer_refusal_keeps_the_request_until_a_new_same_profile_wake_reac let mut writer = HoldingChild::hold_as_writer(&machine.db_path(), &runtime, "worker_writer_holder_child"); writer.wait_until_holding_as_writer(); - std::fs::write( + event_support::write_test_file( barrier.join("work-assessment-release"), b"attempt projection\n", ) .expect("release projection into writer contention"); - let exited = wait_for_state(Duration::from_secs(10), Duration::from_millis(20), || { - worker.try_wait() - }) - .expect("worker exit observation cannot fail"); + let exited = worker.wait().ok(); if exited.is_none() { let _ = worker.kill(); let _ = worker.wait(); diff --git a/crates/cg-cli/tests/event_support/mod.rs b/crates/cg-cli/tests/event_support/mod.rs index aa24539..a510ee9 100644 --- a/crates/cg-cli/tests/event_support/mod.rs +++ b/crates/cg-cli/tests/event_support/mod.rs @@ -14,7 +14,7 @@ use std::os::unix::process::CommandExt; use std::path::{Path, PathBuf}; use std::process::{Child, Command, ExitStatus, Stdio}; use std::sync::{Arc, Mutex}; -use std::time::{Duration, Instant}; +use std::time::Duration; use serde_json::Value; @@ -107,26 +107,47 @@ impl Drop for TestChild { } } -/// Observe state until it completes, errors, or spends its bounded allowance. +struct StateEvents { + _directory: tempfile::TempDir, + path: PathBuf, + receiver: std::os::unix::net::UnixDatagram, +} + +impl StateEvents { + fn new() -> Self { + let directory = tempfile::tempdir_in("/tmp").expect("short state event directory"); + let path = directory.path().join("state.sock"); + let receiver = std::os::unix::net::UnixDatagram::bind(&path).expect("state event socket"); + Self { + _directory: directory, + path, + receiver, + } + } +} + +thread_local! { static STATE_EVENTS: StateEvents = StateEvents::new(); } + +pub fn state_event_path() -> PathBuf { + STATE_EVENTS.with(|events| events.path.clone()) +} + +pub fn write_test_file(path: impl AsRef, contents: impl AsRef<[u8]>) -> std::io::Result<()> { + cg_node_fs::write_test_file_for_test(path, contents) +} + +/// Recheck authoritative state after an explicit fixture write or worker completion. +/// The old allowance arguments remain for callers; the bundle runner owns failure deadlines. pub fn wait_for_state( - allowance: Duration, - interval: Duration, + _allowance: Duration, + _interval: Duration, mut observe: impl FnMut() -> Result, E>, ) -> Result, E> { - assert!( - !interval.is_zero(), - "state wait polling interval must be non-zero" - ); - let started = Instant::now(); loop { if let Some(value) = observe()? { return Ok(Some(value)); } - let remaining = allowance.saturating_sub(started.elapsed()); - if remaining.is_zero() { - return Ok(None); - } - std::thread::sleep(interval.min(remaining)); + STATE_EVENTS.with(|events| events.receiver.recv(&mut [0]).expect("fixture state event")); } } @@ -155,6 +176,7 @@ pub struct Machine { pub home: PathBuf, pub data_root: PathBuf, pub model_cache: PathBuf, + state_events_path: PathBuf, } impl Machine { @@ -167,6 +189,7 @@ impl Machine { let model_cache = base.join("empty-model-cache"); fs::create_dir_all(&model_cache).expect("empty model cache"); Self { + state_events_path: state_event_path(), data_root: base.join("data-root"), home, model_cache, @@ -207,6 +230,7 @@ impl Machine { .args(args) .current_dir(cwd) .env("CG_DATA_ROOT", &self.data_root) + .env("CG_TEST_STATE_EVENTS", &self.state_events_path) .env("HOME", &self.home) .env("CG_MODEL_CACHE_ROOT", &self.model_cache) .env_remove("CLAUDE_CONFIG_DIR") @@ -331,41 +355,45 @@ impl Machine { drop(command); drop(slave); - let transcript = Arc::new(Mutex::new(String::new())); + let transcript = Arc::new(( + Mutex::new((String::new(), false)), + std::sync::Condvar::new(), + )); let reader_transcript = Arc::clone(&transcript); let mut master_reader = fs::File::from(master.try_clone().expect("clone pty master")); let reader = std::thread::spawn(move || { let mut buffer = [0_u8; 1024]; loop { - match master_reader.read(&mut buffer) { - Ok(0) | Err(_) => break, - Ok(read) => reader_transcript - .lock() - .expect("transcript lock") - .push_str(&String::from_utf8_lossy(&buffer[..read])), + let read = master_reader.read(&mut buffer); + let mut state = reader_transcript.0.lock().expect("transcript lock"); + match read { + Ok(0) | Err(_) => { + state.1 = true; + reader_transcript.1.notify_all(); + break; + } + Ok(read) => state.0.push_str(&String::from_utf8_lossy(&buffer[..read])), } + reader_transcript.1.notify_all(); } }); let mut master_writer = fs::File::from(master); let mut seen = 0_usize; for (index, (prompt, reply)) in answers.iter().enumerate() { - let found = wait_for_state(Duration::from_secs(60), Duration::from_millis(20), || { - let found = { - let text = transcript.lock().expect("transcript lock"); - text[seen..] - .find(prompt) - .map(|index| seen + index + prompt.len()) - }; - Ok::<_, ()>(found) - }) - .expect("prompt observation cannot fail"); - let Some(end) = found else { - let _ = child.kill(); - panic!( - "cg never wrote the terminal prompt {prompt:?}; transcript so far: {:?}", - transcript.lock().expect("transcript lock") - ); - }; + let mut state = transcript.0.lock().expect("transcript lock"); + while !state.1 && !state.0[seen..].contains(prompt) { + state = transcript.1.wait(state).expect("terminal output event"); + } + let end = state.0[seen..] + .find(prompt) + .map(|index| seen + index + prompt.len()) + .unwrap_or_else(|| { + panic!( + "cg closed its terminal before prompt {prompt:?}: {:?}", + state.0 + ) + }); + drop(state); seen = end; before_reply(index); master_writer @@ -389,7 +417,7 @@ impl Machine { let status = child.wait().expect("wait for cg"); drop(master_writer); let _ = reader.join(); - let transcript = transcript.lock().expect("transcript lock").clone(); + let transcript = transcript.0.lock().expect("transcript lock").0.clone(); ( Output { code: status.code().unwrap_or(-1), @@ -433,19 +461,11 @@ impl Machine { }) } - /// Poll until the detached search worker process has exited: its per-data-root lock can be - /// taken again. The worker is another process, so its lock is the state there is to observe. + /// The kernel wakes this acquire after the worker releases its lock, even on SIGTERM. pub fn wait_for_search_worker_exit(&self) { let lock_path = self.data_root.join("event-search-worker.lock"); - let released = wait_for_state(Duration::from_secs(120), Duration::from_millis(100), || { - Ok::<_, ()>(cg_node_fs::acquire_worker_lock(&lock_path).ok()) - }) - .expect("worker-lock observation cannot fail"); - assert!( - released.is_some(), - "the detached search worker still holds {} after 120 s", - lock_path.display() - ); + let _released = cg_node_fs::wait_for_worker_lock_for_test(&lock_path) + .expect("detached worker released its lock"); } } diff --git a/crates/cg-cli/tests/fixtures/provider_envelopes/agy_plain.txt b/crates/cg-cli/tests/fixtures/provider_envelopes/agy_plain.txt index 632b64b..5840b76 100644 --- a/crates/cg-cli/tests/fixtures/provider_envelopes/agy_plain.txt +++ b/crates/cg-cli/tests/fixtures/provider_envelopes/agy_plain.txt @@ -1 +1 @@ -Host session proof received. I can answer through the local Agy CLI; capture remains disabled until ingestion ships. +The local Agy CLI returned this completion. diff --git a/crates/cg-cli/tests/fixtures/provider_envelopes/claude_success.json b/crates/cg-cli/tests/fixtures/provider_envelopes/claude_success.json index b46321f..33649b1 100644 --- a/crates/cg-cli/tests/fixtures/provider_envelopes/claude_success.json +++ b/crates/cg-cli/tests/fixtures/provider_envelopes/claude_success.json @@ -6,7 +6,7 @@ "duration_ms": 2427, "duration_api_ms": 1896, "num_turns": 1, - "result": "Host session proof received. I can answer through the logged-in Claude CLI; turn capture should stay off until ingestion ships.", + "result": "The logged-in Claude CLI returned this completion.", "session_id": "6d8b4cb2-7f16-4c84-9e7f-9a43d7a51c30", "total_cost_usd": 0.000278, "usage": { diff --git a/crates/cg-cli/tests/fixtures/provider_envelopes/codex_last_message.txt b/crates/cg-cli/tests/fixtures/provider_envelopes/codex_last_message.txt index 1d1e4d4..80791f0 100644 --- a/crates/cg-cli/tests/fixtures/provider_envelopes/codex_last_message.txt +++ b/crates/cg-cli/tests/fixtures/provider_envelopes/codex_last_message.txt @@ -1 +1 @@ -Host session proof received. I can answer through this logged-in Codex CLI; no turn capture should begin until ingestion ships. +The logged-in Codex CLI returned this completion. diff --git a/crates/cg-cli/tests/import_embedding_workers.rs b/crates/cg-cli/tests/import_embedding_workers.rs index 7c31376..38c7f67 100644 --- a/crates/cg-cli/tests/import_embedding_workers.rs +++ b/crates/cg-cli/tests/import_embedding_workers.rs @@ -105,7 +105,7 @@ const REQUEST_OBSERVER_ENV: &str = "CG_EVENT_SEARCH_TEST_HANDOFF_BARRIER"; fn request_observer(machine: &Machine, name: &str) -> std::path::PathBuf { let path = machine.root.join(name); std::fs::create_dir_all(&path).expect("create request observer directory"); - std::fs::write(path.join("release"), b"do not pause a background worker\n") + event_support::write_test_file(path.join("release"), b"do not pause a background worker\n") .expect("pre-release the existing handoff barrier"); path } @@ -129,7 +129,7 @@ fn observed_request_caps(observer: &std::path::Path) -> Vec { fn batch_observer(machine: &Machine, name: &str) -> std::path::PathBuf { let path = machine.root.join(name); std::fs::create_dir_all(&path).expect("create batch observer directory"); - std::fs::write( + event_support::write_test_file( path.join("observe-embedding-worker-batches"), b"observe and pause\n", ) @@ -195,7 +195,7 @@ fn terminate_observed_background_worker(observer: &std::path::Path) { fn write_config(machine: &Machine, body: &str) { let config_dir = machine.data_root.join("cg"); std::fs::create_dir_all(&config_dir).expect("create cg config directory"); - std::fs::write(config_dir.join("fabric.toml"), body).expect("write cg config"); + event_support::write_test_file(config_dir.join("fabric.toml"), body).expect("write cg config"); } #[test] @@ -212,7 +212,7 @@ fn deferred_import_never_creates_or_changes_the_active_worker_policy() { std::fs::create_dir_all(&config_dir).expect("create cg config directory"); let config_path = config_dir.join("fabric.toml"); let configured = "[search]\nembedding_workers = 1\n"; - std::fs::write(&config_path, configured).expect("write saved worker cap"); + event_support::write_test_file(&config_path, configured).expect("write saved worker cap"); let imported = machine.run( &project, @@ -249,7 +249,8 @@ fn deferred_import_never_creates_or_changes_the_active_worker_policy() { ); let active = r#"{"publication":"active-fixture","embedding_workers":2}"#; - std::fs::write(&policy_path, active).expect("seed another build's active policy"); + event_support::write_test_file(&policy_path, active) + .expect("seed another build's active policy"); let deferred_again = machine.run( &project, &[ @@ -406,7 +407,29 @@ fn detached_background_import_delivers_default_cap_to_its_production_request_bui "background-default-request-session", &turns, ); - let observer = batch_observer(&machine, "background-default-observer"); + // macOS TMPDIR paths can already consume most of a Unix socket address. + let observation = tempfile::Builder::new() + .prefix("cgb") + .tempdir_in("/tmp") + .expect("short batch rendezvous directory"); + let observer = observation.path().to_path_buf(); + event_support::write_test_file( + observer.join("observe-embedding-worker-batches"), + b"observe", + ) + .expect("opt into batch observation"); + let batches = std::os::unix::net::UnixListener::bind(observer.join("batch-events.sock")) + .expect("bind batch rendezvous before spawning the worker"); + let receive_batch = || { + use std::io::Read; + let (mut stream, _) = batches.accept().expect("batch callback connects"); + let mut cap = [0; 8]; + stream.read_exact(&mut cap).expect("batch cap is complete"); + ( + stream, + usize::try_from(u64::from_le_bytes(cap)).expect("cap fits"), + ) + }; let imported = machine.run_with_env( &project, &[ @@ -421,79 +444,43 @@ fn detached_background_import_delivers_default_cap_to_its_production_request_bui &observer, ); assert_eq!(imported.code, 0, "{}", imported.describe()); - let first = wait_for_state( - std::time::Duration::from_secs(10), - std::time::Duration::from_millis(20), - || { - let caps = observed_batch_caps(&observer); - Ok::<_, ()>((!caps.is_empty()).then_some(caps)) - }, - ) - .expect("first batch observation cannot fail"); - let (paused, later) = if first.is_some() { - let paused = wait_for_state( - std::time::Duration::from_secs(10), - std::time::Duration::from_millis(20), - || Ok::<_, ()>(observer.join("entered").exists().then_some(())), - ) - .expect("batch pause observation cannot fail"); - // Replace the execution choice while preserving the accepted request's selected - // frontiers and saved ceiling. A cap update is not a queue acknowledgement. - let mut replacement_policy = worker_policy(&machine.data_root); - replacement_policy["publication"] = "replacement".into(); - replacement_policy["embedding_workers"] = replacement_embedding_workers.into(); - let replacement = - write_event_search_worker_policy(&machine.data_root, &replacement_policy.to_string()); - assert!( - replacement.is_ok(), - "replacement worker policy write failed: {replacement:?}" - ); - std::fs::write( - observer.join("release"), - b"continue after policy replacement\n", - ) - .expect("release owned background worker"); - let later = wait_for_state( - std::time::Duration::from_secs(10), - std::time::Duration::from_millis(20), - || { - let caps = observed_batch_caps(&observer); - Ok::<_, ()>((caps.len() >= 2).then_some(caps)) - }, - ) - .expect("later batch observation cannot fail"); - (paused, later) - } else { - std::fs::write( - observer.join("release"), - b"release after missing callback\n", - ) - .expect("release owned background worker after missing callback"); - (None, None) - }; - terminate_observed_background_worker(&observer); + struct StopObservedWorker<'a>(&'a std::path::Path); + impl Drop for StopObservedWorker<'_> { + fn drop(&mut self) { + terminate_observed_background_worker(self.0); + } + } + + let (mut first_batch, first_cap) = receive_batch(); + let stop_worker = StopObservedWorker(&observer); + assert_eq!(first_cap, initial_embedding_workers); + assert_eq!( + observed_batch_caps(&observer), + vec![initial_embedding_workers] + ); + + // A callback cannot proceed until this policy is durably replaced and the + // test releases it. The next callback announces the resolver's new choice. + let mut replacement_policy = worker_policy(&machine.data_root); + replacement_policy["publication"] = "replacement".into(); + replacement_policy["embedding_workers"] = replacement_embedding_workers.into(); + write_event_search_worker_policy(&machine.data_root, &replacement_policy.to_string()) + .expect("replace worker policy while its first batch is held"); + use std::io::Write; + first_batch.write_all(&[1]).expect("drive the next batch"); + let (_held_second_batch, second_cap) = receive_batch(); + assert_eq!(second_cap, replacement_embedding_workers); + let later = observed_batch_caps(&observer); + drop(stop_worker); machine.wait_for_search_worker_exit(); assert_eq!( observed_request_caps(&observer), vec![initial_embedding_workers], "the detached worker request builder did not carry the available-CPU default" ); - assert_eq!( - first, - Some(vec![initial_embedding_workers]), - "the first detached-worker batch did not receive the available-CPU default" - ); - assert_eq!( - paused, - Some(()), - "the first production batch callback did not pause before policy replacement" - ); assert_eq!( later, - Some(vec![ - initial_embedding_workers, - replacement_embedding_workers - ]), + vec![initial_embedding_workers, replacement_embedding_workers], "the detached worker did not apply the replacement cap at its next batch boundary" ); } @@ -808,7 +795,8 @@ fn already_caught_up_import_does_not_change_an_active_worker_request() { assert_eq!(built.code, 0, "{}", built.describe()); let policy_path = machine.data_root.join(EVENT_SEARCH_WORKER_POLICY); let active = r#"{"publication":"active-fixture","embedding_workers":2}"#; - std::fs::write(&policy_path, active).expect("seed another active worker request"); + event_support::write_test_file(&policy_path, active) + .expect("seed another active worker request"); let unchanged = machine.run( &project, @@ -914,7 +902,7 @@ fn invalid_search_scope_flags_and_config_are_typed_and_refused_before_writes() { ); let config_dir = machine.data_root.join("cg"); std::fs::create_dir_all(&config_dir).expect("create cg config directory"); - std::fs::write( + event_support::write_test_file( config_dir.join("fabric.toml"), "[search]\norder = \"sideways\"\nsessions = 10\n", ) @@ -955,7 +943,7 @@ fn invalid_configured_embedding_workers_are_refused_before_event_or_store_writes ); let config_dir = machine.data_root.join("cg"); std::fs::create_dir_all(&config_dir).expect("create cg config directory"); - std::fs::write(config_dir.join("fabric.toml"), configured) + event_support::write_test_file(config_dir.join("fabric.toml"), configured) .expect("write invalid worker policy"); let before = fingerprint(&machine.data_root); @@ -1188,7 +1176,8 @@ mod request_write_ceilings { &[("B request selected before C exists", "B answer")], ); let text = std::fs::read_to_string(&source).expect("B source"); - std::fs::write(source, text.replace("2026-09-01", "2026-08-01")).expect("B precedes C"); + event_support::write_test_file(source, text.replace("2026-09-01", "2026-08-01")) + .expect("B precedes C"); } fn assert_selected_b_and_c_completed( @@ -1326,8 +1315,11 @@ mod request_write_ceilings { &[("accepted selection", "answer")], ); let text = std::fs::read_to_string(&path).unwrap(); - std::fs::write(path, text.replace("2026-09-01", &format!("2026-09-{day}"))) - .unwrap(); + event_support::write_test_file( + path, + text.replace("2026-09-01", &format!("2026-09-{day}")), + ) + .unwrap(); let imported = import_selected(&machine, &project, "stale-selection", HIGH, "1", &observer); assert_eq!(imported.code, 0, "{}", imported.describe()); @@ -1365,9 +1357,10 @@ mod request_write_ceilings { let mut selected_envelope: serde_json::Value = serde_json::from_str(&envelope()).unwrap(); selected_envelope["work_policy"] = accepted[0]["work_policy"].clone(); - std::fs::write(observer.join("supersede-selected-session"), b"stale-A").unwrap(); + event_support::write_test_file(observer.join("supersede-selected-session"), b"stale-A") + .unwrap(); std::fs::remove_file(observer.join("release")).unwrap(); - std::fs::write(observer.join("numbered-passes"), b"4").unwrap(); + event_support::write_test_file(observer.join("numbered-passes"), b"4").unwrap(); drop(held); let worker = machine.spawn_with_env( &project, @@ -1401,7 +1394,7 @@ mod request_write_ceilings { assert_eq!(arrival.code, 0, "{}", arrival.describe()); assert_eq!(str_at(&arrival.json(), "/search/worker"), "already_running"); } - std::fs::write( + event_support::write_test_file( observer.join(format!("release.{pass}")), b"continue bounded work", ) @@ -1495,8 +1488,9 @@ mod request_write_ceilings { } write_event_search_worker_policy(&machine.data_root, &shared.to_string()).unwrap(); std::fs::remove_file(observer.join("release")).unwrap(); - std::fs::write(observer.join("numbered-passes"), b"2").unwrap(); - std::fs::write(observer.join("refuse-aggregate-read"), b"refuse").unwrap(); + event_support::write_test_file(observer.join("numbered-passes"), b"2").unwrap(); + event_support::write_test_file(observer.join("refuse-aggregate-read"), b"refuse") + .unwrap(); drop(held); let worker = machine.spawn_with_envs( &project, @@ -1504,7 +1498,8 @@ mod request_write_ceilings { &[(REQUEST_OBSERVER_ENV, &observer)], ); await_marker(&observer, "entered.1"); - std::fs::write(observer.join("release.1"), b"second assessment").unwrap(); + event_support::write_test_file(observer.join("release.1"), b"second assessment") + .unwrap(); await_marker(&observer, "entered.2"); let path = write_session( &machine.history_root(), @@ -1513,7 +1508,7 @@ mod request_write_ceilings { &[("accepted at exhausted assessment", "served answer")], ); let text = std::fs::read_to_string(&path).unwrap(); - std::fs::write(path, text.replace("2026-09-01", "2026-09-02")).unwrap(); + event_support::write_test_file(path, text.replace("2026-09-01", "2026-09-02")).unwrap(); let arrival = import_selected( &machine, &project, @@ -1530,7 +1525,8 @@ mod request_write_ceilings { original["selected_work"] ); std::fs::remove_file(observer.join("refuse-aggregate-read")).unwrap(); - std::fs::write(observer.join("release.2"), b"serve new identity").unwrap(); + event_support::write_test_file(observer.join("release.2"), b"serve new identity") + .unwrap(); let output = worker.wait_with_output().unwrap(); assert_eq!(output.status.code(), Some(1), "{output:?}"); assert_eq!( @@ -1587,7 +1583,7 @@ mod request_write_ceilings { "sessionId": "opaque-selected", "cwd": project.display().to_string(), "timestamp": "2026-09-02T09:00:00Z", }); - std::fs::write(path, format!("{opaque}\n")).unwrap(); + event_support::write_test_file(path, format!("{opaque}\n")).unwrap(); let first = import_selected( &machine, &project, @@ -1633,16 +1629,20 @@ mod request_write_ceilings { std::fs::remove_file(observer.join("actual-write-memory-batches")).ok(); std::fs::remove_file(observer.join("release")).unwrap(); std::fs::remove_file(observer.join("entered")).ok(); - std::fs::write(observer.join("numbered-passes"), b"1").unwrap(); + event_support::write_test_file(observer.join("numbered-passes"), b"1").unwrap(); let assessment = machine.root.join("completion-read-assessment"); std::fs::create_dir_all(&assessment).unwrap(); - std::fs::write( + event_support::write_test_file( assessment.join("pre-catch-up-release"), b"initial read may run", ) .unwrap(); if completion_read { - std::fs::write(assessment.join("pause-after-work-assessment"), b"pause").unwrap(); + event_support::write_test_file( + assessment.join("pause-after-work-assessment"), + b"pause", + ) + .unwrap(); } else { let db = contextdb_engine::Database::open(machine.db_path()).unwrap(); db.execute( @@ -1703,7 +1703,7 @@ mod request_write_ceilings { ) .unwrap(); drop(db); - std::fs::write( + event_support::write_test_file( assessment.join("work-assessment-release"), b"read deleted header", ) @@ -1731,7 +1731,7 @@ mod request_write_ceilings { ); assert_eq!(arrival.code, 0, "{}", arrival.describe()); assert_eq!(str_at(&arrival.json(), "/search/worker"), "already_running"); - std::fs::write(observer.join("release.1"), b"serve arrival").unwrap(); + event_support::write_test_file(observer.join("release.1"), b"serve arrival").unwrap(); let output = worker.wait_with_output().unwrap(); assert_eq!(output.status.code(), Some(1), "{output:?}"); assert_eq!( @@ -1787,10 +1787,11 @@ mod request_write_ceilings { "sessionId": session, "cwd": project.display().to_string(), "timestamp": format!("2026-09-{:02}T09:00:00Z", index + 1), }); - std::fs::write(&path, format!("{opaque}\n")).unwrap(); + event_support::write_test_file(&path, format!("{opaque}\n")).unwrap(); } else { let text = std::fs::read_to_string(&path).unwrap(); - std::fs::write(&path, text.replace("2026-09-01", "2026-09-06")).unwrap(); + event_support::write_test_file(&path, text.replace("2026-09-01", "2026-09-06")) + .unwrap(); } let imported = import_selected(&machine, &project, "distinct-no-text", HIGH, "1", &observer); @@ -1904,7 +1905,7 @@ mod request_write_ceilings { let mut arrival_frontier = None; drop(held); if racing { - std::fs::write( + event_support::write_test_file( observer.join("pause-before-selection-cleanup"), b"pause cleanup", ) @@ -2017,19 +2018,14 @@ mod request_write_ceilings { ); assert_selection_row_counts(&machine, 3, if same_context { 5 } else { 4 }); } - std::fs::write( + event_support::write_test_file( observer.join("release-selection-cleanup"), b"serve the arrival", ) .unwrap(); } assert_eq!( - wait_for_state(Duration::from_secs(30), Duration::from_millis(20), || { - worker - .try_wait() - .map(|status| status.map(|status| status.code())) - }) - .unwrap(), + worker.wait().ok().map(|status| status.code()), Some(Some(i32::from(cleanup_failure))), "refused-only worker did not terminate: racing={racing}, cleanup_failure={cleanup_failure}, same_context={same_context}, policy={}, admissions={:?}, cleanup={:?}, projection_error={:?}", worker_policy(&machine.data_root), @@ -2241,12 +2237,23 @@ mod request_write_ceilings { [0] .clone(); std::fs::remove_file(observer.join("release")).unwrap(); - std::fs::write(observer.join("numbered-passes"), b"3").unwrap(); - std::fs::write(observer.join("refuse-selected-coordination-count"), b"10").unwrap(); - std::fs::write(observer.join("hold-writer-during-failure-record"), b"hold").unwrap(); + event_support::write_test_file(observer.join("numbered-passes"), b"3").unwrap(); + event_support::write_test_file( + observer.join("refuse-selected-coordination-count"), + b"10", + ) + .unwrap(); + event_support::write_test_file( + observer.join("hold-writer-during-failure-record"), + b"hold", + ) + .unwrap(); if competing_owner { - std::fs::write(observer.join("pause-before-failed-worker-relock"), b"pause") - .unwrap(); + event_support::write_test_file( + observer.join("pause-before-failed-worker-relock"), + b"pause", + ) + .unwrap(); } drop(held); let worker = machine.spawn_with_env( @@ -2264,14 +2271,18 @@ mod request_write_ceilings { ..Default::default() }) .expect("external writer owns the failure reopen"); - std::fs::write( + event_support::write_test_file( observer.join("failure-record-continue"), b"refuse failure reopen", ) .unwrap(); await_marker(&observer, "failure-record-returned"); drop(writer); - std::fs::write(observer.join("failure-writer-released"), b"released").unwrap(); + event_support::write_test_file( + observer.join("failure-writer-released"), + b"released", + ) + .unwrap(); await_marker(&observer, &format!("entered.{pass}")); for marker in [ "failure-record-entered", @@ -2287,7 +2298,11 @@ mod request_write_ceilings { (10 - pass).to_string() ); if pass < 3 { - std::fs::write(observer.join(format!("release.{pass}")), b"retry").unwrap(); + event_support::write_test_file( + observer.join(format!("release.{pass}")), + b"retry", + ) + .unwrap(); } } let other = if same_context { @@ -2317,15 +2332,18 @@ mod request_write_ceilings { ) .unwrap(); std::fs::remove_file(observer.join("hold-writer-during-failure-record")).unwrap(); - std::fs::write(observer.join("release.3"), b"serve arrival").unwrap(); + event_support::write_test_file(observer.join("release.3"), b"serve arrival").unwrap(); let competing_lock = competing_owner.then(|| { await_marker(&observer, "failed-worker-relock-entered"); let held = acquire_worker_lock(&machine.data_root.join("event-search-worker.lock")) .expect("competing worker owns the released lock"); let startup = machine.run(&project, &["__event-search-worker", &envelope()]); assert_eq!(startup.code, 0, "{}", startup.describe()); - std::fs::write(observer.join("release-failed-worker-relock"), b"lose lock") - .unwrap(); + event_support::write_test_file( + observer.join("release-failed-worker-relock"), + b"lose lock", + ) + .unwrap(); held }); let output = worker.wait_with_output().unwrap(); @@ -2566,7 +2584,7 @@ mod request_write_ceilings { &[("independent physical selected frontier", "answer")], ); let text = std::fs::read_to_string(&path).unwrap(); - std::fs::write(path, text.replace("2026-09-01", "2026-09-02")).unwrap(); + event_support::write_test_file(path, text.replace("2026-09-01", "2026-09-02")).unwrap(); let independent = import_selected( &machine, &project, @@ -2590,7 +2608,7 @@ mod request_write_ceilings { requests[0]["memory_refusal"] = serde_json::json!({"Recorded": {"ordinal": 1, "refusal": {"requested_bytes": 42, "limit_bytes": LOW}}}); } write_event_search_worker_policy(&machine.data_root, &accepted.to_string()).unwrap(); - std::fs::write( + event_support::write_test_file( observer.join("refuse-next-selected-memory"), LOW.to_string(), ) @@ -2659,6 +2677,14 @@ mod request_write_ceilings { #[test] fn a_completed_newest_one_retry_clears_its_refusal_without_embedding_older_history() { + struct ReleaseBackgroundTurn(std::os::unix::net::UnixStream); + impl Drop for ReleaseBackgroundTurn { + fn drop(&mut self) { + use std::io::Write; + let _ = self.0.write_all(&[1]); + } + } + for retired in [false, true] { let machine = Machine::with_nomic(); let project = machine.project("newest-one-refusal"); @@ -2687,13 +2713,13 @@ mod request_write_ceilings { [0] .clone(); assert_selection_row_counts(&machine, 1, 1); - std::fs::write( + event_support::write_test_file( observer.join("refuse-next-selected-memory"), LOW.to_string(), ) .unwrap(); if !retired { - std::fs::write( + event_support::write_test_file( observer.join("pause-before-selection-cleanup"), b"retain frontier", ) @@ -2723,24 +2749,60 @@ mod request_write_ceilings { refused["pending_background_request"]["write_requests"][0]["selection_retired"], retired ); - let retry = machine.run( - &project, - &[ - "import", - ".", - "--context", - "newest-one-refusal", - "--search", - "wait", - "--search-sessions", - "1", - "--embedding-workers", - "1", - "--write-memory-limit-bytes", - &HIGH.to_string(), - ], - ); + // The retry may kick detached housekeeping. Hold that actor at its + // existing pre-catch-up seam while asserting the saved request, so + // selection retirement cannot race the policy snapshot. + let housekeeping_root = tempfile::Builder::new() + .prefix("cgh") + .tempdir_in("/tmp") + .expect("short housekeeping socket directory"); + let barrier = housekeeping_root.path().to_path_buf(); + let housekeeping = (!retired).then(|| { + std::os::unix::net::UnixListener::bind(barrier.join("pre-catch-up.sock")) + .expect("bind housekeeping rendezvous") + }); + let high = HIGH.to_string(); + let arguments = [ + "import", + ".", + "--context", + "newest-one-refusal", + "--search", + "wait", + "--search-sessions", + "1", + "--embedding-workers", + "1", + "--write-memory-limit-bytes", + &high, + ]; + let retry = if retired { + machine.run(&project, &arguments) + } else { + machine.run_with_env( + &project, + &arguments, + "CG_EVENT_SEARCH_TEST_PRE_CATCH_UP_BARRIER", + &barrier, + ) + }; assert_eq!(retry.code, 0, "{}", retry.describe()); + let mut release_background = housekeeping.map(|listener| { + let (mut turn, _) = listener.accept().expect("detached housekeeping arrived"); + use std::io::Read; + let mut entered = [0]; + turn.read_exact(&mut entered).unwrap(); + assert_eq!(entered, [1], "worker holds its turn before cleanup"); + // Hold only this turn. Later turns use the pre-released legacy + // checkpoint after the socket is unlinked; this connection stays held. + std::fs::remove_file(barrier.join("pre-catch-up.sock")).unwrap(); + event_support::write_test_file( + barrier.join("pre-catch-up-release"), + b"later turns pass", + ) + .unwrap(); + ReleaseBackgroundTurn(turn) + }); assert_eq!( worker_policy(&machine.data_root)["pending_background_request"], refused["pending_background_request"] @@ -2764,7 +2826,8 @@ mod request_write_ceilings { )], ); let text = std::fs::read_to_string(&newer).unwrap(); - std::fs::write(newer, text.replace("2026-09-01", "2026-10-01")).unwrap(); + event_support::write_test_file(newer, text.replace("2026-09-01", "2026-10-01")) + .unwrap(); let later = machine.run( &project, &[ @@ -2798,11 +2861,15 @@ mod request_write_ceilings { appended["timestamp"] = "2026-09-02T09:00:00Z".into(); appended["message"]["content"] = "deferred beyond the refused capture boundary".into(); - std::fs::write(&source, format!("{contents}{appended}\n")).unwrap(); + event_support::write_test_file(&source, format!("{contents}{appended}\n")).unwrap(); (4, 5) } else { (2, 2) }; + if let Some(release) = release_background.take() { + drop(release); + machine.wait_for_search_worker_exit(); + } let acknowledged = import_selected( &machine, &project, @@ -2892,7 +2959,7 @@ mod request_write_ceilings { assert_selection_row_counts(&machine, 1, 1); assert_completion_row_counts(&machine, 0, 0); - std::fs::write( + event_support::write_test_file( observer.join("refuse-next-selected-memory"), LOW.to_string(), ) @@ -3084,13 +3151,13 @@ mod request_write_ceilings { .unwrap() ); drop(reader); - std::fs::write( + event_support::write_test_file( observer.join("refuse-next-selected-memory"), LOW.to_string(), ) .unwrap(); if !retired { - std::fs::write( + event_support::write_test_file( observer.join("pause-before-selection-cleanup"), b"retain the accepted frontier", ) @@ -3150,7 +3217,7 @@ mod request_write_ceilings { let suffix = std::fs::read(&history).unwrap(); let mut appended = prefix.clone(); appended.extend(suffix); - std::fs::write(&history, appended).unwrap(); + event_support::write_test_file(&history, appended).unwrap(); assert!(std::fs::read(&history).unwrap().starts_with(&prefix)); let held = acquire_worker_lock(&machine.data_root.join("event-search-worker.lock")) .expect("hold fresh narrow request admission"); @@ -3574,13 +3641,13 @@ mod request_write_ceilings { [0] .clone(); assert_selection_row_counts(&machine, 1, 2); - std::fs::write( + event_support::write_test_file( observer.join("refuse-next-selected-memory"), LOW.to_string(), ) .unwrap(); if !retired { - std::fs::write( + event_support::write_test_file( observer.join("pause-before-selection-cleanup"), b"retain the accepted frontier", ) @@ -3821,7 +3888,7 @@ mod request_write_ceilings { ); assert_eq!(accepted_b.code, 0, "{}", accepted_b.describe()); std::fs::remove_file(observer_b.join("release")).unwrap(); - std::fs::write(observer_b.join("numbered-passes"), b"1").unwrap(); + event_support::write_test_file(observer_b.join("numbered-passes"), b"1").unwrap(); drop(held); let mut worker_b = machine.spawn_with_env( &project_b, @@ -3902,7 +3969,11 @@ mod request_write_ceilings { acquire_worker_lock(&machine.data_root.join("event-search-worker.lock")).is_err() ); assert_selection_counts(&machine, 0); - std::fs::write(observer_b.join("release.1"), b"acknowledge B after unlock").unwrap(); + event_support::write_test_file( + observer_b.join("release.1"), + b"acknowledge B after unlock", + ) + .unwrap(); let output = worker_b.wait_with_output().unwrap(); assert!(output.status.success(), "{output:?}"); assert_ready(&machine, &project_a, "acknowledgement-completed-a"); @@ -4095,7 +4166,7 @@ mod request_write_ceilings { write_event_search_worker_policy(&machine.data_root, &saved.to_string()).unwrap(); let old = refuse_queue(&machine, LOW); let barrier = request_observer(&machine, "older-import"); - std::fs::write( + event_support::write_test_file( barrier.join("pause-before-import-acknowledgement"), b"pause after assessment", ) @@ -4146,7 +4217,7 @@ mod request_write_ceilings { newer_token, old["pending_background_request"]["write_requests"][0]["token"] ); - std::fs::write( + event_support::write_test_file( newer_observer.join("refuse-next-selected-memory"), LOW.to_string(), ) @@ -4175,7 +4246,7 @@ mod request_write_ceilings { newer["pending_background_request"]["write_requests"][0]["refused_memory"], true ); - std::fs::write( + event_support::write_test_file( barrier.join("release-import-acknowledgement"), b"acknowledge only observed identities", ) @@ -4274,6 +4345,9 @@ mod request_write_ceilings { let ready = import(&machine, &old, "old-context", HIGH, &observer); assert_eq!(ready.code, 0, "{}", ready.describe()); machine.wait_for_search_ready(&old, "old-context"); + // The long worker lock is released before queue acknowledgement. The + // initial request must be gone before this case constructs its two new owners. + await_acknowledgement(&machine); machine.wait_for_search_worker_exit(); std::fs::remove_file(observer.join("actual-write-memory-batches")) .expect("reset admissions"); @@ -4383,7 +4457,7 @@ mod request_write_ceilings { &[("D selected in the running request", "D answer")], ); let d_prefix = std::fs::read_to_string(&d).expect("D source"); - std::fs::write(&d, d_prefix.replace("2026-09-01", "2026-08-01")) + event_support::write_test_file(&d, d_prefix.replace("2026-09-01", "2026-08-01")) .expect("D follows B in newest order"); std::fs::create_dir_all(&machine.data_root).expect("queue root"); let held = acquire_worker_lock(&machine.data_root.join("event-search-worker.lock")) @@ -4432,7 +4506,8 @@ mod request_write_ceilings { ); let expanded = std::fs::read_to_string(&source).expect("B suffix"); let suffix = expanded.lines().skip(2).collect::>().join("\n"); - std::fs::write(&source, format!("{prefix}{suffix}\n")).expect("append preserves B prefix"); + event_support::write_test_file(&source, format!("{prefix}{suffix}\n")) + .expect("append preserves B prefix"); let c = write_session( &machine.history_root(), &project, @@ -4440,7 +4515,8 @@ mod request_write_ceilings { &[("newest C request", "C answer")], ); let c_text = std::fs::read_to_string(&c).expect("C source"); - std::fs::write(&c, c_text.replace("2026-09-01", "2026-09-02")).expect("C is newest"); + event_support::write_test_file(&c, c_text.replace("2026-09-01", "2026-09-02")) + .expect("C is newest"); write_config( &machine, "[search.passages]\ntarget_tokens = 8\nhard_tokens = 64\n", @@ -4504,7 +4580,7 @@ mod request_write_ceilings { 2, ); drop(reader); - std::fs::write(barrier.join("release"), b"resume P1 publication") + event_support::write_test_file(barrier.join("release"), b"resume P1 publication") .expect("release embedding"); let result = worker .wait_with_output() @@ -4569,7 +4645,8 @@ mod request_write_ceilings { &[("retained earlier searchable history", "retained answer")], ); let text = std::fs::read_to_string(&a).expect("A source"); - std::fs::write(&a, text.replace("2026-09-01", "2026-07-01")).expect("A precedes B"); + event_support::write_test_file(&a, text.replace("2026-09-01", "2026-07-01")) + .expect("A precedes B"); let observer = request_observer(&machine, "retained-profile-observer"); let imported = import( &machine, @@ -4580,6 +4657,9 @@ mod request_write_ceilings { ); assert_eq!(imported.code, 0, "{}", imported.describe()); machine.wait_for_search_ready(&project, "retained-profile-selection"); + // Ready coverage precedes the initial worker's queue acknowledgment. + // Retire that request before publishing a tail and replacing its profile. + await_acknowledgement(&machine); machine.wait_for_search_worker_exit(); let original = std::fs::read_to_string(&a).expect("captured A prefix"); write_session( @@ -4593,7 +4673,7 @@ mod request_write_ceilings { ); let expanded = std::fs::read_to_string(&a).expect("A appended source"); let suffix = expanded.lines().skip(2).collect::>().join("\n"); - std::fs::write( + event_support::write_test_file( &a, format!("{original}{suffix}\n").replace("2026-09-01", "2026-07-01"), ) @@ -4736,7 +4816,8 @@ mod request_write_ceilings { &[("older unrelated request", "older answer")], ); let text = std::fs::read_to_string(&a).expect("A source"); - std::fs::write(a, text.replace("2026-09-01", "2026-07-01")).expect("A precedes B"); + event_support::write_test_file(a, text.replace("2026-09-01", "2026-07-01")) + .expect("A precedes B"); std::fs::create_dir_all(&machine.data_root).expect("queue root"); let held = acquire_worker_lock(&machine.data_root.join("event-search-worker.lock")) .expect("hold claims"); @@ -4923,7 +5004,8 @@ mod request_write_ceilings { let original = worker_policy(&machine.data_root)["pending_background_request"]["write_requests"][0] .clone(); - std::fs::write(observer.join("refuse-selected-active-time-count"), b"10").unwrap(); + event_support::write_test_file(observer.join("refuse-selected-active-time-count"), b"10") + .unwrap(); drop(held); let refused = machine.run_with_env( &project, @@ -5046,7 +5128,7 @@ mod request_write_ceilings { "pause after saving the assessed identities", ), ] { - std::fs::write(observer.join(marker), content).unwrap(); + event_support::write_test_file(observer.join(marker), content).unwrap(); } drop(held); let worker = machine.spawn_with_env( @@ -5080,7 +5162,7 @@ mod request_write_ceilings { ) }) .collect::>(); - std::fs::write( + event_support::write_test_file( observer.join("release-aggregate-active-time-refusal"), b"refuse observed statement", ) @@ -5105,7 +5187,7 @@ mod request_write_ceilings { } std::fs::remove_file(observer.join("pause-before-aggregate-active-time-refusal")).unwrap(); std::fs::remove_file(observer.join("pause-after-read-timeout")).unwrap(); - std::fs::write( + event_support::write_test_file( observer.join("release-read-timeout-recovery"), b"continue remaining accepted work", ) @@ -5175,7 +5257,8 @@ mod request_write_ceilings { &observer, ); assert_eq!(accepted.code, 0, "{}", accepted.describe()); - std::fs::write(observer.join(marker), failures.to_string()).unwrap(); + event_support::write_test_file(observer.join(marker), failures.to_string()) + .unwrap(); drop(held); let worker = machine.run_with_env( &project, @@ -5288,12 +5371,12 @@ mod request_write_ceilings { &observer, ); assert_eq!(accepted.code, 0, "{}", accepted.describe()); - std::fs::write( + event_support::write_test_file( observer.join("refuse-active-time-after-committed-batch"), b"one typed refusal", ) .expect("refuse after this request commits its first batch"); - std::fs::write( + event_support::write_test_file( observer.join("pause-after-read-timeout"), b"restart after committed progress", ) @@ -5376,19 +5459,22 @@ mod request_write_ceilings { let project = machine.project("foreground-assessment-recovery"); capture_b(&machine, &project); let observer = request_observer(&machine, "foreground-assessment-observer"); - std::fs::write( + event_support::write_test_file( observer.join("refuse-foreground-assessment-active-time-count"), failures.to_string(), ) .unwrap(); - std::fs::write( + event_support::write_test_file( observer.join("foreground-assessment-refusal-after-runs"), after_runs.to_string(), ) .unwrap(); if after_runs == 1 { - std::fs::write(observer.join("refuse-foreground-active-time-count"), b"1") - .unwrap(); + event_support::write_test_file( + observer.join("refuse-foreground-active-time-count"), + b"1", + ) + .unwrap(); } let args = [ "import", @@ -5541,7 +5627,7 @@ mod request_write_ceilings { &[("other request", "other answer")], ); let observer = request_observer(&machine, "large-foreground-refusal-observer"); - std::fs::write( + event_support::write_test_file( observer.join("refuse-foreground-selection-active-time-count"), b"10", ) @@ -5591,7 +5677,7 @@ mod request_write_ceilings { ]; let mut last_ordinal = other_refusal["ordinal"].as_u64().unwrap(); for _ in 0..3 { - std::fs::write( + event_support::write_test_file( observer.join("refuse-foreground-assessment-active-time-count"), b"10", ) @@ -5678,7 +5764,7 @@ mod request_write_ceilings { &[("newer selected request", "newer answer")], ); let observer = request_observer(&machine, "foreground-selection-observer"); - std::fs::write( + event_support::write_test_file( observer.join("refuse-foreground-selection-active-time-count"), b"10", ) @@ -5767,8 +5853,9 @@ mod request_write_ceilings { let project = machine.project("foreground-read-time-bound"); capture_b(&machine, &project); let observer = request_observer(&machine, "foreground-read-time-bound-observer"); - std::fs::write(observer.join("refuse-foreground-active-time-count"), b"10").unwrap(); - std::fs::write( + event_support::write_test_file(observer.join("refuse-foreground-active-time-count"), b"10") + .unwrap(); + event_support::write_test_file( observer.join("pause-after-foreground-read-timeout"), b"pause", ) @@ -5813,20 +5900,15 @@ mod request_write_ceilings { ); assert_eq!(capture.code, 0, "{}", capture.describe()); assert_eq!(u64_at(&capture.json(), "/search/covered_events"), 0); - std::fs::write( + event_support::write_test_file( observer.join(format!("release-foreground-read-timeout-{remaining}")), b"capture changed without coverage progress", ) .unwrap(); } - assert!( - wait_for_state(Duration::from_secs(30), Duration::from_millis(20), || { - foreground.try_wait().map(|status| status.map(|_| ())) - }) - .expect("observe the bounded foreground exit") - .is_some(), - "foreground kept retrying after three refusals without coverage progress" - ); + foreground + .wait() + .expect("bounded foreground exit after three refusals"); let output = foreground .wait_with_output() .expect("foreground terminates at its bound"); @@ -5931,7 +6013,8 @@ mod request_write_ceilings { worker_policy(&machine.data_root)["pending_background_request"]["write_requests"][0] .clone(); assert_eq!(saved["read_timeout_covered_events"], 2); - std::fs::write(observer.join("refuse-selected-active-time-count"), b"10").unwrap(); + event_support::write_test_file(observer.join("refuse-selected-active-time-count"), b"10") + .unwrap(); drop(held); let refused = machine.run_with_env( &project, @@ -5979,7 +6062,7 @@ mod request_write_ceilings { let observer = request_observer(&machine, "read-time-bound-observer"); let accepted = import_selected(&machine, &project, "read-time-bound", LOW, "1", &observer); assert_eq!(accepted.code, 0, "{}", accepted.describe()); - std::fs::write(observer.join("refuse-selected-active-time-count"), b"10") + event_support::write_test_file(observer.join("refuse-selected-active-time-count"), b"10") .expect("more refusals than allowance"); drop(held); let worker = machine.run_with_env( @@ -6053,8 +6136,9 @@ mod request_write_ceilings { worker_policy(&machine.data_root)["pending_background_request"]["write_requests"][0] .clone(); assert_eq!(accepted_request["read_active_time_ms"], 5000); - std::fs::write(observer.join("refuse-selected-active-time-count"), b"10").unwrap(); - std::fs::write( + event_support::write_test_file(observer.join("refuse-selected-active-time-count"), b"10") + .unwrap(); + event_support::write_test_file( observer.join("pause-after-read-timeout"), b"pause after durable recovery count", ) @@ -6171,7 +6255,7 @@ mod request_write_ceilings { &observer, ); assert_eq!(accepted.code, 0, "{}", accepted.describe()); - std::fs::write( + event_support::write_test_file( observer.join("refuse-coordination-after-committed-batch"), b"one typed refusal", ) @@ -6248,7 +6332,7 @@ mod request_write_ceilings { &observer, ); assert_eq!(accepted.code, 0, "{}", accepted.describe()); - std::fs::write(observer.join("refuse-selected-coordination-count"), b"10") + event_support::write_test_file(observer.join("refuse-selected-coordination-count"), b"10") .expect("more refusals than allowance"); drop(held); let worker = machine.run_with_env( @@ -6331,9 +6415,13 @@ mod request_write_ceilings { ); assert_eq!(later.code, 0, "{}", later.describe()); std::fs::remove_file(observer.join("release")).expect("pause after refusal"); - std::fs::write(observer.join("numbered-passes"), b"1").expect("observe first pass"); - std::fs::write(observer.join("refuse-next-selected-work"), b"typed refusal") - .expect("refuse B once"); + event_support::write_test_file(observer.join("numbered-passes"), b"1") + .expect("observe first pass"); + event_support::write_test_file( + observer.join("refuse-next-selected-work"), + b"typed refusal", + ) + .expect("refuse B once"); drop(held); let worker = machine.spawn_with_env( &project, @@ -6366,7 +6454,8 @@ mod request_write_ceilings { str_at(&status.json(), "/history/search/last_failure") .contains("explicit test refusal") ); - std::fs::write(observer.join("release.1"), b"serve C").expect("release worker"); + event_support::write_test_file(observer.join("release.1"), b"serve C") + .expect("release worker"); let output = worker.wait_with_output().expect("worker completes"); assert!(output.status.success(), "{:?}", output); assert!(worker_policy(&machine.data_root)["pending_background_request"].is_null()); @@ -6504,7 +6593,8 @@ mod request_write_ceilings { ); let expanded = std::fs::read_to_string(&source).expect("expanded source"); let suffix = expanded.lines().skip(2).collect::>().join("\n"); - std::fs::write(&source, format!("{original}{suffix}\n")).expect("deferred append"); + event_support::write_test_file(&source, format!("{original}{suffix}\n")) + .expect("deferred append"); let deferred = machine.run( &project, &[ @@ -6704,7 +6794,7 @@ mod request_write_ceilings { ); let expanded = std::fs::read_to_string(&growing).expect("expanded session"); let suffix = expanded.lines().skip(2).collect::>().join("\n"); - std::fs::write(&growing, format!("{prefix}{suffix}\n")) + event_support::write_test_file(&growing, format!("{prefix}{suffix}\n")) .expect("append while preserving source prefix"); let later = import_selected( &appended, @@ -7045,7 +7135,8 @@ mod request_write_ceilings { // The active batch has a claim, but embedding retains no writable Store. A second // independent capture must still succeed before releasing the first batch. let second = import(&machine, &second_project, "second-budget", LOW, &observer); - std::fs::write(observer.join("release"), b"continue\n").expect("release active batch"); + event_support::write_test_file(observer.join("release"), b"continue\n") + .expect("release active batch"); await_acknowledgement(&machine); machine.wait_for_search_worker_exit(); assert_eq!(second.code, 0, "{}", second.describe()); @@ -7082,7 +7173,7 @@ mod request_write_ceilings { &[("older request", "older answer")], ); let old_source = std::fs::read_to_string(&old).expect("read old fixture"); - std::fs::write(old, old_source.replace("2026-09-01", "2026-08-01")) + event_support::write_test_file(old, old_source.replace("2026-09-01", "2026-08-01")) .expect("order old session first"); let middle = write_session( &scoped.history_root(), @@ -7091,7 +7182,7 @@ mod request_write_ceilings { &[("middle request", "middle answer")], ); let middle_source = std::fs::read_to_string(&middle).expect("read middle fixture"); - std::fs::write(middle, middle_source.replace("2026-09-01", "2026-08-15")) + event_support::write_test_file(middle, middle_source.replace("2026-09-01", "2026-08-15")) .expect("keep exclusive work after the first claim"); write_session( &scoped.history_root(), @@ -7145,7 +7236,7 @@ mod request_write_ceilings { REQUEST_OBSERVER_ENV, &scope_observer, ); - std::fs::write(scope_observer.join("release"), b"serve both selectors") + event_support::write_test_file(scope_observer.join("release"), b"serve both selectors") .expect("release scoped claims"); await_acknowledgement(&scoped); scoped.wait_for_search_worker_exit(); @@ -7238,7 +7329,7 @@ mod request_write_ceilings { terminate_observed_background_worker(&orphan_pause); orphan.wait_for_search_worker_exit(); let lease_wait = batch_observer(&orphan, "orphan-lease-wait"); - std::fs::write(lease_wait.join("numbered-passes"), b"1") + event_support::write_test_file(lease_wait.join("numbered-passes"), b"1") .expect("observe the lease-only pass"); let resumed = orphan.spawn_with_env( &orphan_project, @@ -7270,7 +7361,7 @@ mod request_write_ceilings { .expect("expire the orphan claim deterministically"); assert_eq!(changed.rows_affected, 1); } - std::fs::write(lease_wait.join("release"), b"resume expired claim") + event_support::write_test_file(lease_wait.join("release"), b"resume expired claim") .expect("release replacement"); let output = resumed.wait_with_output().expect("replacement completes"); assert_eq!( @@ -7294,9 +7385,9 @@ mod request_write_ceilings { &[first_turn], ); let observer = batch_observer(&machine, "shared-budget-pause"); - std::fs::write(observer.join("numbered-passes"), b"2") + event_support::write_test_file(observer.join("numbered-passes"), b"2") .expect("pause after completed pass"); - std::fs::write(observer.join("release.1"), b"complete first batch\n") + event_support::write_test_file(observer.join("release.1"), b"complete first batch\n") .expect("release first claim"); let first = import( &machine, @@ -7319,7 +7410,7 @@ mod request_write_ceilings { std::fs::read_to_string(&original).expect("read unchanged captured prefix"); let suffix = std::fs::read_to_string(&additional).expect("read new source records"); prefix.push_str(&suffix.replace("additional-session", "shared-session")); - std::fs::write(&original, prefix) + event_support::write_test_file(&original, prefix) .expect("append without changing captured source identity"); std::fs::remove_file(additional).expect("remove temporary source fixture"); let second = import(&machine, &project, "shared-budget", next_limit, &observer); @@ -7444,7 +7535,7 @@ mod request_write_ceilings { } write_event_search_worker_policy(&machine.data_root, &pending.to_string()) .expect("deliver next request ceiling"); - std::fs::write( + event_support::write_test_file( observer.join("release"), b"finish admitted batch and reopen\n", ) @@ -7509,7 +7600,7 @@ mod request_write_ceilings { let suffix = std::fs::read(&history).unwrap(); let mut appended = prefix.clone(); appended.extend(suffix); - std::fs::write(&history, &appended).unwrap(); + event_support::write_test_file(&history, &appended).unwrap(); assert!(std::fs::read(&history).unwrap().starts_with(&prefix)); let retry_observer = request_observer(&machine, "affordable-retry"); let resumed = import( diff --git a/crates/cg-cli/tests/support/mod.rs b/crates/cg-cli/tests/support/mod.rs index 935d170..0f68fbf 100644 --- a/crates/cg-cli/tests/support/mod.rs +++ b/crates/cg-cli/tests/support/mod.rs @@ -8,7 +8,7 @@ use std::path::{Path, PathBuf}; use std::process::{Command, Stdio}; use std::sync::mpsc; use std::thread; -use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH}; +use std::time::{SystemTime, UNIX_EPOCH}; use serde_json::Value; use tempfile::TempDir; @@ -192,6 +192,16 @@ fn run_cg_observing_provider_before_exit_inner( Err(err) => panic!("remove stale provider marker {}: {err}", marker.display()), } } + let release_directory = tempfile::tempdir_in("/tmp").expect("short provider release directory"); + let release_path = release_directory.path().join("provider.sock"); + let mut release_listener = Some( + std::os::unix::net::UnixListener::bind(&release_path).expect("provider release listener"), + ); + fs::write( + provider_started_marker.with_file_name("completion-release-address"), + release_path.as_os_str().as_encoded_bytes(), + ) + .expect("publish provider release address"); let mut command = Command::new(cg_bin()); command .args(args) @@ -222,29 +232,28 @@ fn run_cg_observing_provider_before_exit_inner( let (tx, rx) = mpsc::channel(); let stdout_handle = spawn_stream_reader(stdout, tx.clone()); let stderr_handle = spawn_stream_reader(stderr, tx); - let deadline = Instant::now() + Duration::from_secs(90); let mut observed_before_exit = false; let mut observed_while_provider_running = false; - let status = loop { - while let Ok(chunk) = rx.try_recv() { - if chunk.contains(needle) { - observed_before_exit = true; - if provider_started_marker.exists() && !provider_exited_marker.exists() { - observed_while_provider_running = true; + for chunk in rx { + if chunk.contains(needle) { + observed_before_exit |= child.try_wait().expect("child status").is_none(); + if provider_started_marker.exists() && !provider_exited_marker.exists() { + observed_while_provider_running = true; + if let Some(listener) = release_listener.take() { + let (mut release, _) = + listener.accept().expect("provider is held at completion"); + release + .write_all(&[1]) + .expect("release provider after observing progress"); } } } - if let Some(status) = child.try_wait().expect("poll cg child") { - break status; - } - if Instant::now() >= deadline { - child.kill().expect("kill timed-out cg child"); - break child.wait().expect("collect killed cg child"); - } - thread::sleep(Duration::from_millis(20)); - }; + } + let status = child.wait().expect("collect cg child"); let stdout = stdout_handle.join().expect("join stdout reader"); let stderr = stderr_handle.join().expect("join stderr reader"); + fs::remove_file(provider_started_marker.with_file_name("completion-release-address")) + .expect("remove provider release address"); ObservedCommandOutput { output: CommandOutput { code: status.code().unwrap_or(-1), @@ -284,30 +293,21 @@ pub fn run_cg_writing_stdin_after_output( let (tx, rx) = mpsc::channel(); let stdout_handle = spawn_stream_reader(stdout, tx.clone()); let stderr_handle = spawn_stream_reader(stderr, tx); - let deadline = Instant::now() + Duration::from_secs(10); let mut observed_before_exit = false; let mut held_stdin = None; - let status = loop { - while let Ok(chunk) = rx.try_recv() { - if chunk.contains(needle) { - observed_before_exit = true; - } - } - if observed_before_exit && let Some(mut child_stdin) = stdin.take() { + for chunk in rx { + if chunk.contains(needle) + && let Some(mut child_stdin) = stdin.take() + { + // The command is waiting on the stdin this test still owns. + observed_before_exit = child.try_wait().expect("child status").is_none(); child_stdin .write_all(stdin_after.as_bytes()) .expect("write delayed cg stdin"); held_stdin = Some(child_stdin); } - if let Some(status) = child.try_wait().expect("poll cg child") { - break status; - } - if Instant::now() >= deadline { - child.kill().expect("kill timed-out cg child"); - break child.wait().expect("collect killed cg child"); - } - thread::sleep(Duration::from_millis(20)); - }; + } + let status = child.wait().expect("collect cg child"); let stdout = stdout_handle.join().expect("join stdout reader"); let stderr = stderr_handle.join().expect("join stderr reader"); drop(held_stdin); @@ -426,6 +426,7 @@ impl CompletionBehavior { pub struct FakeProviderRig { _root: TempDir, + driver_interpreter: PathBuf, pub root_path: PathBuf, pub bin_dir: PathBuf, pub fixture_dir: PathBuf, @@ -434,6 +435,8 @@ pub struct FakeProviderRig { impl FakeProviderRig { pub fn new() -> Self { + // Bind the harness dependency before cg receives its restricted provider PATH. + let driver_interpreter = fixture_driver_interpreter(); let root = TempDir::new().expect("fake provider root"); let root_path = root.path().to_path_buf(); let bin_dir = root_path.join("bin"); @@ -442,6 +445,7 @@ impl FakeProviderRig { fs::create_dir_all(&fixture_dir).expect("fake fixture dir"); let rig = Self { _root: root, + driver_interpreter, root_path, bin_dir, fixture_dir, @@ -693,7 +697,8 @@ impl FakeProviderRig { fn write_wrapper(&self, provider: Provider) { let script = format!( - "#!/bin/sh\npython3 {} {} {} \"$@\"\n", + "#!/bin/sh\nexec {} {} {} {} \"$@\"\n", + shell_quote(&self.driver_interpreter), shell_quote(self.fixture_dir.join("fake_provider.py")), shell_quote(provider.binary_name()), shell_quote(&self.fixture_dir), @@ -709,7 +714,8 @@ import json import os import pathlib import sys -import time +import socket +import threading provider = sys.argv[1] root = pathlib.Path(sys.argv[2]) @@ -767,7 +773,7 @@ def write_log(mode, prompt=""): out.write(json.dumps(record, sort_keys=True) + "\n") def mark(name): - (fixture / name).write_text(str(time.time())) + (fixture / name).write_text("reached") def proof(secret, prompt): value = 14695981039346656037 @@ -1175,7 +1181,7 @@ def claude_envelope(reply, is_error=False, status=None): if "--help" in args: write_log("probe") if read("help_probe_behavior", "normal") == "hang": - time.sleep(120) + threading.Event().wait() if provider == "codex" and args and args[0] == "exec": print(read("exec_help_flags", "--output-last-message\n-C")) elif provider == "agy": @@ -1199,7 +1205,7 @@ if "auth" in args and "status" in args: write_log("auth-status-probe") probe_behavior = read("auth_probe_behavior", "normal") if probe_behavior == "hang": - time.sleep(120) + threading.Event().wait() if probe_behavior == "unsupported": print("unknown command: auth status", file=sys.stderr) sys.exit(2) @@ -1271,9 +1277,11 @@ if read("login_state", "valid") == "expired": behavior = read("completion_behavior", "succeed") if behavior == "hang": - time.sleep(120) -if behavior in ("sleep-then-succeed", "sleep-then-extract-harvest"): - time.sleep(1) + threading.Event().wait() +if behavior in ("sleep-then-succeed", "sleep-then-extract-harvest") and (fixture / "completion-release-address").exists(): + with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as release: + release.connect(read("completion-release-address")) + assert release.recv(1) == b"\x01", "parent explicitly releases completion after observing progress" if behavior == "billing-required": mark("completion_exited") if provider == "claude": @@ -1366,24 +1374,17 @@ pub fn run_variadic_fake(argv: &[&str], stdin: &str) -> CommandOutput { LoginState::Valid, CompletionBehavior::Succeed, ); - let mut command = Command::new(rig.bin_dir.join("claude")); + let mut command = Command::new("/bin/sh"); + command.arg(rig.bin_dir.join("claude")); command .args(argv) .env("PATH", rig.path_env()) .stdin(Stdio::piped()) .stdout(Stdio::piped()) .stderr(Stdio::piped()); - let mut attempts = 0; - let mut child = loop { - match command.spawn() { - Ok(child) => break child, - Err(err) if err.raw_os_error() == Some(26) && attempts < 20 => { - attempts += 1; - thread::sleep(Duration::from_millis(10)); - } - Err(err) => panic!("spawn fake claude: {err}"), - } - }; + let mut child = command + .spawn() + .expect("spawn fake claude through its shell interpreter"); if let Some(mut child_stdin) = child.stdin.take() && let Err(err) = child_stdin.write_all(stdin.as_bytes()) { @@ -1475,6 +1476,32 @@ fn walk_files(root: &Path) -> Vec { out } +fn fixture_driver_interpreter() -> PathBuf { + let runner_path = std::env::var_os("PATH") + .expect("fake provider fixture requires python3 on the test runner PATH (PATH is unset)"); + for directory in std::env::split_paths(&runner_path) { + // Canonicalization binds relative PATH entries and follows interpreter symlinks. + let Ok(candidate) = fs::canonicalize(directory.join("python3")) else { + continue; + }; + let Ok(metadata) = fs::metadata(&candidate) else { + continue; + }; + if !metadata.is_file() { + continue; + } + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + if metadata.permissions().mode() & 0o111 == 0 { + continue; + } + } + return candidate; + } + panic!("fake provider fixture requires an executable python3 on the test runner PATH"); +} + fn shell_quote(value: impl AsRef) -> String { let value = value.as_ref().display().to_string(); format!("'{}'", value.replace('\'', "'\\''")) diff --git a/crates/cg-node-fs/AGENTS.md b/crates/cg-node-fs/AGENTS.md index 58a8193..1229f8d 100644 --- a/crates/cg-node-fs/AGENTS.md +++ b/crates/cg-node-fs/AGENTS.md @@ -16,7 +16,7 @@ event identity, SQL, or embedding. This crate consumes no ContextDB read route o | Latest request replacement | `read_event_search_worker_policy`, `write_event_search_worker_policy` | | Published request size ceiling | `EVENT_SEARCH_WORKER_POLICY_MAX_BYTES`; callers reserve space before publication | | Serialized request update | `acquire_event_search_worker_policy_update` | -| Installed/container memory discovery | `machine_memory`; `context_graph::write_budget` chooses the budget | +| Installed/container memory discovery | `machine_memory` reads installed memory from `/proc/meminfo` on Linux and `sysctl hw.memsize` on macOS, plus any container allowance; `context_graph::write_budget` chooses the budget | ## Invariants and guards @@ -30,6 +30,7 @@ Ownership boundaries above are review-only architectural constraints unless a gu | Requests over 4 KiB are refused without replacing the last valid value. | `crates/cg-node-fs/tests/event_search_worker_policy.rs::policy_record_refuses_more_than_four_kibibytes_without_replacing_the_last_value` | | The update guard reads and replaces one bounded record. | `crates/cg-node-fs/tests/event_search_worker_policy.rs::update_guard_reads_and_replaces_one_bounded_record` | | Given host and container memory facts, the core budget consumer sizes from the smaller container allowance. | `crates/context-graph/src/write_budget.rs::a_container_on_a_large_host_is_sized_from_the_container_not_the_host` | +| On Linux and macOS the live probe reports this machine's installed memory, and the budget origin names what was read. | `crates/context-graph/src/write_budget.rs::the_live_probe_reads_this_machine_and_the_origin_names_what_it_read` | ## Review-only authoring rule @@ -37,4 +38,10 @@ When changing the serialized request-update flow, release its guard before openi embedding, starting a process, or acquiring the worker lock. This ordering is an architectural code-review constraint; no automated test currently proves it. +Test-only coordination belongs beside these filesystem operations: `test-support` (or the +internal test cfg) enables marker notifications, explicit release waits, and +`wait_for_worker_lock_for_test`. The blocking OS lock wait also observes release after a killed +child. Notifications only wake an observer to reread real state; they do not acknowledge a request +or establish that its queue entry is gone. Keep these helpers hidden and feature-gated. + Fast check: `cargo test -p cg-node-fs`. Full gate: `scripts/gate.sh` at repository root. diff --git a/crates/cg-node-fs/Cargo.toml b/crates/cg-node-fs/Cargo.toml index e25fc5c..52357c9 100644 --- a/crates/cg-node-fs/Cargo.toml +++ b/crates/cg-node-fs/Cargo.toml @@ -9,6 +9,12 @@ repository.workspace = true description = "Node-deployment filesystem primitives for cg (config file read, per-data-root worker lock)" edition = "2024" +[features] +test-support = [] + +[lints.rust] +unexpected_cfgs = { level = "warn", check-cfg = ['cfg(context_graph_internal_test_support)'] } + [lib] name = "cg_node_fs" path = "src/lib.rs" diff --git a/crates/cg-node-fs/src/lib.rs b/crates/cg-node-fs/src/lib.rs index 90f0dc7..3f70cc7 100644 --- a/crates/cg-node-fs/src/lib.rs +++ b/crates/cg-node-fs/src/lib.rs @@ -99,8 +99,10 @@ pub fn write_event_search_worker_policy(root: &Path, contents: &str) -> std::io: temporary.as_file().sync_all()?; temporary .persist(root.join(EVENT_SEARCH_WORKER_POLICY)) - .map(|_| ()) - .map_err(|error| error.error) + .map_err(|error| error.error)?; + #[cfg(any(context_graph_internal_test_support, feature = "test-support"))] + notify_test_observers_for_test(); + Ok(()) } /// A held per-data-root worker lock. Dropping it releases the lock. The lock is an OS advisory @@ -111,6 +113,15 @@ pub struct WorkerLock { _file: std::fs::File, } +#[cfg(any(context_graph_internal_test_support, feature = "test-support"))] +impl Drop for WorkerLock { + fn drop(&mut self) { + use fs4::fs_std::FileExt; + let _ = FileExt::unlock(&self._file); + notify_test_observers_for_test(); + } +} + /// Why the per-data-root worker lock was not taken. /// /// The two variants are opposite answers to the only question a caller asks — "is my work going to @@ -156,6 +167,21 @@ pub fn acquire_worker_lock(lock_path: &Path) -> Result std::io::Result { + use fs4::fs_std::FileExt; + let file = std::fs::OpenOptions::new() + .create(true) + .truncate(false) + .read(true) + .write(true) + .open(lock_path)?; + file.lock_exclusive()?; + Ok(WorkerLock { _file: file }) +} + /// Load a config file's text, mapping a missing file to `Ok(None)` (an absent config is not an /// error — the node runs with defaults). Any other IO error surfaces as `Err`. pub fn load_text_if_present(path: &Path) -> std::io::Result> { @@ -175,7 +201,8 @@ pub fn load_text_if_present(path: &Path) -> std::io::Result> { /// a container gets killed instead of refused. #[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] pub struct MachineMemory { - /// Installed memory, from `/proc/meminfo`. `None` where the platform does not publish it. + /// Installed memory, from `/proc/meminfo` on Linux or `hw.memsize` on macOS. `None` where the + /// platform does not publish it. pub total_bytes: Option, /// The control group's own memory ceiling, when the kernel enforces a finite one. pub cgroup_limit_bytes: Option, @@ -185,15 +212,48 @@ pub struct MachineMemory { /// /// Every read is best-effort: a platform that publishes neither number answers `None` for both and /// the caller falls back to its own floor rather than failing. Nothing here dereferences a source -/// reference or reaches the network — these are the kernel's own published files. +/// reference or reaches the network — these are the kernel's own published memory facts. pub fn machine_memory() -> MachineMemory { MachineMemory { - total_bytes: read_proc_meminfo_total(), + total_bytes: read_installed_memory(), cgroup_limit_bytes: read_cgroup_memory_limit(), } } +/// Installed physical memory, in bytes, from the platform's own published source. +fn read_installed_memory() -> Option { + #[cfg(target_os = "macos")] + { + read_macos_memsize() + } + #[cfg(not(target_os = "macos"))] + { + read_proc_meminfo_total() + } +} + +/// Darwin publishes installed physical memory in bytes as the `hw.memsize` sysctl. The system +/// utility is addressed by absolute path, so the answer does not depend on the caller's `PATH`, +/// and no shell is involved. +#[cfg(target_os = "macos")] +fn read_macos_memsize() -> Option { + let output = std::process::Command::new("/usr/sbin/sysctl") + .args(["-n", "hw.memsize"]) + .output() + .ok()?; + if !output.status.success() { + return None; + } + let bytes = std::str::from_utf8(&output.stdout) + .ok()? + .trim() + .parse::() + .ok()?; + (bytes > 0).then_some(bytes) +} + /// `MemTotal:` from `/proc/meminfo`, in bytes. The file publishes kibibytes. +#[cfg(not(target_os = "macos"))] fn read_proc_meminfo_total() -> Option { let text = std::fs::read_to_string("/proc/meminfo").ok()?; let kibibytes = text @@ -253,3 +313,61 @@ fn legacy_cgroup_memory_limit() -> Option { // Anything at or above a pebibyte is that sentinel, not a limit anyone set. (bytes < (1 << 50)).then_some(bytes) } + +/// Level notification after a fixture-visible state change. A full socket already +/// carries a wakeup; the observer always rereads authoritative state after waking. +#[cfg(any(context_graph_internal_test_support, feature = "test-support"))] +#[doc(hidden)] +pub fn notify_test_observers_for_test() { + let Some(address) = std::env::var_os("CG_TEST_STATE_EVENTS") else { + return; + }; + let socket = std::os::unix::net::UnixDatagram::unbound().expect("test notification socket"); + socket + .set_nonblocking(true) + .expect("nonblocking test notification"); + let _ = socket.send_to(&[1], address); +} + +/// Replace one fixture marker and wake both its observer and its explicit release waiter. +#[cfg(any(context_graph_internal_test_support, feature = "test-support"))] +#[doc(hidden)] +pub fn write_test_file_for_test( + path: impl AsRef, + contents: impl AsRef<[u8]>, +) -> std::io::Result<()> { + let path = path.as_ref(); + std::fs::write(path, contents)?; + notify_test_observers_for_test(); + if let Some(parent) = path.parent() + && let Ok(address) = std::fs::read_to_string(parent.join(".release-event-address")) + { + let socket = std::os::unix::net::UnixDatagram::unbound()?; + socket.set_nonblocking(true)?; + let _ = socket.send_to(&[1], address); + } + Ok(()) +} + +/// Drive the existing release predicate after explicit writes, without timer polling. +#[cfg(any(context_graph_internal_test_support, feature = "test-support"))] +#[doc(hidden)] +pub fn wait_for_test_release_for_test(directory: &Path, mut released: impl FnMut() -> bool) { + if released() { + return; + } + let socket_dir = tempfile::tempdir_in("/tmp").expect("short release socket directory"); + let address = socket_dir.path().join("release.sock"); + let socket = std::os::unix::net::UnixDatagram::bind(&address).expect("release event socket"); + // Publish before rechecking, closing the race with a parent that has already released. + std::fs::write( + directory.join(".release-event-address"), + address.as_os_str().as_encoded_bytes(), + ) + .expect("publish release address"); + while !released() { + socket + .recv(&mut [0]) + .expect("explicit fixture release event"); + } +} diff --git a/crates/context-graph/AGENTS.md b/crates/context-graph/AGENTS.md index 14ff393..fbe10d8 100644 --- a/crates/context-graph/AGENTS.md +++ b/crates/context-graph/AGENTS.md @@ -69,6 +69,18 @@ Ownership boundaries above are review-only architectural constraints unless a gu `src/store/mod.rs`, and feature-gated bundled artifact IO in `src/embedder/nomic.rs`. Ordinary node filesystem work belongs to `cg-node-fs`; adding core IO requires an explicit placement decision. +Embedding-route validation preserves typed lookup refusals; only an absent space yields +`UnknownEmbeddingSpace`. Guard: `src/store/observation.rs::tests::embedding_route_lookup_is_the_writes_own_and_keeps_its_typed_refusal`. +A write's own lookups (`Store::write_lookup` in `src/store/graph_rows.rs`) are statement reads, never +charged to the caller read budget, so a write during a reindex is not refused for time spent queued +behind its commits. Guard: `src/store/observation.rs::tests::observation_recorded_while_a_reindex_commit_holds_the_commit_boundary_is_not_refused_by_the_read_budget`. +For deterministic reads, adapt ContextDB's existing `DeadlineClock` through the test-only +`src/store/graph_rows.rs` seam. For background capture, use the synchronous test drive in +`src/capture_sync_status.rs`, with the canonical Wallclock guard on the executing thread. +Both timing-site arms in `tests/test_estate_audit.rs` include private test modules; only default-tier +test counting excludes them. Never park a fixture in a pre-commit callback while its sibling +needs an engine read fence; rendezvous before the real statements enter that commit phase. + Fast check: `cargo test -p context-graph --test event_capture --test read_memory_policy`. Full gate: `scripts/gate.sh`. Public contracts live in [library API](../../docs/library-api.md) and [CLI reference](../../docs/cli.md); preserve the README doctest shim in `src/lib.rs`. diff --git a/crates/context-graph/Cargo.toml b/crates/context-graph/Cargo.toml index b570cfd..2be00ef 100644 --- a/crates/context-graph/Cargo.toml +++ b/crates/context-graph/Cargo.toml @@ -19,7 +19,7 @@ path = "src/lib.rs" [features] default = ["nomic-embedder"] # Private controls: activated by this crate's dev dependency, never by release defaults. -test-support = [] +test-support = ["cg-node-fs/test-support", "contextdb-engine/test-seams"] nomic-embedder = [ "dep:candle-core", "dep:candle-nn", @@ -44,7 +44,7 @@ chrono = { workspace = true, features = ["serde"] } contextdb-core.workspace = true contextdb-engine.workspace = true contextdb-parser.workspace = true -contextdb-server = { git = "https://github.com/context-graph-ai/contextdb", rev = "bc9cce955eb85fb1488f5304426eedeb6d97e905", features = ["iroh"], optional = true } +contextdb-server = { git = "https://github.com/context-graph-ai/contextdb", rev = "56dccfdc905653370a7a6775d0998b1053e5926a", features = ["iroh"], optional = true } fs4.workspace = true pollster.workspace = true rayon.workspace = true @@ -71,7 +71,7 @@ candle-nn.workspace = true candle-transformers.workspace = true cg-catalog = { path = "../cg-catalog", features = ["embeddinggemma-test-support"] } cg-connector-claude-code.workspace = true -contextdb-server = { git = "https://github.com/context-graph-ai/contextdb", rev = "bc9cce955eb85fb1488f5304426eedeb6d97e905", features = ["iroh", "test-seams"] } +contextdb-server = { git = "https://github.com/context-graph-ai/contextdb", rev = "56dccfdc905653370a7a6775d0998b1053e5926a", features = ["iroh", "test-seams"] } contextdb-engine = { workspace = true, features = ["test-seams"] } contextdb-tx.workspace = true contextdb-vector = { workspace = true, features = ["test-seams"] } diff --git a/crates/context-graph/src/capture_sync_status.rs b/crates/context-graph/src/capture_sync_status.rs index 24fcf61..7a67a70 100644 --- a/crates/context-graph/src/capture_sync_status.rs +++ b/crates/context-graph/src/capture_sync_status.rs @@ -257,12 +257,63 @@ pub fn estimate_capture( } } +#[cfg(any(context_graph_internal_test_support, feature = "test-support"))] +thread_local! { + static CAPTURE_PROGRESS_TEST_CLOCK: std::cell::Cell = const { std::cell::Cell::new(false) }; +} + pub fn capture_with_progress_observer( + store: &Store, + request: CaptureRequest, + extractor: &dyn ConversationExtractor, + chunk_window: usize, + observer: impl FnMut(CaptureProgressEvent), +) -> Result { + capture_with_progress_elapsed(store, request, extractor, chunk_window, observer, || { + let started = Instant::now(); + #[cfg(any(context_graph_internal_test_support, feature = "test-support"))] + let canonical_start = CAPTURE_PROGRESS_TEST_CLOCK + .with(|enabled| enabled.get().then(|| contextdb_core::Wallclock::now().0)); + move || { + #[cfg(any(context_graph_internal_test_support, feature = "test-support"))] + if let Some(start) = canonical_start { + return Duration::from_millis( + contextdb_core::Wallclock::now().0.saturating_sub(start), + ); + } + started.elapsed() + } + }) +} + +/// Synchronous drive of progress timing using the substrate's canonical thread-local clock. +#[cfg(any(context_graph_internal_test_support, feature = "test-support"))] +#[doc(hidden)] +pub fn capture_with_progress_observer_for_test( + store: &Store, + request: CaptureRequest, + extractor: &dyn ConversationExtractor, + chunk_window: usize, + observer: impl FnMut(CaptureProgressEvent), +) -> Result { + struct Reset(bool); + impl Drop for Reset { + fn drop(&mut self) { + CAPTURE_PROGRESS_TEST_CLOCK.with(|enabled| enabled.set(self.0)); + } + } + let _reset = Reset(CAPTURE_PROGRESS_TEST_CLOCK.with(|enabled| enabled.replace(true))); + // Exercise the public operation, including its input handling and observer delivery. + capture_with_progress_observer(store, request, extractor, chunk_window, observer) +} + +fn capture_with_progress_elapsed Duration>( store: &Store, request: CaptureRequest, extractor: &dyn ConversationExtractor, chunk_window: usize, mut observer: impl FnMut(CaptureProgressEvent), + start_clock: impl FnOnce() -> C, ) -> Result { let text = request.transcript.as_str(); if text.trim().is_empty() { @@ -270,19 +321,19 @@ pub fn capture_with_progress_observer( } let chunks = transcript_chunks(text, chunk_window); let total_chunks = chunks.len(); - let started = Instant::now(); + let elapsed = start_clock(); if total_chunks == 1 { observer(progress_event( 1, total_chunks, - started.elapsed(), + elapsed(), ProgressKind::InFlight, )); let mut report = record_conversation_with_extractor(store, request, extractor)?; observer(progress_event( 1, total_chunks, - started.elapsed(), + elapsed(), ProgressKind::Completed, )); report.resolved_chunk_window = Some(chunk_window.max(1)); @@ -292,11 +343,11 @@ pub fn capture_with_progress_observer( let mut per_chunk_usage: Vec = Vec::new(); for (index, chunk) in chunks.iter().enumerate() { let chunk_index = index + 1; - let chunk_started = Instant::now(); + let chunk_started = elapsed(); observer(progress_event( chunk_index, total_chunks, - started.elapsed(), + elapsed(), ProgressKind::InFlight, )); let harvest = extractor.extract(ExtractionRequest { @@ -304,7 +355,7 @@ pub fn capture_with_progress_observer( transcript: chunk, prior_decisions: &[], })?; - let chunk_elapsed = chunk_started.elapsed(); + let chunk_elapsed = elapsed().saturating_sub(chunk_started); // Drain the extractor's usage records immediately after each chunk. // The extractor's internal chunk_index counter resets on each drain, // so we assign the authoritative chunk_index from the loop variable. @@ -333,7 +384,7 @@ pub fn capture_with_progress_observer( observer(progress_event( chunk_index, total_chunks, - started.elapsed(), + elapsed(), ProgressKind::Completed, )); } diff --git a/crates/context-graph/src/embedder/nomic.rs b/crates/context-graph/src/embedder/nomic.rs index d25ac48..956a590 100644 --- a/crates/context-graph/src/embedder/nomic.rs +++ b/crates/context-graph/src/embedder/nomic.rs @@ -1,7 +1,5 @@ use std::collections::HashMap; use std::sync::{Arc, Mutex, OnceLock}; -#[cfg(feature = "test-support")] -use std::time::Duration; use std::time::Instant; use std::{ fs as file_system, io, @@ -1199,12 +1197,15 @@ fn pause_prepared_embedding_for_test() { return; }; let directory = PathBuf::from(directory); - if file_system::write(directory.join("reached"), b"prepared embedding reached\n").is_err() { + if cg_node_fs::write_test_file_for_test( + directory.join("reached"), + b"prepared embedding reached\n", + ) + .is_err() + { return; } - while !directory.join("release").exists() { - std::thread::sleep(Duration::from_millis(10)); - } + cg_node_fs::wait_for_test_release_for_test(&directory, || directory.join("release").exists()); } #[cfg(all(test, context_graph_internal_test_support))] diff --git a/crates/context-graph/src/lib.rs b/crates/context-graph/src/lib.rs index 5a5bf97..45cefa3 100644 --- a/crates/context-graph/src/lib.rs +++ b/crates/context-graph/src/lib.rs @@ -104,6 +104,7 @@ pub mod test_support { pub use crate::store::event_search_projection::{ after_next_event_search_membership_read_page_for_test, observe_event_search_batch_admission_for_test, + set_event_search_projection_read_clock_for_test, set_event_search_projection_read_deadline_ms_for_test, }; /// How many Stores the event-search runner has opened on the calling thread. diff --git a/crates/context-graph/src/store/digest.rs b/crates/context-graph/src/store/digest.rs index d28eb6d..3af3138 100644 --- a/crates/context-graph/src/store/digest.rs +++ b/crates/context-graph/src/store/digest.rs @@ -35,7 +35,7 @@ impl Store { }); } let space = self - .embedding_space_for_id(&input.embedding.embedding_space_id) + .embedding_space_for_write(&input.embedding.embedding_space_id) .map_err(|_| CgError::UnknownEmbeddingSpace { id: input.embedding.embedding_space_id.clone(), })?; diff --git a/crates/context-graph/src/store/entity_reference_embeddings.rs b/crates/context-graph/src/store/entity_reference_embeddings.rs index 44510c4..048f31f 100644 --- a/crates/context-graph/src/store/entity_reference_embeddings.rs +++ b/crates/context-graph/src/store/entity_reference_embeddings.rs @@ -169,7 +169,7 @@ impl Store { .unwrap_or_else(ContextId::new_v7), }; let space = self - .embedding_space_for_id(embedding_space_id) + .embedding_space_for_write(embedding_space_id) .map_err(|_| CgError::UnknownEmbeddingSpace { id: embedding_space_id.to_string(), })?; diff --git a/crates/context-graph/src/store/event_search_projection.rs b/crates/context-graph/src/store/event_search_projection.rs index f6408fe..4aca013 100644 --- a/crates/context-graph/src/store/event_search_projection.rs +++ b/crates/context-graph/src/store/event_search_projection.rs @@ -277,6 +277,7 @@ thread_local! { const { std::cell::RefCell::new(None) }; static AFTER_MEMBERSHIP_READ_PAGE: std::cell::RefCell>> = const { std::cell::RefCell::new(None) }; + static PROJECTION_READ_CLOCK: std::cell::RefCell>> = const { std::cell::RefCell::new(None) }; static PROJECTION_READ_ACTIVE_MS: std::cell::Cell> = const { std::cell::Cell::new(None) }; static BEFORE_FAILURE_RECORD: std::cell::RefCell>> = const { std::cell::RefCell::new(None) }; @@ -340,6 +341,15 @@ pub fn set_event_search_projection_read_deadline_ms_for_test(active_ms: Option>, +) { + PROJECTION_READ_CLOCK.with(|slot| *slot.borrow_mut() = clock); +} + /// Return and reset this thread's source-membership preparation count. #[cfg(any(context_graph_internal_test_support, feature = "test-support"))] #[doc(hidden)] @@ -3803,10 +3813,7 @@ fn capture_search_page( ordered: &mut Sha256, content: &mut Sha256, ) -> Result<()> { - let session = store - .internal_db() - .read_session(read_limits(store, bounds)) - .map_err(map_engine_err)?; + let session = projection_read_session(store, bounds)?; for row in rows { let raw = captured_raw_from_row(&session, bounds, context_id, columns, row)?; let captured_sha256 = captured_source_sha256( @@ -4840,10 +4847,7 @@ fn load_claimed_member( row: &[Value], bounds: Bounds, ) -> Result { - let session = store - .internal_db() - .read_session(read_limits(store, bounds)) - .map_err(map_engine_err)?; + let session = projection_read_session(store, bounds)?; let membership_id = get_uuid(row, columns, "membership_id")?; let ordinal = get_u64(row, columns, "ordinal")?; let event_id = crate::EventId::from(get_uuid(row, columns, "event_id")?); @@ -7954,10 +7958,7 @@ fn stream_rows( params: &HashMap, mut consume: impl FnMut(&[String], &[Value]) -> Result<()>, ) -> Result<()> { - let session = store - .internal_db() - .read_session(read_limits(store, bounds)) - .map_err(map_engine_err)?; + let session = projection_read_session(store, bounds)?; let mut cursor = session.open_cursor(sql, params).map_err(map_engine_err)?; let read = (|| -> Result<()> { let first = cursor.first_page(); @@ -7987,11 +7988,22 @@ fn query_rows( sql: &str, params: &HashMap, ) -> Result { - let session = store + let session = projection_read_session(store, bounds)?; + session.execute(sql, params).map_err(map_engine_err) +} + +fn projection_read_session(store: &Store, bounds: Bounds) -> Result { + #[cfg(any(context_graph_internal_test_support, feature = "test-support"))] + if let Some(clock) = PROJECTION_READ_CLOCK.with(|slot| slot.borrow().clone()) { + return store + .internal_db() + .read_session_with_clock_for_test(read_limits(store, bounds), clock) + .map_err(map_engine_err); + } + store .internal_db() .read_session(read_limits(store, bounds)) - .map_err(map_engine_err)?; - session.execute(sql, params).map_err(map_engine_err) + .map_err(map_engine_err) } fn read_limits(store: &Store, bounds: Bounds) -> ReadLimits { diff --git a/crates/context-graph/src/store/evidence.rs b/crates/context-graph/src/store/evidence.rs index 605c852..23278f5 100644 --- a/crates/context-graph/src/store/evidence.rs +++ b/crates/context-graph/src/store/evidence.rs @@ -101,14 +101,18 @@ impl Store { } pub fn list_embedding_spaces(&self) -> Result> { - let result = self.read_query( - "SELECT embedding_space_id, dimension, metric, model_family, model_name, model_version, supported_evidence_kinds, default_for_evidence_kind, bindings FROM embedding_spaces", - &HashMap::new(), - )?; + self.embedding_spaces_via(&|sql, params| self.read_query(sql, params)) + } + + fn embedding_spaces_via(&self, query: &LookupQuery<'_>) -> Result> { + let result = query( + "SELECT embedding_space_id, dimension, metric, model_family, model_name, model_version, supported_evidence_kinds, default_for_evidence_kind, bindings FROM embedding_spaces", + &HashMap::new(), + )?; let mut spaces = Vec::with_capacity(result.rows.len()); for row in &result.rows { let embedding_space_id = get_text(row, &result.columns, "embedding_space_id")?; - let mut bindings = self.bindings_for_space(&embedding_space_id)?; + let mut bindings = bindings_for_space(query, &embedding_space_id)?; if bindings.is_empty() { bindings = bindings_from_json( &embedding_space_id, @@ -426,12 +430,13 @@ impl Store { Ok(rows) } + /// What the observation just recorded raised, read back as that write's own lookup. pub(super) fn invalidations_for_observation( &self, context_id: ContextId, observation_id: ObservationId, ) -> Result> { - let result = self.read_query( + let result = self.write_lookup( "SELECT id, status, context_id, affected_decision_id, trigger_observation_id, reason, basis_diff, cascade_from, severity, threatened_intention_ids, intention_now_un_served, detected_at, resolved_at, resolution_action, resolution_decision_id FROM invalidations WHERE context_id = $context_id AND trigger_observation_id = $trigger_observation_id", &HashMap::from([ ("context_id".to_string(), Value::Uuid(context_id.as_uuid())), @@ -501,30 +506,6 @@ impl Store { unreachable!("bounded update_decision_basis retry loop always returns") } - fn bindings_for_space(&self, embedding_space_id: &str) -> Result> { - let result = self.read_query( - "SELECT embedding_space_id, table_name, vector_column FROM embedding_space_bindings WHERE embedding_space_id = $embedding_space_id", - &HashMap::from([( - "embedding_space_id".to_string(), - Value::Text(embedding_space_id.to_string()), - )]), - )?; - let mut bindings = Vec::with_capacity(result.rows.len()); - for row in &result.rows { - bindings.push(EmbeddingSpaceBinding { - embedding_space_id: get_text(row, &result.columns, "embedding_space_id")?, - table_name: get_text(row, &result.columns, "table_name")?, - vector_column: get_text(row, &result.columns, "vector_column")?, - }); - } - bindings.sort_by(|a, b| { - a.table_name - .cmp(&b.table_name) - .then_with(|| a.vector_column.cmp(&b.vector_column)) - }); - Ok(bindings) - } - fn decode_evidence_with_latest_status( &self, columns: &[String], @@ -684,10 +665,16 @@ impl Store { } pub(super) fn embedding_space_for_id(&self, id: &str) -> Result { - self.list_embedding_spaces()? - .into_iter() - .find(|space| space.embedding_space_id == id) - .ok_or_else(|| CgError::UnknownEmbeddingSpace { id: id.to_string() }) + find_embedding_space(self.list_embedding_spaces()?, id) + } + + /// The registered space a write is about to use, looked up as the write's own read: a write + /// is never refused by the read budget a caller's read is held to. + pub(super) fn embedding_space_for_write(&self, id: &str) -> Result { + find_embedding_space( + self.embedding_spaces_via(&|sql, params| self.write_lookup(sql, params))?, + id, + ) } pub(super) fn evidence_status_at_snapshot( @@ -1276,3 +1263,41 @@ pub(super) fn producer_value(producer: &EvidenceProducer) -> Value { pub(super) fn properties_value(properties: &BTreeMap) -> Value { Value::Json(properties_to_json(properties)) } + +/// One registry statement, run through whichever executor the caller's read or write uses. +type LookupQuery<'a> = + dyn Fn(&str, &HashMap) -> Result + 'a; + +fn find_embedding_space(spaces: Vec, id: &str) -> Result { + spaces + .into_iter() + .find(|space| space.embedding_space_id == id) + .ok_or_else(|| CgError::UnknownEmbeddingSpace { id: id.to_string() }) +} + +fn bindings_for_space( + query: &LookupQuery<'_>, + embedding_space_id: &str, +) -> Result> { + let result = query( + "SELECT embedding_space_id, table_name, vector_column FROM embedding_space_bindings WHERE embedding_space_id = $embedding_space_id", + &HashMap::from([( + "embedding_space_id".to_string(), + Value::Text(embedding_space_id.to_string()), + )]), + )?; + let mut bindings = Vec::with_capacity(result.rows.len()); + for row in &result.rows { + bindings.push(EmbeddingSpaceBinding { + embedding_space_id: get_text(row, &result.columns, "embedding_space_id")?, + table_name: get_text(row, &result.columns, "table_name")?, + vector_column: get_text(row, &result.columns, "vector_column")?, + }); + } + bindings.sort_by(|a, b| { + a.table_name + .cmp(&b.table_name) + .then_with(|| a.vector_column.cmp(&b.vector_column)) + }); + Ok(bindings) +} diff --git a/crates/context-graph/src/store/graph_rows.rs b/crates/context-graph/src/store/graph_rows.rs index f1f179c..e318305 100644 --- a/crates/context-graph/src/store/graph_rows.rs +++ b/crates/context-graph/src/store/graph_rows.rs @@ -17,6 +17,28 @@ use contextdb_engine::QueryResult; use crate::Result; +#[cfg(test)] +thread_local! { + static READ_CLOCK: std::cell::RefCell>> = const { std::cell::RefCell::new(None) }; +} + +/// Adapt the substrate's existing deadline seam to a synchronous Store read. +/// Thread-local and unit-test-only: other stores and production sessions keep their clock. +#[cfg(test)] +pub(super) fn with_read_clock_for_test( + clock: std::sync::Arc, + read: impl FnOnce() -> T, +) -> T { + struct Reset(Option>); + impl Drop for Reset { + fn drop(&mut self) { + READ_CLOCK.with(|slot| *slot.borrow_mut() = self.0.take()); + } + } + let _reset = Reset(READ_CLOCK.with(|slot| slot.replace(Some(clock)))); + read() +} + /// One executor of a read statement. pub(crate) trait GraphRows { /// Run one read and hand back the rows the engine answered with. @@ -35,10 +57,19 @@ impl crate::Store { sql: &str, params: &HashMap, ) -> Result { - let session = self - .internal_db() - .read_session(self.read_limits()) - .map_err(super::convert::map_engine_err)?; + #[cfg(test)] + let clock = READ_CLOCK.with(|slot| slot.borrow().clone()); + #[cfg(test)] + let test_session = clock.map(|clock| { + self.internal_db() + .read_session_with_clock_for_test(self.read_limits(), clock) + }); + #[cfg(test)] + let session = + test_session.unwrap_or_else(|| self.internal_db().read_session(self.read_limits())); + #[cfg(not(test))] + let session = self.internal_db().read_session(self.read_limits()); + let session = session.map_err(super::convert::map_engine_err)?; let mut cursor = session .open_cursor(sql, params) .map_err(super::convert::map_engine_err)?; @@ -67,6 +98,41 @@ impl crate::Store { } } +impl crate::Store { + /// Run a lookup a write makes for itself — the embedding-space registry, the entity it + /// attaches to, what its own commit produced — as an ordinary statement read at the latest + /// committed snapshot. + /// + /// The read budget bounds what a caller's read may cost, and on a file-backed store a bounded + /// read also spends that allowance waiting for the commit boundary. A write is not a caller's + /// read: while a reindex commits row after row, a lookup charged against that budget would + /// refuse the write for time it spent queued behind other commits. + pub(crate) fn write_lookup( + &self, + sql: &str, + params: &HashMap, + ) -> Result { + self.internal_db() + .execute(sql, params) + .map_err(super::convert::map_engine_err) + } + + /// The typed graph reads, run as a write's own lookups. + pub(crate) fn write_lookup_rows(&self) -> WriteLookupRows<'_> { + WriteLookupRows(self) + } +} + +/// Executor for a typed graph read made on behalf of a write; see [`crate::Store::write_lookup`]. +pub(crate) struct WriteLookupRows<'a>(&'a crate::Store); + +impl GraphRows for WriteLookupRows<'_> { + fn query_rows(&self, sql: &str, params: &HashMap) -> Result { + self.0.require_application_ready()?; + self.0.write_lookup(sql, params) + } +} + /// A decoded row for the existing library table readers. pub(crate) struct ReadRow { pub(crate) values: HashMap, diff --git a/crates/context-graph/src/store/observation.rs b/crates/context-graph/src/store/observation.rs index 47e947d..bcb1116 100644 --- a/crates/context-graph/src/store/observation.rs +++ b/crates/context-graph/src/store/observation.rs @@ -34,6 +34,8 @@ trait RecordObservationHooks { fn after_state_delta_guards(&mut self) {} fn conditional_update_conflict(&mut self) {} + + fn before_commit(&mut self) {} } struct NoopRecordObservationHooks; @@ -46,8 +48,9 @@ impl Store { } // The hook is private and the public entry point always supplies the no-op implementation. - // It gives the unit test one exact point after strict snapshot guards are staged and before - // the no-op decision, without changing the production operation. + // It gives the unit tests exact points after strict snapshot guards are staged and before + // the no-op decision, and after every lookup the write makes and before its commit, without + // changing the production operation. fn record_observation_with_hooks( &self, mut input: RecordObservation, @@ -68,7 +71,10 @@ impl Store { .iter() .map(|evidence| evidence.id) .collect::>(); - let auto_create_entity = if let Some(entity) = self.get_entity(input.entity_id)? { + self.observe_read("get_entity"); + let auto_create_entity = if let Some(entity) = + super::entity::get_entity(&self.write_lookup_rows(), input.entity_id)? + { if entity.context_id != input.context_id { return Err(CgError::CrossContextObservation); } @@ -249,6 +255,7 @@ impl Store { let _ = self.db.rollback(tx); return Err(e); } + hooks.before_commit(); if let Err(err) = self.db.commit(tx) { match err { EngineError::ConditionalUpdateConflict { .. } => { @@ -553,11 +560,7 @@ impl Store { ) -> Result { match embedding.target { EmbeddingTarget::Evidence(target_id) => { - let space = self - .embedding_space_for_id(&embedding.embedding_space_id) - .map_err(|_| CgError::UnknownEmbeddingSpace { - id: embedding.embedding_space_id.clone(), - })?; + let space = self.embedding_space_for_write(&embedding.embedding_space_id)?; if embedding.vector.len() != space.dimension { return Err(CgError::EmbeddingDimensionMismatch { space_id: embedding.embedding_space_id.clone(), @@ -585,11 +588,7 @@ impl Store { }) } EmbeddingTarget::EntityReference(_, _) => { - let space = self - .embedding_space_for_id(&embedding.embedding_space_id) - .map_err(|_| CgError::UnknownEmbeddingSpace { - id: embedding.embedding_space_id.clone(), - })?; + let space = self.embedding_space_for_write(&embedding.embedding_space_id)?; if embedding.vector.len() != space.dimension { return Err(CgError::EmbeddingDimensionMismatch { space_id: embedding.embedding_space_id.clone(), @@ -612,11 +611,7 @@ impl Store { }) } EmbeddingTarget::Digest(_) => { - let space = self - .embedding_space_for_id(&embedding.embedding_space_id) - .map_err(|_| CgError::UnknownEmbeddingSpace { - id: embedding.embedding_space_id.clone(), - })?; + let space = self.embedding_space_for_write(&embedding.embedding_space_id)?; if embedding.vector.len() != space.dimension { return Err(CgError::EmbeddingDimensionMismatch { space_id: embedding.embedding_space_id.clone(), @@ -1337,7 +1332,7 @@ impl Store { query_vector: &[f32], ) -> Result> { let space = self - .embedding_space_for_id(embedding_space_id) + .embedding_space_for_write(embedding_space_id) .map_err(|_| CgError::UnknownEmbeddingSpace { id: embedding_space_id.to_string(), })?; @@ -2139,6 +2134,259 @@ mod tests { conditional_conflicts: usize, } + #[test] + fn embedding_route_lookup_is_the_writes_own_and_keeps_its_typed_refusal() { + use contextdb_core::read_contract::{DeadlineClock, DeadlineWait}; + use std::sync::{ + Arc, + atomic::{AtomicU64, Ordering}, + }; + + /// Every observation spends a full second, so any read charged to the caller's read + /// budget is refused at once. + struct ExhaustingReadClock { + calls: AtomicU64, + } + impl DeadlineClock for ExhaustingReadClock { + fn now_ms(&self) -> u64 { + self.calls.fetch_add(1, Ordering::SeqCst) * 1_000 + } + + fn wait_until(&self, deadline: u64) -> DeadlineWait<'_> { + Box::pin(std::future::poll_fn(move |context| { + if self.now_ms() >= deadline { + std::task::Poll::Ready(()) + } else { + context.waker().wake_by_ref(); + std::task::Poll::Pending + } + })) + } + } + + let dir = tempfile::tempdir().expect("tempdir"); + let mut store = Store::open(StoreConfig { + db_path: dir.path().join("route-refusal.db"), + default_text_embedder: None, + ..Default::default() + }) + .expect("open store"); + store.read_active_time_ms = std::num::NonZeroU64::new(17).unwrap(); + let input = observation_input(EntityId::new_v7(), ContextId::new_v7(), "test"); + let embeddings = [ + EmbeddingTarget::Evidence(input.evidence[0].id), + EmbeddingTarget::EntityReference(input.entity_id, None), + EmbeddingTarget::Digest(DigestId::new_v7()), + ] + .map(|target| PrecomputedEmbedding { + embedding_space_id: "absent-space".to_string(), + target, + ..Default::default() + }); + for embedding in &embeddings { + let clock = Arc::new(ExhaustingReadClock { + calls: AtomicU64::new(0), + }); + let absent = super::super::graph_rows::with_read_clock_for_test(clock.clone(), || { + store.validate_embedding_route(&input, embedding, &HashMap::new()) + }); + assert!( + matches!(&absent, Err(CgError::UnknownEmbeddingSpace { id }) if id == "absent-space") + && clock.calls.load(Ordering::SeqCst) == 0, + "the route lookup must answer the absent space without being charged to the read \ + budget for {:?}: {absent:?}, read_clock_observations={}", + embedding.target, + clock.calls.load(Ordering::SeqCst) + ); + } + store + .internal_db() + .close() + .expect("close the store's database"); + for embedding in &embeddings { + let refused = store.validate_embedding_route(&input, embedding, &HashMap::new()); + assert!( + matches!(&refused, Err(error) if !matches!(error, CgError::UnknownEmbeddingSpace { .. })), + "a refused lookup must survive validation instead of reading as a missing space \ + for {:?}: {refused:?}", + embedding.target + ); + } + } + + /// Holds the first vector-writing commit after arming inside the commit boundary until the + /// test releases it, and says while it is holding. + struct HoldVectorCommit { + armed: std::sync::atomic::AtomicBool, + holding: std::sync::Arc, + held: std::sync::Mutex>>, + release: std::sync::Mutex>>, + } + + impl contextdb_engine::plugin::DatabasePlugin for HoldVectorCommit { + fn pre_commit( + &self, + ws: &contextdb_tx::WriteSet, + _source: contextdb_engine::plugin::CommitSource, + ) -> contextdb_core::Result<()> { + use std::sync::atomic::Ordering; + if ws.vector_inserts.is_empty() || !self.armed.swap(false, Ordering::SeqCst) { + return Ok(()); + } + self.holding.store(true, Ordering::SeqCst); + if let Some(held) = self.held.lock().expect("held sender").take() { + held.send(()).expect("held commit announced"); + } + if let Some(release) = self.release.lock().expect("release receiver").take() { + release.recv().expect("held commit released"); + } + self.holding.store(false, Ordering::SeqCst); + Ok(()) + } + } + + /// A read clock on which time passes only while the commit boundary is held: a bounded read + /// waiting for that boundary sees its allowance run out, and nothing else does. + struct CommitBoundaryClock { + holding: std::sync::Arc, + observations: std::sync::atomic::AtomicU64, + now: std::sync::atomic::AtomicU64, + } + + impl contextdb_core::read_contract::DeadlineClock for CommitBoundaryClock { + fn now_ms(&self) -> u64 { + use std::sync::atomic::Ordering; + self.observations.fetch_add(1, Ordering::SeqCst); + if self.holding.load(Ordering::SeqCst) { + self.now.fetch_add(1_000, Ordering::SeqCst) + 1_000 + } else { + self.now.load(Ordering::SeqCst) + } + } + + fn wait_until(&self, deadline: u64) -> contextdb_core::read_contract::DeadlineWait<'_> { + Box::pin(std::future::poll_fn(move |context| { + if self.now_ms() >= deadline { + std::task::Poll::Ready(()) + } else { + if self.holding.load(std::sync::atomic::Ordering::SeqCst) { + context.waker().wake_by_ref(); + } + std::task::Poll::Pending + } + })) + } + } + + enum WriterEvent { + ReachedCommit, + Finished, + } + + struct AnnounceCommit(Option>); + + impl RecordObservationHooks for AnnounceCommit { + fn before_commit(&mut self) { + if let Some(events) = self.0.take() { + events + .send(WriterEvent::ReachedCommit) + .expect("commit point announced"); + } + } + } + + #[test] + fn observation_recorded_while_a_reindex_commit_holds_the_commit_boundary_is_not_refused_by_the_read_budget() + { + use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; + use std::sync::{Arc, Mutex}; + + let dir = tempfile::tempdir().expect("tempdir"); + let space = crate::embedding_space::default_text_embedding_space(); + let holding = Arc::new(AtomicBool::new(false)); + let (held_tx, held_rx) = mpsc::channel(); + let (release_tx, release_rx) = mpsc::channel(); + let plugin = Arc::new(HoldVectorCommit { + armed: AtomicBool::new(false), + holding: Arc::clone(&holding), + held: Mutex::new(Some(held_tx)), + release: Mutex::new(Some(release_rx)), + }); + let store = Store::open_with_config_inner_with_plugin( + StoreConfig { + db_path: dir.path().join("reindex-write-budget.db"), + default_text_embedder: None, + ..Default::default() + }, + false, + None, + None, + super::super::DatabaseOpenExtras::with_plugin(plugin.clone()), + vec![crate::EmbedderRegistration { + embedder: Arc::new(crate::embedder::mock::MockEmbedder::new( + space.embedding_space_id.clone(), + space.dimension, + )), + space: space.clone(), + }], + crate::ConsumerSchema::none(), + ) + .expect("open store"); + let context_id = ContextId::new_v7(); + let text_input = |text: &str| { + let mut input = observation_input(EntityId::new_v7(), context_id, "screen"); + input.evidence[0].text_projection = Some(text.to_string()); + input + }; + store + .record_observation(text_input("indexed before the reindex")) + .expect("seed an indexed observation"); + + plugin.armed.store(true, Ordering::SeqCst); + let reindex_store = store.clone(); + let space_id = space.embedding_space_id.clone(); + let reindex = std::thread::spawn(move || { + reindex_store.reindex_space(&space_id, super::super::ReindexOptions::default()) + }); + held_rx + .recv() + .expect("the reindex commit is holding the commit boundary"); + + let (events_tx, events_rx) = mpsc::channel(); + let writer_store = store.clone(); + let writer_holding = Arc::clone(&holding); + let input = text_input("written during the reindex"); + let writer = std::thread::spawn(move || { + let clock = Arc::new(CommitBoundaryClock { + holding: writer_holding, + observations: AtomicU64::new(0), + now: AtomicU64::new(0), + }); + let mut hooks = AnnounceCommit(Some(events_tx.clone())); + let result = super::super::graph_rows::with_read_clock_for_test(clock.clone(), || { + writer_store.record_observation_with_hooks(input, &mut hooks) + }); + let _ = events_tx.send(WriterEvent::Finished); + (result, clock.observations.load(Ordering::SeqCst)) + }); + let first_event = events_rx.recv().expect("writer event"); + release_tx.send(()).expect("release the reindex commit"); + let (written, budget_observations) = writer.join().expect("writer thread"); + let reindexed = reindex.join().expect("reindex thread"); + + assert!( + matches!(first_event, WriterEvent::ReachedCommit) + && written.is_ok() + && budget_observations == 0 + && reindexed.is_ok(), + "a write made while a reindex commit holds the commit boundary must finish its lookups \ + without waiting for that boundary and must never be charged against the read \ + budget: reached_commit_while_held={}, written={written:?}, \ + read_budget_clock_observations={budget_observations}, reindexed={reindexed:?}", + matches!(first_event, WriterEvent::ReachedCommit), + ); + } + impl RecordObservationHooks for SnapshotGuardRaceHook { fn after_state_delta_guards(&mut self) { if !self.paused_once { diff --git a/crates/context-graph/src/store/reindex.rs b/crates/context-graph/src/store/reindex.rs index f3f37ba..c3e28ba 100644 --- a/crates/context-graph/src/store/reindex.rs +++ b/crates/context-graph/src/store/reindex.rs @@ -52,7 +52,7 @@ struct ReindexCommit<'a> { impl Store { pub fn reindex_space(&self, embedding_space_id: &str, options: ReindexOptions) -> Result<()> { let space = self - .embedding_space_for_id(embedding_space_id) + .embedding_space_for_write(embedding_space_id) .map_err(|_| CgError::UnknownEmbeddingSpace { id: embedding_space_id.to_string(), })?; @@ -485,7 +485,7 @@ impl Store { embedding_space_id: &str, started_at: TxId, ) -> Result> { - let space = self.embedding_space_for_id(embedding_space_id)?; + let space = self.embedding_space_for_write(embedding_space_id)?; let bindings = space .bindings .iter() diff --git a/crates/context-graph/src/store/signatures.rs b/crates/context-graph/src/store/signatures.rs index 1bcf105..3518026 100644 --- a/crates/context-graph/src/store/signatures.rs +++ b/crates/context-graph/src/store/signatures.rs @@ -50,7 +50,7 @@ pub struct SignatureProducerLineageRow { pub fn persist_signature_aggregate(store: &Store, aggregate: SignatureAggregate) -> Result<()> { store - .embedding_space_for_id(&aggregate.embedding_space_id) + .embedding_space_for_write(&aggregate.embedding_space_id) .map_err(|_| CgError::UnknownEmbeddingSpace { id: aggregate.embedding_space_id.clone(), })?; diff --git a/crates/context-graph/tests/a_real_store_open_is_refused_by_the_readers_holding_it.rs b/crates/context-graph/tests/a_real_store_open_is_refused_by_the_readers_holding_it.rs index 7a9732e..bd5a5d5 100644 --- a/crates/context-graph/tests/a_real_store_open_is_refused_by_the_readers_holding_it.rs +++ b/crates/context-graph/tests/a_real_store_open_is_refused_by_the_readers_holding_it.rs @@ -43,7 +43,6 @@ use std::os::unix::fs::PermissionsExt; use std::path::{Path, PathBuf}; use std::process::{Child, ChildStdout, Command, Stdio}; use std::sync::mpsc::channel; -use std::time::Duration; use context_graph::{CgError, Store, StoreConfig}; use contextdb_engine::ReadSession; @@ -511,14 +510,12 @@ fn a_real_context_graph_open_is_refused_by_the_live_readers_holding_the_store() store }); observed - .recv_timeout(Duration::from_secs(10)) + .recv() .expect("real reader refusal reached the shared Store wait"); assert!(finished.try_recv().is_err()); first.release(); second.release(); - finished - .recv_timeout(Duration::from_secs(10)) - .expect("writer opens after reader release"); + finished.recv().expect("writer opens after reader release"); let store = writer.join().unwrap(); let shared = Store::open(store_config(store_path)).expect("reuse registered writer"); assert!(std::sync::Arc::ptr_eq( diff --git a/crates/context-graph/tests/acceptance/capture_progress.rs b/crates/context-graph/tests/acceptance/capture_progress.rs index 59f5ef5..3a13070 100644 --- a/crates/context-graph/tests/acceptance/capture_progress.rs +++ b/crates/context-graph/tests/acceptance/capture_progress.rs @@ -1,9 +1,11 @@ +use contextdb_core::Wallclock; +use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::mpsc; use std::sync::{Arc, Mutex}; -use std::time::{Duration, Instant}; +use std::time::Duration; use context_graph::ingest::{ConversationExtractor, ExtractionRequest, StructuredHarvest}; -use context_graph::{CaptureProgressEvent, capture_with_progress_observer}; +use context_graph::{CaptureProgressEvent, capture_with_progress_observer_for_test}; use crate::capture_sync_common::{ TestStore, capture_request, expected_chunk_count, expected_progress_events_for, @@ -45,10 +47,13 @@ fn long_run_emits_observable_chunk_progress() { .into_iter() .find(|event| event.chunk_index == blocked_chunk && event.kind.is_in_flight()) .expect("fixture must expect an in-flight event for the blocked chunk"); - let run_started = Instant::now(); + let run_started = 1_000; + let clock = Arc::new(AtomicU64::new(run_started)); + let worker_clock = Arc::clone(&clock); let worker_transcript = transcript.clone(); let worker = std::thread::spawn(move || { - capture_with_progress_observer( + let _clock = Wallclock::test_clock_guard(move || worker_clock.load(Ordering::SeqCst)); + capture_with_progress_observer_for_test( &worker_store, capture_request("tenant-alpha", &worker_transcript), worker_extractor.as_ref(), @@ -59,20 +64,15 @@ fn long_run_emits_observable_chunk_progress() { ) }); assert!( - blocking_extractor.wait_until_started(Duration::from_secs(2)), + blocking_extractor.wait_until_started(), "capture must reach the blocked chunk extractor before it can complete" ); - let observed_before_release = collect_until_event( - &event_rx, - &expected_blocked_progress, - run_started, - Duration::from_secs(2), - ); + let observed_before_release = + collect_until_event(&event_rx, &expected_blocked_progress, Duration::ZERO); assert_observed_elapsed_is_measured(&observed_before_release); let completed_before_release = worker.is_finished(); - let hold_started = Instant::now(); - std::thread::sleep(Duration::from_millis(150)); - let held_for = hold_started.elapsed(); + let held_for = Duration::from_millis(150); + clock.fetch_add(150, Ordering::SeqCst); blocking_extractor.release(); let report = worker .join() @@ -102,7 +102,7 @@ fn long_run_emits_observable_chunk_progress() { &observed_events, &transcript, chunk_window, - run_started.elapsed(), + Duration::from_millis(clock.load(Ordering::SeqCst) - run_started), ); assert!( completed_blocked_chunk.elapsed >= held_for, @@ -123,8 +123,8 @@ fn long_run_emits_observable_chunk_progress() { let store = TestStore::new(label); let mut progress = Vec::new(); let recording_extractor = RecordingExtractor::recording(); - let run_started = Instant::now(); - let report = capture_with_progress_observer( + let _clock = Wallclock::test_clock_guard(|| 1_000); + let report = capture_with_progress_observer_for_test( &store.store, capture_request("tenant-alpha", &transcript), &recording_extractor, @@ -132,7 +132,7 @@ fn long_run_emits_observable_chunk_progress() { |event| progress.push(event), ) .expect("capture succeeds"); - let wall_elapsed = run_started.elapsed(); + let wall_elapsed = Duration::ZERO; assert_progress_sequence(&progress, &transcript, chunk_window, wall_elapsed); assert_eq!( @@ -147,22 +147,14 @@ fn long_run_emits_observable_chunk_progress() { fn collect_until_event( receiver: &mpsc::Receiver, target: &CaptureProgressEvent, - run_started: Instant, - timeout: Duration, + observed_at: Duration, ) -> Vec<(CaptureProgressEvent, Duration)> { - let deadline = Instant::now() + timeout; let mut observed = Vec::new(); - while Instant::now() < deadline { - match receiver.recv_timeout(Duration::from_millis(50)) { - Ok(event) => { - let is_target = same_progress_shape(&event, target); - observed.push((event, run_started.elapsed())); - if is_target { - break; - } - } - Err(mpsc::RecvTimeoutError::Timeout) => {} - Err(mpsc::RecvTimeoutError::Disconnected) => break, + for event in receiver { + let is_target = same_progress_shape(&event, target); + observed.push((event, observed_at)); + if is_target { + break; } } observed @@ -300,11 +292,11 @@ impl RecordingExtractor { } } - fn wait_until_started(&self, timeout: Duration) -> bool { + fn wait_until_started(&self) -> bool { self.started_rx .lock() .expect("started receiver lock") - .recv_timeout(timeout) + .recv() .map(|call_index| Some(call_index) == self.block_on_call) .unwrap_or(false) } diff --git a/crates/context-graph/tests/acceptance/crud.rs b/crates/context-graph/tests/acceptance/crud.rs index 4b2befd..ffd3841 100644 --- a/crates/context-graph/tests/acceptance/crud.rs +++ b/crates/context-graph/tests/acceptance/crud.rs @@ -1936,7 +1936,22 @@ fn observation_immutable_mutation_rejected() { // Use case: an ingest-pipeline refactor tries to rewrite a Digest's summary; the engine refuses any mutation because every Intention and Decision extracted from that Digest carries a provenance edge back to its bytes. #[test] fn digest_immutable_no_update_symbol() { - trybuild::TestCases::new().compile_fail("tests/ui/no_update_digest.rs"); + // Proves the public API has no `update_digest` on `Store`. The path `Store::update_digest` + // resolves to a public inherent item before any trait item, so it reaches this fallback + // constant only while no public inherent `update_digest` exists. Any public inherent + // function of that name, generic or not, with or without a receiver, is a function item + // that never has this test-private type, and any public inherent constant of that name + // has a type this test does not export; either way this binary stops compiling. Private + // and `pub(crate)` items are invisible here and are not part of this claim; + // `tests/compile_fail.rs` still compiles `tests/ui/no_update_digest.rs` against its + // expected diagnostic. + struct NoUpdateDigest; + #[allow(non_upper_case_globals)] + trait UpdateDigestFallback { + const update_digest: NoUpdateDigest = NoUpdateDigest; + } + impl UpdateDigestFallback for T {} + let _: NoUpdateDigest = Store::update_digest; let (_tempdir, db_path, store) = fixture(); let ctx = ok!( diff --git a/crates/context-graph/tests/acceptance/discovery.rs b/crates/context-graph/tests/acceptance/discovery.rs index fc31719..fd574db 100644 --- a/crates/context-graph/tests/acceptance/discovery.rs +++ b/crates/context-graph/tests/acceptance/discovery.rs @@ -5,7 +5,6 @@ use std::sync::{ atomic::{AtomicBool, Ordering}, }; use std::thread; -use std::time::Duration; use chrono::Utc; use context_graph::*; @@ -2958,7 +2957,11 @@ fn discovery_pull_side_returns_consistent_view_under_concurrent_writer() { let barrier = Arc::new(Barrier::new(2)); let writer_started = Arc::new(AtomicBool::new(false)); let writer_done = Arc::new(AtomicBool::new(false)); - let reader_in_call = Arc::new(AtomicBool::new(false)); + let rounds = writer_pairs.len(); + let round_ready = Arc::new(Barrier::new(2)); + let round_done = Arc::new(Barrier::new(2)); + let reader_round_ready = Arc::clone(&round_ready); + let reader_round_done = Arc::clone(&round_done); let reader_store = reopen(&db_path); let writer_store = reopen(&db_path); let reader_barrier = Arc::clone(&barrier); @@ -2967,8 +2970,6 @@ fn discovery_pull_side_returns_consistent_view_under_concurrent_writer() { let reader_done = Arc::clone(&writer_done); let writer_started_flag = Arc::clone(&writer_started); let writer_done_flag = Arc::clone(&writer_done); - let reader_in_call_flag = Arc::clone(&reader_in_call); - let writer_observed_reader = Arc::clone(&reader_in_call); #[derive(Debug)] struct ObservedRead { @@ -2988,11 +2989,15 @@ fn discovery_pull_side_returns_consistent_view_under_concurrent_writer() { let reader = thread::spawn(move || { reader_barrier.wait(); let mut reads = Vec::new(); - while reads.len() < 150 || !reader_done.load(Ordering::SeqCst) { + for read_index in 0..150 { + // Each of the first rounds overlaps one real supersede. The extra + // rendezvous observes writer completion before the final-state reads. + if read_index <= rounds { + reader_round_ready.wait(); + } let writer_active_at_start = reader_started.load(Ordering::SeqCst) && !reader_done.load(Ordering::SeqCst); let writer_done_before_read = reader_done.load(Ordering::SeqCst); - reader_in_call_flag.store(true, Ordering::SeqCst); let by_intention = reader_store .by_intention(main_intention_id, DiscoveryFilter::default()) .expect("reader by_intention"); @@ -3002,7 +3007,6 @@ fn discovery_pull_side_returns_consistent_view_under_concurrent_writer() { let by_entity_intentions = reader_store .by_entity_intentions(entity_id, DiscoveryFilter::default()) .expect("reader by_entity_intentions"); - reader_in_call_flag.store(false, Ordering::SeqCst); let writer_done_after_read = reader_done.load(Ordering::SeqCst); reads.push(ObservedRead { by_intention_ids: decision_ids(&by_intention), @@ -3023,22 +3027,20 @@ fn discovery_pull_side_returns_consistent_view_under_concurrent_writer() { writer_done_before_read, writer_done_after_read, }); - if !writer_done_before_read { - thread::yield_now(); + if read_index < rounds { + reader_round_done.wait(); } } reads }); let writer = thread::spawn(move || { - writer_barrier.wait(); - while !writer_observed_reader.load(Ordering::SeqCst) { - thread::yield_now(); - } writer_started_flag.store(true, Ordering::SeqCst); + writer_barrier.wait(); let mut replacements = Vec::new(); for (n, (decision_id, old_intention_id, new_intention_id)) in writer_pairs.into_iter().enumerate() { + round_ready.wait(); let replacement = writer_store .supersede_decision( decision_id, @@ -3056,10 +3058,10 @@ fn discovery_pull_side_returns_consistent_view_under_concurrent_writer() { old_intention_id, new_intention_id, )); - thread::sleep(Duration::from_millis(2)); - thread::yield_now(); + round_done.wait(); } writer_done_flag.store(true, Ordering::SeqCst); + round_ready.wait(); replacements }); diff --git a/crates/context-graph/tests/acceptance/evidence.rs b/crates/context-graph/tests/acceptance/evidence.rs index b47b393..b32d4a4 100644 --- a/crates/context-graph/tests/acceptance/evidence.rs +++ b/crates/context-graph/tests/acceptance/evidence.rs @@ -784,9 +784,83 @@ fn collect_rs_files(dir: &std::path::Path, files: &mut Vec) } } +fn core_source_outside_unit_tests(source: &str) -> String { + // rustfmt places a top-level module's closing brace in column zero. Retain + // everything after that brace so a later production item is still audited. + let mut lines = source.lines().peekable(); + let mut production = Vec::new(); + let mut in_unit_tests = false; + while let Some(line) = lines.next() { + if in_unit_tests { + if line == "}" { + in_unit_tests = false; + } + } else if line == "#[cfg(test)]" && lines.peek() == Some(&"mod tests {") { + lines.next(); + in_unit_tests = true; + } else { + production.push(line); + } + } + production.join("\n") +} + +fn sanctioned_test_thread_local_slot( + crate_dir: &std::path::Path, + file: &std::path::Path, + attribute: &str, + line: &str, +) -> bool { + line == "thread_local! {" + && [ + "src/store/event_correlation.rs", + "src/store/event_search_projection.rs", + "src/store/open_retry.rs", + "src/store/event_search_read.rs", + "src/store/graph_rows.rs", + "src/capture_sync_status.rs", + ] + .iter() + .any(|relative| file == crate_dir.join(relative)) + && matches!( + attribute, + "#[cfg(test)]" + | "#[cfg(context_graph_internal_test_support)]" + | "#[cfg(feature = \"test-support\")]" + | "#[cfg(any(context_graph_internal_test_support, feature = \"test-support\"))]" + | "#[cfg(all(test, context_graph_internal_test_support))]" + ) +} + fn assert_no_unsanctioned_app_side_locks_or_global_state_in_core_sources(context: &str) { let crate_dir = std::path::PathBuf::from(env!("CARGO_MANIFEST_DIR")); let src_dir = crate_dir.join("src"); + let with_unit_tests = + "AtomicBefore;\n#[cfg(test)]\nmod tests {\n AtomicInTest;\n}\nAtomicAfter;"; + assert_eq!( + core_source_outside_unit_tests(with_unit_tests), + "AtomicBefore;\nAtomicAfter;" + ); + assert!(core_source_outside_unit_tests("mod tests {\n Atomic;\n}").contains("Atomic")); + assert!( + core_source_outside_unit_tests("#[cfg(any(test, not(test)))]\nmod tests {\n Atomic;\n}") + .contains("Atomic") + ); + // A sanctioned file alone never licenses a production global or a broader cfg. + for attribute in ["", "#[cfg(any(test, not(test)))]"] { + assert!(!sanctioned_test_thread_local_slot( + &crate_dir, + &crate_dir.join("src/capture_sync_status.rs"), + attribute, + "thread_local! {", + )); + } + assert!(!sanctioned_test_thread_local_slot( + &crate_dir, + &crate_dir.join("src/store/evidence.rs"), + "#[cfg(test)]", + "thread_local! {", + )); let allowed_lines = [ "use std::sync::{Arc, Mutex, OnceLock, Weak};", "trace: Arc>,", @@ -843,21 +917,9 @@ fn assert_no_unsanctioned_app_side_locks_or_global_state_in_core_sources(context continue; } let text = std::fs::read_to_string(&file).expect("core source"); - // Sanctioned: `src/store/event_correlation.rs` (the correlation proof observer) and - // `src/store/event_search_projection.rs` (the finalization pause) each keep one test hook - // in a `thread_local!` slot that only `context_graph_internal_test_support` builds compile. - // A hook acts on the production path on that one test thread; it holds no product state, - // serializes no contention, and no default build contains it. - let sanctioned_test_hook_slot = file == crate_dir.join("src/store/event_correlation.rs") - || file == crate_dir.join("src/store/event_search_projection.rs") - // Private admission observer, compiled only by the dev-only test-support feature. - || file == crate_dir.join("src/store/open_retry.rs") - // `src/store/event_search_read.rs` keeps one slot holding the statements the - // event-history reads issued on the calling thread, so a proof can assert what a - // `find` or a `status` COSTS in statements rather than how long it took. It records - // what a read DID; it holds no product state, serializes nothing -- a thread's own - // `RefCell`, never a lock -- and only the dev-only test-support feature compiles it. - || file == crate_dir.join("src/store/event_search_read.rs"); + let text = core_source_outside_unit_tests(&text); + // Only explicit test configurations may carry these thread-local observers. + // They consume canonical clock/read seams and hold no product state. // Sanctioned: `src/store/mod.rs` serializes schema installation inside one process with // `SCHEMA_INSTALL_LOCK`, the process-local role the store `REGISTRY` already plays: store // handles on one path share one Database, and two of them must not interleave DDL. It @@ -878,7 +940,7 @@ fn assert_no_unsanctioned_app_side_locks_or_global_state_in_core_sources(context && (trimmed.contains("HashMap") || trimmed.contains("BTreeMap")); if (global_hash_state || forbidden.iter().any(|forbidden| line.contains(forbidden))) && !allowed_lines.iter().any(|allowed| line.trim() == *allowed) - && !(sanctioned_test_hook_slot && trimmed == "thread_local! {") + && !sanctioned_test_thread_local_slot(&crate_dir, &file, after, trimmed) && !sanctioned_schema_install_lock { panic!( @@ -2812,6 +2874,7 @@ fn replay_under_concurrent_cross_handle_writer_against_same_entity_returns_singl let committed = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); let attempts = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); let (replay_tx, replay_rx) = std::sync::mpsc::channel(); + let (progress_tx, progress_rx) = std::sync::mpsc::channel(); let writers = (0..WRITER_COUNT) .map(|writer| { let config = config.clone(); @@ -2819,6 +2882,7 @@ fn replay_under_concurrent_cross_handle_writer_against_same_entity_returns_singl let stop = std::sync::Arc::clone(&stop); let committed = std::sync::Arc::clone(&committed); let attempts = std::sync::Arc::clone(&attempts); + let progress_tx = progress_tx.clone(); std::thread::spawn(move || { let store = Store::open(config).expect("writer"); barrier.wait(); @@ -2841,35 +2905,18 @@ fn replay_under_concurrent_cross_handle_writer_against_same_entity_returns_singl .is_ok() { committed.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + let _ = progress_tx.send(()); } } }) }) .collect::>(); barrier.wait(); - let progress_deadline = std::time::Instant::now() + std::time::Duration::from_secs(30); - let progress_reached = loop { - if committed.load(std::sync::atomic::Ordering::SeqCst) >= REQUIRED_COMMITS { - break true; - } - if std::time::Instant::now() >= progress_deadline { - break false; - } - std::thread::yield_now(); - }; - match progress_reached { - true => {} - false => { - stop.store(true, std::sync::atomic::Ordering::SeqCst); - for writer in writers { - writer.join().expect("writer joined after progress timeout"); - } - panic!( - "writer flood did not create {REQUIRED_COMMITS} durable observations before replay; committed={}, attempts={}", - committed.load(std::sync::atomic::Ordering::SeqCst), - attempts.load(std::sync::atomic::Ordering::SeqCst) - ); - } + drop(progress_tx); + for _ in 0..REQUIRED_COMMITS { + progress_rx + .recv() + .expect("writer committed a durable observation"); } let reader_config = config.clone(); let reader_committed = std::sync::Arc::clone(&committed); @@ -2883,7 +2930,7 @@ fn replay_under_concurrent_cross_handle_writer_against_same_entity_returns_singl .expect("send replay result"); }); let (before_replay, after_replay, replay) = replay_rx - .recv_timeout(std::time::Duration::from_secs(5)) + .recv() .expect("replay returned while writer flood was still active"); stop.store(true, std::sync::atomic::Ordering::SeqCst); for writer in writers { @@ -3656,6 +3703,7 @@ fn basis_diff_under_concurrent_writer_returns_one_consistent_basis_to_current_dr let committed = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); let attempts = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); let (diff_tx, diff_rx) = std::sync::mpsc::channel(); + let (progress_tx, progress_rx) = std::sync::mpsc::channel(); let writers = (0..4) .map(|_| { let writer_config = config.clone(); @@ -3663,6 +3711,7 @@ fn basis_diff_under_concurrent_writer_returns_one_consistent_basis_to_current_dr let writer_stop = std::sync::Arc::clone(&stop); let writer_committed = std::sync::Arc::clone(&committed); let writer_attempts = std::sync::Arc::clone(&attempts); + let progress_tx = progress_tx.clone(); std::thread::spawn(move || { let store = Store::open(writer_config).expect("writer"); writer_barrier.wait(); @@ -3690,35 +3739,18 @@ fn basis_diff_under_concurrent_writer_returns_one_consistent_basis_to_current_dr .is_ok() { writer_committed.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + let _ = progress_tx.send(()); } } }) }) .collect::>(); barrier.wait(); - let progress_deadline = std::time::Instant::now() + std::time::Duration::from_secs(30); - let progress_reached = loop { - if committed.load(std::sync::atomic::Ordering::SeqCst) >= 50 { - break true; - } - if std::time::Instant::now() >= progress_deadline { - break false; - } - std::thread::yield_now(); - }; - match progress_reached { - true => {} - false => { - stop.store(true, std::sync::atomic::Ordering::SeqCst); - for writer in writers { - writer.join().expect("writer joined after progress timeout"); - } - panic!( - "writer flood did not create 50 durable snapshots before basis_diff; committed={}, attempts={}", - committed.load(std::sync::atomic::Ordering::SeqCst), - attempts.load(std::sync::atomic::Ordering::SeqCst) - ); - } + drop(progress_tx); + for _ in 0..50 { + progress_rx + .recv() + .expect("writer committed a durable snapshot"); } let diff_config = config.clone(); let diff_committed = std::sync::Arc::clone(&committed); @@ -3730,7 +3762,7 @@ fn basis_diff_under_concurrent_writer_returns_one_consistent_basis_to_current_dr diff_tx.send((before, after, diff)).expect("send diff"); }); let (before_diff, after_diff, diff) = diff_rx - .recv_timeout(std::time::Duration::from_secs(5)) + .recv() .expect("basis_diff returned while writer flood was still active"); stop.store(true, std::sync::atomic::Ordering::SeqCst); for writer in writers { diff --git a/crates/context-graph/tests/acceptance/nomic_loads.rs b/crates/context-graph/tests/acceptance/nomic_loads.rs index d575567..d88c2c4 100644 --- a/crates/context-graph/tests/acceptance/nomic_loads.rs +++ b/crates/context-graph/tests/acceptance/nomic_loads.rs @@ -1276,7 +1276,7 @@ fn cg_github_actions_workflow_runs_on_matrix_of_macos_latest_ubuntu_latest_and_d ( "macos-advisory.yml", "macos-latest", - "shard-union-macos", + "shard-union-macos-isolated", "macos-advisory", ), ] { @@ -1292,13 +1292,8 @@ fn cg_github_actions_workflow_runs_on_matrix_of_macos_latest_ubuntu_latest_and_d let rows = yaml["jobs"][job]["strategy"]["matrix"]["include"] .as_sequence() .unwrap(); - let expected_bundles = if system == "ubuntu-latest" { - vec!["acceptance", "import-and-cli", "integration-and-rest"] - } else if job == "features" { - vec!["integration-and-rest + import-and-cli + acceptance"] - } else { - vec!["acceptance", "integration-and-rest + import-and-cli"] - }; + // Both platforms run each logical bundle as its own job. + let expected_bundles = vec!["acceptance", "import-and-cli", "integration-and-rest"]; let mut actual_bundles = rows .iter() .map(|row| row["bundle"].as_str().expect("matrix bundle")) @@ -1330,7 +1325,7 @@ fn cg_readme_ci_section_documents_two_runner_matrix_and_shared_model_cache_root_ assert!(lower.contains("continuous integration") || lower.contains("## ci")); assert!(readme.contains("Required PR test and feature jobs run on `ubuntu-latest`")); assert!(readme.contains( - "separate `macOS advisory` workflow retains the same coverage on `macos-latest`" + "separate `macOS CI (advisory)` workflow retains the same coverage on `macos-latest`" )); assert!(readme.contains("`windows-latest` is excluded deliberately")); assert!(readme.contains("CG_MODEL_CACHE_ROOT` at runner temporary space")); diff --git a/crates/context-graph/tests/acceptance/subscriptions.rs b/crates/context-graph/tests/acceptance/subscriptions.rs index 3b3f4f9..cc3791c 100644 --- a/crates/context-graph/tests/acceptance/subscriptions.rs +++ b/crates/context-graph/tests/acceptance/subscriptions.rs @@ -1,11 +1,11 @@ use std::collections::{BTreeMap, BTreeSet, HashMap}; use std::path::{Path, PathBuf}; -use std::sync::{Arc, Barrier, Mutex}; +use std::sync::{Arc, Barrier, Condvar, Mutex, MutexGuard}; use std::time::{Duration, Instant}; use chrono::Utc; use context_graph::*; -use contextdb_core::{Lsn, TxId, Value as DbValue}; +use contextdb_core::{Lsn, TxId, Value as DbValue, Wallclock}; use contextdb_engine::Database as RawDatabase; use serde_json::json; use tempfile::TempDir; @@ -17,7 +17,7 @@ type NewDecisionCollector = ( ContextId, IntentionId, EntityId, - Arc>>, + Arc>, ); type InvalidationCollector = ( @@ -28,7 +28,7 @@ type InvalidationCollector = ( EntityId, IntentionId, DecisionId, - Arc>>, + Arc>, ); type ConflictCollector = ( @@ -38,7 +38,7 @@ type ConflictCollector = ( EntityId, IntentionId, DecisionId, - Arc>>, + Arc>, ); type PrecedentCollector = ( @@ -47,7 +47,7 @@ type PrecedentCollector = ( ContextId, IntentionId, DecisionId, - Arc>>, + Arc>, ); fn fixture() -> (TempDir, PathBuf, Store) { @@ -143,14 +143,79 @@ fn seed_pending_invalidation_direct( .expect("seed invalidation"); } -fn wait_for_len(items: &Arc>>, expected: usize) { - let deadline = Instant::now() + Duration::from_secs(3); - while Instant::now() < deadline && items.lock().expect("collector").len() < expected { - std::thread::sleep(Duration::from_millis(20)); +/// Collector writes and delivery waits share the same predicate lock. Dropping a +/// callback's collector guard announces its update without polling a clock. +struct Collector { + items: Mutex>, + changed: Condvar, +} + +struct CollectorGuard<'a, T> { + items: MutexGuard<'a, Vec>, + changed: &'a Condvar, +} + +impl std::ops::Deref for CollectorGuard<'_, T> { + type Target = Vec; + + fn deref(&self) -> &Self::Target { + &self.items + } +} + +impl std::ops::DerefMut for CollectorGuard<'_, T> { + fn deref_mut(&mut self) -> &mut Self::Target { + &mut self.items + } +} + +impl Drop for CollectorGuard<'_, T> { + fn drop(&mut self) { + self.changed.notify_all(); + } +} + +impl Collector { + fn new() -> Self { + Self { + items: Mutex::new(Vec::new()), + changed: Condvar::new(), + } + } + + fn lock(&self) -> std::result::Result, &'static str> { + self.items + .lock() + .map(|items| CollectorGuard { + items, + changed: &self.changed, + }) + .map_err(|_| "collector lock poisoned") + } +} + +fn wait_for_len(items: &Arc>, expected: usize) { + let guard = items.items.lock().expect("collector"); + let _guard = items + .changed + .wait_while(guard, |entries| entries.len() < expected) + .expect("delivery notification"); +} + +fn wait_until_each_reaches(counters: &[Arc>], expected: usize) { + for counter in counters { + wait_for_len(counter, expected); } } -fn assert_no_more(items: &Arc>>, expected: usize) { +fn timestamp() -> chrono::DateTime { + chrono::DateTime::from_timestamp_millis( + i64::try_from(Wallclock::now().0).expect("fixture timestamp fits"), + ) + .expect("fixture timestamp is representable") +} + +fn assert_no_more(items: &Arc>, expected: usize) { let deadline = Instant::now() + Duration::from_millis(250); while Instant::now() < deadline { assert_eq!(items.lock().expect("collector").len(), expected); @@ -232,7 +297,7 @@ fn reasoning(label: &str) -> Vec { details: format!("details-{label}"), result: format!("result-{label}"), agent_id: Some("subscription-test-agent".to_string()), - timestamp: Utc::now(), + timestamp: timestamp(), }] } @@ -290,7 +355,7 @@ fn evidence_ref(observation_id: ObservationId) -> EvidenceRef { content_hash: Some( "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa".to_string(), ), - captured_at: Some(Utc::now()), + captured_at: Some(timestamp()), retention_status: RetentionStatus::RetainedExternal, ..Default::default() } @@ -310,7 +375,7 @@ fn observe( context_id, observation_type: "state-change".to_string(), source: "subscription-test".to_string(), - observed_at: Utc::now(), + observed_at: timestamp(), evidence: vec![evidence_ref(observation_id)], observed_properties: BTreeMap::new(), state_delta: changes, @@ -325,7 +390,7 @@ fn collect_new_decision() -> NewDecisionCollector { let ctx = context(&store, "new-decision").id; let intent = intention(&store, ctx, "reload gate policy immediately"); let ent = entity(&store, ctx, "entry", BTreeMap::new()); - let payloads = Arc::new(Mutex::new(Vec::new())); + let payloads = Arc::new(Collector::new()); let seen = Arc::clone(&payloads); store .with_system_principal() @@ -362,7 +427,7 @@ fn collect_invalidation() -> InvalidationCollector { ..Default::default() }) .expect("create decision"); - let payloads = Arc::new(Mutex::new(Vec::new())); + let payloads = Arc::new(Collector::new()); let seen = Arc::clone(&payloads); store .with_system_principal() @@ -390,7 +455,7 @@ fn collect_conflict() -> ConflictCollector { store .record_outcome(older.id, true, Some("worked".to_string())) .expect("older outcome"); - let payloads = Arc::new(Mutex::new(Vec::new())); + let payloads = Arc::new(Collector::new()); let seen = Arc::clone(&payloads); store .with_system_principal() @@ -417,7 +482,7 @@ fn collect_precedent() -> PrecedentCollector { store .record_outcome(prior.id, true, Some("stayed below limit".to_string())) .expect("prior outcome"); - let payloads = Arc::new(Mutex::new(Vec::new())); + let payloads = Arc::new(Collector::new()); let seen = Arc::clone(&payloads); store .with_system_principal() @@ -575,7 +640,7 @@ fn subscribe_invalidation_detected_with_intention_id_filter_only_fires_for_match .expect("create decision"); } - let payloads = Arc::new(Mutex::new(Vec::new())); + let payloads = Arc::new(Collector::new()); let seen = Arc::clone(&payloads); store .with_system_principal() @@ -683,7 +748,7 @@ fn cascade_invalidation_payload_carries_cascade_from_pointer_and_renders_citing_ Vec::new(), vec![direct.id], ); - let payloads = Arc::new(Mutex::new(Vec::new())); + let payloads = Arc::new(Collector::new()); let seen = Arc::clone(&payloads); store .with_system_principal() @@ -828,7 +893,7 @@ fn precedent_match_does_not_fire_for_failed_outcome_precedent() { store .record_outcome(prior.id, false, Some("failed".to_string())) .expect("failed outcome"); - let payloads = Arc::new(Mutex::new(Vec::new())); + let payloads = Arc::new(Collector::new()); let seen = Arc::clone(&payloads); store .with_system_principal() @@ -979,7 +1044,7 @@ fn conflict_detected_scoped_to_context_with_engine_metrics_assertion() { .expect("b outcome"); drop(store); let store = reopen_scoped(&path, [ctx_a]); - let payloads = Arc::new(Mutex::new(Vec::new())); + let payloads = Arc::new(Collector::new()); let seen = Arc::clone(&payloads); let sub = store .with_system_principal() @@ -1080,7 +1145,7 @@ fn precedent_match_trigger_emits_one_event_per_validated_cites_precedent_with_di .record_outcome(parent_b.id, true, None) .expect("parent b outcome"); - let payloads = Arc::new(Mutex::new(Vec::new())); + let payloads = Arc::new(Collector::new()); let seen = Arc::clone(&payloads); store .with_system_principal() @@ -1254,7 +1319,7 @@ fn subscription_filter_context_ids_routes_msp_subscriber_across_authorized_conte denied, Err(SubscriptionError::ContextNotAuthorized(id)) if id == ctx_d )); - let payloads = Arc::new(Mutex::new(Vec::new())); + let payloads = Arc::new(Collector::new()); let seen = Arc::clone(&payloads); let sub = scoped .with_system_principal() @@ -1298,7 +1363,7 @@ fn subscribe_scoped_to_context_blocks_other_context_events_with_engine_route_ass let ctx_b = context(&store, "b").id; drop(store); let scoped = reopen_scoped(&path, [ctx_a]); - let payloads = Arc::new(Mutex::new(Vec::new())); + let payloads = Arc::new(Collector::new()); let seen = Arc::clone(&payloads); let sub = scoped .with_system_principal() @@ -1352,7 +1417,7 @@ fn subscribe_callback_panic_isolated_other_subscribers_continue_at_n_2() { panic!("subscriber panic is isolated by engine"); }) .expect("subscribe panicker"); - let seen = Arc::new(Mutex::new(Vec::new())); + let seen = Arc::new(Collector::new()); let seen_clone = Arc::clone(&seen); store .with_system_principal() @@ -1385,18 +1450,19 @@ fn subscribe_callback_panic_isolated_at_n_10_with_seven_panicking_and_three_coun }) .expect("subscribe panicker"); } - let seen = Arc::new(Mutex::new(Vec::new())); - for _ in 0..3 { - let seen_clone = Arc::clone(&seen); + let counters: Vec>> = + (0..3).map(|_| Arc::new(Collector::new())).collect(); + for counter in &counters { + let seen = Arc::clone(counter); store .with_system_principal() .subscribe_new_decision(SubscriptionFilter::default(), move |payload| { - seen_clone.lock().expect("collector").push(payload.clone()); + seen.lock().expect("collector").push(payload.decision.id); Ok(()) }) .expect("subscribe counter"); } - decision( + let created = decision( &store, ctx, "panic-isolated-scale", @@ -1404,8 +1470,16 @@ fn subscribe_callback_panic_isolated_at_n_10_with_seven_panicking_and_three_coun vec![ent], vec![], ); - wait_for_len(&seen, 3); - assert_eq!(seen.lock().expect("collector").len(), 3); + // Each healthy collector notifies after its callback writes the decision. + // The seven panicking subscribers cannot satisfy a healthy collector's wait. + wait_until_each_reaches(&counters, 1); + for (index, counter) in counters.iter().enumerate() { + assert_eq!( + *counter.lock().expect("collector"), + vec![created.id], + "counting subscriber {index} must receive exactly the one decision" + ); + } } // Use case: two separate Store instances cannot poison each other's dispatchers. @@ -1426,7 +1500,7 @@ fn cross_instance_independence_one_store_panic_does_not_affect_parallel_store() let ctx_y = context(&store_y, "healthy-store").id; let intent_y = intention(&store_y, ctx_y, "y intent"); let ent_y = entity(&store_y, ctx_y, "y entity", BTreeMap::new()); - let seen_y = Arc::new(Mutex::new(Vec::new())); + let seen_y = Arc::new(Collector::new()); let seen_y_clone = Arc::clone(&seen_y); store_y .with_system_principal() @@ -1463,8 +1537,8 @@ fn cross_instance_independence_one_store_panic_does_not_affect_parallel_store() #[test] fn unsubscribe_stops_events_for_removed_subscription_only_with_engine_metrics() { let (_dir, _path, store, ctx, intent, ent, _payloads) = collect_new_decision(); - let a = Arc::new(Mutex::new(Vec::new())); - let b = Arc::new(Mutex::new(Vec::new())); + let a = Arc::new(Collector::new()); + let b = Arc::new(Collector::new()); let a_seen = Arc::clone(&a); let b_seen = Arc::clone(&b); let sub_a = store @@ -1548,7 +1622,7 @@ fn subscription_does_not_fire_when_observation_transaction_rolls_back() { seed_pending_invalidation_direct(&path, ctx, dec.id, seed_observation.observation_id); let store = reopen(&path); - let payloads = Arc::new(Mutex::new(Vec::new())); + let payloads = Arc::new(Collector::new()); let seen = Arc::clone(&payloads); store .with_system_principal() @@ -1565,7 +1639,7 @@ fn subscription_does_not_fire_when_observation_transaction_rolls_back() { context_id: ctx, observation_type: "state-change".to_string(), source: "subscription-test".to_string(), - observed_at: Utc::now(), + observed_at: timestamp(), evidence: vec![evidence_ref(attempted_id)], observed_properties: BTreeMap::new(), state_delta: delta(&[("version", json!("2"))]), @@ -1609,7 +1683,7 @@ fn subscription_does_not_fire_when_observation_transaction_rolls_back() { #[test] fn lost_subscription_acknowledgement_retries_one_identity_and_one_receiver_effect() { let (_dir, _path, store, ctx, ent, _intent, _dec, _payloads) = collect_invalidation(); - let attempts = Arc::new(Mutex::new(Vec::new())); + let attempts = Arc::new(Collector::new()); let visible_effects = Arc::new(Mutex::new(BTreeSet::new())); let attempts_for_handler = Arc::clone(&attempts); let effects_for_handler = Arc::clone(&visible_effects); @@ -1660,7 +1734,7 @@ fn lost_subscription_acknowledgement_retries_one_identity_and_one_receiver_effec #[test] fn external_handler_without_dedup_reports_uncertain_delivery() { let (_dir, _path, store, ctx, ent, _intent, _dec, _payloads) = collect_invalidation(); - let attempts = Arc::new(Mutex::new(Vec::new())); + let attempts = Arc::new(Collector::new()); let seen = Arc::clone(&attempts); let subscription = store .with_system_principal() @@ -1736,7 +1810,7 @@ fn serial_observations_on_same_entity_fire_invalidation_detected_in_observation_ fn cross_event_fifo_order_new_decision_precedes_invalidation_detected_for_same_decision_lifecycle() { let (_dir, _path, store, ctx, ent, intent, _dec, invalidations) = collect_invalidation(); - let new_decisions = Arc::new(Mutex::new(Vec::new())); + let new_decisions = Arc::new(Collector::new()); let seen = Arc::clone(&new_decisions); store .with_system_principal() @@ -1815,7 +1889,7 @@ fn subscription_resume_replays_queued_events_after_reopen_on_same_subscription_i assert_eq!(before_resume.delivered, 0); assert_eq!(before_resume.queued, 1); - let payloads = Arc::new(Mutex::new(Vec::new())); + let payloads = Arc::new(Collector::new()); let seen = Arc::clone(&payloads); let resumed = store .with_system_principal() @@ -1942,7 +2016,7 @@ fn subscription_resume_replays_queued_events_after_reopen_for_invalidation_confl let precedent_before = wait_for_metrics(&store, &precedent_sub, |metrics| metrics.queued == 1); assert_eq!(precedent_before.delivered, 0); - let invalidations = Arc::new(Mutex::new(Vec::new())); + let invalidations = Arc::new(Collector::new()); let seen_invalidations = Arc::clone(&invalidations); let resumed_inv = store .with_system_principal() @@ -1960,7 +2034,7 @@ fn subscription_resume_replays_queued_events_after_reopen_for_invalidation_confl .expect("resume invalidation"); assert_eq!(resumed_inv, inv_sub); - let conflicts = Arc::new(Mutex::new(Vec::new())); + let conflicts = Arc::new(Collector::new()); let seen_conflicts = Arc::clone(&conflicts); let resumed_conflict = store .with_system_principal() @@ -1978,7 +2052,7 @@ fn subscription_resume_replays_queued_events_after_reopen_for_invalidation_confl .expect("resume conflict"); assert_eq!(resumed_conflict, conflict_sub); - let precedents = Arc::new(Mutex::new(Vec::new())); + let precedents = Arc::new(Collector::new()); let seen_precedents = Arc::clone(&precedents); let resumed_precedent = store .with_system_principal() @@ -2097,7 +2171,7 @@ fn lifecycle_audit_chain(with_subscriber: bool) -> Vec { ..Default::default() }) .expect("create decision"); - let payloads = Arc::new(Mutex::new(Vec::new())); + let payloads = Arc::new(Collector::new()); if with_subscriber { let seen = Arc::clone(&payloads); store diff --git a/crates/context-graph/tests/common/ci_stages.rs b/crates/context-graph/tests/common/ci_stages.rs index e654169..7181c2f 100644 --- a/crates/context-graph/tests/common/ci_stages.rs +++ b/crates/context-graph/tests/common/ci_stages.rs @@ -22,7 +22,7 @@ pub fn routes_are_executable(script: &str) -> bool { let lines = active_lines(script); let required = [ "set -euo pipefail", - "shard-union|shard-union-linux|shard-union-macos|shard-union-linux-isolated) run_stage \"$stage\" shard_union ;;", + "shard-union|shard-union-linux|shard-union-macos|shard-union-linux-isolated|shard-union-macos-isolated) run_stage \"$stage\" shard_union ;;", "trace_stage \"$2\" | tee \"$receipt_output\"", "source|lint|docs) run_stage \"$stage\" \"$stage\" ;;", "run_stage \"$stage\" \"$stage\" ;;", @@ -152,7 +152,8 @@ pub fn effective(workflow: &mut Value, script: &str) { "bash scripts/gate.sh shard-union" | "bash scripts/gate.sh shard-union-linux" | "bash scripts/gate.sh shard-union-macos" - | "bash scripts/gate.sh shard-union-linux-isolated" => &["shard_union"], + | "bash scripts/gate.sh shard-union-linux-isolated" + | "bash scripts/gate.sh shard-union-macos-isolated" => &["shard_union"], "bash scripts/gate.sh source" => &["source"], "bash scripts/gate.sh lint" => &["lint_audit", "lint_format", "lint_clippy"], "bash scripts/gate.sh docs" => &["docs"], @@ -254,7 +255,7 @@ fn model_contract_matches( && format!("{:x}", Sha256::digest(helper)) == "347121a931faee2f0e9a2f4bf0ee9baf7cf3859c06e749954b53845d088db5a4" && format!("{:x}", Sha256::digest(manifest)) - == "664292fa142db458743ea06e595756917fadc8ff7a7fb68f4440112c9a863daa" + == "edecf9f1f7ec684692df40dd87be2be24d91d47d780264c5ac40e2575a41e59e" && format!("{:x}", Sha256::digest(launcher)) == "7f8fa9c6efa60180651177d4ea04c75ede06487828a389fe19dceeaa401b97ca" && format!("{:x}", Sha256::digest(receipt)) @@ -262,7 +263,7 @@ fn model_contract_matches( && format!("{:x}", Sha256::digest(core.0)) == "9feafef445a5364998c90178048e781f08b9840595a311f9f7bf620b2c0798d9" && format!("{:x}", Sha256::digest(core.1)) - == "125356acd39f98d17aa3806ea4e94d3ae7ad7ba4d00e9399e6a184cf8cd70867" + == "b7139130b299fd226fcccabab8c06b0e58c3881577e6aa763ce49421b692cc60" } fn shard_contract_is_frozen(script: &str) -> bool { @@ -290,7 +291,7 @@ fn shard_contract_is_frozen(script: &str) -> bool { ), ( "shard_union", - "deea18c7d3ac2a726d5af9f58bfd311143e30e55af0eb4122b77d2f7cfd889ff", + "7b70adc5601f3c36f92929ef31961020ca76df6e36fa8d90b12e557763505900", ), ( "features", @@ -318,11 +319,11 @@ fn shard_contract_is_frozen(script: &str) -> bool { ), ( "feature_qualification_sync", - "664a16d58c155042d7da1f5e2d36dd2fc885ad4fbd88c32a63a03f129fc1eec4", + "ca87a5fc7df6047eb0eb40d215da32e69c33fff1025546acb9fc59d82dcd2e2c", ), ( "features_tests", - "8765584742dc637f7c348fe311cd6677688977b09c6b7eacb2f6183a585394f0", + "121125852235a5d9c71f79edd0ec012d1dbc347b779de9f62b6a3fdaa9e077c9", ), ( "default_wall_isolated", @@ -342,11 +343,11 @@ fn shard_contract_is_frozen(script: &str) -> bool { }) && format!( "{:x}", Sha256::digest(include_bytes!("../../../../scripts/ci-test-shards.py")) - ) == "fb4a3b06a7ec775b9d5b7d7b333920e7dad25859a9f35ca192ca2cffe7276bec" + ) == "02a67a7b222d92aa64144952ceb5d05cbb8069364d85f4311a8557db4c0d294d" && format!( "{:x}", Sha256::digest(include_bytes!("../../../../scripts/ci-test-shards.json")) - ) == "2a21c8fa764e57e3f1b26b0011555415e469ebc3e16c6c3ea5d0a6f0208c03d5" + ) == "0514a0b25b75bdc4d73395b3383cb9870d91218b6ce5ba390af66acda3222459" && format!( "{:x}", Sha256::digest(include_bytes!( @@ -362,42 +363,31 @@ fn shard_contract_is_frozen(script: &str) -> bool { && format!( "{:x}", Sha256::digest(include_bytes!("../../../../Cargo.toml")) - ) == "ab995589c0159a218d0fd370a539c2994b6008e7e64b0d3e171062e6e0b11e85" + ) == "871a8c130d3dbc7e88d4da615dafdd83ec25cfac85b170cd75f673a0a4d376d7" && format!( "{:x}", Sha256::digest(include_bytes!("../../../../Cargo.lock")) - ) == "3d835eba6e63d7839a0eb78ca6715fb805a95509a8dcaf0df7fdc699907a9b84" + ) == "b44b27b83f71c94adff4eddf806d4d5a9ed3f0535f94c44255cdf149bac1afd2" && format!( "{:x}", Sha256::digest(include_bytes!( "../../../../crates/context-graph/tests/test_estate_audit.rs" )) - ) == "f4c2a98e24c6cde0802f91f9bbd6a881ca0fe22557e1ac00d5bf5ae2fd1e5a56" + ) == "5dc41701ce3faf5d2502b71e3a945e73d07158ec859f476fb58b183e3a8dfb3b" } #[allow(dead_code)] pub fn exact_physical_matrix(matrix: &serde_yaml_ng::Mapping) -> bool { - let default: Value = serde_yaml_ng::from_str( + // Default and feature execution run every logical bundle as its own job on each platform. + let physical: Value = serde_yaml_ng::from_str( r#" include: - os: macos-latest bundle: acceptance - os: macos-latest - bundle: integration-and-rest + import-and-cli - - os: ubuntu-latest - bundle: acceptance - - os: ubuntu-latest bundle: integration-and-rest - - os: ubuntu-latest - bundle: import-and-cli -"#, - ) - .expect("default physical matrix"); - let features: Value = serde_yaml_ng::from_str( - r#" -include: - os: macos-latest - bundle: integration-and-rest + import-and-cli + acceptance + bundle: import-and-cli - os: ubuntu-latest bundle: acceptance - os: ubuntu-latest @@ -406,9 +396,8 @@ include: bundle: import-and-cli "#, ) - .expect("feature physical matrix"); - same_matrix(&Value::Mapping(matrix.clone()), &default) - || same_matrix(&Value::Mapping(matrix.clone()), &features) + .expect("physical matrix"); + same_matrix(&Value::Mapping(matrix.clone()), &physical) } pub const CONTROLS_SETUP: &str = "python3 -m venv \"$RUNNER_TEMP/ci-controls-python\"\n\"$RUNNER_TEMP/ci-controls-python/bin/python\" -m pip install PyYAML==6.0.2"; @@ -637,12 +626,13 @@ pub fn exact_shared_command(run: &str) -> bool { Sha256::digest(include_bytes!( "../../../../scripts/ci-cache-post-cleanup.py" )) - ) == "849caa4caee4553e55e5f62c7572e43e29f51b1f901d505ee0b58bbbc0a90844") + ) == "610271f4180992976d687716b7a18eeb94e1d395a26fa7a9421ede505dc47824") || (([ "bash scripts/gate.sh shard-union", "bash scripts/gate.sh shard-union-linux", "bash scripts/gate.sh shard-union-linux-isolated", "bash scripts/gate.sh shard-union-macos", + "bash scripts/gate.sh shard-union-macos-isolated", ] .contains(&run.trim()) || [ @@ -866,7 +856,7 @@ pub fn platform_coverage(mut required: Value, advisory: Value) -> Value { "../../../../.github/workflows/macos-advisory.yml" )) ), - "4ee054c66a544643b33e1285e5524c624c368c452d3ea88407c6b99a1ca726b4" + "e036617f82581348085d77a879a9c5a1e7e87ee23902e10e4af35033062bd4c9" ); let frozen_mac: Value = serde_yaml_ng::from_str(include_str!( "../../../../.github/workflows/macos-advisory.yml" @@ -876,8 +866,8 @@ pub fn platform_coverage(mut required: Value, advisory: Value) -> Value { advisory, frozen_mac, "whole advisory mapping must remain unchanged" ); - assert_eq!(required["name"].as_str(), Some("ci")); - assert_eq!(advisory["name"].as_str(), Some("macOS advisory")); + assert_eq!(required["name"].as_str(), Some("Linux CI")); + assert_eq!(advisory["name"].as_str(), Some("macOS CI (advisory)")); for key in ["on", "permissions", "env", "defaults"] { assert_eq!(required[key], advisory[key], "platform workflow context"); } @@ -950,7 +940,15 @@ pub fn platform_coverage(mut required: Value, advisory: Value) -> Value { actual.sort_unstable(); assert_eq!( actual, - ["controls", "features", "lint", "macos-advisory", "test"] + [ + "controls", + "feature-qualification-core", + "feature-qualification-sync", + "features", + "lint", + "macos-advisory", + "test" + ] ); let compile_core: Value = serde_yaml_ng::from_str(r###"name: Feature compatibility — default and minimal libraries (ubuntu-latest) runs-on: ubuntu-latest @@ -1002,6 +1000,9 @@ steps: CG_MODEL_CACHE_ROOT: ${{ runner.temp }}/cg-model-cache with: cache-bin: false + prefix-key: v1-rust-1.94.1 + add-rust-environment-hash-key: false + save-if: ${{ github.ref == 'refs/heads/dev' }} workspaces: workspace/context-graph -> target key: feature-qualification-core-v1 - name: Run feature compatibility qualification @@ -1082,6 +1083,9 @@ steps: CG_MODEL_CACHE_ROOT: ${{ runner.temp }}/cg-model-cache with: cache-bin: false + prefix-key: v1-rust-1.94.1 + add-rust-environment-hash-key: false + save-if: ${{ github.ref == 'refs/heads/dev' }} workspaces: workspace/context-graph -> target key: feature-qualification-core-v1 - name: Run feature compatibility qualification @@ -1110,9 +1114,25 @@ steps: path: ${{ runner.temp }}/job-receipt retention-days: 14 if-no-files-found: error -"###.replace("default and minimal libraries", "sync, CLI and doctests") +"###.replace("default and minimal libraries", "sync and CLI") .replace("feature-qualification-core", "feature-qualification-sync")).unwrap(); assert_eq!(required["jobs"]["feature-qualification-sync"], compile_sync); + // macOS runs both compatibility build jobs unchanged; only the runner and its label differ. + for job in ["feature-qualification-core", "feature-qualification-sync"] { + let mut expected = required["jobs"][job].clone(); + expected["runs-on"] = Value::String("macos-latest".into()); + expected["name"] = Value::String( + required["jobs"][job]["name"] + .as_str() + .unwrap() + .replace("(ubuntu-latest)", "(macos-latest)"), + ); + assert_ne!(expected["name"], required["jobs"][job]["name"]); + assert_eq!( + advisory["jobs"][job], expected, + "macOS compatibility job {job}" + ); + } for job in ["lint", "controls", "test", "features"] { let linux = &required["jobs"][job]; let mac = &advisory["jobs"][job]; @@ -1121,81 +1141,22 @@ steps: && linux.get("continue-on-error").is_none() && linux.get("needs").is_none() ); + // Every macOS job runs the Linux job's exact steps; only the matrix runner differs. let mut comparable = linux.clone(); - if job == "test" || job == "features" { - for stage in ["source", "model-seed", "public-model"] { - let command = format!("bash scripts/gate.sh {stage}"); - let linux_steps = comparable["steps"].as_sequence_mut().unwrap(); - let matches: Vec<_> = linux_steps - .iter_mut() - .filter(|step| step["run"].as_str().map(str::trim) == Some(command.as_str())) - .collect(); - assert_eq!(matches.len(), 1); - let step = matches.into_iter().next().unwrap(); - let mac_steps: Vec<_> = mac["steps"] - .as_sequence() - .unwrap() - .iter() - .filter(|step| step["run"].as_str().map(str::trim) == Some(command.as_str())) - .collect(); - assert_eq!(mac_steps.len(), 1); - let mac_step = mac_steps[0]; - assert_eq!( - step["env"]["GATE_RECEIPT_DIR"].as_str(), - Some("${{ runner.temp }}/job-receipt/${{ matrix.bundle }}") - ); - assert_eq!( - mac_step["env"]["GATE_RECEIPT_DIR"].as_str(), - Some( - "${{ runner.temp }}/job-receipt/${{ contains(matrix.bundle, 'integration-and-rest') && 'integration-and-rest' || 'acceptance' }}" - ) - ); - step["env"]["GATE_RECEIPT_DIR"] = mac_step["env"]["GATE_RECEIPT_DIR"].clone(); - if job == "test" && stage != "source" { - assert_eq!( - step["env"]["GATE_CI_MODEL_MODE"].as_str(), - Some( - "${{ matrix.bundle == 'acceptance' && 'catalog-narrow' || 'catalog-workspace' }}" - ) - ); - assert_eq!( - mac_step["env"]["GATE_CI_MODEL_MODE"].as_str(), - Some( - "${{ contains(matrix.bundle, 'integration-and-rest') && 'catalog-workspace' || 'catalog-narrow' }}" - ) - ); - step["env"]["GATE_CI_MODEL_MODE"] = - mac_step["env"]["GATE_CI_MODEL_MODE"].clone(); - } - } - } - if job == "features" { - let steps = comparable["steps"].as_sequence_mut().unwrap(); - let selected: Vec<_> = steps - .iter_mut() - .filter(|step| { - step["run"].as_str() - == Some("python3 scripts/ci-test-shards.py bundle features --tests-only") - }) - .collect(); - assert_eq!(selected.len(), 1); - selected.into_iter().next().unwrap()["run"] = - Value::String("python3 scripts/ci-test-shards.py bundle features".into()); - } comparable["strategy"]["matrix"] = mac["strategy"]["matrix"].clone(); assert_eq!(comparable, *mac, "complete platform commands must match"); - let expected_linux: Value = if job == "lint" || job == "controls" { - serde_yaml_ng::from_str("os: [ubuntu-latest]").unwrap() - } else { - serde_yaml_ng::from_str("include:\n - os: ubuntu-latest\n bundle: acceptance\n - os: ubuntu-latest\n bundle: integration-and-rest\n - os: ubuntu-latest\n bundle: import-and-cli").unwrap() - }; - let expected_mac: Value = if job == "lint" || job == "controls" { - serde_yaml_ng::from_str("os: [macos-latest]").unwrap() - } else if job == "test" { - serde_yaml_ng::from_str("include:\n - os: macos-latest\n bundle: acceptance\n - os: macos-latest\n bundle: integration-and-rest + import-and-cli").unwrap() - } else { - serde_yaml_ng::from_str("include:\n - os: macos-latest\n bundle: integration-and-rest + import-and-cli + acceptance").unwrap() + let rows = |system: &str| -> Value { + if job == "lint" || job == "controls" { + serde_yaml_ng::from_str(&format!("os: [{system}]")).unwrap() + } else { + serde_yaml_ng::from_str(&format!( + "include:\n - os: {system}\n bundle: acceptance\n - os: {system}\n bundle: integration-and-rest\n - os: {system}\n bundle: import-and-cli" + )) + .unwrap() + } }; + let expected_linux = rows("ubuntu-latest"); + let expected_mac = rows("macos-latest"); assert!(same_matrix(&linux["strategy"]["matrix"], &expected_linux)); assert!(same_matrix(&mac["strategy"]["matrix"], &expected_mac)); let key = if job == "lint" || job == "controls" { @@ -1207,43 +1168,54 @@ steps: rows.extend(expected_linux[key].as_sequence().unwrap().clone()); comparable["strategy"]["matrix"] = linux["strategy"]["matrix"].clone(); comparable["strategy"]["matrix"][key] = Value::Sequence(rows); - if job == "features" { - for step in comparable["steps"].as_sequence_mut().unwrap() { - if step["run"].as_str() == Some("python3 scripts/ci-test-shards.py bundle features") - { - step["run"] = Value::String( - "python3 scripts/ci-test-shards.py bundle features --tests-only".into(), - ); - } - } - } required["jobs"][job] = comparable; } let mut expected = advisory["jobs"]["macos-advisory"].clone(); expected["name"] = Value::String("PR checks".into()); + let mut mac_needs: Vec<_> = expected["needs"] + .as_sequence() + .unwrap() + .iter() + .map(|v| v.as_str().unwrap().to_string()) + .collect(); + mac_needs.sort_unstable(); + assert_eq!( + mac_needs, + [ + "controls", + "feature-qualification-core", + "feature-qualification-sync", + "features", + "lint", + "test" + ] + ); expected["needs"] = serde_yaml_ng::from_str("[source, lint, docs, test, features, controls, feature-qualification-core, feature-qualification-sync]").unwrap(); let steps = expected["steps"].as_sequence_mut().unwrap(); - let before = steps.len(); - steps.retain(|step| { - ![ - "Free disk space", - "Record disk headroom and remove disposable compiler scratch", - "Remove disposable compiler scratch", - ] - .contains(&step["name"].as_str().unwrap_or("")) - && step["uses"].as_str() - != Some("Swatinem/rust-cache@49a0bdc70d2e1b713ca9e2869b211fcce03d3c1c") - }); - assert_eq!(before - steps.len(), 4); + // The macOS aggregate requires exactly the macOS jobs, which are the Linux jobs other than + // the Linux-only source and documentation checks, and audits the macOS union the same way. + let mac_check = REQUIRED_PR_JOB_AGGREGATE_RUN.replace( + "expected = {'docs', 'source', 'features',", + "expected = {'features',", + ); + assert_ne!(mac_check, REQUIRED_PR_JOB_AGGREGATE_RUN); + let mut checks = 0; + let mut unions = 0; for step in steps { if let Some(run) = step["run"].as_str() { - step["run"] = Value::String(if run.starts_with("python3 - <<'PY_CHECK'") { - REQUIRED_PR_JOB_AGGREGATE_RUN.to_string() - } else { - run.replace("shard-union-macos", "shard-union-linux-isolated") - }); + if run.starts_with("python3 - <<'PY_CHECK'") { + assert_eq!(run.trim(), mac_check.trim(), "macOS aggregate inventory"); + checks += 1; + step["run"] = Value::String(REQUIRED_PR_JOB_AGGREGATE_RUN.to_string()); + } else if run.contains("shard-union") { + assert_eq!(run, "bash scripts/gate.sh shard-union-macos-isolated"); + unions += 1; + step["run"] = + Value::String("bash scripts/gate.sh shard-union-linux-isolated".into()); + } } } + assert_eq!((checks, unions), (1, 1)); let mut actual = required["jobs"]["required"].clone(); for job in [&mut actual, &mut expected] { job["needs"] diff --git a/crates/context-graph/tests/event_context_counts.rs b/crates/context-graph/tests/event_context_counts.rs index eb9e255..c6ce571 100644 --- a/crates/context-graph/tests/event_context_counts.rs +++ b/crates/context-graph/tests/event_context_counts.rs @@ -739,7 +739,7 @@ impl DatabasePlugin for InterleaveOnPurge { .take() .expect("armed once"); committed - .recv_timeout(Duration::from_secs(60)) + .recv() .expect("the concurrent append commits before the purge runs"); } Ok(()) @@ -764,9 +764,7 @@ impl DatabasePlugin for InterleaveOnPurge { .expect("observed lock") .take() .expect("armed once"); - observed - .recv_timeout(Duration::from_secs(60)) - .expect("the observer read the count row"); + observed.recv().expect("the observer read the count row"); } } } @@ -796,9 +794,7 @@ fn append_on_release( committed: Sender<()>, ) -> std::thread::JoinHandle<()> { std::thread::spawn(move || { - release - .recv_timeout(Duration::from_secs(60)) - .expect("the forget reached its purge"); + release.recv().expect("the forget reached its purge"); capture( &store, context_id, @@ -854,9 +850,7 @@ fn an_append_committing_before_a_session_forgets_purge_leaves_no_stale_total() { let store = store.clone(); let between = Arc::clone(&between); std::thread::spawn(move || { - observe_rx - .recv_timeout(Duration::from_secs(60)) - .expect("the purge committed"); + observe_rx.recv().expect("the purge committed"); *between.lock().expect("observation lock") = Some(count_row(&store, context_id)); observed.send(()).expect("report the observation"); }) diff --git a/crates/context-graph/tests/event_core_fault_recovery.rs b/crates/context-graph/tests/event_core_fault_recovery.rs index 902f444..38d723c 100644 --- a/crates/context-graph/tests/event_core_fault_recovery.rs +++ b/crates/context-graph/tests/event_core_fault_recovery.rs @@ -406,9 +406,7 @@ fn run_append_race(conflicts: usize) -> RaceObservation { for attempt in 1..=conflicts { assert_eq!( - arrived - .recv_timeout(Duration::from_secs(15)) - .expect("target reaches real event insert"), + arrived.recv().expect("target reaches real event insert"), attempt, "the controller releases each target attempt only after its INSERT INTO events is staged" ); diff --git a/crates/context-graph/tests/event_search_embedding_parallelism.rs b/crates/context-graph/tests/event_search_embedding_parallelism.rs index 37be80a..f15a18e 100644 --- a/crates/context-graph/tests/event_search_embedding_parallelism.rs +++ b/crates/context-graph/tests/event_search_embedding_parallelism.rs @@ -6,7 +6,6 @@ use std::num::{NonZeroU32, NonZeroU64, NonZeroUsize}; use std::sync::Arc; use std::sync::atomic::{AtomicUsize, Ordering}; use std::sync::mpsc::{Receiver, Sender, channel}; -use std::time::Duration; use cg_node_fs::{read_event_search_worker_policy, write_event_search_worker_policy}; use context_graph::embedder::{ @@ -168,8 +167,8 @@ impl Embedder for ControlledEmbedder { .map_err(|_| { context_graph::CgError::Engine("parallelism observer went away".into()) })?; - held.recv_timeout(Duration::from_secs(30)).map_err(|_| { - context_graph::CgError::Engine("parallelism test release timed out".into()) + held.recv().map_err(|_| { + context_graph::CgError::Engine("parallelism test release disconnected".into()) })?; } else { self.entered @@ -194,11 +193,11 @@ fn concurrent_embedding_calls_run_with_one_rayon_helper_each_for_standalone_and_ let standalone = start_projection(NonZeroUsize::new(2).expect("positive worker cap"), None); let first = standalone .entered - .recv_timeout(Duration::from_secs(10)) + .recv() .expect("standalone first embedding call"); let second = standalone .entered - .recv_timeout(Duration::from_secs(10)) + .recv() .expect("standalone second embedding call"); assert_eq!( first.helper_threads, 1, @@ -220,11 +219,11 @@ fn concurrent_embedding_calls_run_with_one_rayon_helper_each_for_standalone_and_ start_held_store_projection(NonZeroUsize::new(2).expect("positive worker cap")); let first = held_store .entered - .recv_timeout(Duration::from_secs(10)) + .recv() .expect("held-Store first embedding call"); let second = held_store .entered - .recv_timeout(Duration::from_secs(10)) + .recv() .expect("held-Store second embedding call"); assert_eq!( first.helper_threads, 1, @@ -490,7 +489,7 @@ fn finish_projection_with_held_count( call.release(); } while held_observed < expected_held { - let Ok(call) = run.entered.recv_timeout(Duration::from_secs(30)) else { + let Ok(call) = run.entered.recv() else { break; }; if call.release.is_some() { @@ -511,9 +510,9 @@ fn default_runner_enters_two_embedding_calls_before_either_is_released() { return; } let run = start_projection(available, None); - let first = run.entered.recv_timeout(Duration::from_secs(10)).ok(); + let first = run.entered.recv().ok(); let first_observed = first.is_some(); - let second = run.entered.recv_timeout(Duration::from_secs(10)); + let second = run.entered.recv(); let overlap_observed = second.is_ok(); let mut observed = first.into_iter().collect::>(); if let Ok(second) = second { @@ -543,14 +542,9 @@ fn default_runner_enters_two_embedding_calls_before_either_is_released() { #[test] fn embedding_worker_cap_one_keeps_the_second_call_out_until_the_first_returns() { let run = start_projection(NonZeroUsize::MIN, None); - let first = run.entered.recv_timeout(Duration::from_secs(10)).ok(); + let first = run.entered.recv().ok(); let first_observed = first.is_some(); - assert_eq!( - run.waves - .recv_timeout(Duration::from_secs(10)) - .expect("wave admission"), - 1 - ); + assert_eq!(run.waves.recv().expect("wave admission"), 1); let second_before_release = run.entered.try_recv(); let overlap_observed = second_before_release.is_ok(); let mut observed = first.into_iter().collect::>(); @@ -579,15 +573,10 @@ fn embedding_worker_cap_one_keeps_the_second_call_out_until_the_first_returns() #[test] fn embedding_worker_cap_two_enters_two_calls_but_never_a_third_before_release() { let run = start_projection(NonZeroUsize::new(2).unwrap(), None); - let first = run.entered.recv_timeout(Duration::from_secs(10)).ok(); + let first = run.entered.recv().ok(); let first_observed = first.is_some(); - let second = run.entered.recv_timeout(Duration::from_secs(10)); - assert_eq!( - run.waves - .recv_timeout(Duration::from_secs(10)) - .expect("wave admission"), - 2 - ); + let second = run.entered.recv(); + assert_eq!(run.waves.recv().expect("wave admission"), 2); let third_before_release = second .as_ref() .ok() @@ -626,11 +615,11 @@ fn embedding_worker_cap_two_enters_two_calls_but_never_a_third_before_release() #[test] fn held_store_cap_two_overlaps_two_calls_while_its_claim_lease_stays_live() { let run = start_held_store_projection(NonZeroUsize::new(2).unwrap()); - let first = run.entered.recv_timeout(Duration::from_secs(10)).ok(); - let second = run.entered.recv_timeout(Duration::from_secs(10)).ok(); + let first = run.entered.recv().ok(); + let second = run.entered.recv().ok(); assert_eq!( run.waves - .recv_timeout(Duration::from_secs(10)) + .recv() .expect("complete held-Store wave admission"), 2 ); @@ -667,13 +656,8 @@ fn held_store_cap_two_overlaps_two_calls_while_its_claim_lease_stays_live() { #[test] fn held_store_cap_one_serializes_calls_while_its_claim_lease_stays_live() { let run = start_held_store_projection(NonZeroUsize::MIN); - let first = run.entered.recv_timeout(Duration::from_secs(10)).ok(); - assert_eq!( - run.waves - .recv_timeout(Duration::from_secs(10)) - .expect("wave admission"), - 1 - ); + let first = run.entered.recv().ok(); + assert_eq!(run.waves.recv().expect("wave admission"), 1); let second_before_release = run.entered.try_recv(); assert!( @@ -692,7 +676,7 @@ fn held_store_cap_one_serializes_calls_while_its_claim_lease_stays_live() { first.expect("checked first call").release(); let second = run .entered - .recv_timeout(Duration::from_secs(10)) + .recv() .expect("held Store enters the second call after the first returns"); second.release(); assert!( @@ -708,8 +692,8 @@ fn held_store_cap_one_serializes_calls_while_its_claim_lease_stays_live() { fn claim_boundary_resolver_is_called_and_its_worker_cap_is_honored() { let resolutions = Arc::new(AtomicUsize::new(0)); let run = start_projection(NonZeroUsize::MIN, Some(Arc::clone(&resolutions))); - let first = run.entered.recv_timeout(Duration::from_secs(10)).ok(); - let second = run.entered.recv_timeout(Duration::from_secs(10)); + let first = run.entered.recv().ok(); + let second = run.entered.recv(); let overlap_observed = second.is_ok(); let mut observed = first.into_iter().collect::>(); if let Ok(second) = second { @@ -744,7 +728,7 @@ fn standalone_runner_resolves_the_worker_cap_again_for_later_claims() { ); let first = run .entered - .recv_timeout(Duration::from_secs(10)) + .recv() .expect("fixture enters its first embedding call"); let (outcome, held_observed) = finish_projection(run, vec![first]); @@ -774,14 +758,12 @@ fn observe_two_passage_batch( expect_overlap: bool, observed: &mut Vec, ) -> BatchObservation { - let first = entered.recv_timeout(Duration::from_secs(10)).ok(); + let first = entered.recv().ok(); let first_entered = first.is_some(); - let width = waves - .recv_timeout(Duration::from_secs(10)) - .expect("complete wave admission"); + let width = waves.recv().expect("complete wave admission"); assert_eq!(width, if expect_overlap { 2 } else { 1 }); let second = if expect_overlap { - entered.recv_timeout(Duration::from_secs(10)).ok() + entered.recv().ok() } else { entered.try_recv().ok() }; @@ -800,17 +782,12 @@ fn observe_two_passage_batch( call.release(); } let second_after_release = if !overlapped { - entered.recv_timeout(Duration::from_secs(10)).ok() + entered.recv().ok() } else { None }; if second_after_release.is_some() { - assert_eq!( - waves - .recv_timeout(Duration::from_secs(10)) - .expect("serial second wave"), - 1 - ); + assert_eq!(waves.recv().expect("serial second wave"), 1); } let second_entered = overlapped || second_after_release.is_some(); if let Some(second) = second_after_release { @@ -853,7 +830,7 @@ fn changing_cap_run( let mut held = Vec::new(); assert_eq!( claim_boundaries - .recv_timeout(Duration::from_secs(10)) + .recv() .expect("first actual claim reaches worker-cap callback"), expected_caps[0], ); @@ -865,7 +842,7 @@ fn changing_cap_run( )); assert_eq!( claim_boundaries - .recv_timeout(Duration::from_secs(10)) + .recv() .expect("second actual claim reaches worker-cap callback"), expected_caps[1], ); @@ -931,14 +908,14 @@ fn standalone_runner_honors_two_then_one_at_successive_two_passage_claims() { let mut held = Vec::new(); assert_eq!( claim_boundaries - .recv_timeout(Duration::from_secs(10)) + .recv() .expect("first actual claim reaches worker-cap callback"), 2, ); let first = observe_two_passage_batch(&run.entered, &run.waves, true, &mut held); assert_eq!( claim_boundaries - .recv_timeout(Duration::from_secs(10)) + .recv() .expect("second actual claim reaches worker-cap callback"), 1, ); @@ -993,20 +970,15 @@ fn claim_callback_rereads_the_worker_policy_file_before_the_next_two_passage_bat true, ); - let first = run.entered.recv_timeout(Duration::from_secs(10)).ok(); - assert_eq!( - run.waves - .recv_timeout(Duration::from_secs(10)) - .expect("first wave admission"), - 1 - ); + let first = run.entered.recv().ok(); + assert_eq!(run.waves.recv().expect("first wave admission"), 1); let first_overlap = run.entered.try_recv().ok(); let first_overlapped = first_overlap.is_some(); if let Some(call) = first { call.release(); } let first_second = if first_overlap.is_none() { - run.entered.recv_timeout(Duration::from_secs(10)).ok() + run.entered.recv().ok() } else { first_overlap }; @@ -1016,8 +988,8 @@ fn claim_callback_rereads_the_worker_policy_file_before_the_next_two_passage_bat if let Some(call) = first_second { call.release(); } - let second_first = run.entered.recv_timeout(Duration::from_secs(10)).ok(); - let second_second = run.entered.recv_timeout(Duration::from_secs(10)).ok(); + let second_first = run.entered.recv().ok(); + let second_second = run.entered.recv().ok(); let second_first_entered = second_first.is_some(); let second_overlapped = second_second.is_some(); if let Some(call) = second_first { diff --git a/crates/context-graph/tests/event_search_projection_correctness.rs b/crates/context-graph/tests/event_search_projection_correctness.rs index 5948d70..229604a 100644 --- a/crates/context-graph/tests/event_search_projection_correctness.rs +++ b/crates/context-graph/tests/event_search_projection_correctness.rs @@ -87,21 +87,68 @@ fn slow_projection_work_does_not_keep_a_source_read_statement_open() { .expect("append source events"); drop(store); - let (keep_open, receiver) = channel::<()>(); - let receiver = Mutex::new(receiver); - let wait: Observer = Arc::new(move || { - let _ = &keep_open; - let _ = receiver - .lock() - .expect("wait receiver") - .recv_timeout(std::time::Duration::from_millis(700)); + use context_graph::test_support::set_event_search_projection_read_clock_for_test; + use contextdb_core::read_contract::{DeadlineClock, DeadlineWait}; + struct ReadClock { + now: AtomicU64, + reads: AtomicU64, + waiters: Mutex>, + } + impl ReadClock { + fn advance(&self) { + self.now.fetch_add(700, Ordering::SeqCst); + let waiters = std::mem::take(&mut *self.waiters.lock().expect("deadline waiters")); + for waiter in waiters { + waiter.wake(); + } + } + } + impl DeadlineClock for ReadClock { + fn now_ms(&self) -> u64 { + self.reads.fetch_add(1, Ordering::SeqCst); + self.now.load(Ordering::SeqCst) + } + fn wait_until(&self, deadline: u64) -> DeadlineWait<'_> { + Box::pin(std::future::poll_fn(move |context| { + let mut waiters = self.waiters.lock().expect("deadline waiters"); + if self.now_ms() >= deadline { + std::task::Poll::Ready(()) + } else { + waiters.push(context.waker().clone()); + std::task::Poll::Pending + } + })) + } + } + struct Reset; + impl Drop for Reset { + fn drop(&mut self) { + set_event_search_projection_read_clock_for_test(None); + set_event_search_projection_read_deadline_ms_for_test(None); + } + } + let clock = Arc::new(ReadClock { + now: AtomicU64::new(0), + reads: AtomicU64::new(0), + waiters: Mutex::new(Vec::new()), }); + let outside_read = Arc::clone(&clock); + let advance: Observer = Arc::new(move || outside_read.advance()); let embedder = ControlledEmbedder::new(); - *embedder.embed_observer.lock().expect("observer lock") = Some(wait.clone()); - after_next_event_search_membership_read_page_for_test(move || wait()); + *embedder.embed_observer.lock().expect("observer lock") = Some(advance.clone()); + after_next_event_search_membership_read_page_for_test(move || advance()); + set_event_search_projection_read_clock_for_test(Some(clock.clone())); set_event_search_projection_read_deadline_ms_for_test(Some(500)); + let _reset = Reset; let result = run(&config, context_id, &embedder, 2, OBSERVED_LEASE_MS); - set_event_search_projection_read_deadline_ms_for_test(None); + assert!( + clock.reads.load(Ordering::SeqCst) > 0, + "real bounded source reads consumed the substrate deadline clock" + ); + assert!( + clock.now.load(Ordering::SeqCst) >= 1_400, + "both membership work and inference advanced time beyond a read budget" + ); let receipt = result.expect("bounded source pages survive slow work and slow embeddings"); assert_eq!(receipt.embedded_event_count, 6); assert!( @@ -3306,26 +3353,34 @@ fn one_concurrent_failure_waits_for_its_held_sibling_commits_nothing_and_retries let embedder = ControlledEmbedder::new(); *embedder.fail_once_on.lock().expect("failure control") = Some('b'); let (entered_tx, entered_rx) = channel(); + let entered_order = Arc::new(Mutex::new(entered_tx)); let (release_tx, release_rx) = channel(); let release_rx = Arc::new(Mutex::new(release_rx)); let call_index = Arc::new(AtomicUsize::new(0)); let held_active = Arc::new(AtomicBool::new(false)); { + let entered_order = Arc::clone(&entered_order); let release_rx = Arc::clone(&release_rx); let call_index = Arc::clone(&call_index); let held_active = Arc::clone(&held_active); *embedder.embed_observer.lock().expect("observer lock") = Some(Arc::new(move || { - let ordinal = call_index.fetch_add(1, Ordering::SeqCst); - if ordinal == 0 { - held_active.store(true, Ordering::SeqCst); - } - let _ = entered_tx.send(ordinal); + // Take the ordinal and announce it under one lock so the channel order is the + // ordinal order; two workers racing between the two steps made 1 arrive before 0. + let ordinal = { + let announce = entered_order.lock().expect("entry order lock"); + let ordinal = call_index.fetch_add(1, Ordering::SeqCst); + if ordinal == 0 { + held_active.store(true, Ordering::SeqCst); + } + let _ = announce.send(ordinal); + ordinal + }; if ordinal == 0 { - let _ = release_rx.lock().ok().and_then(|receiver| { - receiver - .recv_timeout(std::time::Duration::from_secs(30)) - .ok() - }); + release_rx + .lock() + .expect("release receiver") + .recv() + .expect("explicit sibling release"); held_active.store(false, Ordering::SeqCst); } })); @@ -3356,12 +3411,8 @@ fn one_concurrent_failure_waits_for_its_held_sibling_commits_nothing_and_retries outcome }); - let first = entered_rx - .recv_timeout(std::time::Duration::from_secs(10)) - .ok(); - let second = entered_rx - .recv_timeout(std::time::Duration::from_secs(10)) - .ok(); + let first = entered_rx.recv().ok(); + let second = entered_rx.recv().ok(); let returned_while_held = finished_rx.try_recv().is_ok(); let _ = release_tx.send(()); let outcome = runner.join(); @@ -4418,9 +4469,7 @@ impl HeldWriter { } }); let held = Self(child); - observed - .recv_timeout(std::time::Duration::from_secs(10)) - .expect("real writer owns Store"); + observed.recv().expect("real writer owns Store"); held } fn release(mut self) { @@ -4488,7 +4537,9 @@ fn projection_retries_real_writer_at_claim_renewal_and_commit_without_reembeddin *held.lock().unwrap() = Some(HeldWriter::start(&path)); if phase == "renewal" { clock.fetch_add(LEASE_MS / 2, Ordering::SeqCst); - embed_released.lock().unwrap().recv_timeout(std::time::Duration::from_secs(10)).expect("lease renewal met and released the writer while inference remained held"); + embed_released.lock().unwrap().recv().expect( + "lease renewal met and released the writer while inference remained held", + ); } })); } diff --git a/crates/context-graph/tests/event_search_read_during_projection.rs b/crates/context-graph/tests/event_search_read_during_projection.rs index 5fe219c..bd537e0 100644 --- a/crates/context-graph/tests/event_search_read_during_projection.rs +++ b/crates/context-graph/tests/event_search_read_during_projection.rs @@ -508,31 +508,33 @@ fn consumer_cursor_fetches_unread_passage_versions_across_currency_maintenance() const REOPEN_CHILD_ENV: &str = "CG_READ_DURING_PROJECTION_REOPEN_CHILD"; const REOPEN_TEST: &str = "readers_are_never_refused_while_another_process_reopens_the_store"; -/// How long the child keeps reopening the Store. Bounded so the reads it overlaps stay a handful -/// and the child's own memory stays small. -const REOPEN_WINDOW: std::time::Duration = std::time::Duration::from_secs(3); - /// How many read attempts the parent makes, each through one reader that is dropped before the /// next attempt opens another. const READ_ATTEMPTS: usize = 50; /// The child half of the cross-process test: open and close a plain writable Store on the named /// path in a tight loop, as the detached worker does between claim, render/embed and commit, until -/// the parent has finished its reads or the reopen window closes. +/// the parent explicitly reports completion of its reads. fn run_reopening_child(spec: &str) { - let mut parts = spec.split('\n'); - let (Some(path), Some(started), Some(stop)) = (parts.next(), parts.next(), parts.next()) else { - panic!("child spec is path, start marker and stop marker"); - }; - let path = Path::new(path); - let stop = Path::new(stop); - let cap = std::time::Instant::now() + REOPEN_WINDOW; + use std::io::{Read, Write}; + let (path, socket) = spec + .split_once('\n') + .expect("child spec is store path and control socket"); + let mut control = std::os::unix::net::UnixStream::connect(socket).expect("parent control"); + control + .set_nonblocking(true) + .expect("nonblocking stop check between opens"); let mut opened = 0_u64; - while !stop.exists() && std::time::Instant::now() < cap { - // A writable open may be refused while a reader is hydrating; the worker retries that. + loop { + let mut stop = [0]; + match control.read(&mut stop) { + Ok(_) => break, + Err(error) if error.kind() == std::io::ErrorKind::WouldBlock => {} + Err(error) => panic!("parent control: {error}"), + } if let Ok(store) = Store::open_with_embedder_registrations( StoreConfig { - db_path: path.to_path_buf(), + db_path: Path::new(path).to_path_buf(), extra_embedding_spaces: vec![FixtureEmbedder.embedding_space()], default_text_embedder: None, ..StoreConfig::default() @@ -540,10 +542,12 @@ fn run_reopening_child(spec: &str) { vec![FixtureEmbedder::registration()], ) { opened += 1; - if opened == 1 { - std::fs::write(started, b"started").expect("write the start marker"); - } drop(store); + if opened == 5 { + control + .write_all(&[1]) + .expect("announce five completed reopens"); + } } } println!("child opened the writable Store {opened} times"); @@ -562,27 +566,27 @@ fn readers_are_never_refused_while_another_process_reopens_the_store() { let path = root.path().join("reader-across-processes.db"); let (worker, context_id) = ready_store_with_pending_event(&path); drop(worker); - let started = root.path().join("child-started"); - let stop = root.path().join("parent-finished"); + use std::io::{Read, Write}; + let socket_dir = tempfile::tempdir_in("/tmp").expect("short control socket directory"); + let socket = socket_dir.path().join("control.sock"); + let listener = std::os::unix::net::UnixListener::bind(&socket).expect("control listener"); let child = process::Command::new(std::env::current_exe().expect("test binary path")) .args(["--exact", REOPEN_TEST, "--nocapture", "--test-threads", "1"]) .env( REOPEN_CHILD_ENV, - format!( - "{}\n{}\n{}", - path.display(), - started.display(), - stop.display() - ), + format!("{}\n{}", path.display(), socket.display()), ) .stdout(process::Stdio::piped()) .stderr(process::Stdio::piped()) .spawn() .expect("spawn the reopening child"); - while !started.exists() { - std::thread::yield_now(); - } + let (mut control, _) = listener.accept().expect("reopening child connected"); + let mut ready = [0]; + control + .read_exact(&mut ready) + .expect("child completed five reopens"); + assert_eq!(ready, [1]); let mut refusals = Vec::new(); for attempt in 0..READ_ATTEMPTS { @@ -601,7 +605,9 @@ fn readers_are_never_refused_while_another_process_reopens_the_store() { refusals.push(format!("attempt {attempt}: {refusal}")); } } - std::fs::write(&stop, b"finished").expect("write the stop marker"); + control + .write_all(&[1]) + .expect("stop the reopening child after all reads"); let output = child.wait_with_output().expect("the child finishes"); let child_report = format!( "{}{}", diff --git a/crates/context-graph/tests/fixtures/test_estate_baseline.toml b/crates/context-graph/tests/fixtures/test_estate_baseline.toml index a132284..edec2d3 100644 --- a/crates/context-graph/tests/fixtures/test_estate_baseline.toml +++ b/crates/context-graph/tests/fixtures/test_estate_baseline.toml @@ -15,13 +15,13 @@ "crates/candle-embeddinggemma/src/embeddinggemma.rs" = 1 "crates/cg-catalog/src/lib.rs" = 2 "crates/cg-catalog/src/model_connection.rs" = 5 -"crates/cg-catalog/tests/acceptance/concurrency.rs" = 8 + "crates/cg-catalog/tests/acceptance/generic_llm_endpoint.rs" = 1 "crates/cg-catalog/tests/acceptance/license_gate.rs" = 1 "crates/cg-catalog/tests/acceptance/prefetch_first_touch.rs" = 3 -"crates/cg-catalog/tests/acceptance/swap.rs" = 6 -"crates/cg-catalog/tests/acceptance/sync_arrival.rs" = 3 -"crates/cg-catalog/tests/acceptance/update_check.rs" = 3 +"crates/cg-catalog/tests/acceptance/swap.rs" = 4 + +"crates/cg-catalog/tests/acceptance/update_check.rs" = 1 # Event-history production wait and timing sites. # None synchronizes a test or persists a timestamp: # history/import.rs:133 Instant::now — the import's elapsed time reported in its receipt @@ -34,19 +34,19 @@ # history/worker.rs:273 thread::sleep — 200 ms backoff while the worker lock is held by another worker "crates/cg-cli/src/history/import.rs" = 3 "crates/cg-cli/src/history/worker.rs" = 5 -"crates/cg-cli/src/main.rs" = 12 +"crates/cg-cli/src/main.rs" = 10 # Bounded poll of a separate process's observable state with a deadline (bounded process observation): # :995 thread::sleep — 250 ms pause between reads of `cg status` while a background search build # runs, bounded by the outer `wait_for_search_ready` deadline the caller already holds -"crates/cg-cli/tests/event_history.rs" = 1 + "crates/cg-cli/tests/cli_acceptance.rs" = 2 "crates/cg-cli/tests/cli_model_byok.rs" = 1 # Shared bounded state observation used by prompt, readiness, and lock waits: # :47/:57 Instant::now, sleep — one operation owns its observation budget and capped pause -"crates/cg-cli/tests/event_support/mod.rs" = 2 -"crates/cg-cli/tests/command_workflows_cli.rs" = 2 -"crates/cg-cli/tests/support/mod.rs" = 5 -"crates/context-graph/src/capture_sync_status.rs" = 4 + + +"crates/cg-cli/tests/support/mod.rs" = 1 +"crates/context-graph/src/capture_sync_status.rs" = 3 # consumer_read.rs, the read-only reader's bounded wait across a writer opening or closing the store: # :125 Instant::now — the writer-transition deadline (WRITER_TRANSITION_ALLOWANCE) # :141 Instant::now — start of one reopen attempt @@ -55,7 +55,7 @@ # :527 Instant::now — start of one session call, to tell an owner hand-over from a slow answer # :531 Instant::now — deadline check before replacing the session "crates/context-graph/src/consumer_read.rs" = 6 -"crates/context-graph/src/embedder/nomic.rs" = 2 +"crates/context-graph/src/embedder/nomic.rs" = 1 "crates/context-graph/src/ingest.rs" = 6 # owner_control.rs, the bounded reader-release wait every cg writer uses: # :374 Instant::now — remaining time left before the deadline, checked before waiting at all @@ -70,11 +70,11 @@ "crates/context-graph/tests/a_closed_cursor_answers_its_first_page_by_type_not_by_panic.rs" = 2 "crates/context-graph/tests/a_graph_read_past_the_read_ceiling_answers_what_the_writable_store_answers.rs" = 2 "crates/context-graph/tests/a_stopped_store_answers_the_typed_graph_reads_without_changing.rs" = 6 -"crates/context-graph/tests/acceptance/capture_progress.rs" = 6 + "crates/context-graph/tests/acceptance/context_isolation.rs" = 2 "crates/context-graph/tests/acceptance/crud.rs" = 3 "crates/context-graph/tests/acceptance/discovery.rs" = 3 -"crates/context-graph/tests/acceptance/evidence.rs" = 8 +"crates/context-graph/tests/acceptance/evidence.rs" = 4 "crates/context-graph/tests/acceptance/find_via_digests.rs" = 2 "crates/context-graph/tests/acceptance/full_ontology_extraction.rs" = 2 "crates/context-graph/tests/acceptance/graph_context_isolation.rs" = 1 @@ -90,7 +90,7 @@ "crates/context-graph/tests/acceptance/entity_alert_render.rs" = 1 "crates/context-graph/tests/acceptance/screenshot_memory_distinguishability.rs" = 1 "crates/context-graph/tests/acceptance/screenshot_memory_review.rs" = 3 -"crates/context-graph/tests/acceptance/subscriptions.rs" = 13 +"crates/context-graph/tests/acceptance/subscriptions.rs" = 6 "crates/context-graph/tests/acceptance/temporal.rs" = 1 "crates/context-graph/tests/acceptance/tier0_capture.rs" = 2 "crates/context-graph/tests/common/capture_sync_common.rs" = 2 @@ -98,7 +98,7 @@ "crates/context-graph/tests/compile-fail/fabric_receipt_not_basis.rs" = 1 # Bounded poll of a separate process's observable state with a deadline (bounded process observation): # :333/:335 Instant::now ×2 — the reopening child process loops until the parent's stop file appears or REOPEN_WINDOW ends -"crates/context-graph/tests/event_search_read_during_projection.rs" = 2 + # waiting_for_readers_never_outlasts_the_allowance: building the deadlines the bounded wait takes # (its API is an Instant deadline); nothing measures elapsed time, sleeps, or waits on a lease: # :800 Instant::now — a deadline already spent, to assert an immediate answer @@ -120,10 +120,10 @@ "crates/context-graph/tests/integration/precomputed_embedding_ingest.rs" = 1 "crates/context-graph/tests/integration/producer_diagnostic.rs" = 4 "crates/context-graph/tests/integration/reindex.rs" = 4 -"crates/context-graph/tests/integration/reindex_concurrency.rs" = 12 +"crates/context-graph/tests/integration/reindex_concurrency.rs" = 8 "crates/context-graph/tests/integration/schema.rs" = 1 "crates/context-graph/tests/integration/store_open_runtime_config.rs" = 8 -"crates/context-graph/tests/integration/subscriptions_engine_capabilities.rs" = 10 +"crates/context-graph/tests/integration/subscriptions_engine_capabilities.rs" = 4 "crates/context-graph/tests/one_writable_open_carries_the_owner_handler_and_the_embedders.rs" = 1 "crates/context-graph/tests/ui/byok_reply_phrasing_is_not_a_basis.rs" = 1 "crates/context-graph/tests/ui/no_observation_data_field.rs" = 1 @@ -131,6 +131,24 @@ "crates/context-graph/tests/ui/usage_is_not_a_basis_input.rs" = 1 "crates/context-graph/tests/unit/mock_embedder.rs" = 1 +# Imported thread sleeps and indirect timed waits, including private test modules. +# This arm closes spelling gaps in the original raw-clock arm; it may only lower. +[indirect_timed_wait_sites] +"crates/cg-catalog/src/model_connection.rs" = 4 +"crates/cg-catalog/tests/acceptance/swap.rs" = 1 +"crates/cg-cli/src/history/worker.rs" = 3 +"crates/cg-cli/src/main.rs" = 2 +"crates/context-graph/src/consumer_read.rs" = 1 +"crates/context-graph/src/owner_control.rs" = 1 +"crates/context-graph/src/store/event_search_projection.rs" = 1 +"crates/context-graph/src/store/open_retry.rs" = 1 +"crates/context-graph/tests/acceptance/subscriptions.rs" = 2 +"crates/context-graph/tests/event_search_live_throughput.rs" = 1 +"crates/context-graph/tests/integration/cluster_aggregate_substrate.rs" = 1 +"crates/context-graph/tests/integration/engine_probes/embedder_concurrency.rs" = 4 +"crates/context-graph/tests/integration/reindex_concurrency.rs" = 1 +"crates/context-graph/tests/integration/subscriptions_engine_capabilities.rs" = 2 + [default_tier_test_count] # crate (Cargo.toml package name) -> the exact number of #[test]/#[tokio::test] functions in the # default tier: excludes anything gated behind #[cfg(context_graph_internal_test_support)] and @@ -149,8 +167,8 @@ "cg-node-fs" = 3 "cg-vision-embedder" = 5 # Tests in test_estate_audit.rs are excluded from the default-tier inventory. -"context-graph" = 1338 -"context-graph-cli" = 279 +"context-graph" = 1340 +"context-graph-cli" = 280 [gate_wall_seconds_policy] # A test binary's ceiling is `recorded seconds * tolerance_factor + slack_seconds`. Both terms are diff --git a/crates/context-graph/tests/integration/concurrent_probes_evidence.rs b/crates/context-graph/tests/integration/concurrent_probes_evidence.rs index 71e25d4..f354d6f 100644 --- a/crates/context-graph/tests/integration/concurrent_probes_evidence.rs +++ b/crates/context-graph/tests/integration/concurrent_probes_evidence.rs @@ -704,7 +704,7 @@ fn probe_basis_diff_under_concurrent_writer_returns_mvcc_consistent_horizon() { let after = reader_committed.load(Ordering::SeqCst); diff_tx.send((before, after, diff)).expect("send diff"); }); - let received = diff_rx.recv_timeout(std::time::Duration::from_secs(30)); + let received = diff_rx.recv(); stop.store(true, Ordering::SeqCst); for writer in writers { writer.join().expect("writer joined"); diff --git a/crates/context-graph/tests/integration/engine_probes/embedder_concurrency.rs b/crates/context-graph/tests/integration/engine_probes/embedder_concurrency.rs index 66b45b7..f81e3bc 100644 --- a/crates/context-graph/tests/integration/engine_probes/embedder_concurrency.rs +++ b/crates/context-graph/tests/integration/engine_probes/embedder_concurrency.rs @@ -2,7 +2,7 @@ use std::collections::{BTreeMap, BTreeSet, HashMap}; use std::path::PathBuf; use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; use std::sync::mpsc::{self, TryRecvError}; -use std::sync::{Arc, Barrier, Mutex}; +use std::sync::{Arc, Barrier, Condvar, Mutex}; use std::thread; use std::time::{Duration, Instant}; @@ -43,6 +43,7 @@ struct StagedDuplicateHandleProbe { observed: Mutex>, evidence_insert_queries: AtomicUsize, handle_insert_queries: AtomicUsize, + handle_queries_changed: (Mutex<()>, Condvar), first_full_precommit_seen: AtomicBool, first_full_precommit_open: AtomicBool, concurrent_writer_reached_handle_sql: AtomicBool, @@ -273,14 +274,12 @@ impl StagedDuplicateHandleProbe { .is_ok() { self.first_full_precommit_open.store(true, Ordering::SeqCst); - for _ in 0..200 { - if self.handle_insert_queries.load(Ordering::SeqCst) >= 2 { - self.concurrent_writer_reached_handle_sql - .store(true, Ordering::SeqCst); - break; - } - thread::sleep(Duration::from_millis(1)); - } + // Both real handle statements have reached on_query before either + // may commit; never wait while holding the engine's pre-commit fences. + self.concurrent_writer_reached_handle_sql.store( + self.handle_insert_queries.load(Ordering::SeqCst) >= 2, + Ordering::SeqCst, + ); self.first_full_precommit_open .store(false, Ordering::SeqCst); } @@ -298,11 +297,26 @@ impl DatabasePlugin for StagedDuplicateHandleProbe { self.evidence_insert_queries.fetch_add(1, Ordering::SeqCst); } if sql.starts_with("INSERT INTO evidence_embedding_handles ") { + let mut guard = self + .handle_queries_changed + .0 + .lock() + .expect("handle query gate"); let count = self.handle_insert_queries.fetch_add(1, Ordering::SeqCst) + 1; if count >= 2 && self.first_full_precommit_open.load(Ordering::SeqCst) { self.concurrent_writer_reached_handle_sql .store(true, Ordering::SeqCst); } + self.handle_queries_changed.1.notify_all(); + // Park before executing either handle INSERT, so the second writer + // cannot need a read fence already held by the first pre_commit. + while self.handle_insert_queries.load(Ordering::SeqCst) < 2 { + guard = self + .handle_queries_changed + .1 + .wait(guard) + .expect("both real handle SQL statements admitted"); + } } Ok(()) } diff --git a/crates/context-graph/tests/integration/reindex_concurrency.rs b/crates/context-graph/tests/integration/reindex_concurrency.rs index 5b4130f..e751d76 100644 --- a/crates/context-graph/tests/integration/reindex_concurrency.rs +++ b/crates/context-graph/tests/integration/reindex_concurrency.rs @@ -40,44 +40,20 @@ struct BlockingReindexState { impl BlockingReindexControl { fn wait_until_calls_at_least(&self, expected_calls: usize) -> bool { - let deadline = Instant::now() + Duration::from_millis(500); - let mut state = self.state.lock().expect("blocking state"); - while state.calls < expected_calls { - let now = Instant::now(); - if now >= deadline { - return false; - } - let timeout = deadline.saturating_duration_since(now); - let (next_state, wait) = self - .changed - .wait_timeout(state, timeout) - .expect("call-count wait"); - state = next_state; - if wait.timed_out() && state.calls < expected_calls { - return false; - } - } + let guard = self.state.lock().expect("blocking state"); + let _guard = self + .changed + .wait_while(guard, |state| state.calls < expected_calls) + .expect("call-count event"); true } fn wait_until_blocked_on_second_row(&self) -> bool { - let deadline = Instant::now() + Duration::from_millis(500); - let mut state = self.state.lock().expect("blocking state"); - while !state.blocked_on_second_row { - let now = Instant::now(); - if now >= deadline { - return false; - } - let timeout = deadline.saturating_duration_since(now); - let (next_state, wait) = self - .changed - .wait_timeout(state, timeout) - .expect("blocking wait"); - state = next_state; - if wait.timed_out() && !state.blocked_on_second_row { - return false; - } - } + let guard = self.state.lock().expect("blocking state"); + let _guard = self + .changed + .wait_while(guard, |state| !state.blocked_on_second_row) + .expect("second row event"); true } diff --git a/crates/context-graph/tests/integration/schema.rs b/crates/context-graph/tests/integration/schema.rs index d1a1ed0..f46bebd 100644 --- a/crates/context-graph/tests/integration/schema.rs +++ b/crates/context-graph/tests/integration/schema.rs @@ -1,7 +1,6 @@ use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet}; use std::sync::mpsc; use std::sync::{Arc, Barrier}; -use std::time::Duration; use context_graph::schema::{DIGEST_VECTOR_DIMENSION, TABLES}; use contextdb_core::{Error, TxId, Value}; @@ -3144,7 +3143,7 @@ fn combination_invalidate_decision_concurrent_atomic_one_winner_count_invalidati let ready = (0..WRITER_COUNT) .map(|_| { ready_rx - .recv_timeout(Duration::from_secs(30)) + .recv() .expect("all invalidation writers report readiness") }) .collect::>(); diff --git a/crates/context-graph/tests/integration/subscriptions_engine_capabilities.rs b/crates/context-graph/tests/integration/subscriptions_engine_capabilities.rs index 80a94ff..89c90b2 100644 --- a/crates/context-graph/tests/integration/subscriptions_engine_capabilities.rs +++ b/crates/context-graph/tests/integration/subscriptions_engine_capabilities.rs @@ -1,6 +1,6 @@ use std::collections::{BTreeSet, HashMap}; use std::sync::{ - Arc, Mutex, + Arc, Condvar, Mutex, atomic::{AtomicU64, Ordering}, }; use std::time::{Duration, Instant}; @@ -47,18 +47,20 @@ fn insert_invalidation(db: &Database, context_id: Uuid, severity: &str, reason: .expect("insert invalidation"); } -fn wait_for_count(counter: &AtomicU64, expected: u64) { - let deadline = Instant::now() + Duration::from_secs(5); - while Instant::now() < deadline && counter.load(Ordering::SeqCst) < expected { - std::thread::sleep(Duration::from_millis(20)); - } +fn wait_for_count(counter: &(Mutex, Condvar), expected: u64) { + let guard = counter.0.lock().expect("delivery counter"); + let _guard = counter + .1 + .wait_while(guard, |count| *count < expected) + .expect("delivery event"); } -fn wait_for_log(log: &Arc>>, expected: usize) { - let deadline = Instant::now() + Duration::from_secs(3); - while Instant::now() < deadline && log.lock().expect("sink log").len() < expected { - std::thread::sleep(Duration::from_millis(20)); - } +fn wait_for_log(log: &Arc<(Mutex>, Condvar)>, expected: usize) { + let guard = log.0.lock().expect("sink log"); + let _guard = log + .1 + .wait_while(guard, |events| events.len() < expected) + .expect("sink event"); } fn wait_for_metric(db: &Database, sink: &str, predicate: impl Fn(u64, u64, u64, u64) -> bool) { @@ -92,16 +94,17 @@ fn engine_sink_queue_persists_across_reopen_subscriber_re_registers_resumes_deli } let db = Database::open(&path).expect("reopen database"); - let log: Arc>> = Arc::new(Mutex::new(Vec::new())); + let log = Arc::new((Mutex::new(Vec::new()), Condvar::new())); let seen = Arc::clone(&log); db.register_sink("slack", None, move |event| { - seen.lock().expect("sink log").push(event.clone()); + seen.0.lock().expect("sink log").push(event.clone()); + seen.1.notify_all(); Ok(()) }) .expect("register sink"); wait_for_log(&log, 1); - let captured = log.lock().expect("sink log"); + let captured = log.0.lock().expect("sink log"); assert_eq!(captured.len(), 1); assert_eq!( captured[0].row_values.get("reason"), @@ -147,10 +150,11 @@ fn engine_per_sink_queue_isolation_one_sink_panic_does_not_block_other_sink_deli db.register_sink("slack", None, |_| panic!("expected sink panic")) .expect("register panic sink"); - let audit_count = Arc::new(AtomicU64::new(0)); + let audit_count = Arc::new((Mutex::new(0), Condvar::new())); let seen = Arc::clone(&audit_count); db.register_sink("audit", None, move |_| { - seen.fetch_add(1, Ordering::SeqCst); + *seen.0.lock().expect("delivery counter") += 1; + seen.1.notify_all(); Ok(()) }) .expect("register audit sink"); @@ -167,7 +171,7 @@ fn engine_per_sink_queue_isolation_one_sink_panic_does_not_block_other_sink_deli queued == 50 && retried > 0 && permanent == 0 }); - assert_eq!(audit_count.load(Ordering::SeqCst), 50); + assert_eq!(*audit_count.0.lock().expect("delivery counter"), 50); let audit_metrics = db.sink_metrics_for_test("audit"); assert_eq!(audit_metrics.delivered, 50); assert_eq!(audit_metrics.permanent_failures, 0); @@ -188,11 +192,12 @@ fn cross_context_isolation_enforced_at_engine_route_not_at_typed_wrapper() { declare_event_schema(&db); let scoped = db.scoped_with_contexts(BTreeSet::from([ContextId::new(ctx_a)])); - let log: Arc>> = Arc::new(Mutex::new(Vec::new())); + let log = Arc::new((Mutex::new(Vec::new()), Condvar::new())); let seen = Arc::clone(&log); scoped .register_sink("slack", None, move |event| { - seen.lock().expect("sink log").push(event.clone()); + seen.0.lock().expect("sink log").push(event.clone()); + seen.1.notify_all(); Ok(()) }) .expect("register scoped sink"); @@ -201,7 +206,7 @@ fn cross_context_isolation_enforced_at_engine_route_not_at_typed_wrapper() { insert_invalidation(&db, ctx_b, "warning", "foreign"); wait_for_log(&log, 1); - let captured = log.lock().expect("sink log"); + let captured = log.0.lock().expect("sink log"); assert_eq!( captured.len(), 1, diff --git a/crates/context-graph/tests/interpretation_claim_concurrency.rs b/crates/context-graph/tests/interpretation_claim_concurrency.rs index 07341ce..7c045d5 100644 --- a/crates/context-graph/tests/interpretation_claim_concurrency.rs +++ b/crates/context-graph/tests/interpretation_claim_concurrency.rs @@ -12,7 +12,6 @@ use std::path::PathBuf; use std::sync::mpsc::{Receiver, Sender, channel}; use std::sync::{Arc, Mutex}; use std::thread; -use std::time::Duration; use context_graph::{ AppendEventBatch, CapturedEventAndAnchors, CgError, ConnectorInterpretationRunner, @@ -25,7 +24,6 @@ use serde_json::json; const CONNECTOR: &str = "interpretation-claim-concurrency-red"; const TARGET: &str = "claim-concurrency-v1"; -const CHANNEL_TIMEOUT: Duration = Duration::from_secs(3); fn config(path: PathBuf) -> StoreConfig { StoreConfig { @@ -300,7 +298,7 @@ impl ReleasedConnectorInterpretation for FirstRunnerMapper { self.resume .lock() .expect("A resume lock") - .recv_timeout(CHANNEL_TIMEOUT) + .recv() .expect("test releases A after B has reclaimed"); Err(CgError::Engine( "late A mapper error after B reclaim".into(), @@ -357,7 +355,7 @@ impl ReleasedConnectorInterpretation for SecondRunnerMapper { self.resume .lock() .expect("B resume lock") - .recv_timeout(CHANNEL_TIMEOUT) + .recv() .expect("test releases B to commit"); Ok(effective()) } @@ -382,7 +380,7 @@ fn late_a_error_preserves_b_claim(b_completes_before_a_error: bool) { ConnectorInterpretationRunner::build(request(a_path, context_id, first)) }); let first_claim = a_paused_rx - .recv_timeout(CHANNEL_TIMEOUT) + .recv() .expect("A paused after expiring its real lease with test SQL"); let (b_paused_tx, b_paused_rx) = channel(); @@ -399,7 +397,7 @@ fn late_a_error_preserves_b_claim(b_completes_before_a_error: bool) { ConnectorInterpretationRunner::build(request(b_path, context_id, second)) })); let b_claim = b_paused_rx - .recv_timeout(CHANNEL_TIMEOUT) + .recv() .expect("B reclaimed and paused on the renewed real lease"); if b_completes_before_a_error { diff --git a/crates/context-graph/tests/test_estate_audit.rs b/crates/context-graph/tests/test_estate_audit.rs index 4ea6638..0c5d7a7 100644 --- a/crates/context-graph/tests/test_estate_audit.rs +++ b/crates/context-graph/tests/test_estate_audit.rs @@ -18,9 +18,15 @@ //! `Utc::now(` call sites anywhere under `crates/*/src` or `crates/*/tests` — every one of //! them a persisted-timestamp or synchronization call that bypasses the seamed //! `contextdb_core::Wallclock::now()`. A file's live count rising past its baseline, or a new -//! file carrying any of these calls at all, fails the audit. The fix is the seam +//! file carrying any of these calls at all, fails the audit. Removed sites must also lower +//! their allowances in the same change. The fix is the seam //! (`Wallclock::test_clock_guard` for time, a counter/state/event wait for synchronization), //! never raising the number. +//! The `indirect_timed_wait_sites` subtable also freezes imported thread sleeps, +//! thread parking, and timed channel/condition-variable waits, including private modules. +//! Fully qualified sleeps overlap the original arm; both reject increases and stale +//! allowances. Both site arms include private modules; only the default-tier test count +//! excludes them. //! 2. **Default-tier test count per crate.** `[default_tier_test_count]` freezes, per crate, the //! exact number of `#[test]`/`#[tokio::test]` functions that run in the tier `cargo test` runs //! by default — excluding anything gated behind @@ -62,10 +68,9 @@ use std::path::{Path, PathBuf}; /// Test/src files that never compile in the default tier at all: each is reached only through a /// `mod` declaration in its crate's `tests/acceptance/main.rs` or `tests/integration/main.rs` -/// that itself carries `#[cfg(context_graph_internal_test_support)]`. Their contents — including -/// any `#[test]` functions and any sleep/clock call sites — are excluded from both counted arms -/// for the same reason: code that does not compile by default is not part of the default tier to -/// begin with, not a debt this audit is choosing to forgive. +/// that itself carries `#[cfg(context_graph_internal_test_support)]`. These files are excluded +/// only from default-tier test counts. Both timing-site arms include them, so private fixtures +/// cannot add synchronization debt outside the ratchet. const WHOLE_FILE_EXCLUDE: &[&str] = &[ "crates/context-graph/tests/acceptance/local_entity_face_match_override.rs", "crates/context-graph/tests/acceptance/screenshot_memory_distinguishability.rs", @@ -174,7 +179,35 @@ fn relative(root: &Path, path: &Path) -> String { // Arm 1: sleep and raw-clock sites // --------------------------------------------------------------------------------------------- +fn sleep_clock_site_count(content: &str) -> usize { + SLEEP_CLOCK_NEEDLES + .iter() + .map(|needle| content.matches(needle).count()) + .sum() +} + +fn indirect_wait_site_count(content: &str) -> usize { + // Include fully qualified sleeps here too; the two site arms overlap. + [ + "thread::sleep(", + "thread::park_timeout(", + ".wait_timeout(", + ".recv_timeout(", + ] + .iter() + .map(|needle| content.matches(needle).count()) + .sum() +} + fn count_sleep_clock_sites(root: &Path) -> BTreeMap { + count_sites(root, sleep_clock_site_count, true) +} + +fn count_sites( + root: &Path, + count: fn(&str) -> usize, + include_private: bool, +) -> BTreeMap { let mut counts = BTreeMap::new(); for (_, crate_dir) in crate_dirs(root) { for sub in ["src", "tests"] { @@ -182,16 +215,14 @@ fn count_sleep_clock_sites(root: &Path) -> BTreeMap { walk_rs_files(&crate_dir.join(sub), &mut files); for file in files { let rel = relative(root, &file); - if is_self(&rel) || WHOLE_FILE_EXCLUDE.contains(&rel.as_str()) { + if is_self(&rel) || (!include_private && WHOLE_FILE_EXCLUDE.contains(&rel.as_str())) + { continue; } let Ok(content) = std::fs::read_to_string(&file) else { continue; }; - let total: usize = SLEEP_CLOCK_NEEDLES - .iter() - .map(|needle| content.matches(needle).count()) - .sum(); + let total = count(&content); if total > 0 { counts.insert(rel, total); } @@ -201,32 +232,148 @@ fn count_sleep_clock_sites(root: &Path) -> BTreeMap { counts } -#[test] -fn no_new_sleep_or_raw_clock_sites_beyond_the_frozen_baseline() { - let root = repository_root(); - let baseline = load_baseline(&root); - let live = count_sleep_clock_sites(&root); +fn sleep_clock_violations( + live: &BTreeMap, + baseline: &BTreeMap, +) -> Vec { + site_count_violations(live, baseline, true) +} +fn site_count_violations( + live: &BTreeMap, + baseline: &BTreeMap, + include_private: bool, +) -> Vec { + let files: std::collections::BTreeSet<_> = live.keys().chain(baseline.keys()).collect(); let mut violations = Vec::new(); - for (file, count) in &live { - let allowed = baseline - .sleep_and_clock_sites - .get(file) - .copied() - .unwrap_or(0); - if *count > allowed { + for file in files { + if !include_private && WHOLE_FILE_EXCLUDE.contains(&file.as_str()) { + continue; + } + let count = live.get(file).copied().unwrap_or(0); + let allowed = baseline.get(file).copied().unwrap_or(0); + if count > allowed { violations.push(format!( "{file}: {count} sleep/raw-clock site(s), baseline allows {allowed} — route \ through Wallclock::test_clock_guard (timing) or a counter/state/event wait \ (synchronization), never raise the baseline" )); + } else if count < allowed { + violations.push(format!( + "{file}: {count} sleep/raw-clock site(s), stale baseline allows {allowed} — \ + lower the baseline in the same change that removes a site" + )); } } + violations +} + +#[test] +fn no_new_sleep_or_raw_clock_sites_beyond_the_frozen_baseline() { + for needle in SLEEP_CLOCK_NEEDLES { + assert_eq!(sleep_clock_site_count(&format!("{needle});")), 1); + } + // Mutation controls: a new site in an existing file, a new file carrying a + // site, and a removed site with an unlowered allowance must all be rejected. + let baseline = BTreeMap::from([("existing.rs".to_string(), 1)]); + assert!(sleep_clock_violations(&baseline, &baseline).is_empty()); + let increased = BTreeMap::from([("existing.rs".to_string(), 2)]); + assert_eq!(sleep_clock_violations(&increased, &baseline).len(), 1); + let mut added = baseline.clone(); + added.insert("new.rs".to_string(), 1); + assert_eq!(sleep_clock_violations(&added, &baseline).len(), 1); + assert_eq!(sleep_clock_violations(&BTreeMap::new(), &baseline).len(), 1); + + let root = repository_root(); + // These converted fixtures used indirect polling/timeouts, which a raw-clock + // call-site count cannot see. Keep their synchronization event-driven. + let mut policy_violations = Vec::new(); + for (file, function, forbidden) in [ + ( + "crates/cg-cli/tests/import_embedding_workers.rs", + "detached_background_import_delivers_default_cap_to_its_production_request_builder", + "wait_for_state(", + ), + ( + "crates/context-graph/tests/event_search_projection_correctness.rs", + "one_concurrent_failure_waits_for_its_held_sibling_commits_nothing_and_retries_cleanly", + "recv_timeout(", + ), + ( + "crates/cg-cli/tests/event_support/mod.rs", + "wait_for_search_worker_exit", + "wait_for_state(", + ), + ( + "crates/cg-cli/src/history/worker.rs", + "wait_for_test_release", + "park_timeout(", + ), + ( + "crates/context-graph/src/embedder/nomic.rs", + "pause_prepared_embedding_for_test", + "sleep(", + ), + ] { + let source = std::fs::read_to_string(root.join(file)).expect("guarded fixture source"); + let signature = format!("fn {function}("); + let body = source + .split_once(signature.as_str()) + .expect("guarded fixture is present") + .1 + .split_once("\n}\n") + .expect("guarded top-level fixture closes") + .0; + if body.contains(forbidden) { + policy_violations.push(format!( + "{function} must use explicit events, not the timing helper {forbidden}" + )); + } + } + for needle in [ + "thread::sleep(", + "thread::park_timeout(", + ".wait_timeout(", + ".recv_timeout(", + ] { + assert_eq!(indirect_wait_site_count(needle), 1); + } + assert_eq!(indirect_wait_site_count("std::thread::sleep("), 1); + let private = BTreeMap::from([(WHOLE_FILE_EXCLUDE[0].to_string(), 1)]); + assert_eq!(sleep_clock_violations(&private, &BTreeMap::new()).len(), 1); + assert_eq!(sleep_clock_violations(&BTreeMap::new(), &private).len(), 1); + assert_eq!( + site_count_violations(&private, &BTreeMap::new(), true).len(), + 1 + ); + assert_eq!( + site_count_violations(&BTreeMap::new(), &private, true).len(), + 1 + ); + let fake_provider = std::fs::read_to_string(root.join("crates/cg-cli/tests/support/mod.rs")) + .expect("fake provider source"); + if fake_provider.contains("time.sleep(") { + policy_violations.push("fake providers use explicit completion release or an indefinite held call, never a timed sleep".to_string()); + } + let baseline = load_baseline(&root); + let indirect = count_sites(&root, indirect_wait_site_count, true); + let indirect_violations = + site_count_violations(&indirect, &baseline.indirect_timed_wait_sites, true); + policy_violations.extend( + indirect_violations + .into_iter() + .map(|message| format!("indirect: {message}")), + ); + let live = count_sleep_clock_sites(&root); + policy_violations.extend(sleep_clock_violations( + &live, + &baseline.sleep_and_clock_sites, + )); assert!( - violations.is_empty(), - "test-estate audit: new sleep or raw-clock site(s) beyond \ + policy_violations.is_empty(), + "test-estate audit: new sleep/raw-clock sites or stale allowances in \ tests/fixtures/test_estate_baseline.toml:\n{}", - violations.join("\n") + policy_violations.join("\n") ); } @@ -961,7 +1108,7 @@ fn default_measurement_is_wired(ci: &str, dispatcher: &str, gate: &str, stage: & let union = body("shard_union"); let verify = union.find("python3 scripts/ci-test-shards.py verify"); let copy = union.find("cp \"$RUNNER_TEMP/shard-receipts/default-$system.xml\" \"$repository_root/target/nextest/gate/junit.xml\""); - let wall_name = if stage == "shard-union-linux-isolated" { + let wall_name = if stage.ends_with("-isolated") { "default_wall_isolated" } else { "default_wall" @@ -979,7 +1126,7 @@ fn default_measurement_is_wired(ci: &str, dispatcher: &str, gate: &str, stage: & .contains("python3 scripts/ci-test-shards.py run default \"$GATE_SHARD\"") && union.contains("for system in \"${systems[@]}\"; do") && union.contains("shard-union-linux|shard-union-linux-isolated) systems=(Linux) ;;") - && union.contains("shard-union-macos) systems=(macOS) ;;") + && union.contains("shard-union-macos|shard-union-macos-isolated) systems=(macOS) ;;") && union.contains("--system \"${systems[0]}\"") && matches!((verify, copy, wall), (Some(v), Some(c), Some(w)) if v < c && c < w) && body(wall_name) @@ -988,7 +1135,7 @@ fn default_measurement_is_wired(ci: &str, dispatcher: &str, gate: &str, stage: & .contains("gate_wall_seconds_never_exceed_the_frozen_baseline -- --ignored --exact") && body(wall_name) .contains("python3 scripts/check-gate-wall-clock.py --junit \"$CG_GATE_JUNIT\"") - && (stage != "shard-union-linux-isolated" || body(wall_name).contains( + && (!stage.ends_with("-isolated") || body(wall_name).contains( "cargo test --locked --manifest-path scripts/gate-wall-audit/Cargo.toml --target-dir \"$RUNNER_TEMP/gate-wall-audit-target\" --test test_estate_audit")) } @@ -1062,7 +1209,7 @@ fn the_ignored_wall_clock_arm_is_invoked_by_name_in_the_gate_script_and_ci() { ( std::fs::read_to_string(root.join(".github/workflows/macos-advisory.yml")) .expect("advisory workflow"), - "shard-union-macos", + "shard-union-macos-isolated", ), ] { assert!(default_measurement_is_wired( @@ -1190,6 +1337,7 @@ fn the_readme_ci_snippet_names_the_wall_clock_steps_the_workflow_actually_runs() struct Baseline { sleep_and_clock_sites: BTreeMap, + indirect_timed_wait_sites: BTreeMap, default_tier_test_count: BTreeMap, gate_wall_seconds: BTreeMap, gate_wall_seconds_tolerance_factor: f64, @@ -1204,6 +1352,7 @@ fn load_baseline(root: &Path) -> Baseline { toml::from_str(&text).unwrap_or_else(|e| panic!("parsing {}: {e}", path.display())); let sleep_and_clock_sites = table_of_usize(&value, "sleep_and_clock_sites"); + let indirect_timed_wait_sites = table_of_usize(&value, "indirect_timed_wait_sites"); let default_tier_test_count = table_of_usize(&value, "default_tier_test_count"); let gate_wall_seconds = table_of_f64(&value, "gate_wall_seconds"); let policy = value @@ -1231,6 +1380,7 @@ fn load_baseline(root: &Path) -> Baseline { Baseline { sleep_and_clock_sites, + indirect_timed_wait_sites, default_tier_test_count, gate_wall_seconds, gate_wall_seconds_tolerance_factor, diff --git a/scripts/ci-cache-post-cleanup.py b/scripts/ci-cache-post-cleanup.py index 597baf1..2314cfb 100644 --- a/scripts/ci-cache-post-cleanup.py +++ b/scripts/ci-cache-post-cleanup.py @@ -1,12 +1,18 @@ #!/usr/bin/env python3 -"""Discard only disposable trybuild output before compiler cache post hooks.""" +"""Discard disposable compiler scratch before compiler cache post hooks; keep the trybuild compile cache.""" import os from pathlib import Path import shutil -root = Path(os.environ.get('CARGO_TARGET_DIR', 'target')) -scratch = root / 'tests' +root = Path(os.environ.get("CARGO_TARGET_DIR", "target")) +scratch = root / "tests" if scratch.is_symlink(): scratch.unlink() -elif scratch.exists(): - shutil.rmtree(scratch) +elif scratch.is_dir(): + for entry in scratch.iterdir(): + if entry.name == "trybuild" and not entry.is_symlink(): + continue + if entry.is_symlink() or entry.is_file(): + entry.unlink() + else: + shutil.rmtree(entry) diff --git a/scripts/ci-test-shards.json b/scripts/ci-test-shards.json index 24dc36e..f1ebd9b 100644 --- a/scripts/ci-test-shards.json +++ b/scripts/ci-test-shards.json @@ -6,10 +6,9 @@ ] }, "import-and-cli": { - "filter": "binary_id(=context-graph-cli::event_import_roundtrip) | binary_id(=context-graph-cli::event_history) | binary_id(=context-graph::a_graph_read_past_the_read_ceiling_answers_what_the_writable_store_answers) | binary_id(=context-graph::a_find_costs_the_same_statements_at_any_context_count) | binary_id(=context-graph-cli::import_embedding_workers) | binary_id(=context-graph::event_search_uses_role_preparation_while_digest_record_and_evidence_paths_remain_byte_for_byte_unchanged) | binary_id(=context-graph-cli::write_memory_policy) | binary_id(=context-graph::a_closed_cursor_answers_its_first_page_by_type_not_by_panic) | binary_id(=context-graph) | binary_id(=context-graph-cli::a_read_only_command_answers_a_store_a_writer_is_holding) | binary_id(=context-graph::compile_fail) | binary_id(=context-graph-cli::understanding_actions_cli) | binary_id(=cg-connector-claude-code::source_fidelity) | binary_id(=context-graph::event_forget) | binary_id(=context-graph::literal_source_and_query_text_starting_with_nomic_prefix_words_is_preserved_and_searchable) | binary_id(=context-graph::a_failed_status_fold_leaves_no_cursor_open) | binary_id(=context-graph::read_memory_policy) | binary_id(=context-graph::a_read_declaration_is_verified_against_the_persisted_table) | binary_id(=context-graph::what_a_status_total_examined_is_reported_honestly) | binary_id(=context-graph::event_history_status) | binary_id(=context-graph::context_bindings) | binary_id(=context-graph::event_anchor_head_read) | binary_id(=context-graph-cli::event_forget_terminal) | binary_id(=context-graph-cli::a_failed_status_read_is_never_rendered_as_an_answer) | binary_id(=context-graph::a_stopped_store_answers_the_typed_graph_reads_without_changing) | binary_id(=context-graph::a_store_that_needs_a_writable_reopen_is_not_told_it_is_unreadable) | binary_id(=context-graph::oversized_event_split_uses_selected_tokenizer_and_has_utf8_safe_exact_bidirectional_source_offsets) | binary_id(=context-graph::event_core_regressions) | binary_id(=context-graph::one_writable_open_carries_the_owner_handler_and_the_embedders) | binary_id(=context-graph-cli::cli_model_byok) | binary_id(=context-graph::reading_pass_through_replaces_the_control_socket) | binary_id(=context-graph::a_crash_dirty_store_still_answers_that_no_owner_is_running) | binary_id(=context-graph::a_refusal_with_no_owner_alive_never_names_a_live_owner) | binary_id(=context-graph-cli::import_cli) | binary_id(=context-graph::registration_plus_many_preparations_reads_the_weights_once) | binary_id(=context-graph::event_coverage_receipts) | binary_id(=context-graph::default_nomic_configuration_reports_a_missing_model_as_not_installed_and_never_fetches) | binary_id(=context-graph::event_import_receipt_context) | binary_id(=context-graph::event_search_receipt_order) | binary_id(=context-graph-cli::tier0_cli) | binary_id(=context-graph::event_interpretation) | binary_id(=context-graph::doc_claim_bindings) | binary_id(=context-graph::event_history_wrapper_names_the_actual_unsupported_input_kind) | binary_id(=candle-embeddinggemma::acceptance) | binary_id(=context-graph::event_history_wrapper_refuses_plain_text_embedding_without_an_explicit_role) | binary_id(=context-graph-cli::worker_cli_surface) | binary_id(=cg-vision-embedder::siglip) | binary_id(=context-graph-cli::cg_model_fetch_finds_a_registration_that_cg_model_show_reads_when_data_and_cache_roots_differ) | binary_id(=context-graph::ci_hardening) | binary_id(=context-graph-cli::a_refusal_answers_in_the_format_the_caller_asked_for) | binary_id(=context-graph::event_search_sealing_rejects_an_invalid_prepared_value_without_panicking) | binary_id(=context-graph::event_search_inference_forwarded_from_a_plug_in_is_refused_by_the_bundled_model) | binary_id(=context-graph-cli::help_presents_exactly_the_public_command_list) | binary_id(=context-graph::event_search_preparation_hook_supports_external_embedders) | binary_id(=context-graph::a_busy_store_never_reads_as_an_unexplained_engine_failure) | binary_id(=context-graph::prepared_event_search_document_debug_output_contains_no_token_payload) | binary_id(=context-graph::nomic_real_model_helper_refuses_locally_on_an_empty_cache_and_never_reaches_the_network) | binary_id(=context-graph-cli::capture_smoke_explicit_input) | binary_id(=context-graph::checked_entrypoints_enforce_the_token_limit_for_every_embedder_even_when_the_hook_does_not) | binary_id(=context-graph::interpretation_foundation_regression)", + "filter": "binary_id(=context-graph-cli::event_import_roundtrip) | binary_id(=context-graph::a_graph_read_past_the_read_ceiling_answers_what_the_writable_store_answers) | binary_id(=context-graph::a_find_costs_the_same_statements_at_any_context_count) | binary_id(=context-graph-cli::import_embedding_workers) | binary_id(=context-graph::event_search_uses_role_preparation_while_digest_record_and_evidence_paths_remain_byte_for_byte_unchanged) | binary_id(=context-graph-cli::write_memory_policy) | binary_id(=context-graph::a_closed_cursor_answers_its_first_page_by_type_not_by_panic) | binary_id(=context-graph) | binary_id(=context-graph-cli::a_read_only_command_answers_a_store_a_writer_is_holding) | binary_id(=context-graph::compile_fail) | binary_id(=context-graph-cli::understanding_actions_cli) | binary_id(=cg-connector-claude-code::source_fidelity) | binary_id(=context-graph::event_forget) | binary_id(=context-graph::literal_source_and_query_text_starting_with_nomic_prefix_words_is_preserved_and_searchable) | binary_id(=context-graph::a_failed_status_fold_leaves_no_cursor_open) | binary_id(=context-graph::read_memory_policy) | binary_id(=context-graph::a_read_declaration_is_verified_against_the_persisted_table) | binary_id(=context-graph::what_a_status_total_examined_is_reported_honestly) | binary_id(=context-graph::event_history_status) | binary_id(=context-graph::context_bindings) | binary_id(=context-graph::event_anchor_head_read) | binary_id(=context-graph-cli::event_forget_terminal) | binary_id(=context-graph-cli::a_failed_status_read_is_never_rendered_as_an_answer) | binary_id(=context-graph::a_stopped_store_answers_the_typed_graph_reads_without_changing) | binary_id(=context-graph::a_store_that_needs_a_writable_reopen_is_not_told_it_is_unreadable) | binary_id(=context-graph::oversized_event_split_uses_selected_tokenizer_and_has_utf8_safe_exact_bidirectional_source_offsets) | binary_id(=context-graph::event_core_regressions) | binary_id(=context-graph::one_writable_open_carries_the_owner_handler_and_the_embedders) | binary_id(=context-graph-cli::cli_model_byok) | binary_id(=context-graph::reading_pass_through_replaces_the_control_socket) | binary_id(=context-graph::a_crash_dirty_store_still_answers_that_no_owner_is_running) | binary_id(=context-graph::a_refusal_with_no_owner_alive_never_names_a_live_owner) | binary_id(=context-graph-cli::import_cli) | binary_id(=context-graph::registration_plus_many_preparations_reads_the_weights_once) | binary_id(=context-graph::event_coverage_receipts) | binary_id(=context-graph::default_nomic_configuration_reports_a_missing_model_as_not_installed_and_never_fetches) | binary_id(=context-graph::event_import_receipt_context) | binary_id(=context-graph::event_search_receipt_order) | binary_id(=context-graph-cli::tier0_cli) | binary_id(=context-graph::event_interpretation) | binary_id(=context-graph::doc_claim_bindings) | binary_id(=context-graph::event_history_wrapper_names_the_actual_unsupported_input_kind) | binary_id(=candle-embeddinggemma::acceptance) | binary_id(=context-graph::event_history_wrapper_refuses_plain_text_embedding_without_an_explicit_role) | binary_id(=context-graph-cli::worker_cli_surface) | binary_id(=cg-vision-embedder::siglip) | binary_id(=context-graph-cli::cg_model_fetch_finds_a_registration_that_cg_model_show_reads_when_data_and_cache_roots_differ) | binary_id(=context-graph::ci_hardening) | binary_id(=context-graph-cli::a_refusal_answers_in_the_format_the_caller_asked_for) | binary_id(=context-graph::event_search_sealing_rejects_an_invalid_prepared_value_without_panicking) | binary_id(=context-graph::event_search_inference_forwarded_from_a_plug_in_is_refused_by_the_bundled_model) | binary_id(=context-graph-cli::help_presents_exactly_the_public_command_list) | binary_id(=context-graph::event_search_preparation_hook_supports_external_embedders) | binary_id(=context-graph::a_busy_store_never_reads_as_an_unexplained_engine_failure) | binary_id(=context-graph::prepared_event_search_document_debug_output_contains_no_token_payload) | binary_id(=context-graph::nomic_real_model_helper_refuses_locally_on_an_empty_cache_and_never_reaches_the_network) | binary_id(=context-graph-cli::capture_smoke_explicit_input) | binary_id(=context-graph::checked_entrypoints_enforce_the_token_limit_for_every_embedder_even_when_the_hook_does_not) | binary_id(=context-graph::interpretation_foundation_regression) | binary_id(=context-graph::event_search_embedding_parallelism) | binary_id(=context-graph::audit_outcome_typed_snapshot)", "binaries": [ "context-graph-cli::event_import_roundtrip", - "context-graph-cli::event_history", "context-graph::a_graph_read_past_the_read_ceiling_answers_what_the_writable_store_answers", "context-graph::a_find_costs_the_same_statements_at_any_context_count", "context-graph-cli::import_embedding_workers", @@ -67,11 +66,13 @@ "context-graph::nomic_real_model_helper_refuses_locally_on_an_empty_cache_and_never_reaches_the_network", "context-graph-cli::capture_smoke_explicit_input", "context-graph::checked_entrypoints_enforce_the_token_limit_for_every_embedder_even_when_the_hook_does_not", - "context-graph::interpretation_foundation_regression" + "context-graph::interpretation_foundation_regression", + "context-graph::event_search_embedding_parallelism", + "context-graph::audit_outcome_typed_snapshot" ] }, "integration-and-rest": { - "filter": "not((binary_id(=context-graph::acceptance)) | (binary_id(=context-graph-cli::event_import_roundtrip) | binary_id(=context-graph-cli::event_history) | binary_id(=context-graph::a_graph_read_past_the_read_ceiling_answers_what_the_writable_store_answers) | binary_id(=context-graph::a_find_costs_the_same_statements_at_any_context_count) | binary_id(=context-graph-cli::import_embedding_workers) | binary_id(=context-graph::event_search_uses_role_preparation_while_digest_record_and_evidence_paths_remain_byte_for_byte_unchanged) | binary_id(=context-graph-cli::write_memory_policy) | binary_id(=context-graph::a_closed_cursor_answers_its_first_page_by_type_not_by_panic) | binary_id(=context-graph) | binary_id(=context-graph-cli::a_read_only_command_answers_a_store_a_writer_is_holding) | binary_id(=context-graph::compile_fail) | binary_id(=context-graph-cli::understanding_actions_cli) | binary_id(=cg-connector-claude-code::source_fidelity) | binary_id(=context-graph::event_forget) | binary_id(=context-graph::literal_source_and_query_text_starting_with_nomic_prefix_words_is_preserved_and_searchable) | binary_id(=context-graph::a_failed_status_fold_leaves_no_cursor_open) | binary_id(=context-graph::read_memory_policy) | binary_id(=context-graph::a_read_declaration_is_verified_against_the_persisted_table) | binary_id(=context-graph::what_a_status_total_examined_is_reported_honestly) | binary_id(=context-graph::event_history_status) | binary_id(=context-graph::context_bindings) | binary_id(=context-graph::event_anchor_head_read) | binary_id(=context-graph-cli::event_forget_terminal) | binary_id(=context-graph-cli::a_failed_status_read_is_never_rendered_as_an_answer) | binary_id(=context-graph::a_stopped_store_answers_the_typed_graph_reads_without_changing) | binary_id(=context-graph::a_store_that_needs_a_writable_reopen_is_not_told_it_is_unreadable) | binary_id(=context-graph::oversized_event_split_uses_selected_tokenizer_and_has_utf8_safe_exact_bidirectional_source_offsets) | binary_id(=context-graph::event_core_regressions) | binary_id(=context-graph::one_writable_open_carries_the_owner_handler_and_the_embedders) | binary_id(=context-graph-cli::cli_model_byok) | binary_id(=context-graph::reading_pass_through_replaces_the_control_socket) | binary_id(=context-graph::a_crash_dirty_store_still_answers_that_no_owner_is_running) | binary_id(=context-graph::a_refusal_with_no_owner_alive_never_names_a_live_owner) | binary_id(=context-graph-cli::import_cli) | binary_id(=context-graph::registration_plus_many_preparations_reads_the_weights_once) | binary_id(=context-graph::event_coverage_receipts) | binary_id(=context-graph::default_nomic_configuration_reports_a_missing_model_as_not_installed_and_never_fetches) | binary_id(=context-graph::event_import_receipt_context) | binary_id(=context-graph::event_search_receipt_order) | binary_id(=context-graph-cli::tier0_cli) | binary_id(=context-graph::event_interpretation) | binary_id(=context-graph::doc_claim_bindings) | binary_id(=context-graph::event_history_wrapper_names_the_actual_unsupported_input_kind) | binary_id(=candle-embeddinggemma::acceptance) | binary_id(=context-graph::event_history_wrapper_refuses_plain_text_embedding_without_an_explicit_role) | binary_id(=context-graph-cli::worker_cli_surface) | binary_id(=cg-vision-embedder::siglip) | binary_id(=context-graph-cli::cg_model_fetch_finds_a_registration_that_cg_model_show_reads_when_data_and_cache_roots_differ) | binary_id(=context-graph::ci_hardening) | binary_id(=context-graph-cli::a_refusal_answers_in_the_format_the_caller_asked_for) | binary_id(=context-graph::event_search_sealing_rejects_an_invalid_prepared_value_without_panicking) | binary_id(=context-graph::event_search_inference_forwarded_from_a_plug_in_is_refused_by_the_bundled_model) | binary_id(=context-graph-cli::help_presents_exactly_the_public_command_list) | binary_id(=context-graph::event_search_preparation_hook_supports_external_embedders) | binary_id(=context-graph::a_busy_store_never_reads_as_an_unexplained_engine_failure) | binary_id(=context-graph::prepared_event_search_document_debug_output_contains_no_token_payload) | binary_id(=context-graph::nomic_real_model_helper_refuses_locally_on_an_empty_cache_and_never_reaches_the_network) | binary_id(=context-graph-cli::capture_smoke_explicit_input) | binary_id(=context-graph::checked_entrypoints_enforce_the_token_limit_for_every_embedder_even_when_the_hook_does_not) | binary_id(=context-graph::interpretation_foundation_regression)))", + "filter": "not((binary_id(=context-graph::acceptance)) | (binary_id(=context-graph-cli::event_import_roundtrip) | binary_id(=context-graph::a_graph_read_past_the_read_ceiling_answers_what_the_writable_store_answers) | binary_id(=context-graph::a_find_costs_the_same_statements_at_any_context_count) | binary_id(=context-graph-cli::import_embedding_workers) | binary_id(=context-graph::event_search_uses_role_preparation_while_digest_record_and_evidence_paths_remain_byte_for_byte_unchanged) | binary_id(=context-graph-cli::write_memory_policy) | binary_id(=context-graph::a_closed_cursor_answers_its_first_page_by_type_not_by_panic) | binary_id(=context-graph) | binary_id(=context-graph-cli::a_read_only_command_answers_a_store_a_writer_is_holding) | binary_id(=context-graph::compile_fail) | binary_id(=context-graph-cli::understanding_actions_cli) | binary_id(=cg-connector-claude-code::source_fidelity) | binary_id(=context-graph::event_forget) | binary_id(=context-graph::literal_source_and_query_text_starting_with_nomic_prefix_words_is_preserved_and_searchable) | binary_id(=context-graph::a_failed_status_fold_leaves_no_cursor_open) | binary_id(=context-graph::read_memory_policy) | binary_id(=context-graph::a_read_declaration_is_verified_against_the_persisted_table) | binary_id(=context-graph::what_a_status_total_examined_is_reported_honestly) | binary_id(=context-graph::event_history_status) | binary_id(=context-graph::context_bindings) | binary_id(=context-graph::event_anchor_head_read) | binary_id(=context-graph-cli::event_forget_terminal) | binary_id(=context-graph-cli::a_failed_status_read_is_never_rendered_as_an_answer) | binary_id(=context-graph::a_stopped_store_answers_the_typed_graph_reads_without_changing) | binary_id(=context-graph::a_store_that_needs_a_writable_reopen_is_not_told_it_is_unreadable) | binary_id(=context-graph::oversized_event_split_uses_selected_tokenizer_and_has_utf8_safe_exact_bidirectional_source_offsets) | binary_id(=context-graph::event_core_regressions) | binary_id(=context-graph::one_writable_open_carries_the_owner_handler_and_the_embedders) | binary_id(=context-graph-cli::cli_model_byok) | binary_id(=context-graph::reading_pass_through_replaces_the_control_socket) | binary_id(=context-graph::a_crash_dirty_store_still_answers_that_no_owner_is_running) | binary_id(=context-graph::a_refusal_with_no_owner_alive_never_names_a_live_owner) | binary_id(=context-graph-cli::import_cli) | binary_id(=context-graph::registration_plus_many_preparations_reads_the_weights_once) | binary_id(=context-graph::event_coverage_receipts) | binary_id(=context-graph::default_nomic_configuration_reports_a_missing_model_as_not_installed_and_never_fetches) | binary_id(=context-graph::event_import_receipt_context) | binary_id(=context-graph::event_search_receipt_order) | binary_id(=context-graph-cli::tier0_cli) | binary_id(=context-graph::event_interpretation) | binary_id(=context-graph::doc_claim_bindings) | binary_id(=context-graph::event_history_wrapper_names_the_actual_unsupported_input_kind) | binary_id(=candle-embeddinggemma::acceptance) | binary_id(=context-graph::event_history_wrapper_refuses_plain_text_embedding_without_an_explicit_role) | binary_id(=context-graph-cli::worker_cli_surface) | binary_id(=cg-vision-embedder::siglip) | binary_id(=context-graph-cli::cg_model_fetch_finds_a_registration_that_cg_model_show_reads_when_data_and_cache_roots_differ) | binary_id(=context-graph::ci_hardening) | binary_id(=context-graph-cli::a_refusal_answers_in_the_format_the_caller_asked_for) | binary_id(=context-graph::event_search_sealing_rejects_an_invalid_prepared_value_without_panicking) | binary_id(=context-graph::event_search_inference_forwarded_from_a_plug_in_is_refused_by_the_bundled_model) | binary_id(=context-graph-cli::help_presents_exactly_the_public_command_list) | binary_id(=context-graph::event_search_preparation_hook_supports_external_embedders) | binary_id(=context-graph::a_busy_store_never_reads_as_an_unexplained_engine_failure) | binary_id(=context-graph::prepared_event_search_document_debug_output_contains_no_token_payload) | binary_id(=context-graph::nomic_real_model_helper_refuses_locally_on_an_empty_cache_and_never_reaches_the_network) | binary_id(=context-graph-cli::capture_smoke_explicit_input) | binary_id(=context-graph::checked_entrypoints_enforce_the_token_limit_for_every_embedder_even_when_the_hook_does_not) | binary_id(=context-graph::interpretation_foundation_regression) | binary_id(=context-graph::event_search_embedding_parallelism) | binary_id(=context-graph::audit_outcome_typed_snapshot)))", "binaries": [ "context-graph::integration", "context-graph::event_search_projection_correctness", @@ -85,7 +86,6 @@ "context-graph::a_paged_read_across_a_writer_transition_answers_the_snapshot_it_opened", "context-graph-cli::cli_acceptance", "context-graph::a_second_node_receives_the_whole_startup_schema", - "context-graph::event_search_embedding_parallelism", "context-graph-cli::understand_cli", "context-graph::event_core_policy_recovery", "context-graph::role_prepared_values_share_one_event_history_profile_across_the_raw_embedder_and_its_wrapper", @@ -111,7 +111,6 @@ "context-graph::event_checkpoint_count_contract", "context-graph::a_real_store_open_is_refused_by_the_readers_holding_it", "context-graph::declaring_a_total_on_a_table_the_capture_path_never_writes_costs_the_append_nothing", - "context-graph::audit_outcome_typed_snapshot", "context-graph-cli::fabric_status_cli", "context-graph-cli::fabric_cli_surface", "context-graph::an_owner_ask_with_no_owner_never_touches_the_store_file", @@ -133,7 +132,8 @@ "context-graph::prepared_event_search_document_sealed_under_a_different_profile_is_refused_even_by_an_embedder_designed_to_bypass_the_check", "context-graph-cli::session_reconciliation_wiring", "context-graph::import_error_display", - "context-graph-cli::extraction_prompt_delegation" + "context-graph-cli::extraction_prompt_delegation", + "context-graph-cli::event_history" ] } } diff --git a/scripts/ci-test-shards.py b/scripts/ci-test-shards.py index c45f216..3eff4b8 100644 --- a/scripts/ci-test-shards.py +++ b/scripts/ci-test-shards.py @@ -239,8 +239,8 @@ def workflow_receipt(destination, variant, result): def bundle(variant, tests_only=False): selected = PHYSICAL_BUNDLES[os.environ['GATE_PHYSICAL_BUNDLE']] - if tests_only and (variant != 'features' or os.environ['RUNNER_OS'] != 'Linux' or len(selected) != 1): - raise ValueError('tests-only requires a Linux feature singleton') + if tests_only and (variant != 'features' or len(selected) != 1): + raise ValueError('tests-only requires a feature singleton') if selected == PHYSICAL_BUNDLES['integration-and-rest + import-and-cli + acceptance']: if (variant, os.environ['RUNNER_OS']) != ('features', 'macOS'): raise ValueError('three-stage bundle requires macOS feature execution') diff --git a/scripts/gate.sh b/scripts/gate.sh index 2a80384..26eeb07 100755 --- a/scripts/gate.sh +++ b/scripts/gate.sh @@ -227,14 +227,15 @@ feature_qualification() { } feature_qualification_sync() { feature_builds_sync - feature_docs } features_tests() { - [[ "${RUNNER_OS:-}" == Linux ]] || pending 'feature tests stage requires Linux' case "${GATE_SHARD:-}" in acceptance|integration-and-rest|import-and-cli) ;; *) pending 'feature tests stage requires an exact shard' ;; esac + if [[ "$GATE_SHARD" == acceptance ]]; then + feature_docs + fi python3 scripts/ci-test-shards.py run features "$GATE_SHARD" } @@ -267,7 +268,7 @@ shard_union() { local systems=(Linux macOS) case "$stage" in shard-union-linux|shard-union-linux-isolated) systems=(Linux) ;; - shard-union-macos) systems=(macOS) ;; + shard-union-macos|shard-union-macos-isolated) systems=(macOS) ;; shard-union) ;; *) pending 'unknown shard union stage' ;; esac @@ -279,7 +280,7 @@ shard_union() { mkdir -p "$repository_root/target/nextest/gate" for system in "${systems[@]}"; do cp "$RUNNER_TEMP/shard-receipts/default-$system.xml" "$repository_root/target/nextest/gate/junit.xml" - if [[ "$stage" == shard-union-linux-isolated ]]; then + if [[ "$stage" == shard-union-linux-isolated || "$stage" == shard-union-macos-isolated ]]; then default_wall_isolated else default_wall @@ -530,7 +531,7 @@ print('local package inspected; no publication performed') PY } case "$stage" in - shard-union|shard-union-linux|shard-union-macos|shard-union-linux-isolated) run_stage "$stage" shard_union ;; + shard-union|shard-union-linux|shard-union-macos|shard-union-linux-isolated|shard-union-macos-isolated) run_stage "$stage" shard_union ;; full) cargo fmt --all --check cargo clippy --workspace --all-targets -- -D warnings diff --git a/scripts/tests/test_build_workflow.py b/scripts/tests/test_build_workflow.py index 697c94d..ef62029 100644 --- a/scripts/tests/test_build_workflow.py +++ b/scripts/tests/test_build_workflow.py @@ -524,11 +524,11 @@ def legacy_family(family, workflow, ci, download, readme, advisory=None): required = parse_workflow(ci) advisory = parse_workflow(advisory if advisory is not None else (ROOT/'.github/workflows/macos-advisory.yml').read_text()) - require(required.get('name') == 'ci' and advisory.get('name') == 'macOS advisory', + require(required.get('name') == 'Linux CI' and advisory.get('name') == 'macOS CI (advisory)', 'independent required and advisory workflow roles') for lane, os_name, aggregate, dependencies in ( (required, 'ubuntu-latest', 'required', {'source', 'lint', 'docs', 'controls', 'test', 'features', 'feature-qualification-core', 'feature-qualification-sync'}), - (advisory, 'macos-latest', 'macos-advisory', {'lint', 'controls', 'test', 'features'})): + (advisory, 'macos-latest', 'macos-advisory', {'lint', 'controls', 'test', 'features', 'feature-qualification-core', 'feature-qualification-sync'})): require(set(lane['jobs'].get(aggregate, {}).get('needs', [])) == dependencies, 'exact independent platform aggregate dependencies') require(('required' in lane['jobs']) == (aggregate == 'required'), @@ -537,11 +537,7 @@ def legacy_family(family, workflow, ci, download, readme, advisory=None): job = lane['jobs'].get(name, {}) expected_matrix = {'os': [os_name]} if name in ('test', 'features'): - bundles = ('acceptance', 'integration-and-rest + import-and-cli') - if os_name == 'ubuntu-latest': - bundles = ('acceptance', 'integration-and-rest', 'import-and-cli') - if name == 'features' and os_name == 'macos-latest': - bundles = ('integration-and-rest + import-and-cli + acceptance',) + bundles = ('acceptance', 'integration-and-rest', 'import-and-cli') expected_matrix = {'include': [{'os': os_name, 'bundle': bundle} for bundle in bundles]} require(job.get('runs-on') == '${{ matrix.os }}' and job.get('strategy', {}).get('matrix') == expected_matrix, @@ -672,13 +668,13 @@ def check_triggers(workflow): def check_ci_triggers(workflow): - """Correctness runs on every push, every pull request and manual dispatch.""" + """Correctness runs on dev pushes, every pull request and manual dispatch.""" events = triggers(workflow) require(isinstance(events, dict) and set(events) == {'push', 'pull_request', 'workflow_dispatch'}, 'CI retains push, pull request and manual dispatch triggers') - for event in ('push', 'pull_request'): - require(events[event] in (None, {}), event + ' must be unrestricted') + require(events['push'] == {'branches': ['dev']}, 'automatic pushes select only dev') + require(events['pull_request'] in (None, {}), 'pull requests must be unrestricted') def check_concurrency(workflow): @@ -892,7 +888,7 @@ def admission_run(workflow, event='push', ref='refs/heads/dev', repository=REPOS run.update(run_changes or {}) job_rows = [dict(name=name, conclusion='success') for name in ('source', 'lint (ubuntu-latest)', 'docs', - 'CI controls (ubuntu-latest)', 'Feature compatibility — default and minimal libraries (ubuntu-latest)', 'Feature compatibility — sync, CLI and doctests (ubuntu-latest)', + 'CI controls (ubuntu-latest)', 'Feature compatibility — default and minimal libraries (ubuntu-latest)', 'Feature compatibility — sync and CLI (ubuntu-latest)', *[f'Default tests — {bundle} ({os})' for bundle in ('acceptance', 'integration-and-rest', 'import-and-cli') for os in ('ubuntu-latest',)], @@ -945,7 +941,7 @@ def check_admission(workflow): require(any('actions/workflows/ci.yml/runs' in arg for call in calls for arg in call), 'manual selects ci.yml push runs') require(any('/jobs' in arg and 'filter=latest' in arg for call in calls for arg in call), 'manual verifies latest attempt') required_jobs = ('source', 'docs', 'PR checks', 'lint (ubuntu-latest)', - 'CI controls (ubuntu-latest)', 'Feature compatibility — default and minimal libraries (ubuntu-latest)', 'Feature compatibility — sync, CLI and doctests (ubuntu-latest)', + 'CI controls (ubuntu-latest)', 'Feature compatibility — default and minimal libraries (ubuntu-latest)', 'Feature compatibility — sync and CLI (ubuntu-latest)', 'Default tests — acceptance (ubuntu-latest)', 'Default tests — integration-and-rest (ubuntu-latest)', 'Default tests — import-and-cli (ubuntu-latest)', @@ -1199,7 +1195,7 @@ def positive_admission(): if python3 - <<'JOBS' import json, os, pathlib rows = json.loads(pathlib.Path(os.environ['RUNNER_TEMP'],'jobs').read_text())['jobs'] -expected = {'source', 'docs', 'PR checks', 'lint (ubuntu-latest)', 'CI controls (ubuntu-latest)', 'Feature compatibility — default and minimal libraries (ubuntu-latest)', 'Feature compatibility — sync, CLI and doctests (ubuntu-latest)'} +expected = {'source', 'docs', 'PR checks', 'lint (ubuntu-latest)', 'CI controls (ubuntu-latest)', 'Feature compatibility — default and minimal libraries (ubuntu-latest)', 'Feature compatibility — sync and CLI (ubuntu-latest)'} for bundle in ('acceptance', 'integration-and-rest', 'import-and-cli'): expected.add(f'Default tests — {bundle} (ubuntu-latest)') expected.add(f'Event-search support tests — {bundle} (ubuntu-latest)') @@ -1464,7 +1460,7 @@ def check_smoke(workflow): # Complete parsed mappings are pinned independently of workflow-supplied values. -BUILD_WORKFLOW_SHA256 = '303d86fb46f0072d97b5d2be1485612d9e1c34df65aa8b391049685c4c9ec281' +BUILD_WORKFLOW_SHA256 = '0f95429addfd4037db9754b92b339dceecfb0ce67054cf75590afdd9f0d3cbc4' DEVELOPMENT_BINARIES_JOB_SHA256 = 'c510129efa6b318d4c850a1697338dbf84faff73740c5ce67aa5166e2cedaa5c' @@ -1604,38 +1600,43 @@ def test_complete_delivery_content_controls(self): check_admission(changed) def test_ci_trigger_set(self): - """Correctness runs on every push, every pull request and manual dispatch.""" - check_ci_triggers(parse_workflow((ROOT/'.github/workflows/ci.yml').read_text())) + """Both platforms run on dev pushes, every pull request and exact-SHA dispatch.""" + for filename in ('ci.yml', 'macos-advisory.yml'): + with self.subTest(workflow=filename): + check_ci_triggers(parse_workflow((ROOT/'.github/workflows'/filename).read_text())) def test_ci_trigger_controls(self): - """Branch, tag, path and activity filters cannot narrow correctness admission.""" + """Both platforms reject duplicate candidate pushes and narrowed PR admission.""" import copy - workflow = parse_workflow((ROOT/'.github/workflows/ci.yml').read_text()) - events = triggers(workflow) - events['push'] = None - events['pull_request'] = None - check_ci_triggers(workflow) - check_caller(workflow) - for event in ('push', 'pull_request', 'workflow_dispatch'): - changed = copy.deepcopy(workflow) - del triggers(changed)[event] - with self.subTest(missing=event), self.assertRaises(AssertionError): - check_ci_triggers(changed) - for event in ('push', 'pull_request'): - for key, value in (('branches', ['dev']), ('branches-ignore', ['main']), - ('tags', ['v*']), ('paths', ['src/**']), - ('paths-ignore', ['docs/**']), ('types', ['opened'])): + for filename in ('ci.yml', 'macos-advisory.yml'): + workflow = parse_workflow((ROOT/'.github/workflows'/filename).read_text()) + check_ci_triggers(workflow) + for event in ('push', 'pull_request', 'workflow_dispatch'): changed = copy.deepcopy(workflow) - triggers(changed)[event] = {key: value} - with self.subTest(event=event, filter=key), self.assertRaises(AssertionError): + del triggers(changed)[event] + with self.subTest(workflow=filename, missing=event), self.assertRaises(AssertionError): check_ci_triggers(changed) + for event, configs in ( + ('push', [None, {}, {'branches': ['dev', 'ci-macos-candidate']}, + {'branches': ['**']}, {'branches': ['ci-macos-candidate']}, + {'branches': ['dev'], 'tags': ['v*']}, + {'branches': ['dev'], 'paths-ignore': ['docs/**']}]), + ('pull_request', [{'branches': ['dev']}, {'branches-ignore': ['main']}, + {'paths': ['src/**']}, {'paths-ignore': ['docs/**']}, + {'types': ['opened']}] )): + for config in configs: + changed = copy.deepcopy(workflow) + triggers(changed)[event] = config + with self.subTest(workflow=filename, event=event, config=config), self.assertRaises(AssertionError): + check_ci_triggers(changed) def test_ci_rejects_branch_filtered_push(self): - """A push trigger limited to a named branch is refused for correctness admission.""" - workflow = parse_workflow((ROOT/'.github/workflows/ci.yml').read_text()) - triggers(workflow)['push'] = {'branches': ['feature-integration']} - with self.assertRaises(AssertionError): - check_ci_triggers(workflow) + """Automatic correctness pushes select dev alone, never another named branch.""" + for filename in ('ci.yml', 'macos-advisory.yml'): + workflow = parse_workflow((ROOT/'.github/workflows'/filename).read_text()) + triggers(workflow)['push'] = {'branches': ['feature-integration']} + with self.subTest(workflow=filename), self.assertRaises(AssertionError): + check_ci_triggers(workflow) def test_publisher_execution_settings_controls(self): """Only pinned job fields and inherited workflow execution settings may publish.""" @@ -2307,6 +2308,12 @@ def check(build, required, optional): legacy_family('platforms', yaml.safe_dump(build), yaml.safe_dump(required), '', '', yaml.safe_dump(optional)) check(workflow, ci, advisory) + for key, old_name in (('required', 'ci'), ('advisory', 'macOS advisory')): + required, optional = copy.deepcopy(ci), copy.deepcopy(advisory) + lane = required if key == 'required' else optional + lane['name'] = old_name + with self.subTest(lane=key, old_name=old_name), self.assertRaises(AssertionError): + check(workflow, required, optional) for filename in ('required', 'advisory'): for name in ('lint', 'controls', 'test', 'features'): for systems in ([], ['ubuntu-latest', 'macos-latest'], diff --git a/scripts/tests/test_ci_model_seed.py b/scripts/tests/test_ci_model_seed.py index 99bfd25..322a2e7 100644 --- a/scripts/tests/test_ci_model_seed.py +++ b/scripts/tests/test_ci_model_seed.py @@ -19,7 +19,7 @@ class SeedArtifact(unittest.TestCase): def setUp(self): self.temp = tempfile.TemporaryDirectory() self.addCleanup(self.temp.cleanup) - self.root = Path(self.temp.name) + self.root = Path(self.temp.name).resolve() self.patch = mock.patch.object(seed, 'ROOT', self.root) self.patch.start() self.addCleanup(self.patch.stop) diff --git a/scripts/tests/test_ci_qualification.py b/scripts/tests/test_ci_qualification.py index 24c3b1a..8e073dd 100644 --- a/scripts/tests/test_ci_qualification.py +++ b/scripts/tests/test_ci_qualification.py @@ -743,9 +743,9 @@ def test_compile_qualification_exact_commands_and_failures(self): sync = [ ['build', '--locked', '-p', 'context-graph', '--features', 'fabric-sync', '--lib'], ['build', '--locked', '-p', 'context-graph-cli'], - ['test', '--workspace', '--doc', '--no-fail-fast', '--features', 'context-graph-cli/event-search-test-support'], ] - for stage, expected in [('feature-qualification-core', core), ('feature-qualification-sync', sync), ('feature-qualification', core + sync)]: + docs = [['test', '--workspace', '--doc', '--no-fail-fast', '--features', 'context-graph-cli/event-search-test-support']] + for stage, expected in [('feature-qualification-core', core), ('feature-qualification-sync', sync), ('feature-qualification', core + sync + docs)]: result = self.run_stage(stage) self.assertEqual(result.returncode, 0, result.stderr) self.assertEqual(self.commands(), expected) @@ -753,10 +753,10 @@ def test_compile_qualification_exact_commands_and_failures(self): self.assertEqual(receipt['stage'], stage) self.assertEqual(receipt['test_count'], 0) self.assertEqual(receipt['tools']['nextest'], 'not used') - self.assertEqual(receipt['cargo_test_inventory'], [] if stage == 'feature-qualification-core' else ['src/lib.rs - docs (line 12)']) + self.assertEqual(receipt['cargo_test_inventory'], ['src/lib.rs - docs (line 12)'] if stage == 'feature-qualification' else []) self.assertEqual(receipt['commands'][0], 'bash scripts/gate.sh ' + stage) self.trace.unlink() - for failed in ('build', 'test') if stage != 'feature-qualification-core' else ('build',): + for failed in ('build', 'test') if stage == 'feature-qualification' else ('build',): result = self.run_stage(stage, FAIL_SUBCOMMAND=failed) self.assertEqual(result.returncode, 23, result.stderr) receipt = json.loads((self.root / 'job-receipt' / (stage + '.json')).read_text()) @@ -779,7 +779,18 @@ def test_features_tests_identity_inventory_and_stale_junit(self): self.assertEqual(receipt['commands'][0], 'bash scripts/gate.sh features-tests') self.assertEqual(receipt['test_count'], 2) self.assertEqual(len(receipt['executed_feature_inventory']), 2) - for system, shard in [('macOS', 'acceptance'), ('Linux', ''), ('Linux', 'wrong')]: + # The acceptance bundle alone runs the complete feature doctests, before its tests. + doc = 'cargo test --workspace --doc --no-fail-fast --features context-graph-cli/event-search-test-support' + self.assertEqual(receipt['commands'][1], doc) + self.assertEqual(receipt['cargo_test_inventory'], ['src/lib.rs - docs (line 12)']) + other = self.run_stage('features-tests', **dict(env, GATE_SHARD='import-and-cli')) + self.assertEqual(other.returncode, 0, other.stderr) + other_receipt = json.loads((self.root / 'job-receipt/features-tests.json').read_text()) + self.assertNotIn(doc, other_receipt['commands']) + self.assertEqual(other_receipt['cargo_test_inventory'], []) + result = self.run_stage('features-tests', **dict(env, RUNNER_OS='macOS')) + self.assertEqual(result.returncode, 0, result.stderr) + for system, shard in [('macOS', ''), ('macOS', 'wrong'), ('Linux', ''), ('Linux', 'wrong')]: result = self.run_stage('features-tests', **dict(env, RUNNER_OS=system, GATE_SHARD=shard)) self.assertNotEqual(result.returncode, 0) # A successful dispatcher without a new JUnit cannot reuse the preceding measurement. @@ -816,11 +827,30 @@ def test_isolated_wall_runs_only_after_complete_six_receipt_union(self): 'p=pathlib.Path(os.environ["CG_GATE_JUNIT"])\n' 'assert len(list(ET.parse(p).getroot().iter("testcase"))) == 3\n' 'print("complete measurement checked",sys.argv[1])\n') + isolated_audit = [['test', '--locked', '--manifest-path', 'scripts/gate-wall-audit/Cargo.toml', '--target-dir', str(self.root / 'gate-wall-audit-target'), '--test', 'test_estate_audit', 'gate_wall_seconds_never_exceed_the_frozen_baseline', '--', '--ignored', '--exact']] result = self.run_stage('shard-union-linux-isolated') self.assertEqual(result.returncode, 0, result.stdout + result.stderr) self.assertIn('Linux features: complete disjoint union', result.stdout) - self.assertEqual(self.commands(), [['test', '--locked', '--manifest-path', 'scripts/gate-wall-audit/Cargo.toml', '--target-dir', str(self.root / 'gate-wall-audit-target'), '--test', 'test_estate_audit', 'gate_wall_seconds_never_exceed_the_frozen_baseline', '--', '--ignored', '--exact']]) + self.assertEqual(self.commands(), isolated_audit) self.trace.unlink() + # The macOS route verifies only macOS receipts, so Linux receipts alone cannot satisfy it. + result = self.run_stage('shard-union-macos-isolated') + self.assertNotEqual(result.returncode, 0) + self.assertFalse(self.trace.exists()) + for directory in (path for path in receipts.iterdir() if path.is_dir()): + row = json.loads((directory / 'shard.json').read_text()); row['os'] = 'macOS' + (directory / 'shard.json').write_text(json.dumps(row)) + result = self.run_stage('shard-union-macos-isolated') + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertIn('macOS features: complete disjoint union', result.stdout) + self.assertEqual(self.commands(), isolated_audit) + self.trace.unlink() + result = self.run_stage('shard-union-linux-isolated') + self.assertNotEqual(result.returncode, 0) + self.assertFalse(self.trace.exists()) + for directory in (path for path in receipts.iterdir() if path.is_dir()): + row = json.loads((directory / 'shard.json').read_text()); row['os'] = 'Linux' + (directory / 'shard.json').write_text(json.dumps(row)) # A missing or failing sixth feature receipt must stop before any Rust audit. path = receipts / 'features-integration-and-rest/shard.json'; original = path.read_text() for mutation in ('missing', 'failed', 'wrong identity'): @@ -838,7 +868,7 @@ def hosted_receipt_program(self, workflow, job): if job in ('test', 'features'): variant = 'default' if job == 'test' else 'features' command = 'python3 scripts/ci-test-shards.py bundle ' + variant - if job == 'features' and workflow['name'] == 'ci': command += ' --tests-only' + if job == 'features': command += ' --tests-only' step = next(s for s in workflow['jobs'][job]['steps'] if s.get('run') == command) self.assertNotIn('if', step) self.assertNotIn('continue-on-error', step) @@ -1185,7 +1215,7 @@ def valid(document): check = workflow['jobs'][aggregate]['steps'][0]['run'] program = check.split("<<'PY_CHECK'\n", 1)[1].split('PY_CHECK', 1)[0] expected = set(workflow['jobs'][aggregate]['needs']) - self.assertEqual(expected, {'lint', 'controls', 'test', 'features'} | ({'source', 'docs', 'feature-qualification-core', 'feature-qualification-sync'} if aggregate == 'required' else set())) + self.assertEqual(expected, {'lint', 'controls', 'test', 'features', 'feature-qualification-core', 'feature-qualification-sync'} | ({'source', 'docs'} if aggregate == 'required' else set())) def succeeds(rows): return subprocess.run(['python3', '-I', '-c', program], env=dict(os.environ, NEEDS_JSON=json.dumps(rows)), capture_output=True).returncode == 0 rows = {name: {'result': 'success'} for name in expected} @@ -1247,5 +1277,58 @@ def admitted(selected): self.assertFalse(admitted(rows + [{'name': 'lint (macos-latest)', 'conclusion': 'success'}])) +def run_each_test_in_its_own_process(): + """Run every discovered test once, each in its own interpreter, several at a time. + + The suite is unchanged: every test class this module collects is enumerated, each test + runs in a fresh process with the same interpreter and environment, and the run fails + unless every enumerated test reports exactly one executed and passing test. + """ + import concurrent.futures + import sys + module = sys.modules[__name__] + names = [] + + def collect(suite): + for item in suite: + if isinstance(item, unittest.TestSuite): + collect(item) + else: + cls = type(item) + if getattr(module, cls.__name__, None) is not cls: + raise SystemExit('test class is not addressable from this module: ' + cls.__qualname__) + names.append(cls.__name__ + '.' + item._testMethodName) + + collect(unittest.defaultTestLoader.loadTestsFromModule(module)) + if not names or len(set(names)) != len(names): + raise SystemExit('empty or duplicate test inventory') + + def execute(name): + result = subprocess.run([sys.executable, str(pathlib.Path(__file__).resolve()), name], + capture_output=True, text=True) + # unittest reports on stderr; the test's own prints stay on stdout. + lines = [line for line in result.stderr.splitlines() if line.strip()] + summary = [line for line in lines if line.startswith('Ran ')] + passed = (result.returncode == 0 and bool(summary) and summary[-1].startswith('Ran 1 test in ') + and (lines[-1] == 'OK' or lines[-1].startswith('OK ('))) + return name, passed, result.stdout + result.stderr + + workers = max(2, os.cpu_count() or 2) + failures = [] + with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as pool: + for name, passed, output in pool.map(execute, names): + print(('ok ' if passed else 'FAIL ') + name, flush=True) + if not passed: + failures.append(name) + print(output, flush=True) + print(f'Ran {len(names)} tests in separate processes with {workers} workers; {len(failures)} failed', flush=True) + if failures: + raise SystemExit('failed tests: ' + ', '.join(failures)) + + if __name__ == '__main__': - unittest.main() + import sys + if len(sys.argv) > 1: + unittest.main() + else: + run_each_test_in_its_own_process() diff --git a/scripts/tests/test_ci_shards.py b/scripts/tests/test_ci_shards.py index 8156fa8..187be38 100644 --- a/scripts/tests/test_ci_shards.py +++ b/scripts/tests/test_ci_shards.py @@ -4,6 +4,7 @@ import importlib.util import json from pathlib import Path +import shutil import tempfile import subprocess import os @@ -251,12 +252,16 @@ def test_cache_cleanup_preserves_dependencies_and_does_not_follow_scratch_links( target = self.root / 'compiler-cache' (target / 'tests').mkdir(parents=True) (target / 'tests/case').write_text('scratch') + (target / 'tests/trybuild').mkdir() + (target / 'tests/trybuild/compiled').write_text('trybuild') (target / 'deps').mkdir() (target / 'deps/keep').write_text('dependency') command = ['python3', str(ROOT / 'scripts/ci-cache-post-cleanup.py')] subprocess.run(command, env=dict(os.environ, CARGO_TARGET_DIR=str(target)), check=True) - self.assertFalse((target / 'tests').exists()) + self.assertFalse((target / 'tests/case').exists()) + self.assertEqual((target / 'tests/trybuild/compiled').read_text(), 'trybuild') self.assertEqual((target / 'deps/keep').read_text(), 'dependency') + shutil.rmtree(target / 'tests') (target / 'tests').symlink_to(target / 'deps', target_is_directory=True) subprocess.run(command, env=dict(os.environ, CARGO_TARGET_DIR=str(target)), check=True) self.assertFalse((target / 'tests').is_symlink()) @@ -423,14 +428,18 @@ def test_catalog_seed_source_and_workflow_bindings(self): self.assertNotIn(hashlib.sha256(changed.encode()).hexdigest(), shared) for filename in ('ci.yml', 'macos-advisory.yml'): workflow = yaml.safe_load((ROOT / '.github/workflows' / filename).read_text()) - default_mode = ("${{ matrix.bundle == 'acceptance' && 'catalog-narrow' || 'catalog-workspace' }}" if filename == 'ci.yml' else - "${{ contains(matrix.bundle, 'integration-and-rest') && 'catalog-workspace' || 'catalog-narrow' }}") + default_mode = "${{ matrix.bundle == 'acceptance' && 'catalog-narrow' || 'catalog-workspace' }}" + # Shards that compile the identical selection share one compiler cache per platform: + # default acceptance alone compiles the narrow selection. + keys = {'test': "test-catalog-seed-v3-${{ matrix.bundle == 'acceptance' && 'narrow' || 'workspace' }}", + 'features': 'features-catalog-seed-v3-workspace'} for job, expected in [('test', default_mode), ('features', 'catalog-features')]: for stage in ('model-seed', 'public-model'): step = next(step for step in workflow['jobs'][job]['steps'] if step.get('run', '').strip() == 'bash scripts/gate.sh ' + stage) self.assertEqual(step['env']['GATE_CI_MODEL_MODE'], expected) cache = next(step for step in workflow['jobs'][job]['steps'] if step.get('uses', '').startswith('Swatinem/rust-cache@')) - self.assertEqual(cache['with']['key'], job + '-catalog-seed-v2-${{ matrix.bundle }}') + self.assertEqual(cache['with']['key'], keys[job]) + self.assertIs(cache['with']['cache-on-failure'], True) def test_singleton_routing_and_platform_mutations_are_rejected(self): """Every singleton retains its own setup evidence and full platform commands.""" @@ -468,38 +477,40 @@ def test_singleton_routing_and_platform_mutations_are_rejected(self): else: step['env']['GATE_CI_MODEL_MODE'] = "${{ matrix.bundle == '" + shard + "' && 'catalog-narrow' || 'catalog-features' }}" def load(text): parsed = parser(text) - if isinstance(parsed, dict) and parsed.get('name') == 'ci': return changed['ci.yml'] - if isinstance(parsed, dict) and parsed.get('name') == 'macOS advisory': return changed['macos-advisory.yml'] + if isinstance(parsed, dict) and parsed.get('name') == 'Linux CI': return changed['ci.yml'] + if isinstance(parsed, dict) and parsed.get('name') == 'macOS CI (advisory)': return changed['macos-advisory.yml'] return parsed with mock.patch.object(yaml, 'safe_load', side_effect=load), self.assertRaises(AssertionError): self.test_catalog_seed_source_and_workflow_bindings() self.test_workflow_covers_both_variants_and_platforms_before_aggregate() - def test_tests_only_requires_linux_feature_singleton(self): - """Only explicit feature singleton jobs may select the tests-only stage.""" + def test_tests_only_requires_feature_singleton(self): + """Only explicit feature singleton jobs may select the tests-only stage, on either platform.""" for variant, system, bundle in [('default', 'Linux', 'acceptance'), - ('features', 'macOS', 'acceptance'), + ('default', 'macOS', 'acceptance'), ('features', 'Linux', 'integration-and-rest + import-and-cli'), + ('features', 'macOS', 'integration-and-rest + import-and-cli'), ('features', 'macOS', 'integration-and-rest + import-and-cli + acceptance')]: with self.subTest(variant=variant, system=system, bundle=bundle), mock.patch.dict(os.environ, dict(RUNNER_OS=system, GATE_PHYSICAL_BUNDLE=bundle)), mock.patch.object(shards.subprocess, 'run') as runner: with self.assertRaises(ValueError): shards.bundle(variant, tests_only=True) runner.assert_not_called() - with tempfile.TemporaryDirectory() as temporary, mock.patch.dict(os.environ, - dict(RUNNER_OS='Linux', GATE_PHYSICAL_BUNDLE='acceptance', RUNNER_TEMP=temporary)), \ - mock.patch.object(shards.subprocess, 'run', return_value=mock.Mock(returncode=0)) as runner, \ - mock.patch.object(shards, 'workflow_receipt') as receipt: - shards.bundle('features', tests_only=True) - self.assertEqual(runner.call_args.args[0], ['bash', 'scripts/gate.sh', 'features-tests']) - self.assertEqual(runner.call_args.kwargs['env']['GATE_SHARD'], 'acceptance') - receipt.assert_called_once_with(Path(temporary) / 'job-receipt/acceptance', 'features', 0) + for system in ('Linux', 'macOS'): + with self.subTest(system=system), tempfile.TemporaryDirectory() as temporary, mock.patch.dict(os.environ, + dict(RUNNER_OS=system, GATE_PHYSICAL_BUNDLE='acceptance', RUNNER_TEMP=temporary)), \ + mock.patch.object(shards.subprocess, 'run', return_value=mock.Mock(returncode=0)) as runner, \ + mock.patch.object(shards, 'workflow_receipt') as receipt: + shards.bundle('features', tests_only=True) + self.assertEqual(runner.call_args.args[0], ['bash', 'scripts/gate.sh', 'features-tests']) + self.assertEqual(runner.call_args.kwargs['env']['GATE_SHARD'], 'acceptance') + receipt.assert_called_once_with(Path(temporary) / 'job-receipt/acceptance', 'features', 0) def test_compile_jobs_are_exact_required_source_bound_commands(self): """Both compatibility jobs retain their complete reviewed execution and admission mapping.""" import hashlib - pins = {'feature-qualification-core': '1c9d7ae7fa1ff4bc553de26b01d99c976729e4cdaea18fda5da48ccc3967a06c', 'feature-qualification-sync': 'a483f8e5d5c5ae4bd1a1c9df60ef2111b693ba5272ba9b2f51f73417090b317f'} + pins = {'feature-qualification-core': '14a7f84756b8f77e292d4db534418f58e954124e34e2180e659ca04ff6a6f665', 'feature-qualification-sync': 'fc7523f5837733e7f82635d4921a6ad3ed682ef46d05461367245983f5adf280'} def valid(workflow): for name, digest in pins.items(): if hashlib.sha256(json.dumps(workflow['jobs'].get(name), sort_keys=True, separators=(',', ':'), ensure_ascii=False).encode()).hexdigest() != digest: @@ -542,7 +553,7 @@ def test_isolated_audit_manifest_lock_and_source_boundary(self): lock_digest = 'e6a98ee802ad0f1e9e3696f87daa40c6b7a78c6bb9424f6472f1afd4a798a55f' def valid(body, lock_bytes, root_manifest): return (body == expected and hashlib.sha256(lock_bytes).hexdigest() == lock_digest - and hashlib.sha256(root_manifest).hexdigest() == 'ab995589c0159a218d0fd370a539c2994b6008e7e64b0d3e171062e6e0b11e85') + and hashlib.sha256(root_manifest).hexdigest() == '871a8c130d3dbc7e88d4da615dafdd83ec25cfac85b170cd75f673a0a4d376d7') self.assertTrue(valid(expected, lock.read_bytes(), (ROOT / 'Cargo.toml').read_bytes())) for key, value in [('edition', '2021'), ('autotests', True), ('publish', True)]: changed = copy.deepcopy(expected); changed['package'][key] = value @@ -570,27 +581,69 @@ def valid(body, lock_bytes, root_manifest): for path in (manifest, lock, ROOT / 'Cargo.toml', ROOT / 'Cargo.lock', ROOT / 'crates/context-graph/tests/test_estate_audit.rs'): self.assertIn(hashlib.sha256(path.read_bytes()).hexdigest(), helper) + def test_macos_workflow_runs_the_linux_jobs_on_macos(self): + """macOS runs every Linux correctness job with identical steps; only the runner differs.""" + import copy + def mirrored(linux, mac): + def as_mac(job): + text = json.dumps(job, sort_keys=True) + for old, new in (('"os": "ubuntu-latest"', '"os": "macos-latest"'), ('"os": ["ubuntu-latest"]', '"os": ["macos-latest"]'), + ('"runs-on": "ubuntu-latest"', '"runs-on": "macos-latest"'), ('(ubuntu-latest)', '(macos-latest)')): + text = text.replace(old, new) + return json.loads(text) + jobs = ('lint', 'controls', 'test', 'features', 'feature-qualification-core', 'feature-qualification-sync') + if set(mac['jobs']) != set(jobs) | {'macos-advisory'}: + return False + if any('ubuntu-latest' in json.dumps(mac['jobs'][name]) or as_mac(linux['jobs'][name]) != mac['jobs'][name] for name in jobs): + return False + aggregate = copy.deepcopy(linux['jobs']['required']) + aggregate['name'] = 'macOS advisory results' + aggregate['needs'] = [name for name in aggregate['needs'] if name not in ('source', 'docs')] + check = aggregate['steps'][0]['run'] + aggregate['steps'][0]['run'] = check.replace("{'docs', 'source', 'features',", "{'features',") + union = next(step for step in aggregate['steps'] if step.get('run') == 'bash scripts/gate.sh shard-union-linux-isolated') + union['run'] = 'bash scripts/gate.sh shard-union-macos-isolated' + return (aggregate['steps'][0]['run'] != check and aggregate == mac['jobs']['macos-advisory'] + and all(linux[key] == mac[key] for key in (True, 'permissions', 'env', 'defaults'))) # YAML reads `on` as True + linux = yaml.safe_load((ROOT / '.github/workflows/ci.yml').read_text()) + mac = yaml.safe_load((ROOT / '.github/workflows/macos-advisory.yml').read_text()) + self.assertTrue(mirrored(linux, mac)) + for name in ('lint', 'controls', 'test', 'features', 'feature-qualification-core', 'feature-qualification-sync'): + with self.subTest(job=name): + changed = copy.deepcopy(mac); del changed['jobs'][name] + self.assertFalse(mirrored(linux, changed)) + changed = copy.deepcopy(mac); changed['jobs'][name]['steps'].pop() + self.assertFalse(mirrored(linux, changed)) + changed = copy.deepcopy(mac); changed['jobs'][name]['timeout-minutes'] = 5 + self.assertFalse(mirrored(linux, changed)) + for mutation in ('ubuntu row', 'combined bundle', 'full features', 'union', 'needs'): + with self.subTest(mutation=mutation): + changed = copy.deepcopy(mac) + if mutation == 'ubuntu row': changed['jobs']['test']['strategy']['matrix']['include'][0]['os'] = 'ubuntu-latest' + elif mutation == 'combined bundle': changed['jobs']['test']['strategy']['matrix']['include'][1]['bundle'] = 'integration-and-rest + import-and-cli' + elif mutation == 'full features': + step = next(step for step in changed['jobs']['features']['steps'] if step.get('run', '').startswith('python3 scripts/ci-test-shards.py')) + step['run'] = 'python3 scripts/ci-test-shards.py bundle features' + elif mutation == 'union': + step = next(step for step in changed['jobs']['macos-advisory']['steps'] if 'shard-union' in step.get('run', '')) + step['run'] = 'bash scripts/gate.sh shard-union-linux-isolated' + else: changed['jobs']['macos-advisory']['needs'].remove('feature-qualification-sync') + self.assertFalse(mirrored(linux, changed)) + def test_workflow_covers_both_variants_and_platforms_before_aggregate(self): for filename, system in [('ci.yml', 'ubuntu-latest'), ('macos-advisory.yml', 'macos-latest')]: workflow = yaml.safe_load((ROOT / '.github/workflows' / filename).read_text()) for job in ('test', 'features'): body = workflow['jobs'][job] rows = body['strategy']['matrix']['include'] - expected = ([ - {'os': 'macos-latest', 'bundle': 'acceptance'}, - {'os': 'macos-latest', 'bundle': 'integration-and-rest + import-and-cli'}, - ] if job == 'test' else [ - {'os': 'macos-latest', 'bundle': 'integration-and-rest + import-and-cli + acceptance'}, - ]) + [ - {'os': 'ubuntu-latest', 'bundle': 'acceptance'}, - {'os': 'ubuntu-latest', 'bundle': 'integration-and-rest'}, - {'os': 'ubuntu-latest', 'bundle': 'import-and-cli'}, - ] - self.assertEqual(rows, [row for row in expected if row['os'] == system]) + # Each platform runs every logical shard as its own job. + expected = [{'os': system, 'bundle': bundle} + for bundle in ('acceptance', 'integration-and-rest', 'import-and-cli')] + self.assertEqual(rows, expected) self.assertEqual(body['env']['GATE_PHYSICAL_BUNDLE'], '${{ matrix.bundle }}') self.assertFalse(body['strategy']['fail-fast']) command = f'python3 scripts/ci-test-shards.py bundle {"default" if job == "test" else job}' - if job == 'features' and system == 'ubuntu-latest': command += ' --tests-only' + if job == 'features': command += ' --tests-only' runs = [step for step in body['steps'] if step.get('run') == command] self.assertEqual(len(runs), 1) self.assertNotIn('if', runs[0]) @@ -598,9 +651,7 @@ def test_workflow_covers_both_variants_and_platforms_before_aggregate(self): for stage in ('source', 'model-seed', 'public-model'): self.assertEqual(sum(step.get('run', '').strip() == f'bash scripts/gate.sh {stage}' for step in body['steps']), 1) step = next(step for step in body['steps'] if step.get('run', '').strip() == f'bash scripts/gate.sh {stage}') - self.assertEqual(step['env']['GATE_RECEIPT_DIR'], - '${{ runner.temp }}/job-receipt/${{ matrix.bundle }}' if system == 'ubuntu-latest' else - "${{ runner.temp }}/job-receipt/${{ contains(matrix.bundle, 'integration-and-rest') && 'integration-and-rest' || 'acceptance' }}") + self.assertEqual(step['env']['GATE_RECEIPT_DIR'], '${{ runner.temp }}/job-receipt/${{ matrix.bundle }}') uploads = [step for step in body['steps'] if step.get('uses', '').startswith('actions/upload-artifact@')] self.assertEqual(len(uploads), 3) for shard, upload in zip(('integration-and-rest', 'import-and-cli', 'acceptance'), uploads):