Skip to content

CI: improve macOS test portability and diagnostics #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
lucidprogrammer wants to merge 1 commit into from
Closed

CI: improve macOS test portability and diagnostics #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 43 additions & 9 deletions .github/CI.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,13 +22,19 @@ contributor runs the same commands locally.

| Path | Workflow | Trigger | What a result means |
|---|---|---|---|
| PR checks | [`ci.yml`](workflows/ci.yml) (`ci`) | every push and every pull request; manual dispatch with exact `cg_sha` and `contextdb_sha` | Correctness of this exact commit. The aggregate job **`PR checks`** passes only if every required Linux, source and documentation job succeeded. |
| macOS advisory | [`macos-advisory.yml`](workflows/macos-advisory.yml) (`macOS advisory`) | same push, pull request and exact-SHA dispatch triggers | Full macOS coverage with a failing aggregate on failure; nonblocking for Linux and delivery. |
| PR checks | [`ci.yml`](workflows/ci.yml) (`Linux CI`) | pushes to `dev` and every pull request; manual dispatch with exact `cg_sha` and `contextdb_sha` | Correctness of this exact commit. The aggregate job **`PR checks`** passes only if every required Linux, source and documentation job succeeded. |
| macOS advisory | [`macos-advisory.yml`](workflows/macos-advisory.yml) (`macOS CI (advisory)`) | pushes to `dev`, every pull request and exact-SHA dispatch | Full macOS coverage with a failing aggregate on failure; nonblocking for Linux and delivery. |
| Extended integration | [`extended-integration.yml`](workflows/extended-integration.yml) (`extended integration`) | callable; Mondays 03:23 UTC; manual dispatch with exact SHAs | Public Nomic model and event-search discovery on Linux and macOS (aggregate `extended-required`). A manual dispatch with `trusted: true` additionally runs the protected trusted jobs (aggregate `trusted-qualification`). |
| Performance measurement | [`performance.yml`](workflows/performance.yml) (`Performance measurement (manual guide)`) | Never — dormant placeholder | **Manual today.** The workflow cannot start: its only trigger is a push filter that ignores every branch and every tag, and manual dispatch is commented out. It checks out nothing, runs no benchmark and never reports a status. The manual journey is [`docs/performance/README.md`](../docs/performance/README.md). |
| Development downloads | [`build.yml`](workflows/build.yml) (`Build development binaries`) | called after PR checks for each push to dev; maintainer dispatch on dev | Packages and smokes exact-source binaries, then publishes the rolling development prerelease. |
| Publication preparation | [`publication.yml`](workflows/publication.yml) (`publication preparation`) | manual dispatch with exact SHAs | Locked optimized CLI build and local package inspection (aggregate `publication-receipt`). It publishes nothing, creates no tag and performs no crates.io action. |

Automatic pushes run only on `dev`, preserving development publication. Candidate
branches receive both platform workflows through pull requests or exact-SHA manual
dispatch, without duplicate push runs. Display-name changes apply to future runs;
existing runs retain their recorded names. Job names, `PR checks`, and
`macOS advisory results` are unchanged.

## PR checks (`ci.yml`)

Every job first validates the requested ContextDB SHA, checks out the selected
Expand Down Expand Up @@ -80,6 +86,29 @@ debug assertions and overflow checks enabled. Local Cargo profile defaults stay
specified in `Cargo.toml`. Linux jobs reclaim preinstalled SDKs with sudo and print
root disk headroom before and after linking. Disposable `target/tests` trybuild output
is removed before compiler cache post hooks, including on failure.

The macOS default acceptance bundle temporarily records the digest immutability
compile proof's inner Cargo boundaries under its existing receipt artifact's
`trybuild-diagnostics/` directory. A native nextest runner sets `CARGO` only for
`crud::digest_immutable_no_update_symbol`; other cases execute with their original
Cargo environment. The wrapper forwards to the pinned toolchain's Cargo without
changing arguments, compiler flags, output streams, test selection or timeout.
Generated manifests and locks, selected compiler fingerprint fields and original
fingerprint hashes are saved before inner cleanup and after execution. Fingerprint
environment entries are excluded. A start without an end records interrupted work;
it is never a successful command receipt. Scratch cleanup still runs afterward.
The complete Python controls entrypoint also runs the wrapper and diagnostic admission
controls. The diagnostic source closure binds the dispatcher, executable helper,
controls entrypoint and wrapper controls separately from the seven model sources.
`trybuild-diagnostics/completeness.json` records `accepted`, `incomplete` or `fail`
without replacing the original nextest exit status or JUnit. Admission requires matching
Cargo, rustc, nextest, Python, helper and test executable hashes; consecutive writer receipts;
actual arguments, flag/cfg state and input snapshots; matched start/spawn/end boundaries;
and a successful final capture. Check commands require generated manifest, lock and
compiler fingerprints. Earlier metadata commands record absent generated inputs explicitly.
A single interrupted inner command requires a matching digest-case JUnit timeout to
identify the last active command; it remains an incomplete trace with causal admission
refused. Missing writes, identities, snapshots or helper byte changes fail admission.
These changes require hosted evidence to establish end-to-end completion under
30 minutes; queue time, cold model downloads, repeated workspace compilation and the
aggregate audit build remain part of that measurement.
Expand Down Expand Up @@ -296,7 +325,7 @@ unit-reuse evidence and complete cold/warm hosted measurements.

## macOS qualification follow-up

Linux is the current required qualification platform. The separate `macOS advisory`
Linux is the current required qualification platform. The separate `macOS CI (advisory)`
workflow preserves macOS lint, default tests, feature tests and complete Python controls:
five physical jobs (two default, one feature, lint and controls) produce six logical
Rust receipts. Its `macOS advisory results` aggregate fails on failed, cancelled,
Expand All @@ -306,12 +335,17 @@ The legacy local `shard-union` stage still requires twelve receipts; explicit Li
and macOS stages each require exactly six, with no fallback to available platforms.

Qualification remains pending: seven macOS test failures remain unresolved.
The default owner-service locator also remains broken: the native user temporary
root plus the full socket key can exceed macOS's 103-byte socket pathname limit.
A writable database open can succeed while the separate owner service fails to start.
A repair and independent native owner/reader proof are still pending; prior reader
authentication results do not prove the default locator. These are open follow-up
items, not passing evidence. Completion requires identifying and fixing each failure in source, verifying
The pinned ContextDB macOS default locator now canonicalizes the fixed system
temporary root `/tmp` and uses a private per-effective-UID directory with the full
socket key. Focused upstream engine coverage consists of five integration cases
actually run on the pinned revision and three unchanged library cases carried
from an older revision, not eight newly executed cases. The only reader
authentication proof is from a separate earlier ContextDB candidate, not the
pinned revision; prior reader authentication results do not prove the pinned
default locator or full Context Graph qualification. Full Context Graph macOS
consumer qualification, including the integrated default owner/reader locator,
remains pending. These are open follow-up items, not passing evidence.
Completion requires identifying and fixing each failure in source, verifying
locator behavior, running all five macOS jobs with all six valid receipts on the exact
integrated source, and reviewing the resulting evidence before changing qualification.
No issue number or hosted green result is claimed.
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
name: ci
name: Linux CI

on:
pull_request:
push:
branches: [dev]
workflow_dispatch:
inputs:
cg_sha:
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/macos-advisory.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
name: macOS advisory
name: macOS CI (advisory)

on:
pull_request:
push:
branches: [dev]
workflow_dispatch:
inputs:
cg_sha:
Expand Down
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ Without cargo-nextest, the script runs workspace tests but skips the two checks

### Continuous integration

Required PR test and feature jobs run on `ubuntu-latest`; the separate `macOS advisory` workflow retains the same coverage on `macos-latest`.
The `Linux CI` and `macOS CI (advisory)` workflows run on pushes to `dev`, every pull request and exact-SHA manual dispatch. Candidate branches use pull requests or manual dispatch for validation. Required PR test and feature jobs run on `ubuntu-latest`; the separate `macOS CI (advisory)` workflow retains the same coverage on `macos-latest`.
`windows-latest` is excluded deliberately because the ContextDB sync substrate requires Unix descriptor-relative filesystem operations.
The workflows install the repository's Rust 1.94.1 toolchain explicitly, use immutable action pins, and verify the selected Context Graph head and locked ContextDB revision. Base, head and event SHAs are recorded separately. Fork PR jobs use public inputs and no provider secrets.

Expand All @@ -86,7 +86,7 @@ A cold miss fetches the default model; the document, query and event-path Nomic
| Performance measurement | Manual; dormant `performance.yml` never runs; [performance guide](docs/performance/README.md) | Manual-first: build the exact baseline and candidate on one controlled host, keep raw evidence private, publish only a reviewed summary; nothing hosted measures performance |
| Publication preparation | Manual `publication.yml`; `bash scripts/gate.sh publication-check` | Locked optimized CLI build and local package inspection; no publishing, tags or crates.io action |

The PR source job always records `qualification.json` and a step summary bound to the checked source and locked dependency SHA. `scripts/ci-impact.py` contains path-map version 1: Markdown under `docs/` and `skills/` and the named root documentation/license files are unaffected; runtime, build, scripts, tests and CI paths require performance and trusted integration qualification. Unknown paths, unreadable or empty diffs and initial parentless roots require both classes. All Linux checks remain required. The same macOS commands and full controls run in the separate, nonblocking `macOS advisory` workflow; failures remain visible. See the [macOS qualification follow-up](.github/CI.md#macos-qualification-follow-up). Affected changes report `qualification: pending maintainer native/trusted run`; public check success does not imply native or trusted qualification. Fork PRs use no secrets and never run these privileged workloads. For performance, the pending label means a manual measurement may be wanted; see the [performance guide](docs/performance/README.md).
The PR source job always records `qualification.json` and a step summary bound to the checked source and locked dependency SHA. `scripts/ci-impact.py` contains path-map version 1: Markdown under `docs/` and `skills/` and the named root documentation/license files are unaffected; runtime, build, scripts, tests and CI paths require performance and trusted integration qualification. Unknown paths, unreadable or empty diffs and initial parentless roots require both classes. All Linux checks remain required. The same macOS commands and full controls run in the separate, nonblocking `macOS CI (advisory)` workflow; failures remain visible. See the [macOS qualification follow-up](.github/CI.md#macos-qualification-follow-up). Affected changes report `qualification: pending maintainer native/trusted run`; public check success does not imply native or trusted qualification. Fork PRs use no secrets and never run these privileged workloads. For performance, the pending label means a manual measurement may be wanted; see the [performance guide](docs/performance/README.md).

A maintainer can explicitly supply `--qualification-receipt <path>` when running `python3 scripts/ci-impact.py --base <sha> --output <path>`. Receipt schema 1 requires the exact `cg_sha` and `contextdb_sha`, `result: "qualified"`, the ordered `qualified_classes` array (`performance`, `trusted-integration`) and an `evidence_sha256` digest for each class. Mismatched or incomplete receipts fail and retain the pending status. Scheduled results are never discovered or accepted implicitly.

Expand Down
22 changes: 11 additions & 11 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 5 additions & 5 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,11 +44,11 @@ candle-core = { path = "vendor/candle-core", version = "=0.10.2" }
candle-nn = { path = "vendor/candle-nn", version = "=0.10.2" }
candle-transformers = { path = "vendor/candle-transformers", version = "=0.10.2" }
criterion = { version = "0.8", features = ["html_reports"] }
contextdb-core = { git = "https://github.com/context-graph-ai/contextdb", rev = "bc9cce955eb85fb1488f5304426eedeb6d97e905" }
contextdb-engine = { git = "https://github.com/context-graph-ai/contextdb", rev = "bc9cce955eb85fb1488f5304426eedeb6d97e905" }
contextdb-parser = { git = "https://github.com/context-graph-ai/contextdb", rev = "bc9cce955eb85fb1488f5304426eedeb6d97e905" }
contextdb-tx = { git = "https://github.com/context-graph-ai/contextdb", rev = "bc9cce955eb85fb1488f5304426eedeb6d97e905" }
contextdb-vector = { git = "https://github.com/context-graph-ai/contextdb", rev = "bc9cce955eb85fb1488f5304426eedeb6d97e905" }
contextdb-core = { git = "https://github.com/context-graph-ai/contextdb", rev = "a883cb73b787958be20ef1e712ffbfd26093f587" }
contextdb-engine = { git = "https://github.com/context-graph-ai/contextdb", rev = "a883cb73b787958be20ef1e712ffbfd26093f587" }
contextdb-parser = { git = "https://github.com/context-graph-ai/contextdb", rev = "a883cb73b787958be20ef1e712ffbfd26093f587" }
contextdb-tx = { git = "https://github.com/context-graph-ai/contextdb", rev = "a883cb73b787958be20ef1e712ffbfd26093f587" }
contextdb-vector = { git = "https://github.com/context-graph-ai/contextdb", rev = "a883cb73b787958be20ef1e712ffbfd26093f587" }
safetensors = "0.7.0"
sha2 = "0.10.9"
tempfile = "3.23.0"
Expand Down
Loading
Loading