feat(ae): /dx/evidence_manifest ingest → forensic_db.dx_evidence_manifest by ConstanzeTU · Pull Request #78 · k8sstormcenter/pixie

ConstanzeTU · 2026-07-02T14:10:13Z

Stacked on #77 (dx graph rename) → #73 → #68. Closes the AE-side wire gap for dx's §9 completeness contract. Full 'what's included' + TTL test image posted as a comment below once the image is pushed.

…manifest Closes the AE-side wire gap for dx's §9 completeness contract (stacked on the dx_evidence_graph rename). dx already POSTs manifests via aeclient.WriteEvidenceManifest → /dx/evidence_manifest; aeprod26 404s that path, so manifest_rows_exported=0 / graph_write_failures{kind=manifest}>0. Adds, mirroring the evidence_graph path: - control: manifestWriter iface + SetManifestWriter + POST /dx/evidence_manifest handler. Accepts one manifest.Manifest per verdict; scalars map to typed columns, nested collections (case_window/findings/orders/seeds/chain) are rendered as JSON text so the JSONEachRow insert is CH-version independent. - clickhouse: forensic_db.dx_evidence_manifest table (columns = manifest.Manifest JSON tags; event_time nanos + hostname read-path like dx_evidence_graph), WriteEvidenceManifest sink, KnownTables + OperatorOwnedTables (created on boot). - main: wire SetManifestWriter(applier). - tests: endpoint 501/202/502 + nested-as-text flattening; table-set guard.

coderabbitai · 2026-07-02T14:10:22Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 988c93ed-c43d-47c2-a5d9-37dd3cf2ce62

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feat/ae-evidence-manifest

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

ConstanzeTU · 2026-07-02T14:11:30Z

🧪 TTL test image for the dx-agent

ttl.sh/px-51d5e18d5/vizier-adaptive_export_image:1d

digest sha256:e3bc91aa75d3ad8a4921fbd6a4dc12db29606f8c6a5a55220528a8731d89732c · AE image only, built from 51d5e18d5 (this branch), TTL 24h. Patch just the AE deployment:

kubectl set image ds/<ae-daemonset> adaptive_export=ttl.sh/px-51d5e18d5/vizier-adaptive_export_image:1d

Exactly what's in this AE

New endpoint — POST /dx/evidence_manifest (mirrors /dx/evidence_graph):

accepts one manifest.Manifest per call (the object aeclient.WriteEvidenceManifest already sends),
202 on success · 501 if the sink is unwired · 400 bad JSON · 502 on CH write failure,
behind the same bearer-JWT auth as the rest of the control surface.

New table — forensic_db.dx_evidence_manifest, created on boot (it's in OperatorOwnedTables, so no manual DDL). Columns are the manifest.Manifest JSON tags:

column	type	notes
`investigation_id`	String
`event_time`	UInt64	unix nanoseconds (same read-path convention as `dx_evidence_graph`)
`hostname`	String
`condition`	String
`verdict`	String	`ruled_in` \| `metastasis`
`confidence`	Float64
`posterior`	Float64
`catalog_version`	String
`case_window`	String	JSON text (`[2]float64`)
`findings`	String	JSON text (`[]FindingRef`)
`orders`	String	JSON text (`[]Order`)
`seeds`	String	JSON text (`[]Seed`)
`chain`	String	JSON text (`*Chain`, `""` when absent)
`evidence_hash`	String

Nested-as-text: the handler pre-renders case_window/findings/orders/seeds/chain to JSON strings before the INSERT … FORMAT JSONEachRow, so the write is ClickHouse-version independent (no reliance on input_format_json_read_objects_as_strings). Scalars stay typed; event_time round-trips as an integer literal (no sci-notation).

Engine: MergeTree ORDER BY (event_time, hostname), monthly partitions via fromUnixTimestamp64Nano, 30-day TTL — px-readable, same as dx_evidence_graph.

Quick verify

POST /dx/evidence_manifest → expect 202 (was 404 on aeprod26).
SELECT investigation_id, verdict, findings FROM forensic_db.dx_evidence_manifest → your consulted rows land; manifest_rows_exported_total should climb and graph_write_failures{kind=manifest} stop.

A full vizier_release (aeprod27) with this is being cut too; I'll post that tag here when it lands.

ConstanzeTU · 2026-07-02T15:02:33Z

✅ Full release: `aeprod27`

The complete vizier image set (with the /dx/evidence_manifest change) is now in GHCR under 0.14.19-aeprod27:

release/vizier/v0.14.19-aeprod27
ghcr.io/k8sstormcenter/vizier-adaptive_export_image:0.14.19-aeprod27
  digest sha256:8b78d60ba06e650627e25a3ff857a02a16ac8d51cd7d98f0a944058c2814de67

query_broker, pem, and the rest of the set are published at the same tag — use this for a full roll (vs. the ttl.sh image which patches only AE). Built from 51d5e18d5 (this branch tip).

Same behavior as the TTL image: POST /dx/evidence_manifest → 202, rows land in forensic_db.dx_evidence_manifest.

entlein · 2026-07-17T15:50:17Z

Superseded by #85 — the three PRs (#73/#77/#78) were a linear one-commit-each stack on the pre-squash ae-followup-auth base (now inside main's #68 squash). #85 is their combined content cherry-picked cleanly onto current main, to be reviewed/tested as one PR. Keeping this open for reference; close once #85 lands.

entlein · 2026-07-17T15:59:02Z

Correction: folded into #73 (not #85 — that was closed). #73 now carries the combined content on main.

…manifest (folds #77/#78) (#73) * feat(adaptive_export): wire the /query runner — dx OrderQuery → forensic capture (dx#93) The control surface shipped with control.New(activeSet, nil) — '/query runner wired later'. So every dx OrderQuery 501'd and the table dx read for a verdict (e.g. the jndi-in-http) never reached forensic_db unless a kubescape-anomaly window happened to push it. The forensic export steered to noise, not to dx's evidence. Adds Controller.OrderQuery(target, table, window, queryID): a single-shot QueryFor → querier.Query → sink.WritePixieRows (the same path pushPixieRows uses, plus globalSem + reconcile accounting), independent of any anomaly window. Wires control.New(activeSet, ctl) in main.go so POST /query executes it. When the operator-side querier is disabled, OrderQuery errors and /query 502s; start/stop + dx_attack_graph are unaffected. Pairs with entlein/dx#96 which orders the full consulted read-set through this runner. Tests: writes-rows / no-querier-errors / empty-no-write / sink-error. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(ae): rename dx_attack_graph → dx_evidence_graph (malignant view) Restacks the dx graph rename onto the #73 /query-runner tip. Renames across the AE forensic path: - table forensic_db.dx_attack_graph → dx_evidence_graph - view dx_attack_graph_malicious → dx_evidence_graph_malignant (medical terminology: malignant, not malicious) - endpoint /dx/attack_graph → /dx/evidence_graph - Go WriteAttackGraph/handleDXAttackGraph → *EvidenceGraph Done mechanically on #73 so it also covers order_query_test.go (absent from the original rename branch's lineage). Also fixes two pre-existing #73 lint nits surfaced by touching the package: gofumpt blank lines in order_query_test.go and the missing order_query_test.go entry in controller/BUILD.bazel's controller_test srcs (gazelle). * feat(ae): add /dx/evidence_manifest ingest → forensic_db.dx_evidence_manifest Closes the AE-side wire gap for dx's §9 completeness contract (stacked on the dx_evidence_graph rename). dx already POSTs manifests via aeclient.WriteEvidenceManifest → /dx/evidence_manifest; aeprod26 404s that path, so manifest_rows_exported=0 / graph_write_failures{kind=manifest}>0. Adds, mirroring the evidence_graph path: - control: manifestWriter iface + SetManifestWriter + POST /dx/evidence_manifest handler. Accepts one manifest.Manifest per verdict; scalars map to typed columns, nested collections (case_window/findings/orders/seeds/chain) are rendered as JSON text so the JSONEachRow insert is CH-version independent. - clickhouse: forensic_db.dx_evidence_manifest table (columns = manifest.Manifest JSON tags; event_time nanos + hostname read-path like dx_evidence_graph), WriteEvidenceManifest sink, KnownTables + OperatorOwnedTables (created on boot). - main: wire SetManifestWriter(applier). - tests: endpoint 501/202/502 + nested-as-text flattening; table-set guard. * lint(#73): fix arc-lint failures + copy.bara cleanup - .arclint: exclude the adaptive_export_loadtest/suite/ nested go module (module 'aeloadsuite') from linters — golangci-lint typecheck failed on it from the root module, same as the already-excluded tools/loadgen/. - skaffold.yaml (loadtest): reindent the requires: block to indent-sequences:false (yamllint: expected 0/4, was 2/6). yamllint clean now. - e2e_calibrate_soc.yaml: strip trailing whitespace (arc autofix). - copy.bara.sky: drop the malformed duplicate line (md_udtfs/** + shared/manager BUILD.bazel concatenated behind a comment; both already listed above it). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017k7uYSNUctQvkTAZYJbaB3 * genfiles(#73): .bazelignore the aeloadsuite nested module run-genfiles failed because gazelle generated src/e2e_test/adaptive_export_loadtest/ suite/BUILD.bazel (untracked → dirty tree). suite/ is a separate go module ('aeloadsuite'), not part of the root px.dev/pixie build — gazelle's generated importpath/deps for it are wrong. Ignore it like the sibling tools/loadgen module (already in .bazelignore) so gazelle skips it. Verified: gazelle fix leaves the tree clean afterward. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017k7uYSNUctQvkTAZYJbaB3 --------- Co-authored-by: Entlein <eineintlein@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: pixie-agent <croedig@sba-research.org>

entlein mentioned this pull request Jul 17, 2026

refactor(ae): rename dx_attack_graph → dx_evidence_graph (malignant view) #77

Closed

entlein closed this Jul 18, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(ae): /dx/evidence_manifest ingest → forensic_db.dx_evidence_manifest#78

feat(ae): /dx/evidence_manifest ingest → forensic_db.dx_evidence_manifest#78
ConstanzeTU wants to merge 1 commit into
refactor/ae-evidence-graph-renamefrom
feat/ae-evidence-manifest

ConstanzeTU commented Jul 2, 2026

Uh oh!

coderabbitai Bot commented Jul 2, 2026

Review skipped

Uh oh!

ConstanzeTU commented Jul 2, 2026

Uh oh!

ConstanzeTU commented Jul 2, 2026

Uh oh!

entlein commented Jul 17, 2026

Uh oh!

entlein commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ConstanzeTU commented Jul 2, 2026

Uh oh!

coderabbitai Bot commented Jul 2, 2026

Review skipped

Uh oh!

ConstanzeTU commented Jul 2, 2026

🧪 TTL test image for the dx-agent

Exactly what's in this AE

Quick verify

Uh oh!

ConstanzeTU commented Jul 2, 2026

✅ Full release: aeprod27

Uh oh!

entlein commented Jul 17, 2026

Uh oh!

entlein commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

✅ Full release: `aeprod27`