Skip to content

fix: preserve native model attribution across all harnesses - #487

Open
sox8502 wants to merge 12 commits into
mainfrom
daniel/model-attribution-coverage
Open

sox8502 wants to merge 12 commits into
mainfrom
daniel/model-attribution-coverage

Conversation

@sox8502

@sox8502 sox8502 commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Successful harness executions could lose their model identity when token samples were absent or children finished later. Capture each harness's native model evidence independently of tokens and carry it into chat and terminal experiment reports. Auto routing, not executed, manual launches, and unresolved identity remain explicit outcomes.

This revision replaces the earlier large implementation with the existing usage ledger, native thread/turn IDs, native message databases, hooks, and outbox. OpenCode V1/V2 share one database reader. Removed custom parent holds, broad ownership reconciliation, command matching, and duplicated harness/test machinery. No new dependency or UI changes.

Companion service: https://github.com/alphaXiv/openresearch.sh/pull/637 (merged). Deploy migration 0130 and its matching API/projector before releasing this client. Keep this CLI PR unmerged pending approval.

Verification

Primary used the Codex in-app browser in isolated slot 7. Official-build telemetry was directed exclusively to a local receiver and local service database. Native records, ledger samples, emitted reports, and service projections were reconciled; each listed experiment event was ingested and projected once.

Case Observed result
OpenCode V2 parent + different-model child; local runs big-pickle parent, gpt-6.1-sol child; both done, exact
OpenCode V1 background child after parent completion; native parent wakeup mimo-v2.6-flash-free child run done/exact; own native completion and parent wakeup accounted
V1 cancellation after tool execution Cancelled with native identity retained
Codex resume + Luna child run + parent cancellation Child run gpt-6-luna, parent gpt-6-sol; cancelled chat; no duplicate old-root execution
Antigravity no-tool response + child experiment Native parent/child identities retained, including identity-only child samples
Claude message + Haiku child + parent experiment Native Haiku/Sonnet identities retained; parent run done/exact
Cursor provider refusal Failed, not_executed; live success retest blocked by account usage limit
Controlled lifecycle/fault ordering Automated native sample deduplication, identity-only → measured completion, late children, interrupted/restarted reporting, delayed launch output, native-server liveness, exactly-once terminal events
Privacy Automated opt-out/development suppression; local-only official-build telemetry

Local shipping checks passed on the simplified candidate: Rust + Windows launcher formatting, clippy, locked build, 1,009 unit + 15 integration tests (3 ignored), and development-build channel assertion. These checks include the final review fixes; Clippy 1.99 also passes. Current-head platform CI and Greptile are running.

Known limits: Cursor's hidden child identity remains explicitly unresolved; Antigravity child token counts are not exposed by its stream. OpenCode V2 was exercised using official beta 19271. Codex's account-denied default model cannot be tested successfully; explicit Sol/Luna work. Windows/Linux verification is CI, not hands-on OS testing. Earlier PR revisions' larger coverage matrix is historical evidence, not a claim that every row was rerun on this rewrite.

  • Final current-head Greptile 5/5 with no actionable findings
  • Final Windows/Linux CI
  • User hands-on acceptance in slot 7

@greptile-apps

greptile-apps Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 4/5

[Medium risk] Refactors model attribution tracking across harness implementations.

The PR should not merge until an Antigravity hook captured before transcript flush can be joined to its eventual planner step.

Findings

  1. P1 Unflushed planner loses attribution ▶
  2. P1 Pending reports stop migrating ▶
Fix with agent prompt
### Issue 1
src/commands/mcp_gate.rs:253-257
If PostInvocation runs before a successful planner row appears in the transcript, this branch saves the model under an `invocation:` key. The later streamed usage and completed tool use a `step:` key, so they cannot find that captured identity. The executed model or the experiment run it launched can remain unresolved even though the hook recorded the model.

### Issue 2
src/store.rs:undefined-675
When an existing database has a terminal experiment report still waiting for its launcher, this migration adds `pending_since` but leaves it null for that report. The new settlement path only reads rows where `pending_since` is non-null, so the report is never staged or retried.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The PR adds durable, token-independent native model attribution across chat harnesses and experiment runs, including late-work reconciliation and run-invoker binding. The latest revision separates unmatched Antigravity hooks from planner-step identities and keeps forwarded child steps from replacing the root conversation. One unflushed-transcript case still leaves a captured hook identity disconnected from its planner step.

Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart LR
  H[PostInvocation hook] --> K{Planner row visible?}
  K -->|Yes| S[Step-key identity]
  K -->|No| I[Invocation-key identity]
  S --> U[Streamed usage and tool lookup]
  I -. no step-key match .-> U
Loading

Reviews (10) · Last reviewed commit: "Keep Antigravity child and failed-invoca..."

Comment thread src/local/harness/opencode_v2.rs Outdated
Comment thread src/local/harness/opencode.rs Outdated
Comment thread src/local/harness/opencode_v2.rs Outdated
@sox8502

sox8502 commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai review

Please review current head a7e4347. Native recovery now retries incomplete OpenCode reads, verifies missing native history before settling an explicit exception, and preserves ownership across child continuations. Codex child ownership follows native root-turn IDs, and the observed pre-execution model refusal is reported as not executed. Windows database handles are closed before test cleanup.

Validation: formatting, clippy, locked build, 1,051 unit tests and 14 integration tests passed; the isolated official-build finalization test also passed. The primary is continuing actual-app verification against this revision. Both linked PRs remain unmerged.

Comment thread src/local/harness/opencode.rs Outdated
Comment thread src/local/harness/opencode.rs Outdated
Comment thread src/compute.rs Outdated
@sox8502

sox8502 commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai review

Please review current head 8995241. The prior findings are addressed: V1 background watches retain newly discovered descendants and continuation parents; retiring trees are natively interrupted and drained, and busy/failed reads stay durably held; the compute test restores CODEX_THREAD_ID via the existing environment guard.

Primary app verification also found and fixed official OpenCode V2 beta recognition and fresh-store schema readiness. V1 history is refused before beta launch because that migration is unverified; no marker is fabricated or history dropped. The actual V2 app now executes chat/tools and agent-launched done, failed, cancelled and repeated experiments against a local receiver. Full fmt/clippy/build/tests passed on this exact tree after merging current main80724f4a. Native-failure checks retain exact Codex/Antigravity identities after output.

Please check the full PR and these follow-ups, including durable ownership, retirement, native beta compatibility and migration safety. Hands-on coverage expansion is still underway; the PR remains unmerged.

@sox8502

sox8502 commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai review

Comment thread src/local/harness/opencode.rs Outdated
Comment thread src/local/harness/opencode.rs Outdated
@sox8502

sox8502 commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai review

Comment thread src/local/harness/opencode.rs Outdated
@sox8502

sox8502 commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai review

Comment thread src/local/harness/opencode.rs Outdated
@sox8502

sox8502 commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai review

@sox8502

sox8502 commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai review

Please review current head 24c421a. Ponytail simplification reuses native OpenCode history instead of duplicate live-stream accounting, consolidates its woken-run traversal, shares Codex capture writers/rollout parsing, and removes unused helpers. Net -287 production / -36 test lines; full local fmt/clippy/build/tests passed (1075 unit + 15 integration, 4 pre-existing ignored). Required native child/reroute/recovery behavior is retained; focused regressions passed. Primary app rechecks are in progress. Review the full current diff and flag any regression in ownership, cancellation or finalization.

@sox8502

sox8502 commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai review

Comment thread src/store.rs
Comment thread src/commands/mcp_gate.rs Outdated
@greptile-apps

This comment has been minimized.

@sox8502

sox8502 commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai review

40336a3 addresses the two Antigravity findings: unmatched hooks no longer claim future planner-step keys, and forwarded child steps cannot replace the root conversation or lend their planner identity to another conversation's tool. A focused regression test covers both failures. Primary app recheck passed child run → parent tool → resumed parent, retaining native parent/child identities.

The pending_since migration finding applies only to unmerged drafts of this PR. Current origin/main a87bb4c has no terminal/link_json/pending_since deferred-report schema or records; no released state needs conversion. Adding a draft-only migration would preserve machinery specifically removed by this simplification.

Final local gates: fmt, launcher fmt, clippy, locked build, 1,009 unit + 15 integration tests; 3 ignored. Clippy 1.99 also passes with a four-line, trait-scoped workaround for async-trait-generated attributes. No new dependency or toolchain pin.

Comment thread src/commands/mcp_gate.rs
Comment on lines +253 to +257
let key = match planner {
Some(planner) => {
crate::local::harness::antigravity::invocation_sample_id(conversation, planner)
}
None => format!("antigravity:{conversation}:invocation:{step}"),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Unflushed planner loses attribution

If PostInvocation runs before a successful planner row appears in the transcript, this branch saves the model under an invocation: key. The later streamed usage and completed tool use a step: key, so they cannot find that captured identity. The executed model or the experiment run it launched can remain unresolved even though the hook recorded the model.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/commands/mcp_gate.rs
Line: 253-257

Comment:
**Unflushed planner loses attribution**

If PostInvocation runs before a successful planner row appears in the transcript, this branch saves the model under an `invocation:` key. The later streamed usage and completed tool use a `step:` key, so they cannot find that captured identity. The executed model or the experiment run it launched can remain unresolved even though the hook recorded the model.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant