[stavp-star]

Closes #44 · Part of epic #49 · Stack position 14/15. Base: deep-agent/47-utils-mcp (#62), auto-retargets on merge.

What

uv run deep-agent-eval (deep_agent/evaluation.py) is the migration's go/no-go harness.

• Runs both systems on the same items:
  • DeepAgentRunner reuses the benchmark run path ([Deep Agent 10/15] Bound the agent loop and cut latency: limits, tool surface, connection reuse, caching (#42) #59), so it gets limits, timeout, tracing, and the verified final_result rows ([Deep Agent 9/15] Structured final answer bound to the actually-executed SQL (#41) #58).
  • LegacyGraphRunner invokes agent_graph exactly like /chat (non-interactive). When the legacy graph returns only SQL, that SQL is executed through the read-only guard, so the two are compared like-for-like.
• Executes each item's gold SQL (guarded, up to 1,000 rows) and scores execution accuracy on result sets:
  • order-insensitive unless the gold query has a top-level ORDER BY (a subquery's ORDER BY doesn't count)
  • floats to 6 decimals; Decimal and numeric strings normalised; whitespace trimmed
  • duplicates count (multiset); permuted columns still match
• Reports per system: execution accuracy, p50/p95 latency, avg input/output tokens, avg tool calls, accuracy per tag, and status counts. Output is a markdown comparison table plus JSON (--out).
• --gate exits 1 unless deep accuracy ≥ legacy accuracy and deep p95 ≤ --p95-slo (default 60 s).
• Deep-agent traces get a deep-agent.execution_accuracy score in Langfuse.
• --langfuse-dataset NAME runs as a Langfuse Experiment (dataset.run_experiment with an execution_accuracy evaluator), which creates a linked dataset run you can compare across prompt versions ([Deep Agent 12/15] Specialised subagents + Langfuse prompt management (#48) #61). Items: input={"question"}, expected_output={"sql"}.

Dataset seed: agent/benchmarks/eval_dataset.json, 10 real questions, each tagged (simple_filter, categorical, large_categorical, join, per_capita, temporal, anti_join, ratio, temporal_latest, deduplication, ...).

⚠️ Only 2 items have gold SQL, taken from past agent runs that converged on the same query, and marked gold_source: "agent_run_unverified". I didn't invent gold SQL for a schema I can't query; the other 8 are gold_sql: null and reported as no_gold. A domain expert needs to supply and verify the gold set. Utils-Server langfuse_create_dataset_from_traces can bootstrap candidates from real traces.

CI (.github/workflows/agent-tests.yml, the repo's first workflow): on PRs and pushes touching agent/ or core/, it runs uv sync --frozen, then ruff (pyflakes) on deep_agent, then the unit tests with an 80% coverage gate on deep_agent. Real-service e2e tests are excluded.

• The private python-core-utils dependency is fetched over HTTPS with a token. Action needed: add the repository secret PYTHON_CORE_UTILS_TOKEN (fine-grained, read access to matzpen-agency/python-core-utils).
• Simulated locally with no agent/.env and dummy Langfuse settings: the suite doesn't depend on local config (417 passed).

Acceptance criteria

• ≥ 90% line coverage on deep_agent/tools.py (93%); agent-construction tests present ([Deep Agent 2/15] Actually load skills and AGENTS.md memory (#34) #51 onward). Package total 82%.
• deep-agent-eval runs the dataset against both agents and reports execution accuracy, p50/p95, avg tokens, and avg tool calls, with Langfuse dataset runs via --langfuse-dataset
• Dataset ≥ 50 questions with gold SQL covering all listed categories. Not met: 10 tagged questions, 2 with unverified gold. Needs domain input and DB access.
• [~] CI runs unit tests on every PR, once PYTHON_CORE_UTILS_TOKEN is added. The workflow can't be exercised from here.

Definition of done: pending

• First comparison report (legacy vs deep), from a live run with the expert gold set
• Agree the go/no-go thresholds. The defaults implemented are: deep accuracy ≥ legacy, p95 ≤ 60 s.

Housekeeping note

agent/.coverage is a tracked file (since 2a304ac), so every coverage run dirties it. I left it untouched; suggest git rm --cached agent/.coverage and ignoring it.

Testing

• tests/deep_agent/test_evaluation.py: 31 tests (comparison edge cases, top-level ORDER BY detection, scoring statuses, summaries, gate, both-systems end-to-end, Langfuse task/evaluator, dataset validity including read-only gold SQL)
• Full suite (CI command): 417 passed, 2 skipped; coverage gate met

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: ASSERTIVE
• Plan: Advanced
• Run ID: 2544c487-ac29-4e06-b43a-05ebdfb4f25a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}