Repository navigation
[Epic] Deep Agent migration: review findings & hardening plan #49
Description
Tracking epic for the architecture and code review of feature/deep-agent-updated: migrating the LangGraph workflow agent (agent/src/agent) to a deepagents-based agent (agent/src/deep_agent).
Headline findings
- The skills and
AGENTS.mdnever load. All four skills and the memory file are silently ignored, on every machine ([Deep Agent] Skills and AGENTS.md memory are never loaded (hardcoded paths + StateBackend + wrong skill source layout) #34). - The agent runs arbitrary LLM SQL with no read-only guard, and the LIMIT safety net is easy to bypass ([Deep Agent] Enforce read-only SQL, AST-level LIMIT and safe identifiers in Trino tools #35).
enrich_sql_filtersmakes the model re-type the whole catalog as a tool argument ([Deep Agent] enrich_sql_filters forces the LLM to re-emit the full catalog as an argument and drops tracing config #36).- The deep agent has no Langfuse tracing ([Deep Agent] Add end-to-end Langfuse tracing (callbacks, nested tool config, span metadata, scores) #37).
- There are no tests and no eval harness, so there is no evidence yet that the migration improves accuracy ([Deep Agent] Unit tests + offline eval harness (legacy graph vs deep agent) as the migration go/no-go gate #44).
Suggested order
P0: fix before any further evaluation (current measurements don't reflect the intended design)
- [Deep Agent] Skills and AGENTS.md memory are never loaded (hardcoded paths + StateBackend + wrong skill source layout) #34 Skills/memory loading
- [Deep Agent] Enforce read-only SQL, AST-level LIMIT and safe identifiers in Trino tools #35 Read-only SQL guard, LIMIT, identifiers
- [Deep Agent] enrich_sql_filters forces the LLM to re-emit the full catalog as an argument and drops tracing config #36 Catalog argument / tracing config in
enrich_sql_filters - [Deep Agent] Add end-to-end Langfuse tracing (callbacks, nested tool config, span metadata, scores) #37 End-to-end Langfuse tracing
P1: required for production parity
- [Deep Agent] Remove process-global request state (capability flags, connection singleton, frozen date) via runtime context #38 Remove process-global request state
- [Deep Agent] Classify Trino errors from TrinoUserError.error_name instead of keyword matching #39 Trino error classification from
error_name - [Deep Agent] Resolve contradictory and incorrect guidance across system prompt, AGENTS.md, tool docstrings and skills #40 Contradictory / incorrect prompt and skill guidance
- [Deep Agent] Structured final answer (response_format) bound to the actually-executed SQL #41 Structured final answer bound to executed SQL
- [Deep Agent] Bound the agent loop and cut latency (call limits, tool surface, connection reuse, caching) #42 Loop bounds and latency
- [Deep Agent] Wire the deep agent into the API/UI behind a flag with a persistent checkpointer and HITL #43 API/UI integration, persistent checkpointer, HITL
- [Deep Agent] Unit tests + offline eval harness (legacy graph vs deep agent) as the migration go/no-go gate #44 Tests + eval harness (go/no-go gate)
P2: quality and modernization
- [Deep Agent] Standardize tool return contracts, surface MCP errors, fix null/Hebrew handling, cut token waste #45 Tool return contracts and token efficiency
- [Deep Agent] Remove committed logs/scratch files and turn run_deep_agent.py into a proper CLI entry point #46 Repo hygiene / CLI entry point
- [Deep Agent] Integrate Utils-Server tools via Jarvis Registry MCP (Trino guard, Hebrew text, datetime, geo, eval) #47 Utils-Server tools via Jarvis Registry MCP
- [Deep Agent] Specialized subagents + Langfuse prompt management for the deep agent #48 Specialized subagents + Langfuse prompt management
Migration exit criteria
- Deep agent execution accuracy ≥ legacy graph on the shared eval dataset ([Deep Agent] Unit tests + offline eval harness (legacy graph vs deep agent) as the migration go/no-go gate #44)
- p95 latency within the agreed SLO ([Deep Agent] Bound the agent loop and cut latency (call limits, tool surface, connection reuse, caching) #42)
- Feature-flagged rollout complete, legacy graph removable ([Deep Agent] Wire the deep agent into the API/UI behind a flag with a persistent checkpointer and HITL #43)
🤖 Generated with Claude Code
Activity
stavp-star commentedon Oct 20, 2026
Implementation: 15 stacked PRs (merge top to bottom)
PR 1 targets
feature/deep-agent-updated; each later PR targets the one above it, and GitHub retargets it automatically as you merge.uv run deep-agent), committed logs and scratch files removedAGENTS.mdactually load (they never did)errorName, with line/column{ok, data, error}tool envelope; MCP errors surfaceddeep-agent-benchdeep-agent-eval --gate) and CIchat_with_agent: engine switch, Postgres threads, streaming, HITLSuite on #64: agent 434 passed, 3 skipped; backend 14 passed.
Needs you / infra
PYTHON_CORE_UTILS_TOKENso CI ([Deep Agent 14/15] Execution-accuracy eval harness (deep vs legacy) + CI (#44) #63) can install the private dependencyJEEN_API_KEY/JEEN_METADATA_MCP_KEY: real-looking defaults are committed inagent/src/agent/config.py(predates this work)deep-agent-bench, thendeep-agent-evalwith an expert-verified gold set. Only 2 of 10 seed items have (unverified) gold SQL.agent/scripts/sync_deep_agent_prompts.py --label staging) and approve the 5 Utils-Server tools in the registryDEEP_AGENT_CHECKPOINT_DB_URLfor durable threads in deployed environments🤖 Generated with Claude Code
StavPonte11 commentedon Oct 21, 2026
Added 16/16: #66 (location-analyst subagent, closes #65), stacked on #64.