Repository navigation
feat: 5-Agent Agentic RAG 리포트 생성 (Strands Graph) - #80
Merged
Merged
Conversation
Path(key).name can collide or contain characters that break the download path; the document_id is the stable identifier for the chunk.
01_production_5_agent_agentic_rag_design.md requires the report generation agent loop to be owned by a Strands Graph (5 specialized agents + deterministic nodes), not delegated to AgentCore Managed Harness. ADR-002's rationale (Memory-based metric trends, Cedar as a safety guarantee) is exactly what the new design forbids, so this supersedes it for the report generation agent only; report_editing_chatbot and insight_map (ADR-003) are unaffected.
Pydantic schemas, GraphState/RetryBudget, shared enums, and repository/retriever protocols per 01_production_5_agent_agentic_rag_design.md Phase 1. No agent logic or wiring yet - contracts only.
Authorization service (mirrors BE's ensure_session_access / ensure_patient_ref_access as read-only checks), context loader (LOAD_CONTEXT logic), metric trend calculator, five deterministic validators (schema/citation/metric/safety/state-transition), and SQLAlchemy-backed implementations of the Phase 1 repository protocols. Verified end-to-end against a local Postgres seeded with fixture data (session/transcript/patient history read paths, run/draft/event write paths, authorization allow+deny, duplicate-run guard) before committing - caught and fixed a real bug where JSONB writes need json.dumps() + CAST(... AS jsonb) rather than relying on psycopg to adapt dicts, matching the existing app/storage/rds.py convention.
Five Strands Agent objects (Clinical Context, Evidence Research,
Evidence Critic, Clinical Report, Quality Assurance), each with its
own system prompt and tool allowlist per
01_production_5_agent_agentic_rag_design.md §8-13. Research/Critic
tools are real implementations layered over the existing KURE-v1 +
pgvector RAG infra (app/rag/retriever.py, vector_store.py,
semantic_layer.py) rather than stubs.
QA Agent's PASS decision is re-checked deterministically against
configured thresholds and CRITICAL/BLOCKING issues before being
trusted, per §0.2 ("LLM 평가만으로 저장 조건을 판정하지 않는다").
Verified against real Bedrock (Claude Haiku) end-to-end, not mocked:
- Clinical Context Agent correctly used pre-computed trend_direction
facts instead of inventing its own
- Evidence Research Agent ran a real 10-tool agentic loop against a
seeded local pgvector corpus (search/hybrid_search/rerank/fetch)
- Evidence Critic Agent correctly returned RESEARCH_MORE on genuinely
incomplete coverage
- Clinical Report Agent produced schema-valid claims with proper
evidence refs
- QA Agent's own narrative said the report passed, but its actual
scores (safety=0.96, patient_context_fidelity=0.9) were below
threshold - the deterministic gate correctly downgraded PASS to
REWRITE, confirming the defense-in-depth design works as intended
…ration Wires the 5 agents (Phase 3) and deterministic logic (Phase 2) into a real strands.multiagent.graph.Graph. The actual installed strands-agents API differs from the design doc's pseudo-code (GraphBuilder + per-edge boolean conditions, nodes must be MultiAgentBase, not Graph()/add_conditional_edges with plain functions) - adapted accordingly per the doc's own note that the pseudo-code is a "structural contract" to adapt to the real SDK version. Domain state (ReportGenerationState, renamed from GraphState to avoid colliding with strands.multiagent.graph.GraphState) flows through invocation_state rather than Strands' ContentBlock task-passing, confirmed reliable with a minimal real-Graph experiment before committing to the design. Deterministic nodes (LOAD_CONTEXT, DETERMINISTIC_VALIDATION, FINAL_VALIDATION, SAVE_DRAFT, HUMAN_REVIEW, FAILURE) are thin MultiAgentBase wrappers around Phase 2 functions; routing.py holds pure predicate edge conditions. Verified against real Bedrock + a live local Postgres end-to-end (not mocked), running the actual graph rather than individual agents: - Found and fixed a real ordering bug: create_run() was inserting the QUEUED row before the duplicate-run check, so every run detected itself as a duplicate. Moved the check to before row creation and removed the now-redundant check from context_loader. - With a deliberately sparse 4-chunk RAG corpus, watched the real Evidence Research/Critic cycle run 3 full research rounds (correct retry_round increments each time), then correctly route to HUMAN_REVIEW once the research budget was exhausted rather than looping forever or auto-approving - the exact safety behavior the design requires. - Verified deterministic_validation -> quality_assurance -> (QA's own narrative said pass, deterministic gate correctly caught safety=0.9-0.96 never hitting the required exact 1.00 and downgraded to REWRITE, twice) -> confirms the REWRITE cycle and gate both work. - Directly verified final_validation -> save_draft with a constructed PASS state: report_drafts row written correctly with patient/literature evidence namespaces kept separate (§17), run marked SUCCEEDED.
- API: POST .../sessions/{session_id}/reports/generate (queues via SQS,
returns immediately, mirrors jobs.py's create_job pattern) and
GET .../report-generation-runs/{run_id} (real status lookup, not a
stub - includes report_draft_id once saved).
- SQS worker (report_generation_worker.py): mirrors cpu_worker.py's
heartbeat/visibility-extension pattern since a run can take minutes
to tens of minutes across retry cycles.
- runtime_entrypoint.invoke() + deployment.py: single entrypoint shared
between local CLI execution (--fixture) and AgentCore Runtime
deployment, per §21.3.
- New sqs_report_generation_queue_url setting + .env.example entry.
Verified for real against a live local Postgres (not mocked):
- POST endpoint via FastAPI TestClient (SQS mocked) - correct run_id
and message body shape.
- Worker's SQS message Pydantic model round-trips the exact JSON the
API produces.
- invoke() end-to-end with a real non-owning therapist_id - fails fast
with UNAUTHORIZED, correctly serialized, no Bedrock calls needed.
- Found and fixed a real gap: a nonexistent therapist_id trips a raw
ForeignKeyViolation during create_run() that propagated as an
unhandled SQLAlchemy exception instead of a clean FAILED result.
Wrapped it into InvalidSessionStateError, and invoke() now catches
ReportGenerationError broadly so AgentCore/local callers always get
a JSON result instead of a raw traceback.
- GET status endpoint against the real SUCCEEDED run row from Phase 4's
save-path test - correctly returns report_draft_id, quality_scores,
and 404s for a non-owning therapist_id instead of leaking the row.
174 tests, full suite runs in ~2s (no Bedrock/DB calls) per
01_production_5_agent_agentic_rag_design.md §25:
- unit: routing predicates, metric trend calculator, 5 deterministic
validators, repository JSONB serialization (regression guard for the
Phase 2 json.dumps/CAST bug), authorization_service allow/deny paths
- agent contracts: tool allowlists per agent, Clinical Context's
retry-once-then-raise behavior, QA's deterministic pass-gate
(regression test for the real safety<1.0 downgrade behavior observed
live in Phase 3/4)
- integration: full real strands Graph engine with agent-calling
functions and repos mocked - covers the ACCEPT happy path, critic
RESEARCH_MORE cycle, research budget exhaustion -> HUMAN_REVIEW, QA
REWRITE cycle, QA FAIL -> FAILURE, and critic HUMAN_REVIEW routing.
Same scenarios verified live with real Bedrock in Phase 4, now
permanent and free to re-run.
- security: IDOR/cross-therapist access (session + patient_ref +
run ownership), HTML/script injection stripping, foreign-ID leakage
detection, oversized payload rejection, trace/event redaction
- golden: 15 fixture inputs per §24.3 (scaffolding only, per user
decision - LLM output isn't deterministic enough for exact-match
golden regression, so these define scenarios + reuse the real
validators as the invariant checker)
Found and fixed two real gaps while writing these:
- graph.py: model_config was always {} on every run; now records
actual model IDs + prompt/schema version per §15.
- report_editing_chatbot/validators.py (shared safety validator):
the design doc's own flagship forbidden example, "자폐가 의심됩니다",
wasn't caught by the existing regex patterns. Added a targeted
pattern for disorder-name + "의심" without breaking the existing
test that protects "발달 지연이 의심됩니다" as a legitimate clinical
hedge - the distinction is diagnosis-name vs. general observation.
…etry
Discovered running a real local demo: with the research retry budget
already exhausted, the Evidence Critic Agent got stuck retrying
EvidenceCriticOutput structured-output generation 148+ times without
completing or failing, burning real Bedrock calls indefinitely. Our
own retry budgets (max_research_rounds etc.) only bound the Graph-level
cycles - they never bounded Strands' internal per-invocation retry
loop for structured output validation.
Fixed by passing limits={"turns": N} to every agent(...) call (Strands
Agent.__call__ supports this natively - trips the cap at the top of the
next loop iteration and returns cleanly with structured_output=None and
stop_reason="limit_turns" instead of spinning). Verified against real
Bedrock that a tiny limit produces the expected clean stop. Existing
"structured_output is None" handling in each agent's run function
already covers this path, so no other logic changed.
Turn budgets differ by agent: tool-using agents (Research/Critic) get
generous caps (40/30) since legitimate multi-tool work needs many
turns; the three no-tool agents get a small cap (6).
Added regression tests asserting all 5 agents pass the limits kwarg.
…ogy pipeline search_korean_evidence, search_international_evidence, and search_clinical_guidelines were calling app.rag.retriever.retrieve_evidence(), a simpler function built for the Lambda/legacy-Bedrock path that embeds the raw query directly - no Kiwi tokenization, no ontology.yaml synonym expansion, no metadata filtering. Only hybrid_search attempted any of this, and even it used naive .split() instead of real Kiwi. All four tools now go through app.rag.retriever.Retriever, which wraps the existing rag_graph.py LangGraph pipeline (Kiwi morphological analysis -> ontology-based query expansion -> language_area/metric metadata filter -> pgvector search -> filter-free fallback retry on weak results). hybrid_search now delegates to the same pipeline rather than duplicating a partial version of it. Verified against real seeded pgvector data that Kiwi tokenization is actually invoked (not silently falling back) and that the four search tools return results consistent with the fixed pipeline.
Observed in a real end-to-end verification run: with many evidence candidates to check (14 fetch_document_section + 8 verify_source_metadata calls in one critic invocation), 30 turns wasn't always enough for the Critic to reach valid structured output, tripping the turns cap and falling back to HUMAN_REVIEW before the research retry budget was even exhausted. The fallback itself worked safely (no hang, no crash) - this is a tuning adjustment, not a bug fix.
…etrieval
Adds scripts/eval_phoenix_experiment.py, which runs the existing GUI-authored
Phoenix dataset (utterai_phoenix_eval_dataset_v1) through two independent
experiments - the legacy app.rag.retriever.retrieve_evidence() path and the
new Kiwi/ontology/metadata-filter pipeline in evidence_research/tools.py's
_search() - with hit@k/recall@k (code) plus document_relevance/faithfulness/
clinical_answer_quality (LLM judge) evaluators. clinical_answer_quality also
checks expected_behavior so answers that should refuse/abstain get penalized
if they instead confirm a diagnosis.
--task {legacy,new,both} lets a pre-merge run on main log a legacy-only
baseline and a post-merge run log both for comparison, tagged with
branch@sha (or EVAL_TAG when there's no .git, e.g. running from inside a
deployed pod).
Verified for real against the actual prod corpus (port-forwarded, read-only):
found 4 of the dataset's expected_document_ids don't exist under those names
in the real corpus (doc_mlu_guide/doc_ttr_ndw map to differently-named real
docs; doc_latency/doc_soap_template have no dedicated document at all yet).
_retrieval_scoped_dataset() corrects the resolvable ones and excludes the
LAT-*/SOAP-* cases from this retrieval-only scope client-side, without
mutating the stored Phoenix dataset - those cases move to a future
end-to-end v2 dataset once that content exists.
Also fixed two bugs found by actually running this against prod: Phoenix's
sync Client rejects async task callables (switched tasks to sync + internal
asyncio.run), and APP_ENV=local forces sslmode=disable which RDS refuses
(need APP_ENV=dev for real Postgres SSL).
…Lambda Previously the Lambda only wrapped the old retrieve_evidence() (no Kiwi/ ontology/filter/rerank). Rewire it to reuse the same 7 @tool functions the Evidence Research Agent calls in-process (search_korean_evidence, search_international_evidence, search_clinical_guidelines, hybrid_search, rerank_evidence, fetch_document_section, resolve_ontology_synonyms) - @tool-decorated functions remain plain-callable, so this is a single source of truth for the retrieval pipeline shared between local Strands execution and Lambda/AgentCore Gateway execution. Kept 7 granular tools instead of consolidating to one, so the Evidence Research Agent's choice of search strategy stays visible as distinct tool calls (the point of this exercise being to explore multi-agent design, not just retrieval quality). Dispatch reads context.client_context.custom['bedrockAgentCoreToolName'] (format "${target_name}_${tool_name}" per AWS docs) and matches by suffix since one Lambda now serves all 7 Gateway targets. Verified locally against docker-compose Postgres: dispatch routes correctly for a DB-backed tool (search_korean_evidence) and a non-DB tool (resolve_ontology_synonyms, real ontology.yaml expansion), and returns 400 for an unrecognized tool name.
…aseline) Companion to eval_phoenix_experiment.py, using the GUI-authored utterai_phoenix_e2e_dataset_v1 (8 cases with utterances/metrics/session/ sections) to run the legacy "retrieve_evidence -> build_template_report_prompt -> Bedrock Claude" path end-to-end and score the generated report with hit@k/recall@k, section_structure_compliance (all requested sections non-empty), and LLM-judge document_relevance/faithfulness/clinical_answer_quality. The new 5-agent path is out of scope here: run_report_generation() reads real session/patient_ref/therapist rows via repositories, and seeding those into prod was explicitly ruled out (read-only access only). Local-DB fixture seeding for the 5-agent path is a separate follow-up. Verified for real against prod (read-only, same tunnel pattern as the retrieval eval): 8/8 task runs succeeded, 48/48 evaluations completed.
gpffh20
changed the title
jeongmin1999
self-requested a review
July 3, 2026 10:56
jeongmin1999
approved these changes
Jul 3, 2026
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
개요
01_production_5_agent_agentic_rag_design.md설계에 따라 ADR-002(단일 AgentCore 에이전트)를 대체하는 5-Agent Strands Graph 기반 리포트 생성 파이프라인을 구현했습니다. (ADR-004로 문서화)HUMAN_REVIEW로 안전하게 격하limits={"turns": N}): 실제 데모 중 구조화 출력 생성이 148회 연속 실패하며 멈추지 않던 버그를 이 방식으로 해결 (Strands의 내부 재시도가 무제한이었음)검증
배포 영향
/internal/ai)는 기존 API에 영향 없는 순수 추가report_generation_worker.py를 실행하는 k8s Deployment가 아직 없어 머지해도 SQS 소비자가 살아나지 않음 (worker 배포는 별도 Infra 작업)남은 작업 (별도 후속)
Test plan
pytest tests/179 passedruff check