examples: add auditable evaluation optimization loop

examples/optimization/eval_optimize_loop/DESIGN.md

@@ -0,0 +1,49 @@
+# Evaluation + Optimization 闭环设计
+
+## 方案说明
+
+本示例复用 `AgentEvaluator`、`AgentOptimizer` 和 `TargetPrompt`，不修改生产
+源码。Pipeline 先分别评测 train/validation，保存每条 case 的 metric、状态、
+失败原因和关键轨迹；再按执行异常、回复不匹配、工具名称、工具参数、rubric、
+知识召回和格式问题进行确定性归因。优化阶段只修改 working copy，候选完成
+train/validation 回放后，按 case 输出新增通过、新增失败、分数提升、分数下降
+和 unchanged。Gate 检查验证集提升阈值、无新增 hard fail、critical case 不退化、
+验证集退化和成本/耗时预算；训练提升而验证退化直接判定过拟合。fake-model、
+fake-judge 和 trace mode 使用同一比较与 gate 链路，保证无 API Key 也能复现。
+报告 JSON/Markdown 保存输入 hash、候选、逐 case delta、归因、成本、耗时和理由；
+默认不回写源 prompt；仅 `real` 模式显式 `--write-back` 且 gate 接受时才写回，
+fake/trace 模式拒绝回写，避免合成候选污染检入文件。
+
+## 阶段与 Review
+
+- A：模型、输入校验和 evaluator 适配；Review A 检查结果保留、泄漏和边界。
+- B：优化、归因、逐 case diff、gate 和 working-copy；Review B 检查过拟合、
+  hard fail、成本和 prompt 恢复。
+- C：CLI、fake/trace、报告和样例；Review C 对照 Issue #91 核对交付物。
+- D：目标测试、覆盖率、flake8、函数复杂度和最终 diff；Review D 做交付审查。
+
+## 主要文件
+
+实现：`loop/models.py`、`loop/evaluation.py`、`loop/analysis.py`、
+`loop/pipeline.py`、`loop/reporting.py`、`agent/agent.py`、`run_pipeline.py`。
+
+资源：`data/train.evalset.json`、`data/val.evalset.json`、
+`data/fake_trace.json`、`optimizer.json`、`gate.json`、
+`optimization_report.json`、`README.md`。
+
+`optimization_report.json` 是 Issue #91 要求的示例输出，不是稳定契约。
+其中时间戳、Git SHA、Python 版本和耗时仅展示审计字段，实际运行由 pipeline
+在输出目录重新生成，测试不依赖这些环境相关值。
+
+测试：`tests/evaluation/test_eval_optimize_loop_*.py`。
+
+## 验收
+
+```bash
+uv run pytest tests/evaluation/test_eval_optimize_loop_*.py \
+  --cov=examples.optimization.eval_optimize_loop --cov-fail-under=90
+uv run flake8 --max-complexity=15 --max-line-length=120 \
+  examples/optimization/eval_optimize_loop
+uv run python examples/optimization/eval_optimize_loop/run_pipeline.py \
+  --fake-model --fake-judge
+```

examples/optimization/eval_optimize_loop/README.md

@@ -0,0 +1,16 @@
+# Evaluation + Optimization Loop
+
+本示例把 `AgentEvaluator`、`AgentOptimizer` 和 `TargetPrompt` 组合成可审计的
+baseline → optimize → candidate → gate 闭环。默认 `fake-model` 不需要 API Key，
+运行时间通常小于 30 秒；`real` 模式读取 `TRPC_AGENT_API_KEY`、
+`TRPC_AGENT_BASE_URL` 和 `TRPC_AGENT_MODEL_NAME`（也可用 `--model-name` 覆盖）。
+`trace` 模式读取预录制的
+baseline/candidate evalset，适合离线回归。
+
+```bash
+uv run python examples/optimization/eval_optimize_loop/run_pipeline.py
+```
+
+输出目录包含 `optimization_report.json`、`optimization_report.md` 和临时工作副本。
+只有 `real` 模式显式传入 `--write-back` 且 gate 接受时才会更新 prompt 源文件；
+fake/trace 模式会拒绝回写，避免合成候选污染源文件。

examples/optimization/eval_optimize_loop/agent/agent.py

@@ -0,0 +1,88 @@
+"""Real and deterministic agents used by the optimization loop example."""
+
+from __future__ import annotations
+
+import os
+import uuid
+from pathlib import Path
+
+from trpc_agent_sdk.agents import LlmAgent
+from trpc_agent_sdk.models import OpenAIModel
+from trpc_agent_sdk.runners import Runner
+from trpc_agent_sdk.sessions import InMemorySessionService
+from trpc_agent_sdk.types import Content
+from trpc_agent_sdk.types import Part
+
+APP_NAME = "eval_optimize_loop_agent"
+USER_ID = "eval-optimize-loop"
+CANDIDATE_MARKER = "OPTIMIZED_CANDIDATE"
+UNKNOWN_RESPONSE = '{"queue":"unknown"}'
+
+
+async def fake_call_agent(prompt_path: Path, query: str) -> str:
+    """Return a prompt-sensitive deterministic response without an API key."""
+    prompt = prompt_path.read_text(encoding="utf-8")
+    if CANDIDATE_MARKER in prompt and ("1006" in query or "download" in query):
+        return '{"queue":"billing"}'
+    if "refund" in query and CANDIDATE_MARKER not in prompt:
+        return UNKNOWN_RESPONSE
+    if "download" in query:
+        return UNKNOWN_RESPONSE
+    return _expected_queue(query)
+
+
+async def real_call_agent(prompt_path: Path, query: str) -> str:
+    """Run one real OpenAI-compatible Agent invocation."""
+    agent = _create_agent(prompt_path)
+    sessions = InMemorySessionService()
+    runner = Runner(app_name=APP_NAME, agent=agent, session_service=sessions)
+    session_id = uuid.uuid4().hex
+    await sessions.create_session(
+        app_name=APP_NAME,
+        user_id=USER_ID,
+        session_id=session_id,
+        state={},
+    )
+    message = Content(role="user", parts=[Part.from_text(text=query)])
+    return await _consume_final_text(runner, session_id, message)
+
+
+def _create_agent(prompt_path: Path) -> LlmAgent:
+    api_key = _required_env("TRPC_AGENT_API_KEY")
+    base_url = _required_env("TRPC_AGENT_BASE_URL")
+    model_name = _required_env("TRPC_AGENT_MODEL_NAME")
+    model = OpenAIModel(model_name=model_name, api_key=api_key, base_url=base_url)
+    return LlmAgent(
+        name=APP_NAME,
+        description="Support queue classifier.",
+        model=model,
+        instruction=prompt_path.read_text(encoding="utf-8"),
+    )
+
+
+def _required_env(name: str) -> str:
+    value = os.getenv(name)
+    if not value:
+        raise RuntimeError(f"required environment variable is missing: {name}")
+    return value
+
+
+async def _consume_final_text(runner: Runner, session_id: str, message: Content) -> str:
+    output = []
+    async for event in runner.run_async(
+            user_id=USER_ID,
+            session_id=session_id,
+            new_message=message,
+    ):
+        if not event.is_final_response() or not event.content:
+            continue
+        output.extend(part.text or "" for part in event.content.parts or [] if not part.thought)
+    return "".join(output).strip()
+
+
+def _expected_queue(query: str) -> str:
+    if "invoice" in query or "payment" in query or "refund" in query:
+        return '{"queue":"billing"}'
+    if "password" in query:
+        return '{"queue":"account"}'
+    return '{"queue":"technical"}'

examples/optimization/eval_optimize_loop/agent/prompts/system.md

@@ -0,0 +1,3 @@
+Classify support requests.
+
+Return a short answer containing the selected queue.

examples/optimization/eval_optimize_loop/data/fake_trace.json

@@ -0,0 +1,54 @@
+{
+  "baseline": {
+    "train": [
+      {
+        "eval_id": "trace_train_1",
+        "eval_mode": "trace",
+        "conversation": [
+          {
+            "user_content": {"role": "user", "parts": [{"text": "trace invoice"}]},
+            "final_response": {"role": "model", "parts": [{"text": "{\"queue\":\"billing\"}"}]}
+          }
+        ]
+      }
+    ],
+    "validation": [
+      {
+        "eval_id": "trace_validation_1",
+        "eval_mode": "trace",
+        "conversation": [
+          {
+            "user_content": {"role": "user", "parts": [{"text": "trace password"}]},
+            "final_response": {"role": "model", "parts": [{"text": "{\"queue\":\"account\"}"}]}
+          }
+        ]
+      }
+    ]
+  },
+  "candidate": {
+    "train": [
+      {
+        "eval_id": "trace_train_1",
+        "eval_mode": "trace",
+        "conversation": [
+          {
+            "user_content": {"role": "user", "parts": [{"text": "trace invoice"}]},
+            "final_response": {"role": "model", "parts": [{"text": "{\"queue\":\"billing\"}"}]}
+          }
+        ]
+      }
+    ],
+    "validation": [
+      {
+        "eval_id": "trace_validation_1",
+        "eval_mode": "trace",
+        "conversation": [
+          {
+            "user_content": {"role": "user", "parts": [{"text": "trace password"}]},
+            "final_response": {"role": "model", "parts": [{"text": "{\"queue\":\"account\"}"}]}
+          }
+        ]
+      }
+    ]
+  }
+}

examples/optimization/eval_optimize_loop/data/train.evalset.json

@@ -0,0 +1,54 @@
+{
+  "eval_set_id": "eval_optimize_train",
+  "name": "Evaluation optimization loop training cases",
+  "eval_cases": [
+    {
+      "eval_id": "train_billing_format",
+      "conversation": [
+        {
+          "invocation_id": "train-1",
+          "user_content": {
+            "role": "user",
+            "parts": [{"text": "I was charged twice for invoice 42."}]
+          },
+          "final_response": {
+            "role": "model",
+            "parts": [{"text": "{\"queue\":\"billing\"}"}]
+          }
+        }
+      ]
+    },
+    {
+      "eval_id": "train_account_recall",
+      "conversation": [
+        {
+          "invocation_id": "train-2",
+          "user_content": {
+            "role": "user",
+            "parts": [{"text": "I cannot reset my account password."}]
+          },
+          "final_response": {
+            "role": "model",
+            "parts": [{"text": "{\"queue\":\"account\"}"}]
+          }
+        }
+      ]
+    },
+    {
+      "eval_id": "train_technical_router",
+      "conversation": [
+        {
+          "invocation_id": "train-3",
+          "user_content": {
+            "role": "user",
+            "parts": [{"text": "The SDK times out while opening a stream."}]
+          },
+          "final_response": {
+            "role": "model",
+            "parts": [{"text": "{\"queue\":\"technical\"}"}]
+          }
+        }
+      ]
+    }
+  ]
+}

examples/optimization/eval_optimize_loop/data/val.evalset.json

@@ -0,0 +1,54 @@
+{
+  "eval_set_id": "eval_optimize_validation",
+  "name": "Evaluation optimization loop validation cases",
+  "eval_cases": [
+    {
+      "eval_id": "val_candidate_improves",
+      "conversation": [
+        {
+          "invocation_id": "validation-1",
+          "user_content": {
+            "role": "user",
+            "parts": [{"text": "Please refund the duplicate subscription payment."}]
+          },
+          "final_response": {
+            "role": "model",
+            "parts": [{"text": "{\"queue\":\"billing\"}"}]
+          }
+        }
+      ]
+    },
+    {
+      "eval_id": "val_candidate_no_effect",
+      "conversation": [
+        {
+          "invocation_id": "validation-2",
+          "user_content": {
+            "role": "user",
+            "parts": [{"text": "Where can I download last year's invoices?"}]
+          },
+          "final_response": {
+            "role": "model",
+            "parts": [{"text": "{\"queue\":\"billing\"}"}]
+          }
+        }
+      ]
+    },
+    {
+      "eval_id": "val_candidate_regresses",
+      "conversation": [
+        {
+          "invocation_id": "validation-3",
+          "user_content": {
+            "role": "user",
+            "parts": [{"text": "My API connection closes with error 1006."}]
+          },
+          "final_response": {
+            "role": "model",
+            "parts": [{"text": "{\"queue\":\"technical\"}"}]
+          }
+        }
+      ]
+    }
+  ]
+}

examples/optimization/eval_optimize_loop/gate.json

@@ -0,0 +1,12 @@
+{
+  "primary_metric": "final_response_avg_score",
+  "min_score_delta": 0.1,
+  "critical_case_ids": [],
+  "max_critical_regression": 0.0,
+  "hard_case_ids": [],
+  "hard_metric_names": [],
+  "max_total_cost": null,
+  "max_duration_seconds": 180.0,
+  "train_epsilon": 0.000001,
+  "validation_epsilon": 0.000001
+}