diff --git a/AGENTS.md b/AGENTS.md index 4152a4c..38d3c53 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -11,7 +11,7 @@ ## 擴充 AI 供應商 -目前正式啟用 Gemini。供應商透過 `ai/contracts.py` 的 `AiProvider` Protocol 接入， +目前註冊 Gemini 與 Groq（固定 `openai/gpt-oss-120b`）。供應商透過 `ai/contracts.py` 的 `AiProvider` Protocol 接入， 不繼承 Gemini 實作。新增時只需實作 adapter 並在 `app.py` 註冊： 1. 提供 `id`、`name`、`model`，實作 `generate(key, system, prompt, schema, *, timeout)`。 @@ -40,4 +40,5 @@ Schema v7 只新增供應商欄位，原有 v6 Key 標記為 `gemini`，加密內容、排序及冷卻狀態保留。 Schema v8 保存 AI 編輯時間的追問狀態；schema 與版本紀錄必須在同一交易內提交。 Schema v9 新增 `ai_keys.last_error`；既有 Key 保留，舊紀錄的失敗原因預設為 `NULL`。 -第二供應商的隔離與備援透過測試替身驗證；不代表已正式支援另一個雲端服務。 +Groq 使用 strict JSON Schema，沒有可讀模型名稱 metadata，回傳 `None` 並顯示共用 AI 標籤。 +供應商隔離、備援與確認流程透過模擬 HTTP 回應驗證；不代表已完成真實 Groq Key 實測。 diff --git a/CHANGELOG.md b/CHANGELOG.md index 100aea3..396cf22 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,12 @@ Remindly 的重要變更會記錄在這裡。 ## Unreleased +- 修正週期提醒的本機時間約束：只有時鐘時依週期計算首次日期，避免每週一、三、五被覆寫成週六；保留明確起始日期。 + +- Groq 請求明確設定 User-Agent，避免預設 Python urllib 被 Cloudflare 以 HTTP 403／1010 擋下。 + +- 新增 Groq adapter，固定 GPT-OSS 120B 與 strict JSON Schema；沿用個人 Key、冷卻、確認及 18 秒共用預算，Gemini 仍為預設供應商。 + - 修正單組 AI Key 提前在 6 秒逾時：請求改用剩餘共用等待預算，仍最多嘗試三組 Key，預算 18 秒。 - AI 供應商失敗原因依 Key 分類顯示並持久化（schema v9），供 `/ai` 查詢；成功後清除，不保存上游錯誤原文。 - AI 供應商介面、共用備援與標準 JSON Schema；Gemini adapter 獨立負責 API 格式與錯誤轉換。 diff --git a/README.md b/README.md index 0b23e10..b3142ba 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ - 🔎 **自然語言建立提醒**：支援「明天下午三點提醒我倒垃圾」這類中文輸入，拆出提醒時間、提醒事項與參與者。 -- 🧠 **自帶 Gemini 語意解析**：個人可設定多組 API Key，以 Gemini 3.5 Flash-Lite 理解口語、不同日期及草稿修正；不明確時追問，仍需確認才建立。未設定或關閉 AI 時使用原本規則解析。 +- 🧠 **自帶 Key 的 AI 語意解析**：個人可設定多組 Gemini／Groq API Key，以 Gemini 3.5 Flash-Lite 或 GPT-OSS 120B 理解口語、不同日期及草稿修正；不明確時追問，仍需確認才建立。未設定或關閉 AI 時使用原本規則解析。 - 🔁 **週期性提醒**：`每天 09:00`、`每週一 09:00`、`每個月 1, 18, 25 號 09:00`、`每年 12/25 08:00`、`每 15 分鐘 / 每 2 小時` 都能一句話建立，到期後自動排下一次。內建解析遇到無效語法會直接拒絕；AI 解析遇到缺漏或歧義則追問。 @@ -78,33 +78,36 @@ Telegram Update | `/cancel R-8F3K` | 取消提醒。 | | `/timezone Asia/Taipei` | 設定時區。 | | `/groupmode on\|off\|status` | 查看或切換群組自然語言模式。 | -| `/ai` | 私聊管理個人 Gemini Key、開關及備援順序。 | +| `/ai` | 私聊管理個人 Gemini／Groq Key、開關及備援順序。 | ### 個人 AI 設定（選用） **不設定也能用。** 所有使用者功能相同，不分免費／付費；預設使用內建規則解析。 -想用 AI 理解口語提醒，可自備 Gemini API Key，平台不提供 Key。 +想用 AI 理解口語提醒，可自備 Gemini 或 Groq API Key，平台不提供 Key。 **開啟：** 與 Bot 私聊，輸入： ```text -/ai add <你的 Gemini API Key> +/ai add gemini <你的 Gemini API Key> +/ai add groq <你的 Groq API Key> ``` +選一個供應商新增即可；省略供應商仍預設 Gemini。Groq 固定使用 `openai/gpt-oss-120b`，不提供模型切換。 首次新增後自動開啟；若之前手動關閉過，請再輸入 `/ai on`。 接著直接說「明晚八點叫我拿包裹」，確認後才會建立提醒。 確認卡會標示「使用 … 解析」，方便核對本次使用的模型或內建規則。 +Groq 模型 metadata 沒有可讀名稱，確認卡顯示「使用 AI 解析」；供應商與模型 ID 仍存於解析紀錄。 | 想做什麼 | 指令 | | --- | --- | | 查看狀態與 Key ID | `/ai` | | 關閉／重新開啟 AI | `/ai off` ／ `/ai on` | -| 新增備援 Key | `/ai add <另一組 Key>` | +| 新增備援 Key | `/ai add gemini ` ／ `/ai add groq ` | | 將某組 Key 優先使用 | `/ai move 1` | | 刪除 Key | `/ai delete ` | -- **費用自己掌握：** 依你的 Google 專案計費；付費專案可能直接收費，不會先跳出付費確認。同專案多組 Key 共用額度。 -- **資料會傳出：** Key 經 Telegram 傳至 Bot 加密保存；啟用後，一般私聊文字、群組觸發訊息及必要上下文會送至 Google。 +- **費用自己掌握：** 依你的供應商帳號／專案計費；付費帳號可能直接收費，不會先跳出付費確認。同帳號／專案多組 Key 可能共用額度。 +- **資料會傳出：** Key 經 Telegram 傳至 Bot 加密保存；啟用後，一般私聊文字、群組觸發訊息及必要上下文會送至實際使用的供應商（Google 或 Groq）；備援時可能依序送至兩者。 - **時間與事項約束：** 可明確辨識的時間及事項區由共用本機規則定案，事項保留原文；只補時間時保留草稿日期、事項、對象與週期。已確定欄位的多餘追問會移除，未解條件則保留；無效或已過的時間會追問，不接受模型換成其他日期。其他口語歧義仍須確認。 - **AI 暫時不能用：** 草稿會保留；稍後重試，或 `/ai off` 後重送，改用內建解析。 - **等待時間：** 最多嘗試三組 Key，共用 18 秒等待預算；逾時不代表 Key 失效或免費額度用盡。 @@ -196,7 +199,7 @@ cp .env.example .env | `SCHEDULER_INTERVAL_SECONDS` | 否 | `10` | 到期提醒掃描間隔。 | | `DRAFT_TTL_MINUTES` | 否 | `10` | 草稿與修改 session 過期時間。 | | `LOG_LEVEL` | 否 | `INFO` | Logging level。 | -| `AI_ENCRYPTION_KEY` | AI 設定必填 | 無 | Fernet 主金鑰，加密個人 Gemini Key；需固定並另行備份。 | +| `AI_ENCRYPTION_KEY` | AI 設定必填 | 無 | Fernet 主金鑰，加密個人 AI Key；需固定並另行備份。 | ### 啟用個人 AI（選用） @@ -311,7 +314,7 @@ Runtime 新增 `cryptography` 供 Key 加密。更新 runtime dependencies 後 uv export --locked --no-dev --no-emit-project --format requirements-txt --output-file requirements.txt ``` -`tests/test_ai.py` 與 `tests/test_ai_flow.py` 使用模擬 Gemini 回應及暫存 SQLite，檢查 REST +`tests/test_ai.py`、`tests/test_groq.py` 與 `tests/test_ai_flow.py` 使用模擬供應商回應及暫存 SQLite，檢查 REST 請求、結構化驗證、Key 備援與 Telegram 完整流程，不消耗真實 API 額度。這些檢查不代表 已量測模型中文準確率；實際模型表現需另用真實 Key 驗證。 diff --git a/scripts/groq_live_eval.py b/scripts/groq_live_eval.py new file mode 100644 index 0000000..76df07a --- /dev/null +++ b/scripts/groq_live_eval.py @@ -0,0 +1,221 @@ +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import time +from dataclasses import asdict +from datetime import datetime +from pathlib import Path +from zoneinfo import ZoneInfo + +from remindly.ai.contracts import AiUnavailable, GeneratedResponse, ProviderError +from remindly.ai.groq import GroqProvider +from remindly.ai.parser import SCHEMA, SYSTEM, SemanticParser, validate_result +from remindly.reminders.models import MentionKind, ParseResult, Participant, ReminderDraft +from remindly.settings import load_dotenv + +ROOT = Path(__file__).resolve().parents[1] +CASES = ROOT / "tests/fixtures/groq_live_cases.jsonl" +PERSON = Participant("Orange", MentionKind.TEXT_MENTION, user_id=7, username="orange") +ALICE = Participant("Alice", MentionKind.TEXT_MENTION, user_id=8, username="alice") + + +def summarize(result: ParseResult) -> dict: + return { + "title": result.title, + "at": result.remind_at.isoformat() if result.remind_at else None, + "missing": sorted(result.missing_fields), + "complete": bool(result.title and result.remind_at and not result.missing_fields), + "participants": sorted(p.stable_key for p in result.participants), + "recurrence": json.loads(json.dumps(asdict(result.recurrence))) + if result.recurrence + else None, + "question": result.raw.get("question"), + "intent": result.raw.get("intent"), + } + + +def differences(actual: dict, expected: dict) -> dict: + failures = {} + for field, wanted in expected.items(): + value = actual.get(field) + if field == "recurrence" and wanted is not None and isinstance(value, dict): + value = {key: value.get(key) for key in wanted} + if field in {"missing", "participants"}: + wanted = sorted(wanted) + if value != wanted: + failures[field] = {"expected": wanted, "actual": value} + return failures + + +def context_for(case: dict) -> ReminderDraft | None: + seed = case.get("seed") + if not seed: + return None + return ReminderDraft( + id="seed", + chat_id=100, + chat_type="private", + creator_user_id=7, + timezone=case["timezone"], + source_text=seed["source"], + title=seed["title"], + remind_at=datetime.fromisoformat(seed["at"]) if seed.get("at") else None, + participants=[PERSON], + missing_fields=seed["missing"] + (["clarification"] if seed.get("condition") else []), + parse_result={ + "grain": "ai", + "question": seed.get("question"), + "condition_question": seed.get("question"), + }, + ) + + +class RecordingGenerator: + def __init__(self, key: str): + self.key = key + self.provider = GroqProvider() + self.raw = None + + def enabled(self, user_id: int) -> bool: + return True + + def generate(self, user_id, system, prompt, schema): + self.raw = self.provider.generate(self.key, system, prompt, schema) + return GeneratedResponse(self.raw, self.provider.id, self.provider.model) + + +def evaluate(case: dict, generator: RecordingGenerator) -> dict: + now = datetime.fromisoformat(case["now"]).astimezone(ZoneInfo(case["timezone"])) + participants = [PERSON, ALICE] if case.get("alice") else [PERSON] + generator.raw = None + local_error = None + try: + local = summarize( + SemanticParser(generator).parse( + case["text"], 7, now, participants, context=context_for(case) + ) + ) + except AiUnavailable: + local, local_error = None, "validation" + raw_error = None + try: + raw = summarize(validate_result(generator.raw, now, participants)) + except (ValueError, TypeError, KeyError, OverflowError): + raw, raw_error = None, "validation" + return { + "raw": generator.raw, + "raw_result": raw, + "local_result": local, + "raw_failures": differences(raw, case["expected"]) if raw else {"error": raw_error}, + "local_failures": differences(local, case["expected"]) if local else {"error": local_error}, + } + + +def wait_until(deadline: float) -> None: + while (remaining := deadline - time.time()) > 0: + print(f"cooldown: {remaining:.0f}s remaining", flush=True) + time.sleep(min(30, remaining)) + + +def main() -> None: + parser = argparse.ArgumentParser( + description="Groq live regression evaluation; uses personal quota" + ) + parser.add_argument( + "--output", type=Path, required=True, help="Append-only JSONL; resumes this run" + ) + args = parser.parse_args() + load_dotenv(ROOT / ".env") + key = os.environ.get("GROQ_API_KEY", "").strip() + if not key: + raise SystemExit("GROQ_API_KEY missing") + cases = [json.loads(line) for line in CASES.read_text().splitlines() if line.strip()] + digest = hashlib.sha256(CASES.read_bytes() + SYSTEM.encode() + json.dumps(SCHEMA).encode()) + for relative in ( + "ai/groq.py", + "ai/parser.py", + "reminders/parser.py", + "reminders/recurrence.py", + ): + digest.update((ROOT / "src/remindly" / relative).read_bytes()) + meta = { + "event": "meta", + "signature": digest.hexdigest(), + "model": GroqProvider.model, + "cases": len(cases), + "requests": sum(case["repeats"] for case in cases), + } + previous = ( + [json.loads(line) for line in args.output.read_text().splitlines()] + if args.output.exists() + else [] + ) + if previous and previous[0] != meta: + raise SystemExit("Suite or code changed; choose a new output path") + args.output.parent.mkdir(parents=True, exist_ok=True) + generator = RecordingGenerator(key) + with args.output.open("a") as stream: + + def save(row): + stream.write(json.dumps(row, ensure_ascii=False).replace(key, "[REDACTED]") + "\n") + stream.flush() + + if not previous: + save(meta) + done = {(row["id"], row["run"]) for row in previous if row["event"] == "result"} + if previous and previous[-1]["event"] == "retry": + wait_until(previous[-1]["retry_at"]) + for case in cases: + for run in range(1, case["repeats"] + 1): + if (case["id"], run) in done: + continue + while True: + started = time.monotonic() + try: + result = evaluate(case, generator) + except ProviderError as exc: + row = { + "event": "retry", + "id": case["id"], + "run": run, + "kind": exc.kind, + "cooldown": exc.cooldown, + "retry_at": time.time() + exc.cooldown, + "seconds": round(time.monotonic() - started, 3), + } + if exc.kind not in {"quota", "rate", "daily_quota"}: + row["event"] = "fatal" + save(row) + raise SystemExit(f"Stopped: {exc.kind}; progress saved") from None + save(row) + print(f"429 {exc.kind}: cooling down {exc.cooldown}s", flush=True) + wait_until(row["retry_at"]) + continue + row = { + "event": "result", + "id": case["id"], + "run": run, + "source": case["source"], + "text": case["text"], + "now": case["now"], + "expected": case["expected"], + "seconds": round(time.monotonic() - started, 3), + **result, + } + save(row) + done.add((case["id"], run)) + print( + f"{len(done)}/{meta['requests']} {case['id']} #{run}: " + f"raw={'FAIL' if result['raw_failures'] else 'PASS'} " + f"local={'FAIL' if result['local_failures'] else 'PASS'} " + f"{row['seconds']}s", + flush=True, + ) + break + + +if __name__ == "__main__": + main() diff --git a/src/remindly/ai/groq.py b/src/remindly/ai/groq.py new file mode 100644 index 0000000..0c7f3d6 --- /dev/null +++ b/src/remindly/ai/groq.py @@ -0,0 +1,118 @@ +from __future__ import annotations + +import http.client +import json +import math +import urllib.error +import urllib.request +from collections.abc import Callable +from contextlib import suppress +from typing import Any + +from remindly.ai.contracts import AI_TIME_BUDGET_SECONDS, ProviderError +from remindly.ai.gemini import NoRedirect + + +class GroqProvider: + id = "groq" + name = "Groq" + model = "openai/gpt-oss-120b" + + def __init__(self, *, opener: Callable[..., Any] | None = None) -> None: + self._opener = opener or urllib.request.build_opener(NoRedirect()).open + + def model_display_name(self, key: str, *, timeout: float) -> str | None: + # Groq 的模型 metadata 沒有可讀名稱欄位，沿用共用 AI 標籤。 + return None + + def generate( + self, + key: str, + system: str, + prompt: dict[str, object], + schema: dict[str, object], + *, + timeout: float = AI_TIME_BUDGET_SECONDS, + ) -> dict[str, object]: + payload = { + "model": self.model, + "messages": [ + {"role": "system", "content": system}, + {"role": "user", "content": json.dumps(prompt, ensure_ascii=False)}, + ], + "response_format": { + "type": "json_schema", + "json_schema": {"name": "reminder", "strict": True, "schema": groq_schema(schema)}, + }, + "reasoning_effort": "low", + "max_completion_tokens": 2048, + "temperature": 0, + } + request = urllib.request.Request( + "https://api.groq.com/openai/v1/chat/completions", + data=json.dumps(payload).encode(), + method="POST", + headers={ + "Content-Type": "application/json", + "Authorization": f"Bearer {key}", + "User-Agent": "Remindly", + }, + ) + try: + with self._opener(request, timeout=timeout) as response: + body = json.loads(response.read(131073)) + except urllib.error.HTTPError as exc: + # 不讀取可能回顯 Key 的錯誤內文，只採用狀態碼與結構化 headers。 + try: + if exc.code == 401: + raise ProviderError("invalid") from None + if exc.code == 403: + raise ProviderError("permission") from None + if exc.code == 429: + headers = exc.headers or {} + cooldown = 60 + with suppress(TypeError, ValueError, OverflowError): + cooldown = max( + 60, min(90001, math.ceil(float(headers.get("retry-after", "60")))) + ) + if headers.get("x-ratelimit-remaining-requests") == "0": + raise ProviderError("daily_quota", cooldown) from None + if headers.get("x-ratelimit-remaining-tokens") == "0": + raise ProviderError("rate", cooldown) from None + raise ProviderError("quota", cooldown) from None + if exc.code >= 500: + raise ProviderError("temporary") from None + raise ProviderError("request") from None + finally: + exc.close() + except TimeoutError: + raise ProviderError("timeout") from None + except urllib.error.URLError as exc: + kind = "timeout" if isinstance(exc.reason, TimeoutError) else "connection" + raise ProviderError(kind) from None + except (OSError, http.client.HTTPException): + raise ProviderError("connection") from None + except (ValueError, UnicodeError): + raise ProviderError("response") from None + try: + choice = body["choices"][0] + if choice["finish_reason"] != "stop" or choice["message"].get("refusal"): + raise ValueError("incomplete") + result = json.loads(choice["message"]["content"]) + if not isinstance(result, dict): + raise ValueError("not an object") + return result + except (KeyError, IndexError, TypeError, ValueError, AttributeError): + raise ProviderError("response") from None + + +def groq_schema(schema: dict[str, object]) -> dict[str, object]: + result = dict(schema) + if "properties" in schema: + result["properties"] = { + name: groq_schema(value) for name, value in schema["properties"].items() + } + result["additionalProperties"] = False + if "items" in schema: + result["items"] = groq_schema(schema["items"]) + return result diff --git a/src/remindly/app.py b/src/remindly/app.py index 1e82827..a6a16ac 100644 --- a/src/remindly/app.py +++ b/src/remindly/app.py @@ -7,6 +7,7 @@ from remindly import __version__ from remindly.ai.gateway import AiGateway from remindly.ai.gemini import GeminiProvider +from remindly.ai.groq import GroqProvider from remindly.ai.keys import ApiKeyStore from remindly.ai.parser import SemanticParser from remindly.bot.ai_commands import AiCommands @@ -67,7 +68,9 @@ def main() -> None: repository = ReminderRepository(settings.database_path) repository.migrate() ai_keys = ApiKeyStore(repository, settings.ai_encryption_key) - ai_gateway = AiGateway(ai_keys, {"gemini": GeminiProvider()}, default_provider="gemini") + ai_gateway = AiGateway( + ai_keys, {"gemini": GeminiProvider(), "groq": GroqProvider()}, default_provider="gemini" + ) client = TelegramClient(settings.token) renderer = ReminderRenderer() diff --git a/src/remindly/reminders/parser.py b/src/remindly/reminders/parser.py index 3c76cb1..be68580 100644 --- a/src/remindly/reminders/parser.py +++ b/src/remindly/reminders/parser.py @@ -2,7 +2,7 @@ import calendar import re -from dataclasses import dataclass +from dataclasses import dataclass, replace from datetime import UTC, datetime, timedelta, tzinfo from zoneinfo import ZoneInfo @@ -289,7 +289,26 @@ def explicit_time( and marker[1] != RecurrencePeriod.INTERVAL and prefix.startswith(marker[0]) ): - return self.explicit_time(cleaned.replace(marker[0], "", 1), now, existing_date) + constraint = self.explicit_time(cleaned.replace(marker[0], "", 1), now, existing_date) + if constraint: + temporal, time_only = constraint + consumed = LEADING_ZAI_RE.sub("", temporal.consumed_text).strip() + if ( + temporal.remind_at + and not temporal.missing_time + and (TIME_RE.match(consumed) or PART_OF_DAY_RE.match(consumed)) + ): + rule = RecurrenceRule( + period=marker[1], + hour=temporal.remind_at.hour, + minute=temporal.remind_at.minute, + **marker[2], + ) + # 單純時鐘的日期由週期計算，不能沿用一次性提醒的今天／明天。 + return replace( + temporal, remind_at=next_fire(rule, now), is_past=False + ), time_only + return constraint prompt = PROMPT_SPLIT_RE.search(cleaned) if prompt: left = cleaned[: prompt.start()].strip() diff --git a/tests/fixtures/README.md b/tests/fixtures/README.md index 3c4cd75..5889af3 100644 --- a/tests/fixtures/README.md +++ b/tests/fixtures/README.md @@ -107,3 +107,58 @@ uv run python -B -m unittest discover -s tests -p 'test_time_corpus.py' -v 並確認舊確認按鈕與未完成的新草稿都不能建立提醒。 有草稿的變更指令即使模型誤標 `create`／`not_reminder`，仍是更新；未解條件必須保留。 明確另開提醒的文字不套用這項更新約束。 + +## Groq 真實 API 回歸測試 + +`groq_live_cases.jsonl` 從上述既有資料集挑選 36 個不同案例；每筆以 `source` 指回原始 +檔名與 ID，預期值沿用已核對的資料，不從 Groq 或本機解析輸出產生。 +單獨時間案例補成「提醒我〈時間〉買湯」。8 個曾出現事項改寫、日期或追問漂移的案例 +各執行 3 次，其餘各 1 次，共 52 次請求。草稿案例使用固定 seed，不以當輪模型輸出 +作為下一輪測試前提，避免前一輪錯誤污染獨立案例。 + +這是手動實測，不會在 `make check`／CI 呼叫 API。需要專案 `.env` 的個人 +`GROQ_API_KEY`，會消耗該帳號額度： + +```sh +uv run python scripts/groq_live_eval.py --output /tmp/groq-live-results.jsonl +``` + +每次請求使用正式 `GroqProvider`、中文 prompt、schema 與 18 秒 timeout；先保留模型 +原始 JSON，以 `validate_result()` 檢查 schema／欄位，再獨立比對經 `SemanticParser` +處理後的結果。比對事項、首次時間、缺漏欄位、完整性、對象與週期；因此本機修正了 +模型錯誤也不會把模型原始結果算成通過。週期只比較該筆明列的欄位。 + +工具直接呼叫 adapter，不經正式 Key store／Gateway，不讀寫正式資料庫、不連 Telegram。 +HTTP 429 使用 adapter 提供的冷卻秒數後重試同一筆，不計為已完成的語意測試；等待每 +30 秒顯示一次進度。其他供應商錯誤停止並保存進度，避免憑證失效或連線問題持續消耗請求。 +輸出逐行追加原始回應、兩層結果、差異及耗時；不保存 Key、URL 或上游錯誤內文。 +使用相同輸出路徑可續跑，已完成的案例／輪次不會重送，最後一次 429 的冷卻亦會保留。 +案例、prompt、schema 或主要解析程式碼變更時須改用新的輸出路徑，避免混合不同版本結果。 + +`tests/test_groq_live_eval.py` 離線驗證資料來源、模型與本機結果分開計分、429 冷卻及續跑。 + +### 2026-10-30 實測結果 + +`groq_live_results_2026-10-30.jsonl` 保存這輪完整原始回應與比對結果，模型為 +`openai/gpt-oss-120b`；所有輸入為上述回歸案例，未發送 Telegram 或建立正式提醒。 +`tests/test_groq_live_eval.py` 會離線重播全部 52 筆原始回應，比對各筆既有預期值。 + +| 指標 | 結果 | +| --- | --- | +| 不同案例／完成語意請求 | 36／52 | +| 模型原始結果符合預期 | 26/52（50%） | +| 本機處理後符合預期 | 52/52（100%） | +| 模型不符預期的不同案例 | 14 | +| 實際 API 請求 | 58：52 次取得回應、5 次 429、1 次供應商 request 錯誤 | +| 成功取得回應的耗時 | 平均 1.154 秒；中位數 1.109 秒；P95 1.820 秒；範圍 0.704–2.258 秒 | + +5 次 429 都依 adapter 提供的 60 秒冷卻後續跑；另一次 `request` 錯誤使工具停止， +重新執行同一輸出路徑後從該筆續跑成功，沒有重送已完成案例。只保存固定錯誤代碼， +未讀取／保存上游錯誤內文，因此無法判定該次 request 錯誤的根因。 +上述耗時涵蓋 adapter 與本機解析，不含失敗請求與冷卻等待。 + +模型錯誤包含下下週一日期、未指定上午／下午的五點、過期月日跨年、早上十二點、 +事項內「或」被當成時間選項、事項被改寫，以及草稿修正遺失既有條件。 +8 個重複案例中，曆法名詞與補事項案例的原始結果各 3/3 通過；其餘 6 個案例 +原始結果各 0/3，本機結果均為 3/3。這批案例刻意集中既有錯誤與邊界， +50%／100% 僅代表本資料集的嚴格欄位比對，不能推估一般輸入準確率或正式環境可用率。 diff --git a/tests/fixtures/groq_live_cases.jsonl b/tests/fixtures/groq_live_cases.jsonl new file mode 100644 index 0000000..4367dbe --- /dev/null +++ b/tests/fixtures/groq_live_cases.jsonl @@ -0,0 +1,36 @@ +{"id":"time-slash-date","source":"time_regressions.jsonl#slash-date","now":"2026-10-23T12:10:00+08:00","timezone":"Asia/Taipei","text":"提醒我11/17 23:55買湯","repeats":1,"expected":{"title":"買湯","at":"2026-11-17T23:55:00+08:00","complete":true,"missing":[],"participants":["user:7"],"recurrence":null}} +{"id":"time-chinese-date","source":"time_regressions.jsonl#chinese-date","now":"2026-10-23T12:10:00+08:00","timezone":"Asia/Taipei","text":"提醒我十一月十七號晚上九點買湯","repeats":1,"expected":{"title":"買湯","at":"2026-11-17T21:00:00+08:00","complete":true,"missing":[],"participants":["user:7"],"recurrence":null}} +{"id":"time-iso-date","source":"time_regressions.jsonl#iso-date","now":"2026-10-23T12:10:00+08:00","timezone":"Asia/Taipei","text":"提醒我2026-11-17 23:55買湯","repeats":1,"expected":{"title":"買湯","at":"2026-11-17T23:55:00+08:00","complete":true,"missing":[],"participants":["user:7"],"recurrence":null}} +{"id":"time-tomorrow-alias","source":"time_regressions.jsonl#tomorrow-alias","now":"2026-10-23T12:10:00+08:00","timezone":"Asia/Taipei","text":"提醒我明日 23:55買湯","repeats":1,"expected":{"title":"買湯","at":"2026-10-24T23:55:00+08:00","complete":true,"missing":[],"participants":["user:7"],"recurrence":null}} +{"id":"time-next-next-monday","source":"time_regressions.jsonl#next-next-monday","now":"2026-10-23T12:10:00+08:00","timezone":"Asia/Taipei","text":"提醒我下下週一九點買湯","repeats":1,"expected":{"title":"買湯","at":"2026-11-02T09:00:00+08:00","complete":true,"missing":[],"participants":["user:7"],"recurrence":null}} +{"id":"time-relative-hours","source":"time_regressions.jsonl#relative-hours","now":"2026-10-23T12:10:00+08:00","timezone":"Asia/Taipei","text":"提醒我三小時後買湯","repeats":1,"expected":{"title":"買湯","at":"2026-10-23T15:10:00+08:00","complete":true,"missing":[],"participants":["user:7"],"recurrence":null}} +{"id":"time-relative-half-hour","source":"time_regressions.jsonl#relative-half-hour","now":"2026-10-23T12:10:00+08:00","timezone":"Asia/Taipei","text":"提醒我半小時後買湯","repeats":1,"expected":{"title":"買湯","at":"2026-10-23T12:40:00+08:00","complete":true,"missing":[],"participants":["user:7"],"recurrence":null}} +{"id":"time-half-hour-clock","source":"time_regressions.jsonl#half-hour-clock","now":"2026-10-23T12:10:00+08:00","timezone":"Asia/Taipei","text":"提醒我下午三點半買湯","repeats":1,"expected":{"title":"買湯","at":"2026-10-23T15:30:00+08:00","complete":true,"missing":[],"participants":["user:7"],"recurrence":null}} +{"id":"time-invalid-day","source":"time_regressions.jsonl#invalid-day","now":"2026-10-23T12:10:00+08:00","timezone":"Asia/Taipei","text":"提醒我11/31 23:55買湯","repeats":1,"expected":{"title":"買湯","at":null,"complete":false,"missing":["time"],"participants":["user:7"],"recurrence":null}} +{"id":"time-invalid-hour","source":"time_regressions.jsonl#invalid-hour","now":"2026-10-23T12:10:00+08:00","timezone":"Asia/Taipei","text":"提醒我25:00買湯","repeats":1,"expected":{"title":"買湯","at":null,"complete":false,"missing":["time"],"participants":["user:7"],"recurrence":null}} +{"id":"time-explicit-past","source":"time_regressions.jsonl#explicit-past","now":"2026-10-23T12:10:00+08:00","timezone":"Asia/Taipei","text":"提醒我2026-10-22 09:00買湯","repeats":1,"expected":{"title":"買湯","at":"2026-10-22T09:00:00+08:00","complete":false,"missing":["time"],"participants":["user:7"],"recurrence":null}} +{"id":"time-year-rollover","source":"time_regressions.jsonl#year-rollover","now":"2026-12-31T23:00:00+08:00","timezone":"Asia/Taipei","text":"提醒我1/1 09:00買湯","repeats":1,"expected":{"title":"買湯","at":"2027-01-01T09:00:00+08:00","complete":true,"missing":[],"participants":["user:7"],"recurrence":null}} +{"id":"time-leap-day","source":"time_regressions.jsonl#leap-day","now":"2028-02-28T12:10:00+08:00","timezone":"Asia/Taipei","text":"提醒我2/29 09:00買湯","repeats":1,"expected":{"title":"買湯","at":"2028-02-29T09:00:00+08:00","complete":true,"missing":[],"participants":["user:7"],"recurrence":null}} +{"id":"time-new-york","source":"time_regressions.jsonl#new-york","now":"2026-10-23T23:00:00-04:00","timezone":"America/New_York","text":"提醒我明早九點買湯","repeats":1,"expected":{"title":"買湯","at":"2026-10-24T09:00:00-04:00","complete":true,"missing":[],"participants":["user:7"],"recurrence":null}} +{"id":"reminder-beef-soup-report","source":"time_regressions.jsonl#beef-soup-report","now":"2026-10-23T12:10:00+08:00","timezone":"Asia/Taipei","text":"提醒我在五點的時候買兩份清燉牛肉湯","repeats":1,"alice":false,"expected":{"title":"買兩份清燉牛肉湯","at":"2026-10-24T05:00:00+08:00","missing":[],"complete":true,"participants":["user:7"],"recurrence":null}} +{"id":"reminder-suffix-in-title","source":"time_regressions.jsonl#suffix-in-title","now":"2026-10-23T12:10:00+08:00","timezone":"Asia/Taipei","text":"提醒我在五點的時候買喝湯的時候要用的湯匙","repeats":1,"alice":false,"expected":{"title":"買喝湯的時候要用的湯匙","at":"2026-10-24T05:00:00+08:00","missing":[],"complete":true,"participants":["user:7"],"recurrence":null}} +{"id":"reminder-title-period","source":"time_regressions.jsonl#title-period","now":"2026-10-23T12:10:00+08:00","timezone":"Asia/Taipei","text":"五點買晚上要用的湯匙","repeats":1,"alice":false,"expected":{"title":"買晚上要用的湯匙","at":"2026-10-24T05:00:00+08:00","missing":[],"complete":true,"participants":["user:7"],"recurrence":null}} +{"id":"reminder-title-date-slash","source":"time_regressions.jsonl#title-date-slash","now":"2026-10-23T12:10:00+08:00","timezone":"Asia/Taipei","text":"7/30 21:00提醒我08/01要去家樂福","repeats":1,"alice":false,"expected":{"title":"08/01要去家樂福","at":"2027-07-30T21:00:00+08:00","missing":[],"complete":true,"participants":["user:7"],"recurrence":null}} +{"id":"reminder-multiple-participants","source":"time_regressions.jsonl#multiple-participants","now":"2026-10-23T12:10:00+08:00","timezone":"Asia/Taipei","text":"明天下午三點提醒我和 @alice 倒垃圾","repeats":1,"alice":true,"expected":{"title":"倒垃圾","at":"2026-10-24T15:00:00+08:00","missing":[],"complete":true,"participants":["user:7","user:8"],"recurrence":null}} +{"id":"reminder-missing-title","source":"time_regressions.jsonl#missing-title","now":"2026-10-23T12:10:00+08:00","timezone":"Asia/Taipei","text":"提醒我明天下午三點","repeats":1,"alice":false,"expected":{"title":null,"at":"2026-10-24T15:00:00+08:00","missing":["title"],"complete":false,"participants":["user:7"],"recurrence":null}} +{"id":"reminder-breakfast","source":"time_regressions.jsonl#breakfast","now":"2026-10-23T12:10:00+08:00","timezone":"Asia/Taipei","text":"提醒我買早餐","repeats":1,"alice":false,"expected":{"title":"買早餐","at":null,"missing":["time"],"complete":false,"participants":["user:7"],"recurrence":null}} +{"id":"reminder-daily","source":"time_regressions.jsonl#daily","now":"2026-10-23T12:10:00+08:00","timezone":"Asia/Taipei","text":"每天九點提醒我買湯","repeats":1,"alice":false,"expected":{"title":"買湯","at":"2026-10-24T09:00:00+08:00","missing":[],"complete":true,"participants":["user:7"],"recurrence":{"period":"daily","hour":9,"minute":0}}} +{"id":"reminder-weekly","source":"time_regressions.jsonl#weekly","now":"2026-10-23T12:10:00+08:00","timezone":"Asia/Taipei","text":"每週一、三、五九點提醒我買湯","repeats":1,"alice":false,"expected":{"title":"買湯","at":"2026-10-26T09:00:00+08:00","missing":[],"complete":true,"participants":["user:7"],"recurrence":{"period":"weekly","hour":9,"minute":0,"weekdays":[0,2,4]}}} +{"id":"reminder-monthly","source":"time_regressions.jsonl#monthly","now":"2026-10-23T12:10:00+08:00","timezone":"Asia/Taipei","text":"每月1號九點提醒我買湯","repeats":1,"alice":false,"expected":{"title":"買湯","at":"2026-11-01T09:00:00+08:00","missing":[],"complete":true,"participants":["user:7"],"recurrence":{"period":"monthly","hour":9,"minute":0,"month_days":[1]}}} +{"id":"reminder-yearly","source":"time_regressions.jsonl#yearly","now":"2026-10-23T12:10:00+08:00","timezone":"Asia/Taipei","text":"每年11/17九點提醒我買湯","repeats":1,"alice":false,"expected":{"title":"買湯","at":"2026-11-17T09:00:00+08:00","missing":[],"complete":true,"participants":["user:7"],"recurrence":{"period":"yearly","hour":9,"minute":0,"year_month":11,"year_day":17}}} +{"id":"reminder-interval","source":"time_regressions.jsonl#interval","now":"2026-10-23T12:10:00+08:00","timezone":"Asia/Taipei","text":"每15分鐘提醒我喝水","repeats":1,"alice":false,"expected":{"title":"喝水","at":"2026-10-23T12:25:00+08:00","missing":[],"complete":true,"participants":["user:7"],"recurrence":{"period":"interval","interval_seconds":900}}} +{"id":"content-morning-twelve-round-1","source":"gemini_content_regressions.jsonl#morning-twelve-round-1","now":"2026-10-24T12:10:00+08:00","timezone":"Asia/Taipei","text":"提醒我 明天早上十二點 買湯","repeats":3,"expected":{"title":"買湯","at":"2026-10-25T00:00:00+08:00","complete":true,"missing":[],"participants":["user:7"],"recurrence":null}} +{"id":"content-alternative-in-subject-round-1","source":"gemini_content_regressions.jsonl#alternative-in-subject-round-1","now":"2026-10-24T12:10:00+08:00","timezone":"Asia/Taipei","text":"提醒我11/17 23:55買湯或11/18要用的湯匙","repeats":3,"expected":{"title":"買湯或11/18要用的湯匙","at":"2026-11-17T23:55:00+08:00","complete":true,"missing":[],"participants":["user:7"],"recurrence":null}} +{"id":"content-exact-current-clock-rolls-round-2","source":"gemini_content_regressions.jsonl#exact-current-clock-rolls-round-2","now":"2026-10-24T12:10:00+08:00","timezone":"Asia/Taipei","text":"提醒我 12:10 買湯","repeats":1,"expected":{"title":"買湯","at":"2026-10-25T12:10:00+08:00","complete":true,"missing":[],"participants":["user:7"],"recurrence":null}} +{"id":"content-chinese-date-extra-day-digit-round-2","source":"gemini_content_regressions.jsonl#chinese-date-extra-day-digit-round-2","now":"2026-10-24T12:10:00+08:00","timezone":"Asia/Taipei","text":"提醒我 11月170日 23:55 買湯","repeats":1,"expected":{"title":"買湯","at":null,"complete":false,"missing":["time"],"participants":["user:7"],"recurrence":null}} +{"id":"content-recorded-calendar-noun-question","source":"gemini_content_regressions.jsonl#recorded-calendar-noun-question","now":"2026-10-24T12:10:00+08:00","timezone":"Asia/Taipei","text":"提醒我明天15:00工作日行事曆","repeats":3,"expected":{"title":"工作日行事曆","at":"2026-10-25T15:00:00+08:00","complete":true,"missing":[],"participants":["user:7"],"recurrence":null}} +{"id":"content-recorded-literal-title-rewritten-round-1","source":"gemini_content_regressions.jsonl#recorded-literal-title-rewritten-round-1","now":"2026-10-24T12:10:00+08:00","timezone":"Asia/Taipei","text":"提醒我明天15:00之前的會議紀錄","repeats":3,"expected":{"title":"之前的會議紀錄","at":"2026-10-25T15:00:00+08:00","complete":true,"missing":[],"participants":["user:7"],"recurrence":null}} +{"id":"followup-recorded-garbled-context-title-rewrite","source":"gemini_followup_regressions.jsonl#recorded-garbled-context-title-rewrite","now":"2026-10-24T12:10:00+08:00","timezone":"Asia/Taipei","text":"寄出年度報表","repeats":3,"seed":{"title":null,"at":"2026-10-25T15:00:00+08:00","missing":["title"],"source":"提醒我明天15:00","question":"職耇與䐂蕩耇䐂略耇䐂耇䐂","condition":false},"expected":{"title":"寄出年度報表","at":"2026-10-25T15:00:00+08:00","complete":true,"missing":[],"participants":["user:7"],"recurrence":null}} +{"id":"correction-ordinary-7-round-1","source":"gemini_correction_regressions.jsonl#ordinary-7-round-1","now":"2026-10-25T12:10:00+08:00","timezone":"Asia/Taipei","text":"延後寄包裹","repeats":3,"seed":{"title":null,"at":"2026-10-26T15:00:00+08:00","missing":["title"],"question":null,"source":"提醒我明天15:00","condition":false},"expected":{"title":"延後寄包裹","at":"2026-10-26T15:00:00+08:00","complete":true,"missing":[],"participants":["user:7"],"recurrence":null}} +{"id":"correction-known-title-existing-condition-round-1","source":"gemini_correction_regressions.jsonl#known-title-existing-condition-round-1","now":"2026-10-25T12:10:00+08:00","timezone":"Asia/Taipei","text":"改成明天九點","repeats":3,"seed":{"title":"只在工作日喝水","at":"2026-10-26T15:00:00+08:00","missing":[],"question":"是否只限週一至週五？","source":"只在工作日，明天15:00提醒我","condition":true},"expected":{"title":"只在工作日喝水","at":"2026-10-26T09:00:00+08:00","complete":false,"missing":["clarification"],"participants":["user:7"],"recurrence":null}} +{"id":"correction-deadline-definition-round-1","source":"gemini_correction_regressions.jsonl#deadline-definition-round-1","now":"2026-10-25T12:10:00+08:00","timezone":"Asia/Taipei","text":"提醒我明天下午三點前交報表","repeats":3,"seed":null,"expected":{"title":"前交報表","at":"2026-10-26T15:00:00+08:00","complete":false,"missing":["clarification"],"participants":["user:7"],"recurrence":null}} diff --git a/tests/fixtures/groq_live_results_2026-10-30.jsonl b/tests/fixtures/groq_live_results_2026-10-30.jsonl new file mode 100644 index 0000000..9f3fb79 --- /dev/null +++ b/tests/fixtures/groq_live_results_2026-10-30.jsonl @@ -0,0 +1,59 @@ +{"event": "meta", "signature": "709a7872214675da85af48e64315f116f79ac54480ff79075349b7e7ce8c858a", "model": "openai/gpt-oss-120b", "cases": 36, "requests": 52} +{"event": "result", "id": "time-slash-date", "run": 1, "source": "time_regressions.jsonl#slash-date", "text": "提醒我11/17 23:55買湯", "now": "2026-10-23T12:10:00+08:00", "expected": {"title": "買湯", "at": "2026-11-17T23:55:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 1.028, "raw": {"intent": "create", "title": "買湯", "date": "2026-11-17", "time": "23:55", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "買湯", "at": "2026-11-17T23:55:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "買湯", "at": "2026-11-17T23:55:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {}, "local_failures": {}} +{"event": "result", "id": "time-chinese-date", "run": 1, "source": "time_regressions.jsonl#chinese-date", "text": "提醒我十一月十七號晚上九點買湯", "now": "2026-10-23T12:10:00+08:00", "expected": {"title": "買湯", "at": "2026-11-17T21:00:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 0.732, "raw": {"intent": "create", "title": "買湯", "date": "2026-11-17", "time": "21:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "買湯", "at": "2026-11-17T21:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "買湯", "at": "2026-11-17T21:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {}, "local_failures": {}} +{"event": "result", "id": "time-iso-date", "run": 1, "source": "time_regressions.jsonl#iso-date", "text": "提醒我2026-11-17 23:55買湯", "now": "2026-10-23T12:10:00+08:00", "expected": {"title": "買湯", "at": "2026-11-17T23:55:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 0.933, "raw": {"intent": "create", "title": "買湯", "date": "2026-11-17", "time": "23:55", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "買湯", "at": "2026-11-17T23:55:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "買湯", "at": "2026-11-17T23:55:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {}, "local_failures": {}} +{"event": "result", "id": "time-tomorrow-alias", "run": 1, "source": "time_regressions.jsonl#tomorrow-alias", "text": "提醒我明日 23:55買湯", "now": "2026-10-23T12:10:00+08:00", "expected": {"title": "買湯", "at": "2026-10-24T23:55:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 0.825, "raw": {"intent": "create", "title": "買湯", "date": "2026-10-24", "time": "23:55", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "買湯", "at": "2026-10-24T23:55:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "買湯", "at": "2026-10-24T23:55:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {}, "local_failures": {}} +{"event": "result", "id": "time-next-next-monday", "run": 1, "source": "time_regressions.jsonl#next-next-monday", "text": "提醒我下下週一九點買湯", "now": "2026-10-23T12:10:00+08:00", "expected": {"title": "買湯", "at": "2026-11-02T09:00:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 1.268, "raw": {"intent": "create", "title": "買湯", "date": "2026-11-08", "time": "09:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "買湯", "at": "2026-11-08T09:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "買湯", "at": "2026-11-02T09:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {"at": {"expected": "2026-11-02T09:00:00+08:00", "actual": "2026-11-08T09:00:00+08:00"}}, "local_failures": {}} +{"event": "result", "id": "time-relative-hours", "run": 1, "source": "time_regressions.jsonl#relative-hours", "text": "提醒我三小時後買湯", "now": "2026-10-23T12:10:00+08:00", "expected": {"title": "買湯", "at": "2026-10-23T15:10:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 1.923, "raw": {"intent": "create", "title": "買湯", "date": null, "time": null, "relative_minutes": 180, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "買湯", "at": "2026-10-23T15:10:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "買湯", "at": "2026-10-23T15:10:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {}, "local_failures": {}} +{"event": "result", "id": "time-relative-half-hour", "run": 1, "source": "time_regressions.jsonl#relative-half-hour", "text": "提醒我半小時後買湯", "now": "2026-10-23T12:10:00+08:00", "expected": {"title": "買湯", "at": "2026-10-23T12:40:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 1.37, "raw": {"intent": "create", "title": "買湯", "date": null, "time": null, "relative_minutes": 30, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "買湯", "at": "2026-10-23T12:40:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "買湯", "at": "2026-10-23T12:40:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {}, "local_failures": {}} +{"event": "retry", "id": "time-half-hour-clock", "run": 1, "kind": "quota", "cooldown": 60, "retry_at": 1793372028.3569212, "seconds": 0.177} +{"event": "result", "id": "time-half-hour-clock", "run": 1, "source": "time_regressions.jsonl#half-hour-clock", "text": "提醒我下午三點半買湯", "now": "2026-10-23T12:10:00+08:00", "expected": {"title": "買湯", "at": "2026-10-23T15:30:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 0.948, "raw": {"intent": "create", "title": "買湯", "date": "2026-10-23", "time": "15:30", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "買湯", "at": "2026-10-23T15:30:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "買湯", "at": "2026-10-23T15:30:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {}, "local_failures": {}} +{"event": "result", "id": "time-invalid-day", "run": 1, "source": "time_regressions.jsonl#invalid-day", "text": "提醒我11/31 23:55買湯", "now": "2026-10-23T12:10:00+08:00", "expected": {"title": "買湯", "at": null, "complete": false, "missing": ["time"], "participants": ["user:7"], "recurrence": null}, "seconds": 1.222, "raw": {"intent": "create", "title": "買湯", "date": null, "time": null, "relative_minutes": null, "recurrence": null, "participant_indices": [0], "question": "您提供的日期 11/31 不存在，請問您想設定哪一天的提醒？"}, "raw_result": {"title": "買湯", "at": null, "missing": ["time"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": "您提供的日期 11/31 不存在，請問您想設定哪一天的提醒？", "intent": "create"}, "local_result": {"title": "買湯", "at": null, "missing": ["time"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": "日期或時間無效，請重新輸入正確日期與時間。", "intent": "create"}, "raw_failures": {}, "local_failures": {}} +{"event": "result", "id": "time-invalid-hour", "run": 1, "source": "time_regressions.jsonl#invalid-hour", "text": "提醒我25:00買湯", "now": "2026-10-23T12:10:00+08:00", "expected": {"title": "買湯", "at": null, "complete": false, "missing": ["time"], "participants": ["user:7"], "recurrence": null}, "seconds": 1.016, "raw": {"intent": "create", "title": "買湯", "date": "2026-10-23", "time": null, "relative_minutes": null, "question": "請提供有效的時間（例如 08:30 或 20:45）", "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "買湯", "at": "2026-10-23T00:00:00+08:00", "missing": ["time"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": "請提供有效的時間（例如 08:30 或 20:45）", "intent": "create"}, "local_result": {"title": "買湯", "at": null, "missing": ["time"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": "日期或時間無效，請重新輸入正確日期與時間。", "intent": "create"}, "raw_failures": {"at": {"expected": null, "actual": "2026-10-23T00:00:00+08:00"}}, "local_failures": {}} +{"event": "result", "id": "time-explicit-past", "run": 1, "source": "time_regressions.jsonl#explicit-past", "text": "提醒我2026-10-22 09:00買湯", "now": "2026-10-23T12:10:00+08:00", "expected": {"title": "買湯", "at": "2026-10-22T09:00:00+08:00", "complete": false, "missing": ["time"], "participants": ["user:7"], "recurrence": null}, "seconds": 0.915, "raw": {"intent": "create", "title": "買湯", "date": "2026-10-22", "time": "09:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "買湯", "at": "2026-10-22T09:00:00+08:00", "missing": ["time"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "買湯", "at": "2026-10-22T09:00:00+08:00", "missing": ["time"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": "這個時間已經過去，請重新輸入未來的日期與時間。", "intent": "create"}, "raw_failures": {}, "local_failures": {}} +{"event": "result", "id": "time-year-rollover", "run": 1, "source": "time_regressions.jsonl#year-rollover", "text": "提醒我1/1 09:00買湯", "now": "2026-12-31T23:00:00+08:00", "expected": {"title": "買湯", "at": "2027-01-01T09:00:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 1.07, "raw": {"intent": "create", "title": "買湯", "date": "2027-01-01", "time": "09:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "買湯", "at": "2027-01-01T09:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "買湯", "at": "2027-01-01T09:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {}, "local_failures": {}} +{"event": "result", "id": "time-leap-day", "run": 1, "source": "time_regressions.jsonl#leap-day", "text": "提醒我2/29 09:00買湯", "now": "2028-02-28T12:10:00+08:00", "expected": {"title": "買湯", "at": "2028-02-29T09:00:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 1.125, "raw": {"intent": "create", "title": "買湯", "date": "2028-02-29", "time": "09:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "買湯", "at": "2028-02-29T09:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "買湯", "at": "2028-02-29T09:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {}, "local_failures": {}} +{"event": "retry", "id": "time-new-york", "run": 1, "kind": "quota", "cooldown": 60, "retry_at": 1793372094.8977962, "seconds": 0.236} +{"event": "result", "id": "time-new-york", "run": 1, "source": "time_regressions.jsonl#new-york", "text": "提醒我明早九點買湯", "now": "2026-10-23T23:00:00-04:00", "expected": {"title": "買湯", "at": "2026-10-24T09:00:00-04:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 0.832, "raw": {"intent": "create", "title": "買湯", "date": "2026-10-24", "time": "09:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "買湯", "at": "2026-10-24T09:00:00-04:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "買湯", "at": "2026-10-24T09:00:00-04:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {}, "local_failures": {}} +{"event": "result", "id": "reminder-beef-soup-report", "run": 1, "source": "time_regressions.jsonl#beef-soup-report", "text": "提醒我在五點的時候買兩份清燉牛肉湯", "now": "2026-10-23T12:10:00+08:00", "expected": {"title": "買兩份清燉牛肉湯", "at": "2026-10-24T05:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null}, "seconds": 0.834, "raw": {"intent": "create", "title": "買兩份清燉牛肉湯", "date": "2026-10-23", "time": "17:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "買兩份清燉牛肉湯", "at": "2026-10-23T17:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "買兩份清燉牛肉湯", "at": "2026-10-24T05:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {"at": {"expected": "2026-10-24T05:00:00+08:00", "actual": "2026-10-23T17:00:00+08:00"}}, "local_failures": {}} +{"event": "result", "id": "reminder-suffix-in-title", "run": 1, "source": "time_regressions.jsonl#suffix-in-title", "text": "提醒我在五點的時候買喝湯的時候要用的湯匙", "now": "2026-10-23T12:10:00+08:00", "expected": {"title": "買喝湯的時候要用的湯匙", "at": "2026-10-24T05:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null}, "seconds": 1.395, "raw": {"intent": "create", "title": "買喝湯的時候要用的湯匙", "date": "2026-10-23", "time": "17:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "買喝湯的時候要用的湯匙", "at": "2026-10-23T17:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "買喝湯的時候要用的湯匙", "at": "2026-10-24T05:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {"at": {"expected": "2026-10-24T05:00:00+08:00", "actual": "2026-10-23T17:00:00+08:00"}}, "local_failures": {}} +{"event": "result", "id": "reminder-title-period", "run": 1, "source": "time_regressions.jsonl#title-period", "text": "五點買晚上要用的湯匙", "now": "2026-10-23T12:10:00+08:00", "expected": {"title": "買晚上要用的湯匙", "at": "2026-10-24T05:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null}, "seconds": 1.202, "raw": {"intent": "create", "title": "買晚上要用的湯匙", "date": null, "time": null, "relative_minutes": null, "question": "請問您想要在什麼日期的五點提醒您購買晚上要用的湯匙？", "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "買晚上要用的湯匙", "at": null, "missing": ["time"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": "請問您想要在什麼日期的五點提醒您購買晚上要用的湯匙？", "intent": "create"}, "local_result": {"title": "買晚上要用的湯匙", "at": "2026-10-24T05:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {"at": {"expected": "2026-10-24T05:00:00+08:00", "actual": null}, "missing": {"expected": [], "actual": ["time"]}, "complete": {"expected": true, "actual": false}}, "local_failures": {}} +{"event": "result", "id": "reminder-title-date-slash", "run": 1, "source": "time_regressions.jsonl#title-date-slash", "text": "7/30 21:00提醒我08/01要去家樂福", "now": "2026-10-23T12:10:00+08:00", "expected": {"title": "08/01要去家樂福", "at": "2027-07-30T21:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null}, "seconds": 1.071, "raw": {"intent": "create", "title": "08/01要去家樂福", "date": "2026-07-30", "time": "21:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "08/01要去家樂福", "at": "2026-07-30T21:00:00+08:00", "missing": ["time"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "08/01要去家樂福", "at": "2027-07-30T21:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {"at": {"expected": "2027-07-30T21:00:00+08:00", "actual": "2026-07-30T21:00:00+08:00"}, "missing": {"expected": [], "actual": ["time"]}, "complete": {"expected": true, "actual": false}}, "local_failures": {}} +{"event": "result", "id": "reminder-multiple-participants", "run": 1, "source": "time_regressions.jsonl#multiple-participants", "text": "明天下午三點提醒我和 @alice 倒垃圾", "now": "2026-10-23T12:10:00+08:00", "expected": {"title": "倒垃圾", "at": "2026-10-24T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7", "user:8"], "recurrence": null}, "seconds": 0.95, "raw": {"intent": "create", "title": "倒垃圾", "date": "2026-10-24", "time": "15:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0, 1]}, "raw_result": {"title": "倒垃圾", "at": "2026-10-24T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7", "user:8"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "倒垃圾", "at": "2026-10-24T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7", "user:8"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {}, "local_failures": {}} +{"event": "result", "id": "reminder-missing-title", "run": 1, "source": "time_regressions.jsonl#missing-title", "text": "提醒我明天下午三點", "now": "2026-10-23T12:10:00+08:00", "expected": {"title": null, "at": "2026-10-24T15:00:00+08:00", "missing": ["title"], "complete": false, "participants": ["user:7"], "recurrence": null}, "seconds": 1.155, "raw": {"intent": "create", "title": null, "date": "2026-10-24", "time": "15:00", "relative_minutes": null, "question": "請告訴我要提醒的內容是什麼？", "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": null, "at": "2026-10-24T15:00:00+08:00", "missing": ["title"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": "請告訴我要提醒的內容是什麼？", "intent": "create"}, "local_result": {"title": null, "at": "2026-10-24T15:00:00+08:00", "missing": ["title"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": "請告訴我要提醒的內容是什麼？", "intent": "create"}, "raw_failures": {}, "local_failures": {}} +{"event": "result", "id": "reminder-breakfast", "run": 1, "source": "time_regressions.jsonl#breakfast", "text": "提醒我買早餐", "now": "2026-10-23T12:10:00+08:00", "expected": {"title": "買早餐", "at": null, "missing": ["time"], "complete": false, "participants": ["user:7"], "recurrence": null}, "seconds": 1.105, "raw": {"intent": "create", "title": "買早餐", "date": null, "time": null, "relative_minutes": null, "question": "請問您想在什麼時間收到提醒？", "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "買早餐", "at": null, "missing": ["time"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": "請問您想在什麼時間收到提醒？", "intent": "create"}, "local_result": {"title": "買早餐", "at": null, "missing": ["time"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": "請問您想在什麼時間收到提醒？", "intent": "create"}, "raw_failures": {}, "local_failures": {}} +{"event": "retry", "id": "reminder-daily", "run": 1, "kind": "quota", "cooldown": 60, "retry_at": 1793372163.7314072, "seconds": 0.284} +{"event": "result", "id": "reminder-daily", "run": 1, "source": "time_regressions.jsonl#daily", "text": "每天九點提醒我買湯", "now": "2026-10-23T12:10:00+08:00", "expected": {"title": "買湯", "at": "2026-10-24T09:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": {"period": "daily", "hour": 9, "minute": 0}}, "seconds": 1.185, "raw": {"intent": "create", "title": "買湯", "date": "2026-10-24", "time": "09:00", "relative_minutes": null, "question": null, "recurrence": {"period": "daily", "hour": 9, "minute": 0, "interval_seconds": null, "weekdays": [], "month_days": [], "year_month": null, "year_day": null}, "participant_indices": [0]}, "raw_result": {"title": "買湯", "at": "2026-10-24T09:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": {"period": "daily", "hour": 9, "minute": 0, "weekdays": [], "month_days": [], "year_month": null, "year_day": null, "interval_seconds": null}, "question": null, "intent": "create"}, "local_result": {"title": "買湯", "at": "2026-10-24T09:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": {"period": "daily", "hour": 9, "minute": 0, "weekdays": [], "month_days": [], "year_month": null, "year_day": null, "interval_seconds": null}, "question": null, "intent": "create"}, "raw_failures": {}, "local_failures": {}} +{"event": "result", "id": "reminder-weekly", "run": 1, "source": "time_regressions.jsonl#weekly", "text": "每週一、三、五九點提醒我買湯", "now": "2026-10-23T12:10:00+08:00", "expected": {"title": "買湯", "at": "2026-10-26T09:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": {"period": "weekly", "hour": 9, "minute": 0, "weekdays": [0, 2, 4]}}, "seconds": 1.045, "raw": {"intent": "create", "title": "買湯", "date": "2026-10-26", "time": "09:00", "relative_minutes": null, "question": null, "recurrence": {"period": "weekly", "hour": 9, "minute": 0, "weekdays": [0, 2, 4], "interval_seconds": null, "month_days": [], "year_month": null, "year_day": null}, "participant_indices": [0]}, "raw_result": {"title": "買湯", "at": "2026-10-26T09:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": {"period": "weekly", "hour": 9, "minute": 0, "weekdays": [0, 2, 4], "month_days": [], "year_month": null, "year_day": null, "interval_seconds": null}, "question": null, "intent": "create"}, "local_result": {"title": "買湯", "at": "2026-10-26T09:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": {"period": "weekly", "hour": 9, "minute": 0, "weekdays": [0, 2, 4], "month_days": [], "year_month": null, "year_day": null, "interval_seconds": null}, "question": null, "intent": "create"}, "raw_failures": {}, "local_failures": {}} +{"event": "result", "id": "reminder-monthly", "run": 1, "source": "time_regressions.jsonl#monthly", "text": "每月1號九點提醒我買湯", "now": "2026-10-23T12:10:00+08:00", "expected": {"title": "買湯", "at": "2026-11-01T09:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": {"period": "monthly", "hour": 9, "minute": 0, "month_days": [1]}}, "seconds": 1.575, "raw": {"intent": "create", "title": "買湯", "date": "2026-11-01", "time": "09:00", "relative_minutes": null, "question": null, "recurrence": {"period": "monthly", "hour": 9, "minute": 0, "interval_seconds": null, "weekdays": [], "month_days": [1], "year_month": null, "year_day": null}, "participant_indices": [0]}, "raw_result": {"title": "買湯", "at": "2026-11-01T09:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": {"period": "monthly", "hour": 9, "minute": 0, "weekdays": [], "month_days": [1], "year_month": null, "year_day": null, "interval_seconds": null}, "question": null, "intent": "create"}, "local_result": {"title": "買湯", "at": "2026-11-01T09:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": {"period": "monthly", "hour": 9, "minute": 0, "weekdays": [], "month_days": [1], "year_month": null, "year_day": null, "interval_seconds": null}, "question": null, "intent": "create"}, "raw_failures": {}, "local_failures": {}} +{"event": "result", "id": "reminder-yearly", "run": 1, "source": "time_regressions.jsonl#yearly", "text": "每年11/17九點提醒我買湯", "now": "2026-10-23T12:10:00+08:00", "expected": {"title": "買湯", "at": "2026-11-17T09:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": {"period": "yearly", "hour": 9, "minute": 0, "year_month": 11, "year_day": 17}}, "seconds": 1.141, "raw": {"intent": "create", "title": "買湯", "date": "2026-11-17", "time": "09:00", "relative_minutes": null, "question": null, "recurrence": {"period": "yearly", "hour": 9, "minute": 0, "year_month": 11, "year_day": 17, "interval_seconds": null, "weekdays": [], "month_days": []}, "participant_indices": [0]}, "raw_result": {"title": "買湯", "at": "2026-11-17T09:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": {"period": "yearly", "hour": 9, "minute": 0, "weekdays": [], "month_days": [], "year_month": 11, "year_day": 17, "interval_seconds": null}, "question": null, "intent": "create"}, "local_result": {"title": "買湯", "at": "2026-11-17T09:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": {"period": "yearly", "hour": 9, "minute": 0, "weekdays": [], "month_days": [], "year_month": 11, "year_day": 17, "interval_seconds": null}, "question": null, "intent": "create"}, "raw_failures": {}, "local_failures": {}} +{"event": "result", "id": "reminder-interval", "run": 1, "source": "time_regressions.jsonl#interval", "text": "每15分鐘提醒我喝水", "now": "2026-10-23T12:10:00+08:00", "expected": {"title": "喝水", "at": "2026-10-23T12:25:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": {"period": "interval", "interval_seconds": 900}}, "seconds": 1.287, "raw": {"intent": "create", "title": "喝水", "date": null, "time": null, "relative_minutes": null, "question": null, "recurrence": {"period": "interval", "interval_seconds": 900, "hour": null, "minute": null, "weekdays": [], "month_days": [], "year_month": null, "year_day": null}, "participant_indices": [0]}, "raw_result": {"title": "喝水", "at": "2026-10-23T12:25:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": {"period": "interval", "hour": null, "minute": null, "weekdays": [], "month_days": [], "year_month": null, "year_day": null, "interval_seconds": 900}, "question": null, "intent": "create"}, "local_result": {"title": "喝水", "at": "2026-10-23T12:25:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": {"period": "interval", "hour": null, "minute": null, "weekdays": [], "month_days": [], "year_month": null, "year_day": null, "interval_seconds": 900}, "question": null, "intent": "create"}, "raw_failures": {}, "local_failures": {}} +{"event": "result", "id": "content-morning-twelve-round-1", "run": 1, "source": "gemini_content_regressions.jsonl#morning-twelve-round-1", "text": "提醒我 明天早上十二點 買湯", "now": "2026-10-24T12:10:00+08:00", "expected": {"title": "買湯", "at": "2026-10-25T00:00:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 1.763, "raw": {"intent": "create", "title": "買湯", "date": "2026-10-25", "time": "12:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "買湯", "at": "2026-10-25T12:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "買湯", "at": "2026-10-25T00:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {"at": {"expected": "2026-10-25T00:00:00+08:00", "actual": "2026-10-25T12:00:00+08:00"}}, "local_failures": {}} +{"event": "result", "id": "content-morning-twelve-round-1", "run": 2, "source": "gemini_content_regressions.jsonl#morning-twelve-round-1", "text": "提醒我 明天早上十二點 買湯", "now": "2026-10-24T12:10:00+08:00", "expected": {"title": "買湯", "at": "2026-10-25T00:00:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 0.704, "raw": {"intent": "create", "title": "買湯", "date": "2026-10-25", "time": "12:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "買湯", "at": "2026-10-25T12:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "買湯", "at": "2026-10-25T00:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {"at": {"expected": "2026-10-25T00:00:00+08:00", "actual": "2026-10-25T12:00:00+08:00"}}, "local_failures": {}} +{"event": "result", "id": "content-morning-twelve-round-1", "run": 3, "source": "gemini_content_regressions.jsonl#morning-twelve-round-1", "text": "提醒我 明天早上十二點 買湯", "now": "2026-10-24T12:10:00+08:00", "expected": {"title": "買湯", "at": "2026-10-25T00:00:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 0.868, "raw": {"intent": "create", "title": "買湯", "date": "2026-10-25", "time": "12:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "買湯", "at": "2026-10-25T12:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "買湯", "at": "2026-10-25T00:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {"at": {"expected": "2026-10-25T00:00:00+08:00", "actual": "2026-10-25T12:00:00+08:00"}}, "local_failures": {}} +{"event": "result", "id": "content-alternative-in-subject-round-1", "run": 1, "source": "gemini_content_regressions.jsonl#alternative-in-subject-round-1", "text": "提醒我11/17 23:55買湯或11/18要用的湯匙", "now": "2026-10-24T12:10:00+08:00", "expected": {"title": "買湯或11/18要用的湯匙", "at": "2026-11-17T23:55:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 1.385, "raw": {"intent": "create", "title": null, "date": null, "time": null, "relative_minutes": null, "question": "您想設定哪一個提醒？是 11/17 23:55 的「買湯」還是 11/18 的「要用的湯匙」？", "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": null, "at": null, "missing": ["time", "title"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": "您想設定哪一個提醒？是 11/17 23:55 的「買湯」還是 11/18 的「要用的湯匙」？", "intent": "create"}, "local_result": {"title": "買湯或11/18要用的湯匙", "at": "2026-11-17T23:55:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {"title": {"expected": "買湯或11/18要用的湯匙", "actual": null}, "at": {"expected": "2026-11-17T23:55:00+08:00", "actual": null}, "complete": {"expected": true, "actual": false}, "missing": {"expected": [], "actual": ["time", "title"]}}, "local_failures": {}} +{"event": "result", "id": "content-alternative-in-subject-round-1", "run": 2, "source": "gemini_content_regressions.jsonl#alternative-in-subject-round-1", "text": "提醒我11/17 23:55買湯或11/18要用的湯匙", "now": "2026-10-24T12:10:00+08:00", "expected": {"title": "買湯或11/18要用的湯匙", "at": "2026-11-17T23:55:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 1.233, "raw": {"intent": "create", "title": null, "date": null, "time": null, "relative_minutes": null, "question": "請問您想先建立哪一個提醒？是 11/17 23:55 的「買湯」還是 11/18 的「要用的湯匙」？", "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": null, "at": null, "missing": ["time", "title"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": "請問您想先建立哪一個提醒？是 11/17 23:55 的「買湯」還是 11/18 的「要用的湯匙」？", "intent": "create"}, "local_result": {"title": "買湯或11/18要用的湯匙", "at": "2026-11-17T23:55:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {"title": {"expected": "買湯或11/18要用的湯匙", "actual": null}, "at": {"expected": "2026-11-17T23:55:00+08:00", "actual": null}, "complete": {"expected": true, "actual": false}, "missing": {"expected": [], "actual": ["time", "title"]}}, "local_failures": {}} +{"event": "retry", "id": "content-alternative-in-subject-round-1", "run": 3, "kind": "quota", "cooldown": 60, "retry_at": 1793372236.0985951, "seconds": 0.17} +{"event": "result", "id": "content-alternative-in-subject-round-1", "run": 3, "source": "gemini_content_regressions.jsonl#alternative-in-subject-round-1", "text": "提醒我11/17 23:55買湯或11/18要用的湯匙", "now": "2026-10-24T12:10:00+08:00", "expected": {"title": "買湯或11/18要用的湯匙", "at": "2026-11-17T23:55:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 1.169, "raw": {"intent": "create", "title": null, "date": null, "time": null, "relative_minutes": null, "question": "請問您想分別建立兩筆提醒，還是只設定其中一筆？請說明日期與內容。", "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": null, "at": null, "missing": ["time", "title"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": "請問您想分別建立兩筆提醒，還是只設定其中一筆？請說明日期與內容。", "intent": "create"}, "local_result": {"title": "買湯或11/18要用的湯匙", "at": "2026-11-17T23:55:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {"title": {"expected": "買湯或11/18要用的湯匙", "actual": null}, "at": {"expected": "2026-11-17T23:55:00+08:00", "actual": null}, "complete": {"expected": true, "actual": false}, "missing": {"expected": [], "actual": ["time", "title"]}}, "local_failures": {}} +{"event": "result", "id": "content-exact-current-clock-rolls-round-2", "run": 1, "source": "gemini_content_regressions.jsonl#exact-current-clock-rolls-round-2", "text": "提醒我 12:10 買湯", "now": "2026-10-24T12:10:00+08:00", "expected": {"title": "買湯", "at": "2026-10-25T12:10:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 0.876, "raw": {"intent": "create", "title": "買湯", "date": "2026-10-24", "time": "12:10", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "買湯", "at": "2026-10-24T12:10:00+08:00", "missing": ["time"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "買湯", "at": "2026-10-25T12:10:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {"at": {"expected": "2026-10-25T12:10:00+08:00", "actual": "2026-10-24T12:10:00+08:00"}, "complete": {"expected": true, "actual": false}, "missing": {"expected": [], "actual": ["time"]}}, "local_failures": {}} +{"event": "result", "id": "content-chinese-date-extra-day-digit-round-2", "run": 1, "source": "gemini_content_regressions.jsonl#chinese-date-extra-day-digit-round-2", "text": "提醒我 11月170日 23:55 買湯", "now": "2026-10-24T12:10:00+08:00", "expected": {"title": "買湯", "at": null, "complete": false, "missing": ["time"], "participants": ["user:7"], "recurrence": null}, "seconds": 1.113, "raw": {"intent": "create", "title": "買湯", "date": "2026-11-17", "time": "23:55", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "買湯", "at": "2026-11-17T23:55:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "買湯", "at": null, "missing": ["time"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": "日期或時間無效，請重新輸入正確日期與時間。", "intent": "create"}, "raw_failures": {"at": {"expected": null, "actual": "2026-11-17T23:55:00+08:00"}, "complete": {"expected": false, "actual": true}, "missing": {"expected": ["time"], "actual": []}}, "local_failures": {}} +{"event": "result", "id": "content-recorded-calendar-noun-question", "run": 1, "source": "gemini_content_regressions.jsonl#recorded-calendar-noun-question", "text": "提醒我明天15:00工作日行事曆", "now": "2026-10-24T12:10:00+08:00", "expected": {"title": "工作日行事曆", "at": "2026-10-25T15:00:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 1.18, "raw": {"intent": "create", "title": "工作日行事曆", "date": "2026-10-25", "time": "15:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "工作日行事曆", "at": "2026-10-25T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "工作日行事曆", "at": "2026-10-25T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {}, "local_failures": {}} +{"event": "result", "id": "content-recorded-calendar-noun-question", "run": 2, "source": "gemini_content_regressions.jsonl#recorded-calendar-noun-question", "text": "提醒我明天15:00工作日行事曆", "now": "2026-10-24T12:10:00+08:00", "expected": {"title": "工作日行事曆", "at": "2026-10-25T15:00:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 1.253, "raw": {"intent": "create", "title": "工作日行事曆", "date": "2026-10-25", "time": "15:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "工作日行事曆", "at": "2026-10-25T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "工作日行事曆", "at": "2026-10-25T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {}, "local_failures": {}} +{"event": "result", "id": "content-recorded-calendar-noun-question", "run": 3, "source": "gemini_content_regressions.jsonl#recorded-calendar-noun-question", "text": "提醒我明天15:00工作日行事曆", "now": "2026-10-24T12:10:00+08:00", "expected": {"title": "工作日行事曆", "at": "2026-10-25T15:00:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 1.015, "raw": {"intent": "create", "title": "工作日行事曆", "date": "2026-10-25", "time": "15:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "工作日行事曆", "at": "2026-10-25T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "工作日行事曆", "at": "2026-10-25T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {}, "local_failures": {}} +{"event": "result", "id": "content-recorded-literal-title-rewritten-round-1", "run": 1, "source": "gemini_content_regressions.jsonl#recorded-literal-title-rewritten-round-1", "text": "提醒我明天15:00之前的會議紀錄", "now": "2026-10-24T12:10:00+08:00", "expected": {"title": "之前的會議紀錄", "at": "2026-10-25T15:00:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 1.017, "raw": {"intent": "create", "title": "會議紀錄", "date": "2026-10-25", "time": "15:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "會議紀錄", "at": "2026-10-25T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "之前的會議紀錄", "at": "2026-10-25T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {"title": {"expected": "之前的會議紀錄", "actual": "會議紀錄"}}, "local_failures": {}} +{"event": "result", "id": "content-recorded-literal-title-rewritten-round-1", "run": 2, "source": "gemini_content_regressions.jsonl#recorded-literal-title-rewritten-round-1", "text": "提醒我明天15:00之前的會議紀錄", "now": "2026-10-24T12:10:00+08:00", "expected": {"title": "之前的會議紀錄", "at": "2026-10-25T15:00:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 0.92, "raw": {"intent": "create", "title": "會議紀錄", "date": "2026-10-25", "time": "15:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "會議紀錄", "at": "2026-10-25T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "之前的會議紀錄", "at": "2026-10-25T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {"title": {"expected": "之前的會議紀錄", "actual": "會議紀錄"}}, "local_failures": {}} +{"event": "retry", "id": "content-recorded-literal-title-rewritten-round-1", "run": 3, "kind": "quota", "cooldown": 60, "retry_at": 1793372304.830623, "seconds": 0.185} +{"event": "result", "id": "content-recorded-literal-title-rewritten-round-1", "run": 3, "source": "gemini_content_regressions.jsonl#recorded-literal-title-rewritten-round-1", "text": "提醒我明天15:00之前的會議紀錄", "now": "2026-10-24T12:10:00+08:00", "expected": {"title": "之前的會議紀錄", "at": "2026-10-25T15:00:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 0.818, "raw": {"intent": "create", "title": "會議紀錄", "date": "2026-10-25", "time": "15:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "會議紀錄", "at": "2026-10-25T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "之前的會議紀錄", "at": "2026-10-25T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "raw_failures": {"title": {"expected": "之前的會議紀錄", "actual": "會議紀錄"}}, "local_failures": {}} +{"event": "result", "id": "followup-recorded-garbled-context-title-rewrite", "run": 1, "source": "gemini_followup_regressions.jsonl#recorded-garbled-context-title-rewrite", "text": "寄出年度報表", "now": "2026-10-24T12:10:00+08:00", "expected": {"title": "寄出年度報表", "at": "2026-10-25T15:00:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 1.091, "raw": {"intent": "update", "title": "寄出年度報表", "date": "2026-10-25", "time": "15:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "寄出年度報表", "at": "2026-10-25T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "update"}, "local_result": {"title": "寄出年度報表", "at": "2026-10-25T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "update"}, "raw_failures": {}, "local_failures": {}} +{"event": "result", "id": "followup-recorded-garbled-context-title-rewrite", "run": 2, "source": "gemini_followup_regressions.jsonl#recorded-garbled-context-title-rewrite", "text": "寄出年度報表", "now": "2026-10-24T12:10:00+08:00", "expected": {"title": "寄出年度報表", "at": "2026-10-25T15:00:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 1.058, "raw": {"intent": "update", "title": "寄出年度報表", "date": "2026-10-25", "time": "15:00", "relative_minutes": null, "recurrence": null, "question": null, "participant_indices": [0]}, "raw_result": {"title": "寄出年度報表", "at": "2026-10-25T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "update"}, "local_result": {"title": "寄出年度報表", "at": "2026-10-25T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "update"}, "raw_failures": {}, "local_failures": {}} +{"event": "result", "id": "followup-recorded-garbled-context-title-rewrite", "run": 3, "source": "gemini_followup_regressions.jsonl#recorded-garbled-context-title-rewrite", "text": "寄出年度報表", "now": "2026-10-24T12:10:00+08:00", "expected": {"title": "寄出年度報表", "at": "2026-10-25T15:00:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 1.183, "raw": {"intent": "update", "title": "寄出年度報表", "date": "2026-10-25", "time": "15:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "寄出年度報表", "at": "2026-10-25T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "update"}, "local_result": {"title": "寄出年度報表", "at": "2026-10-25T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "update"}, "raw_failures": {}, "local_failures": {}} +{"event": "result", "id": "correction-ordinary-7-round-1", "run": 1, "source": "gemini_correction_regressions.jsonl#ordinary-7-round-1", "text": "延後寄包裹", "now": "2026-10-25T12:10:00+08:00", "expected": {"title": "延後寄包裹", "at": "2026-10-26T15:00:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 1.102, "raw": {"intent": "update", "title": "寄包裹", "date": null, "time": null, "relative_minutes": null, "recurrence": null, "participant_indices": [0], "question": "請提供要延後的具體日期與時間。"}, "raw_result": {"title": "寄包裹", "at": null, "missing": ["time"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": "請提供要延後的具體日期與時間。", "intent": "update"}, "local_result": {"title": "延後寄包裹", "at": "2026-10-26T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "update"}, "raw_failures": {"title": {"expected": "延後寄包裹", "actual": "寄包裹"}, "at": {"expected": "2026-10-26T15:00:00+08:00", "actual": null}, "complete": {"expected": true, "actual": false}, "missing": {"expected": [], "actual": ["time"]}}, "local_failures": {}} +{"event": "result", "id": "correction-ordinary-7-round-1", "run": 2, "source": "gemini_correction_regressions.jsonl#ordinary-7-round-1", "text": "延後寄包裹", "now": "2026-10-25T12:10:00+08:00", "expected": {"title": "延後寄包裹", "at": "2026-10-26T15:00:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 1.82, "raw": {"intent": "update", "title": "寄包裹", "date": null, "time": null, "relative_minutes": null, "question": "請提供新的提醒時間", "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "寄包裹", "at": null, "missing": ["time"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": "請提供新的提醒時間", "intent": "update"}, "local_result": {"title": "延後寄包裹", "at": "2026-10-26T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "update"}, "raw_failures": {"title": {"expected": "延後寄包裹", "actual": "寄包裹"}, "at": {"expected": "2026-10-26T15:00:00+08:00", "actual": null}, "complete": {"expected": true, "actual": false}, "missing": {"expected": [], "actual": ["time"]}}, "local_failures": {}} +{"event": "result", "id": "correction-ordinary-7-round-1", "run": 3, "source": "gemini_correction_regressions.jsonl#ordinary-7-round-1", "text": "延後寄包裹", "now": "2026-10-25T12:10:00+08:00", "expected": {"title": "延後寄包裹", "at": "2026-10-26T15:00:00+08:00", "complete": true, "missing": [], "participants": ["user:7"], "recurrence": null}, "seconds": 1.441, "raw": {"intent": "update", "title": "寄包裹", "date": null, "time": null, "relative_minutes": null, "question": "請問您想將提醒時間改為何時？", "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "寄包裹", "at": null, "missing": ["time"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": "請問您想將提醒時間改為何時？", "intent": "update"}, "local_result": {"title": "延後寄包裹", "at": "2026-10-26T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "update"}, "raw_failures": {"title": {"expected": "延後寄包裹", "actual": "寄包裹"}, "at": {"expected": "2026-10-26T15:00:00+08:00", "actual": null}, "complete": {"expected": true, "actual": false}, "missing": {"expected": [], "actual": ["time"]}}, "local_failures": {}} +{"event": "result", "id": "correction-known-title-existing-condition-round-1", "run": 1, "source": "gemini_correction_regressions.jsonl#known-title-existing-condition-round-1", "text": "改成明天九點", "now": "2026-10-25T12:10:00+08:00", "expected": {"title": "只在工作日喝水", "at": "2026-10-26T09:00:00+08:00", "complete": false, "missing": ["clarification"], "participants": ["user:7"], "recurrence": null}, "seconds": 1.146, "raw": {"intent": "update", "title": "只在工作日喝水", "date": "2026-10-26", "time": "09:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "只在工作日喝水", "at": "2026-10-26T09:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "update"}, "local_result": {"title": "只在工作日喝水", "at": "2026-10-26T09:00:00+08:00", "missing": ["clarification"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": "是否只限週一至週五？", "intent": "update"}, "raw_failures": {"complete": {"expected": false, "actual": true}, "missing": {"expected": ["clarification"], "actual": []}}, "local_failures": {}} +{"event": "fatal", "id": "correction-known-title-existing-condition-round-1", "run": 2, "kind": "request", "cooldown": 60, "retry_at": 1793372375.974089, "seconds": 1.478} +{"event": "result", "id": "correction-known-title-existing-condition-round-1", "run": 2, "source": "gemini_correction_regressions.jsonl#known-title-existing-condition-round-1", "text": "改成明天九點", "now": "2026-10-25T12:10:00+08:00", "expected": {"title": "只在工作日喝水", "at": "2026-10-26T09:00:00+08:00", "complete": false, "missing": ["clarification"], "participants": ["user:7"], "recurrence": null}, "seconds": 1.044, "raw": {"intent": "update", "title": "只在工作日喝水", "date": "2026-10-26", "time": "09:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "只在工作日喝水", "at": "2026-10-26T09:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "update"}, "local_result": {"title": "只在工作日喝水", "at": "2026-10-26T09:00:00+08:00", "missing": ["clarification"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": "是否只限週一至週五？", "intent": "update"}, "raw_failures": {"complete": {"expected": false, "actual": true}, "missing": {"expected": ["clarification"], "actual": []}}, "local_failures": {}} +{"event": "result", "id": "correction-known-title-existing-condition-round-1", "run": 3, "source": "gemini_correction_regressions.jsonl#known-title-existing-condition-round-1", "text": "改成明天九點", "now": "2026-10-25T12:10:00+08:00", "expected": {"title": "只在工作日喝水", "at": "2026-10-26T09:00:00+08:00", "complete": false, "missing": ["clarification"], "participants": ["user:7"], "recurrence": null}, "seconds": 2.258, "raw": {"intent": "update", "title": "只在工作日喝水", "date": "2026-10-26", "time": "09:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "只在工作日喝水", "at": "2026-10-26T09:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "update"}, "local_result": {"title": "只在工作日喝水", "at": "2026-10-26T09:00:00+08:00", "missing": ["clarification"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": "是否只限週一至週五？", "intent": "update"}, "raw_failures": {"complete": {"expected": false, "actual": true}, "missing": {"expected": ["clarification"], "actual": []}}, "local_failures": {}} +{"event": "result", "id": "correction-deadline-definition-round-1", "run": 1, "source": "gemini_correction_regressions.jsonl#deadline-definition-round-1", "text": "提醒我明天下午三點前交報表", "now": "2026-10-25T12:10:00+08:00", "expected": {"title": "前交報表", "at": "2026-10-26T15:00:00+08:00", "complete": false, "missing": ["clarification"], "participants": ["user:7"], "recurrence": null}, "seconds": 1.201, "raw": {"intent": "create", "title": "交報表", "date": "2026-10-26", "time": "15:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "交報表", "at": "2026-10-26T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "前交報表", "at": "2026-10-26T15:00:00+08:00", "missing": ["clarification"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": "請確認額外提醒條件，並一次提供一筆明確提醒。", "intent": "create"}, "raw_failures": {"title": {"expected": "前交報表", "actual": "交報表"}, "complete": {"expected": false, "actual": true}, "missing": {"expected": ["clarification"], "actual": []}}, "local_failures": {}} +{"event": "result", "id": "correction-deadline-definition-round-1", "run": 2, "source": "gemini_correction_regressions.jsonl#deadline-definition-round-1", "text": "提醒我明天下午三點前交報表", "now": "2026-10-25T12:10:00+08:00", "expected": {"title": "前交報表", "at": "2026-10-26T15:00:00+08:00", "complete": false, "missing": ["clarification"], "participants": ["user:7"], "recurrence": null}, "seconds": 0.971, "raw": {"intent": "create", "title": "交報表", "date": "2026-10-26", "time": "15:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "交報表", "at": "2026-10-26T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "前交報表", "at": "2026-10-26T15:00:00+08:00", "missing": ["clarification"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": "請確認額外提醒條件，並一次提供一筆明確提醒。", "intent": "create"}, "raw_failures": {"title": {"expected": "前交報表", "actual": "交報表"}, "complete": {"expected": false, "actual": true}, "missing": {"expected": ["clarification"], "actual": []}}, "local_failures": {}} +{"event": "result", "id": "correction-deadline-definition-round-1", "run": 3, "source": "gemini_correction_regressions.jsonl#deadline-definition-round-1", "text": "提醒我明天下午三點前交報表", "now": "2026-10-25T12:10:00+08:00", "expected": {"title": "前交報表", "at": "2026-10-26T15:00:00+08:00", "complete": false, "missing": ["clarification"], "participants": ["user:7"], "recurrence": null}, "seconds": 1.225, "raw": {"intent": "create", "title": "交報表", "date": "2026-10-26", "time": "15:00", "relative_minutes": null, "question": null, "recurrence": null, "participant_indices": [0]}, "raw_result": {"title": "交報表", "at": "2026-10-26T15:00:00+08:00", "missing": [], "complete": true, "participants": ["user:7"], "recurrence": null, "question": null, "intent": "create"}, "local_result": {"title": "前交報表", "at": "2026-10-26T15:00:00+08:00", "missing": ["clarification"], "complete": false, "participants": ["user:7"], "recurrence": null, "question": "請確認額外提醒條件，並一次提供一筆明確提醒。", "intent": "create"}, "raw_failures": {"title": {"expected": "前交報表", "actual": "交報表"}, "complete": {"expected": false, "actual": true}, "missing": {"expected": ["clarification"], "actual": []}}, "local_failures": {}} diff --git a/tests/test_ai.py b/tests/test_ai.py index 5de5957..790d7bd 100644 --- a/tests/test_ai.py +++ b/tests/test_ai.py @@ -681,6 +681,44 @@ def test_pending_recurrence_uses_original_weekdays(self): self.assertEqual((0,), result.recurrence.weekdays) self.assertEqual(datetime(2026, 10, 26, 9, tzinfo=NOW.tzinfo), result.remind_at) + def test_weekly_first_fire_is_not_overwritten_by_clock_rollover(self): + now = datetime(2026, 10, 30, 12, tzinfo=NOW.tzinfo) + for text in ( + "每週一、三、五早上九點提醒我吃藥", + "提醒我每週一、三、五早上九點吃藥", + "每週一、三、五09:00提醒我吃藥", + ): + with self.subTest(text=text): + result = self.parse( + parsed( + title="吃藥", + date="2026-11-02", + time="09:00", + recurrence=rule(period="weekly", weekdays=[0, 2, 4]), + ), + text, + now=now, + ) + self.assertEqual(datetime(2026, 11, 2, 9, tzinfo=NOW.tzinfo), result.remind_at) + self.assertEqual((0, 2, 4), result.recurrence.weekdays) + self.assertEqual("吃藥", result.title) + self.assertEqual((), result.missing_fields) + + def test_weekly_explicit_start_date_is_preserved(self): + now = datetime(2026, 10, 30, 12, tzinfo=NOW.tzinfo) + result = self.parse( + parsed( + title="吃藥", + date="2026-11-02", + time="09:00", + recurrence=rule(period="weekly", weekdays=[0, 2, 4]), + ), + "每週一、三、五11/4早上九點提醒我吃藥", + now=now, + ) + self.assertEqual(datetime(2026, 11, 4, 9, tzinfo=NOW.tzinfo), result.remind_at) + self.assertEqual((), result.missing_fields) + class SemanticValidationTest(unittest.TestCase): def test_separate_dates_title_and_absolute_time(self) -> None: diff --git a/tests/test_ai_flow.py b/tests/test_ai_flow.py index 70e381f..bbffe06 100644 --- a/tests/test_ai_flow.py +++ b/tests/test_ai_flow.py @@ -354,6 +354,50 @@ def test_registered_second_provider_works_through_unchanged_bot_confirmation(sel self.assertEqual(SECOND, other.calls[0][0]) self.assertEqual([], self.gemini.calls) + def test_groq_fallback_uses_remaining_budget_and_requires_confirmation(self) -> None: + import io + import json + + from remindly.ai.groq import GroqProvider + + calls = [] + + def opener(request, *, timeout): + calls.append((request.get_header("Authorization"), timeout)) + return io.BytesIO( + json.dumps( + { + "choices": [ + { + "finish_reason": "stop", + "message": {"content": json.dumps(parsed(title="拿包裹"))}, + } + ] + } + ).encode() + ) + + self.gateway.providers["groq"] = GroqProvider(opener=opener) + self.send(f"/ai add gemini {SECRET}") + self.send(f"/ai add groq {SECOND}") + self.assertIn("groq · Groq", self.telegram.messages[-1].text) + self.gemini.responses.append(ProviderError("timeout")) + with patch("remindly.ai.gateway.time.monotonic", side_effect=[100, 100, 105, 106]): + self.send("明天下午三點叫我去拿包裹") + self.assertEqual([(f"Bearer {SECOND}", 13)], calls) + self.assertEqual([18], self.gemini.timeouts) + first, second = self.keys.list(USER.id) + self.assertEqual(("cooldown", "timeout"), (first.status, first.last_error)) + self.assertEqual(("ready", None), (second.status, second.last_error)) + draft = self.service.peek_draft(CHAT.id, USER.id, NOW) + self.assertEqual("groq", draft.parse_result["provider"]) + self.assertEqual("openai/gpt-oss-120b", draft.parse_result["model"]) + self.assertIn("使用 AI 解析", self.telegram.messages[-1].text) + self.assertEqual([], self.repo.list_pending(CHAT.id)) + self.confirm(draft.id) + self.assertEqual("拿包裹", self.repo.list_pending(CHAT.id)[0].title) + self.assertNotIn(SECOND, repr(self.telegram.messages)) + def test_failed_key_message_deletion_is_reported_without_echoing(self) -> None: with patch.object( self.telegram, "delete_message", side_effect=TelegramApiError("deleteMessage", "no") diff --git a/tests/test_groq.py b/tests/test_groq.py new file mode 100644 index 0000000..d8d58c0 --- /dev/null +++ b/tests/test_groq.py @@ -0,0 +1,148 @@ +from __future__ import annotations + +import http.client +import io +import json +import unittest +import urllib.error +from copy import deepcopy +from email.message import Message + +from test_ai import SECRET, parsed + +from remindly.ai.contracts import ProviderError +from remindly.ai.groq import GroqProvider, groq_schema +from remindly.ai.parser import SCHEMA + + +def completion(data=None, *, finish="stop") -> bytes: + return json.dumps( + {"choices": [{"finish_reason": finish, "message": {"content": json.dumps(data)}}]} + ).encode() + + +class GroqProviderTest(unittest.TestCase): + def test_truncated_chunked_response_is_a_sanitized_connection_error(self) -> None: + class Socket: + def makefile(self, *args): + return io.BytesIO( + b"HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n" + b"100\r\n" + SECRET.encode() + ) + + response = http.client.HTTPResponse(Socket()) + response.begin() + provider = GroqProvider(opener=lambda *args, **kwargs: response) + with self.assertRaises(ProviderError) as error: + provider.generate(SECRET, "system", {}, SCHEMA) + self.assertEqual("connection", error.exception.kind) + self.assertNotIn(SECRET, str(error.exception)) + self.assertTrue(error.exception.__suppress_context__) + self.assertTrue(response.isclosed()) + + def test_request_contract_and_schema_conversion(self) -> None: + before = deepcopy(SCHEMA) + + def opener(request, *, timeout): + self.assertEqual(12.5, timeout) + self.assertEqual("POST", request.get_method()) + self.assertEqual("https://api.groq.com/openai/v1/chat/completions", request.full_url) + self.assertEqual(f"Bearer {SECRET}", request.get_header("Authorization")) + self.assertEqual("Remindly", request.get_header("User-agent")) + self.assertNotIn(SECRET, request.full_url) + body = json.loads(request.data) + self.assertNotIn(SECRET, request.data.decode()) + self.assertEqual("openai/gpt-oss-120b", body["model"]) + self.assertEqual("system", body["messages"][0]["content"]) + self.assertEqual({"text": "提醒我"}, json.loads(body["messages"][1]["content"])) + response_format = body["response_format"] + self.assertEqual("json_schema", response_format["type"]) + self.assertTrue(response_format["json_schema"]["strict"]) + converted = response_format["json_schema"]["schema"] + self.assertEqual(groq_schema(SCHEMA), converted) + self.assertFalse(converted["additionalProperties"]) + recurrence = converted["properties"]["recurrence"] + self.assertFalse(recurrence["additionalProperties"]) + self.assertEqual(["object", "null"], recurrence["type"]) + self.assertEqual(["integer", "null"], recurrence["properties"]["hour"]["type"]) + return io.BytesIO(completion(parsed())) + + provider = GroqProvider(opener=opener) + self.assertEqual( + parsed(), provider.generate(SECRET, "system", {"text": "提醒我"}, SCHEMA, timeout=12.5) + ) + self.assertEqual(before, SCHEMA) + self.assertIsNone(provider.model_display_name(SECRET, timeout=2)) + array = {"type": "array", "items": SCHEMA} + self.assertFalse(groq_schema(array)["items"]["additionalProperties"]) + + def test_http_errors_are_sanitized_and_quota_uses_only_headers(self) -> None: + cases = [ + (401, {}, "invalid", 60), + (403, {}, "permission", 60), + (400, {}, "request", 60), + (302, {}, "request", 60), + (503, {}, "temporary", 60), + (429, {}, "quota", 60), + (429, {"retry-after": "120.5"}, "quota", 121), + (429, {"retry-after": "nan"}, "quota", 60), + (429, {"retry-after": "inf"}, "quota", 60), + (429, {"retry-after": "bad"}, "quota", 60), + (429, {"retry-after": "999999"}, "quota", 90001), + (429, {"x-ratelimit-remaining-requests": "0"}, "daily_quota", 60), + (429, {"x-ratelimit-remaining-tokens": "0"}, "rate", 60), + (429, {"x-ratelimit-remaining-requests": "2"}, "quota", 60), + ] + for status, fields, kind, cooldown in cases: + headers = Message() + for name, value in fields.items(): + headers[name] = value + stream = io.BytesIO(SECRET.encode()) + + def opener(*args, status=status, headers=headers, stream=stream, **kwargs): + raise urllib.error.HTTPError(SECRET, status, SECRET, headers, stream) + + with ( + self.subTest(status=status, fields=fields), + self.assertRaises(ProviderError) as error, + ): + GroqProvider(opener=opener).generate(SECRET, "system", {}, SCHEMA) + self.assertEqual((kind, cooldown), (error.exception.kind, error.exception.cooldown)) + self.assertNotIn(SECRET, str(error.exception)) + self.assertTrue(error.exception.__suppress_context__) + self.assertTrue(stream.closed) + + def test_transport_errors_and_bad_responses(self) -> None: + for failure, kind in ( + (TimeoutError(SECRET), "timeout"), + (urllib.error.URLError(TimeoutError(SECRET)), "timeout"), + (urllib.error.URLError(SECRET), "connection"), + (OSError(SECRET), "connection"), + ): + + def opener(*args, failure=failure, **kwargs): + raise failure + + with self.subTest(kind=kind), self.assertRaises(ProviderError) as error: + GroqProvider(opener=opener).generate(SECRET, "system", {}, SCHEMA) + self.assertEqual(kind, error.exception.kind) + self.assertNotIn(SECRET, str(error.exception)) + + for body in ( + b"bad", + b"[]", + b"{}", + b'{"choices": []}', + b'{"choices": [null]}', + completion([]), + completion(parsed(), finish="length"), + b'{"choices": [{"finish_reason": "stop", "message": null}]}', + b'{"choices": [{"finish_reason": "stop", "message": {"content": "bad"}}]}', + b'{"choices": [{"finish_reason": "stop", ' + b'"message": {"refusal": "no", "content": "{}"}}]}', + ): + with self.subTest(body=body), self.assertRaises(ProviderError) as error: + GroqProvider(opener=lambda *a, body=body, **kw: io.BytesIO(body)).generate( + SECRET, "system", {}, SCHEMA + ) + self.assertEqual("response", error.exception.kind) diff --git a/tests/test_groq_live_eval.py b/tests/test_groq_live_eval.py new file mode 100644 index 0000000..89545c2 --- /dev/null +++ b/tests/test_groq_live_eval.py @@ -0,0 +1,112 @@ +from __future__ import annotations + +import json +import tempfile +import unittest +from pathlib import Path +from unittest.mock import patch + +from test_ai import parsed + +from remindly.ai.contracts import GeneratedResponse, ProviderError +from scripts import groq_live_eval as evaluation + + +class GroqLiveEvaluationTest(unittest.TestCase): + def case(self): + return { + "id": "test", + "source": "test", + "text": "提醒我明天下午三點買湯", + "now": "2026-10-30T12:00:00+08:00", + "timezone": "Asia/Taipei", + "repeats": 1, + "expected": { + "title": "買湯", + "at": "2026-10-31T15:00:00+08:00", + "missing": [], + "complete": True, + "participants": ["user:7"], + "recurrence": None, + }, + } + + def test_suite_has_distinct_sources_and_repeats_drift_cases(self): + cases = [json.loads(line) for line in evaluation.CASES.read_text().splitlines()] + self.assertEqual(36, len(cases)) + self.assertEqual(36, len({case["id"] for case in cases})) + self.assertEqual(52, sum(case["repeats"] for case in cases)) + self.assertEqual(8, sum(case["repeats"] == 3 for case in cases)) + for case in cases: + source_file, source_id = case["source"].split("#") + source = [ + json.loads(line) + for line in (evaluation.CASES.parent / source_file).read_text().splitlines() + ] + self.assertIn(source_id, {row["id"] for row in source}) + + def test_raw_failure_is_preserved_when_local_rules_repair_it(self): + generator = evaluation.RecordingGenerator("fake-key") + with patch.object( + generator.provider, + "generate", + return_value=parsed(title="模型改寫", date="2027-01-01", time="15:00"), + ): + result = evaluation.evaluate(self.case(), generator) + self.assertIn("title", result["raw_failures"]) + self.assertIn("at", result["raw_failures"]) + self.assertEqual({}, result["local_failures"]) + self.assertEqual("模型改寫", result["raw"]["title"]) + + def test_recorded_live_responses_satisfy_local_regression_expectations(self): + cases = { + case["id"]: case for case in map(json.loads, evaluation.CASES.read_text().splitlines()) + } + recording = evaluation.CASES.with_name("groq_live_results_2026-10-30.jsonl") + rows = [ + row + for row in map(json.loads, recording.read_text().splitlines()) + if row["event"] == "result" + ] + self.assertEqual(52, len(rows)) + self.assertEqual(52, len({(row["id"], row["run"]) for row in rows})) + for row in rows: + with self.subTest(case=row["id"], run=row["run"]): + generator = evaluation.RecordingGenerator("fake-key") + with patch.object(generator.provider, "generate", return_value=row["raw"]): + result = evaluation.evaluate(cases[row["id"]], generator) + self.assertEqual({}, result["local_failures"]) + + def test_quota_cooldown_and_resume_do_not_repeat_completed_calls(self): + case = self.case() + calls = [] + + class FakeGenerator: + raw = None + + def generate(self, *args, **kwargs): + calls.append(True) + if len(calls) == 1: + raise ProviderError("quota", 120) + self.raw = parsed(title="買湯", date="2026-10-31", time="15:00") + return GeneratedResponse(self.raw, "groq", "test") + + with tempfile.TemporaryDirectory() as directory: + suite = Path(directory) / "cases.jsonl" + suite.write_text(json.dumps(case) + "\n") + output = Path(directory) / "results.jsonl" + with ( + patch.object(evaluation, "CASES", suite), + patch.object(evaluation, "RecordingGenerator", return_value=FakeGenerator()), + patch.object(evaluation, "wait_until") as wait, + patch.dict("os.environ", {"GROQ_API_KEY": "fake-key"}), + patch("sys.argv", ["eval", "--output", str(output)]), + ): + evaluation.main() + evaluation.main() + rows = [json.loads(line) for line in output.read_text().splitlines()] + self.assertEqual(["meta", "retry", "result"], [row["event"] for row in rows]) + self.assertEqual(120, rows[1]["cooldown"]) + wait.assert_called_once_with(rows[1]["retry_at"]) + self.assertEqual(2, len(calls)) + self.assertNotIn("fake-key", output.read_text())