Skip to content

feat: add session memory replay consistency harness#237

Open
mingri31164 wants to merge 1 commit into
trpc-group:mainfrom
mingri31164:feat/issue-89-session-replay-consistency
Open

feat: add session memory replay consistency harness#237
mingri31164 wants to merge 1 commit into
trpc-group:mainfrom
mingri31164:feat/issue-89-session-replay-consistency

Conversation

@mingri31164

Copy link
Copy Markdown

Summary

  • add a reusable Session / Memory / Summary replay harness with 10 public JSONL trajectories
  • compare normalized events, state, memory, summary content and strict summary metadata across InMemory, SQLite, SQL, and Redis
  • emit field-level JSON differences with session ID, event index or summary ID, field path, and both backend values
  • cover duplicate appends and failed summary replacement with reload-and-recovery checks
  • support InMemory-only, default InMemory + SQLite, and environment-enabled SQL / Redis modes
  • fix Redis hash round trips discovered by the harness: redis-py mapping-based HSET plus typed HGETALL decoding for standalone and cluster storage

Consistency model

Only non-business variance is normalized: physical IDs, wall-clock timestamps, JSON key order, summary Unicode/case/whitespace, and contract-free memory result ordering. Summary ownership, version, supersedes relation, update ordering, anchor count, event order, content, state, and memory membership remain strict. Allowed differences are limited to memory order and recovery mechanism and are reported explicitly.

Verification

  • default InMemory + SQLite report: 10/10 cases, 0 unexpected differences, 0 invariant failures, 0.521s
  • real MySQL 8.0.37 + Redis 7.4.9 integration: 10/10 cases, 0 unexpected differences, 0 invariant failures, 1.209s
  • replay acceptance suite with real services: 15 passed, including 10/10 injected inconsistencies and all 3 required summary defect classes
  • Redis / Redis Cluster / Session targeted regression: 91 passed
  • repository suite: 9,431 passed, 1 existing xfail, 2 unrelated baseline failures (Python 3.11 enum compatibility and charset detection)
  • package build: wheel and sdist succeeded

Evidence

The screenshots below are rendered directly from the generated integration JSON and JUnit XML artifacts. The second screenshot shows a controlled summary ownership fault resolved to the exact field and backend values.

trpc-replay-issue-89-integration-full trpc-replay-issue-89-fault-detection-full Fixes #89

@github-actions

Copy link
Copy Markdown

CLA Assistant Lite bot:
Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.


I have read the CLA Document and I hereby sign the CLA


You can retrigger this bot by commenting recheck in this Pull Request

@codecov

codecov Bot commented Jul 25, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.87879% with 4 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@6a2f7f9). Learn more about missing BASE report.

Files with missing lines Patch % Lines
trpc_agent_sdk/storage/_redis.py 87.50000% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main        #237   +/-   ##
==========================================
  Coverage        ?   87.89509%           
==========================================
  Files           ?         482           
  Lines           ?       45180           
  Branches        ?           0           
==========================================
  Hits            ?       39711           
  Misses          ?        5469           
  Partials        ?           0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@helloopenworld

Copy link
Copy Markdown
Contributor

AI Code Review

审查完成。我已逐文件核对 diff 内容,并交叉验证了被调用方(RedisSessionService/RedisClusterStorage/SummarizerSessionManager/SessionSummarizer 等)、配置(SessionServiceConfigdecode_responses)、SQL 持久化字段以及测试断言路径。

发现的问题

🚨 Critical

无。

⚠️ Warning

  • session_memory_summary_diff_report.json:1-3457:将 3457 行的生成物作为源文件提交进仓库

    • 该文件由 replay_harness.py 生成,含 generated_at(已为 2026-07-25T16:36:37...,提交即过期)和 report_sha256。测试(test_all_public_cases_match_without_false_positives)只写 tmp_path 并做自一致性校验,从不与这份提交件比对,因此它既无回归保护作用,又会随每次手动运行 python -m tests.sessions.replay_harness(默认 --output 指向仓库根同名文件,见 tests/sessions/replay_harness.py:984)产生噪声 diff。建议将该 JSON 加入 .gitignore,或改为提交一份固定夹具并在测试中校验。
  • tests/sessions/replay_harness.py:550-565fail_summary 仅用 except RuntimeError 捕获注入失败,未覆盖其他异常类型

    • try 内调用 _perform_summary(会触发真实存储写入),若存储/摘要逻辑抛出非 RuntimeError(如 ValueErrorIntegrityError、SQL 序列化错误),异常会直接逃逸,跳过 update_session(session_checkpoint) 与缓存恢复,使该用例的"失败后恢复"语义未被验证。当前因确定性 model 未触发,但削弱了用例对恢复路径的覆盖。建议捕获更宽的异常范围(仍按错误消息匹配注入哨兵)。

💡 Suggestion

总结

整体风险较低:_redis.py/_redis_cluster.py 的 hset→mapping 与 hgetall 反序列化改动是针对 decode_responses=True 场景的真实修复,且与现有 _update_app_state/_update_user_state 调用方和 query 路径兼容,无破坏性回归;replay harness 自身逻辑自洽。主要需修复的是提交生成物与 fail_summary 异常覆盖两项 Warning,不存在必须阻塞合并的 Critical 问题。

测试建议

  • fail_summary 注入一个非 RuntimeError 的存储失败(如 mock update_sessionValueError),断言恢复分支仍被执行且 recovered 正确。
  • 增加一条针对 decode_responses=True 下 Redis(非集群)hgetall/hset 多字段状态读写往返的测试,锁定本次 _deserialize_value/_serialize_hash_value 的修复行为。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

构建 Session / Memory 多后端回放一致性测试框架

2 participants