Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3,457 changes: 3,457 additions & 0 deletions session_memory_summary_diff_report.json

Large diffs are not rendered by default.

31 changes: 31 additions & 0 deletions tests/sessions/replay_cases/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Replay Consistency Design

The replay harness loads ten JSONL traces and applies each operation to a paired
SessionService and MemoryService. InMemory and SQLite run by default, so CI gets
a real persistence round trip without external infrastructure. Set
`TRPC_REPLAY_BACKENDS=inmemory` for the smallest mode. A disposable SQL or Redis
backend can be added with `TRPC_REPLAY_SQL_URL` or `TRPC_REPLAY_REDIS_URL`;
explicit backend selection uses `TRPC_REPLAY_BACKENDS=inmemory,sql,redis`.

Comparison removes only non-business variance. Physical event IDs map to logical
replay IDs, wall-clock timestamps become a typed marker, JSON objects are
compared structurally, and memory matches are sorted because equal-keyword
ranking is not part of the MemoryService contract. Summary text uses Unicode
NFKC, case folding, and whitespace collapse. Summary ownership, version,
supersedes relation, event counts, and active anchor count remain exact. The
report records any raw memory-order or recovery-mechanism difference under a
narrow `allowed_diffs` rule; content, count, state, event order, and summary
metadata are never globally ignored.

Duplicate append and failed summary replacement cases use compensating replay
recovery, then reload the backend to detect duplicate events, dirty state, stale
anchors, or an incorrect cached summary. Per-backend expectations keep
InMemory-only runs meaningful, while pairwise comparison exposes backend drift.
Every unexpected difference includes the logical session ID, JSON field path,
event index or summary ID when applicable, and both backend values. Run the
harness directly to refresh the report:

```bash
python -m tests.sessions.replay_harness \
--output session_memory_summary_diff_report.json
```
10 changes: 10 additions & 0 deletions tests/sessions/replay_cases/session_memory_summary.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{"case_id":"single_turn","description":"One user message followed by one assistant response.","session_id":"session-single","operations":[{"op":"append_event","event":{"id":"single-user-1","timestamp":1700000001.0,"author":"user","role":"user","text":"Hello, replay harness."}},{"op":"append_event","event":{"id":"single-agent-1","timestamp":1700000002.0,"author":"assistant","role":"model","text":"Hello. How can I help?"}}],"expect":{"active_event_count":2,"historical_event_count":0,"state":{},"summary_present":false}}
{"case_id":"multi_turn","description":"Three complete user and assistant turns preserve event order.","session_id":"session-multi","operations":[{"op":"append_event","event":{"id":"multi-user-1","timestamp":1700000011.0,"author":"user","role":"user","text":"My name is Ada."}},{"op":"append_event","event":{"id":"multi-agent-1","timestamp":1700000012.0,"author":"assistant","role":"model","text":"Nice to meet you, Ada."}},{"op":"append_event","event":{"id":"multi-user-2","timestamp":1700000013.0,"author":"user","role":"user","text":"What is my name?"}},{"op":"append_event","event":{"id":"multi-agent-2","timestamp":1700000014.0,"author":"assistant","role":"model","text":"Your name is Ada."}},{"op":"append_event","event":{"id":"multi-user-3","timestamp":1700000015.0,"author":"user","role":"user","text":"Remember that."}},{"op":"append_event","event":{"id":"multi-agent-3","timestamp":1700000016.0,"author":"assistant","role":"model","text":"I will remember it."}}],"expect":{"active_event_count":6,"historical_event_count":0,"state":{},"summary_present":false}}
{"case_id":"tool_call","description":"A function call and matching function response round trip through storage.","session_id":"session-tool","operations":[{"op":"append_event","event":{"id":"tool-user-1","timestamp":1700000021.0,"author":"user","role":"user","text":"What is the weather in Shenzhen?"}},{"op":"append_event","event":{"id":"tool-call-1","timestamp":1700000022.0,"author":"assistant","role":"model","function_call":{"id":"call-weather-1","name":"get_weather","args":{"city":"Shenzhen","unit":"celsius"}}}},{"op":"append_event","event":{"id":"tool-response-1","timestamp":1700000023.0,"author":"tool","role":"function","function_response":{"id":"call-weather-1","name":"get_weather","response":{"temperature":31,"condition":"sunny"}}}},{"op":"append_event","event":{"id":"tool-agent-1","timestamp":1700000024.0,"author":"assistant","role":"model","text":"It is 31 C and sunny in Shenzhen."}}],"expect":{"active_event_count":4,"historical_event_count":0,"state":{},"summary_present":false}}
{"case_id":"state_overwrite","description":"Session, user, and app state updates use last-write-wins semantics.","session_id":"session-state","initial_state":{"theme":"light","user:locale":"en","app:policy":"v1"},"operations":[{"op":"state_update","event":{"id":"state-update-1","timestamp":1700000031.0,"author":"system","state_delta":{"theme":"dark","counter":1}}},{"op":"state_update","event":{"id":"state-update-2","timestamp":1700000032.0,"author":"system","state_delta":{"theme":"solarized","counter":2,"user:locale":"fr","app:policy":"v2","temp:request":"not-persisted"}}}],"expect":{"active_event_count":2,"historical_event_count":0,"state":{"theme":"solarized","counter":2,"user:locale":"fr","app:policy":"v2"},"summary_present":false}}
{"case_id":"memory_write_read","description":"Preference and fact memories are stored and retrieved by keyword.","session_id":"session-memory","operations":[{"op":"append_event","event":{"id":"memory-user-1","timestamp":1700000041.0,"author":"user","role":"user","text":"I prefer green tea without sugar."}},{"op":"append_event","event":{"id":"memory-agent-1","timestamp":1700000042.0,"author":"assistant","role":"model","text":"I will remember your tea preference."}},{"op":"append_event","event":{"id":"memory-user-2","timestamp":1700000043.0,"author":"user","role":"user","text":"My timezone is Asia Shanghai."}},{"op":"append_event","event":{"id":"memory-agent-2","timestamp":1700000044.0,"author":"assistant","role":"model","text":"Your timezone preference is saved."}},{"op":"store_memory"},{"op":"search_memory","label":"tea_preference","query":"tea"},{"op":"search_memory","label":"timezone_fact","query":"timezone"}],"expect":{"active_event_count":4,"historical_event_count":0,"state":{},"memory_counts":{"tea_preference":2,"timezone_fact":2},"summary_present":false}}
{"case_id":"summary_create_update","description":"A second summary supersedes the first revision without losing ownership metadata.","session_id":"session-summary-update","operations":[{"op":"append_event","event":{"id":"summary-u1","timestamp":1700000051.0,"author":"user","role":"user","text":"Plan a release checklist."}},{"op":"append_event","event":{"id":"summary-a1","timestamp":1700000052.0,"author":"assistant","role":"model","text":"Start with tests and changelog."}},{"op":"append_event","event":{"id":"summary-u2","timestamp":1700000053.0,"author":"user","role":"user","text":"Add a security review."}},{"op":"append_event","event":{"id":"summary-a2","timestamp":1700000054.0,"author":"assistant","role":"model","text":"Security review added."}},{"op":"append_event","event":{"id":"summary-u3","timestamp":1700000055.0,"author":"user","role":"user","text":"Keep the latest decisions."}},{"op":"append_event","event":{"id":"summary-a3","timestamp":1700000056.0,"author":"assistant","role":"model","text":"The decisions are retained."}},{"op":"summarize","summary_id":"summary-update-v1","version":1,"text":"The release checklist includes tests, changelog, and a security review."},{"op":"append_event","event":{"id":"summary-u4","timestamp":1700000057.0,"author":"user","role":"user","text":"The release owner is Lin."}},{"op":"append_event","event":{"id":"summary-a4","timestamp":1700000058.0,"author":"assistant","role":"model","text":"Lin is the release owner."}},{"op":"summarize","summary_id":"summary-update-v2","version":2,"supersedes":"summary-update-v1","text":"The release needs tests, changelog, and security review. Lin owns the release."}],"expect":{"active_event_count":3,"historical_event_count":7,"state":{},"summary_present":true,"summary_id":"summary-update-v2","summary_version":2,"summary_supersedes":"summary-update-v1","summary_text":"the release needs tests, changelog, and security review. lin owns the release.","summary_session_id":"session-summary-update","summary_anchor_count":1}}
{"case_id":"summary_event_truncation","description":"A summary anchor, retained events, and later events reconstruct compressed context.","session_id":"session-summary-truncate","operations":[{"op":"append_event","event":{"id":"truncate-u1","timestamp":1700000061.0,"author":"user","role":"user","text":"The project codename is Orion."}},{"op":"append_event","event":{"id":"truncate-a1","timestamp":1700000062.0,"author":"assistant","role":"model","text":"Codename Orion noted."}},{"op":"append_event","event":{"id":"truncate-u2","timestamp":1700000063.0,"author":"user","role":"user","text":"The target date is Friday."}},{"op":"append_event","event":{"id":"truncate-a2","timestamp":1700000064.0,"author":"assistant","role":"model","text":"Friday is the target date."}},{"op":"append_event","event":{"id":"truncate-u3","timestamp":1700000065.0,"author":"user","role":"user","text":"Keep two recent events."}},{"op":"append_event","event":{"id":"truncate-a3","timestamp":1700000066.0,"author":"assistant","role":"model","text":"Two recent events will remain."}},{"op":"summarize","summary_id":"summary-truncate-v1","version":1,"text":"Project Orion targets Friday."},{"op":"append_event","event":{"id":"truncate-u4","timestamp":1700000067.0,"author":"user","role":"user","text":"Add a deployment rehearsal."}},{"op":"append_event","event":{"id":"truncate-a4","timestamp":1700000068.0,"author":"assistant","role":"model","text":"Deployment rehearsal added."}}],"expect":{"active_event_count":5,"historical_event_count":4,"state":{},"summary_present":true,"summary_id":"summary-truncate-v1","summary_version":1,"summary_text":"project orion targets friday.","summary_session_id":"session-summary-truncate","summary_anchor_count":1}}
{"case_id":"summary_semantic_normalization","description":"Summary comparison normalizes Unicode case and whitespace but preserves metadata.","session_id":"session-summary-semantic","operations":[{"op":"append_event","event":{"id":"semantic-u1","timestamp":1700000071.0,"author":"user","role":"user","text":"I chose tea."}},{"op":"append_event","event":{"id":"semantic-a1","timestamp":1700000072.0,"author":"assistant","role":"model","text":"Tea choice recorded."}},{"op":"append_event","event":{"id":"semantic-u2","timestamp":1700000073.0,"author":"user","role":"user","text":"Brew it next."}},{"op":"append_event","event":{"id":"semantic-a2","timestamp":1700000074.0,"author":"assistant","role":"model","text":"Brewing is next."}},{"op":"summarize","summary_id":"summary-semantic-v1","version":1,"text":" User chose TEA.\nNext: brew. "}],"expect":{"active_event_count":3,"historical_event_count":2,"state":{},"summary_present":true,"summary_id":"summary-semantic-v1","summary_version":1,"summary_text":"user chose tea. next: brew.","summary_session_id":"session-summary-semantic","summary_anchor_count":1}}
{"case_id":"duplicate_write_recovery","description":"A retried event write is detected and recovered to one logical event.","session_id":"session-duplicate","operations":[{"op":"append_event","event":{"id":"duplicate-u1","timestamp":1700000081.0,"author":"user","role":"user","text":"Run once."}},{"op":"append_event","event":{"id":"duplicate-a1","timestamp":1700000082.0,"author":"assistant","role":"model","text":"Completed once."}},{"op":"duplicate_append","event":{"id":"duplicate-a1","timestamp":1700000082.0,"author":"assistant","role":"model","text":"Completed once."}}],"expect":{"active_event_count":2,"historical_event_count":0,"state":{},"summary_present":false,"unique_event_ids":true,"recovery_kinds":["duplicate_append"]}}
{"case_id":"failed_summary_recovery","description":"A failure after summary replacement restores events, state, and the prior summary revision.","session_id":"session-failed-summary","operations":[{"op":"append_event","event":{"id":"failure-u1","timestamp":1700000091.0,"author":"user","role":"user","text":"The migration starts Monday."}},{"op":"append_event","event":{"id":"failure-a1","timestamp":1700000092.0,"author":"assistant","role":"model","text":"Monday start recorded."}},{"op":"append_event","event":{"id":"failure-u2","timestamp":1700000093.0,"author":"user","role":"user","text":"Backups are mandatory."}},{"op":"append_event","event":{"id":"failure-a2","timestamp":1700000094.0,"author":"assistant","role":"model","text":"Backups are mandatory."}},{"op":"append_event","event":{"id":"failure-u3","timestamp":1700000095.0,"author":"user","role":"user","text":"Keep the newest turn."}},{"op":"append_event","event":{"id":"failure-a3","timestamp":1700000096.0,"author":"assistant","role":"model","text":"Newest turn retained."}},{"op":"summarize","summary_id":"summary-failure-v1","version":1,"text":"Migration starts Monday and requires backups."},{"op":"append_event","event":{"id":"failure-u4","timestamp":1700000097.0,"author":"user","role":"user","text":"Notify operations."}},{"op":"append_event","event":{"id":"failure-a4","timestamp":1700000098.0,"author":"assistant","role":"model","text":"Operations will be notified."}},{"op":"fail_summary","summary_id":"summary-failure-v2","version":2,"supersedes":"summary-failure-v1","text":"Migration starts Monday, requires backups, and operations must be notified."}],"expect":{"active_event_count":5,"historical_event_count":4,"state":{},"summary_present":true,"summary_id":"summary-failure-v1","summary_version":1,"summary_text":"migration starts monday and requires backups.","summary_session_id":"session-failed-summary","summary_anchor_count":1,"recovery_kinds":["summary_update"]}}
Loading
Loading