Skip to content

Fix active-turn flag leak with TTL-based stale eviction (issue #2456) - #2484

Closed
TrueFurina wants to merge 2 commits into
TencentCloud:mainfrom
TrueFurina:fix/turn-active-flag-leak-2456
Closed

TrueFurina wants to merge 2 commits into
TencentCloud:mainfrom
TrueFurina:fix/turn-active-flag-leak-2456

Conversation

@TrueFurina

Copy link
Copy Markdown

Summary

This PR fixes the active-turn flag leak reported in issue #2456, where a Dashboard thread could stay urn_active=true forever after a dropped WebSocket handler task, a client disconnect mid-stream, or the process being interrupted. Once stuck, the composer stays disabled ("???") and the user cannot type a new message.

Root cause

WebSocketHub._active_turns was a plain set. mark_turn_idle only runs inside the inally block of WebSocketChannel.handle_inbound. When the coroutine is dropped (connection loss, process crash, task cancellation), that inally is unreachable - the flag leaks for the lifetime of the process.

Fix

  1. _active_turns is now dict[str, float] mapping hread_id -> time.monotonic() of the last pushed frame.
  2. push_to_thread auto-refreshes the TTL via a new �ump_turn_activity helper, so any thread that still receives chunks stays "warm".
  3. New evict_stale_turns(max_quiet_seconds=30.0) drops entries that have not been bumped for more than 30 s, and logs which thread_ids were pruned. It is called lazily from is_turn_active, so reconnecting clients (or the ping machinery) heal the leak without any scheduler.
  4. mark_turn_active / mark_turn_idle keep their public signatures; only the backing store changes, so no caller code is affected.

Test plan

  • Manual: drop a WS handler task mid-stream ? after the default 30 s quiet window, is_turn_active(thread_id) returns False.
  • Manual: a live stream that keeps producing chunks keeps its entry alive across the entire run.
  • mark_turn_idle still removes entries immediately (the happy path is unchanged).

30s races normal tool latency: ws_channel pushes frames only when the
processor emits a chunk, and awaiting_tools is silent, so a single slow
shell/MCP call can look like an orphan. is_turn_active() feeds the
turn_status frame sent on reconnect, so evicting a live turn reports
"idle" while it is still running. 15 min still heals a genuine leak;
tighten once a heartbeat frame exists.
@jubaoliang jubaoliang closed this Oct 31, 2026
TrueFurina added a commit to TrueFurina/Octop that referenced this pull request Nov 2, 2026
- TencentCloud#2495 fix(oauth): preserve existing query params on authorization_endpoint (TencentCloud#2391)
- TencentCloud#2529 fix(execute_env): fold admin env into local_shell backend (close TencentCloud#179)
- TencentCloud#2543 [CLOSED BY AUTHOR] accidental empty blob commit - re-opening with fix
- TencentCloud#2570 fix(core): guard POSIX-only os.chmod/os.umask on Windows (TencentCloud#2370)
- TencentCloud#2571 fix(tls): guard POSIX-only os.chmod on Windows (TencentCloud#2370)
- TencentCloud#2572 fix(core): guard final POSIX-only os.chmod call sites (TencentCloud#2370)
- TencentCloud#2573 fix(experts): guard POSIX-only os.chmod in skill scripts (TencentCloud#2370)
- TencentCloud#2575 fix(agents): reuse _token_int in _usage_has_segments for safe coercion
- TencentCloud#2580 fix(shell): guard bare int() in shell_ws resize handler
- TencentCloud#2605 fix(xlsx): suppress PermissionError during soffice shim temp cleanup
- TencentCloud#2648 fix(security): prevent path-traversal in tccli OAuth helper --profile
- TencentCloud#2665 fix(octop): guard credential marker reads and backup file read with OSError
- TencentCloud#2692 fix(cli): read shell rc file as UTF-8 when installing completion
- TencentCloud#2706 fix(security): write tool guard rules atomically so a torn save cannot disable the guard
- TencentCloud#2731 fix(gateway): keep TLS verification when sending the Feishu greeting
- TencentCloud#2596 fix(clinical): avoid TOCTOU in atomic_write temp cleanup
- TencentCloud#2597 fix(service): suppress OSError in _write_managed_file finally cleanup
- TencentCloud#2599 fix(skill-manager): use unlink(missing_ok=True) to close TOCTOU in restore
- TencentCloud#2600 fix(users): add missing_ok=True to avatar unlink in delete_profile_avatar
- TencentCloud#2601 fix(api): add missing_ok=True to daily-memory delete unlink
- TencentCloud#2604 fix(agently): add missing_ok=True to confirmation-file unlinks
- TencentCloud#2606 fix(ldap): narrow _safe_unbind suppress(Exception) to LDAPException
- TencentCloud#2607 fix(skills): narrow suppress(Exception) around proc.wait to specific errors
- TencentCloud#2635 fix(cli): narrow suppress(Exception) in _close_cli_session to OSError+RuntimeError
- TencentCloud#2636 fix(feishu): narrow suppress(Exception) in creator timeout finally to OSError
- TencentCloud#2637 fix(gateway): narrow suppress(Exception) in usage_record to OSError
- TencentCloud#2639 fix(terminal): narrow suppress(Exception) in agent-fallback lookup to OSError
- TencentCloud#2640 fix(backend): narrow suppress(Exception) in probe.py docker path to OSError+RuntimeError
- TencentCloud#2641 fix(browse): narrow _close_backend suppress(Exception) to OSError+RuntimeError
- TencentCloud#2643 fix(backup): add timeout=900 to pg_dump and pg_restore subprocess.run
- TencentCloud#2645 fix(pptx): add missing_ok=True to all bare unlink/rmdir in clean.py
- TencentCloud#2646 fix(soffice): add missing_ok=True to gcc shim temp-file cleanup in docx and pptx skills
- TencentCloud#2647 fix(composer): narrow suppress(Exception) in write_files adelete to OSError+RuntimeError
- TencentCloud#2652 Add encoding=utf-8 to pdf skill scripts and timeout to ollama pkill
- TencentCloud#2690 fix(backup): unlink symlinked/junctioned dirs during replace import
- TencentCloud#2693 fix(env): write ~/.octop/env atomically
- TencentCloud#2478 fix(cli): guard bundled import when --force clears site-packages (TencentCloud#2458)
- TencentCloud#2541 guard seed_bundled so init --force never wipes data then crashes (TencentCloud#2458)
- TencentCloud#2566 fix(cron): add in-process idempotency guard against duplicate deliveries (TencentCloud#2401)
- TencentCloud#2577 fix(config): guard bare int() on http_port and database port parsing
- TencentCloud#2578 fix(voice): guard bare int() on tencent voice_type coercion
- TencentCloud#2585 fix(cli): guard remaining bare asyncio.run() in skills.py and launch.py
- TencentCloud#2586 fix(utils): specify encoding=utf-8 on subprocess text=True calls
- TencentCloud#2588 fix(infra): add encoding=utf-8 to subprocess text=True in control
- TencentCloud#2593 fix(experts): add encoding=utf-8 to macro read_text calls
- TencentCloud#2653 office skills: add explicit encoding=utf-8 to open() in PDF and validator scripts
- TencentCloud#2654 patrol_server.py: add explicit encoding=utf-8 to JSONL file reader
- TencentCloud#2656 Guard Path.iterdir() with OSError in browser_media and skill_package_store
- TencentCloud#2661 fix(octop): guard glob/rglob with OSError in expert catalog discovery
- TencentCloud#2664 fix(octop): guard slim_control endpoint read with OSError and JSON parse errors
- TencentCloud#2170 docs: complete the ~/.octop layout tree and note the wizard password location
- TencentCloud#2369 docs: fix wizard password path in .env.example (post-TencentCloud#2174)
- TencentCloud#2480 fix(thread-artifacts): prefer tool result path over request args (TencentCloud#2464)
- TencentCloud#2484 Fix active-turn flag leak with TTL-based stale eviction (issue TencentCloud#2456)
- TencentCloud#2490 fix(cli): handle EXDEV when backup output crosses filesystems
- TencentCloud#2494 fix(grep_safety): search only target file path, not parent dir siblings (TencentCloud#2352)
- TencentCloud#2509 fix(sso): only treat same-origin opener as Octop SSO popup (TencentCloud#2347)
- TencentCloud#2519 fix(provider-probe): skip disabled models when picking default probe target (TencentCloud#2312)
- TencentCloud#2526 fix(knowledge): emit citations only for hits that made it into context (TencentCloud#1490)
- TencentCloud#2527 fix(hitl): isolate pending approvals by thread, not just session (TencentCloud#1897)
- TencentCloud#2539 fix(message_keys): group-chat sessions key by room not by sender (TencentCloud#2067)
- TencentCloud#2542 return 409 PROVIDER_NAME_TAKEN on duplicate provider creation (TencentCloud#1967)
- TencentCloud#2548 fix(skillhub): retry HTTP request on transient CDN IPv4 failure (TencentCloud#1819)
- TencentCloud#2562 fix(cron): preserve interval next_run_time across rebuilds (TencentCloud#2292)
- TencentCloud#2581 fix(gateway): use time.monotonic() for bot scan-poll deadline loop
- TencentCloud#2592 fix(core): make cleanup and datetime parsing resilient
- TencentCloud#2631 fix(users): add ignore_errors=True to checkpoint rmtree
- TencentCloud#2699 fix(media): serialize inbound attachment path allocation and upload
- TencentCloud#2714 fix(gateway): parse creator events from a text-mode subprocess pipe
- TencentCloud#2720 fix(preferences): keep bookmarks with upper-case URL scheme
TrueFurina added a commit to TrueFurina/Octop that referenced this pull request Nov 2, 2026
- TencentCloud#2495 fix(oauth): preserve existing query params on authorization_endpoint (TencentCloud#2391)
- TencentCloud#2529 fix(execute_env): fold admin env into local_shell backend (close TencentCloud#179)
- TencentCloud#2570 fix(core): guard POSIX-only os.chmod/os.umask on Windows (TencentCloud#2370)
- TencentCloud#2571 fix(tls): guard POSIX-only os.chmod on Windows (TencentCloud#2370)
- TencentCloud#2572 fix(core): guard final POSIX-only os.chmod call sites (TencentCloud#2370)
- TencentCloud#2573 fix(experts): guard POSIX-only os.chmod in skill scripts (TencentCloud#2370)
- TencentCloud#2575 fix(agents): reuse _token_int in _usage_has_segments for safe coercion
- TencentCloud#2580 fix(shell): guard bare int() in shell_ws resize handler
- TencentCloud#2605 fix(xlsx): suppress PermissionError during soffice shim temp cleanup
- TencentCloud#2648 fix(security): prevent path-traversal in tccli OAuth helper --profile
- TencentCloud#2665 fix(octop): guard credential marker reads and backup file read with OSError
- TencentCloud#2692 fix(cli): read shell rc file as UTF-8 when installing completion
- TencentCloud#2706 fix(security): write tool guard rules atomically so a torn save cannot disable the guard
- TencentCloud#2731 fix(gateway): keep TLS verification when sending the Feishu greeting
- TencentCloud#2596 fix(clinical): avoid TOCTOU in atomic_write temp cleanup
- TencentCloud#2597 fix(service): suppress OSError in _write_managed_file finally cleanup
- TencentCloud#2599 fix(skill-manager): use unlink(missing_ok=True) to close TOCTOU in restore
- TencentCloud#2600 fix(users): add missing_ok=True to avatar unlink in delete_profile_avatar
- TencentCloud#2601 fix(api): add missing_ok=True to daily-memory delete unlink
- TencentCloud#2604 fix(agently): add missing_ok=True to confirmation-file unlinks
- TencentCloud#2606 fix(ldap): narrow _safe_unbind suppress(Exception) to LDAPException
- TencentCloud#2607 fix(skills): narrow suppress(Exception) around proc.wait to specific errors
- TencentCloud#2635 fix(cli): narrow suppress(Exception) in _close_cli_session to OSError+RuntimeError
- TencentCloud#2636 fix(feishu): narrow suppress(Exception) in creator timeout finally to OSError
- TencentCloud#2637 fix(gateway): narrow suppress(Exception) in usage_record to OSError
- TencentCloud#2639 fix(terminal): narrow suppress(Exception) in agent-fallback lookup to OSError
- TencentCloud#2640 fix(backend): narrow suppress(Exception) in probe.py docker path to OSError+RuntimeError
- TencentCloud#2641 fix(browse): narrow _close_backend suppress(Exception) to OSError+RuntimeError
- TencentCloud#2643 fix(backup): add timeout=900 to pg_dump and pg_restore subprocess.run
- TencentCloud#2645 fix(pptx): add missing_ok=True to all bare unlink/rmdir in clean.py
- TencentCloud#2646 fix(soffice): add missing_ok=True to gcc shim temp-file cleanup in docx and pptx skills
- TencentCloud#2647 fix(composer): narrow suppress(Exception) in write_files adelete to OSError+RuntimeError
- TencentCloud#2652 Add encoding=utf-8 to pdf skill scripts and timeout to ollama pkill
- TencentCloud#2690 fix(backup): unlink symlinked/junctioned dirs during replace import
- TencentCloud#2693 fix(env): write ~/.octop/env atomically
- TencentCloud#2478 fix(cli): guard bundled import when --force clears site-packages (TencentCloud#2458)
- TencentCloud#2541 guard seed_bundled so init --force never wipes data then crashes (TencentCloud#2458)
- TencentCloud#2566 fix(cron): add in-process idempotency guard against duplicate deliveries (TencentCloud#2401)
- TencentCloud#2577 fix(config): guard bare int() on http_port and database port parsing
- TencentCloud#2578 fix(voice): guard bare int() on tencent voice_type coercion
- TencentCloud#2585 fix(cli): guard remaining bare asyncio.run() in skills.py and launch.py
- TencentCloud#2586 fix(utils): specify encoding=utf-8 on subprocess text=True calls
- TencentCloud#2588 fix(infra): add encoding=utf-8 to subprocess text=True in control
- TencentCloud#2593 fix(experts): add encoding=utf-8 to macro read_text calls
- TencentCloud#2653 office skills: add explicit encoding=utf-8 to open() in PDF and validator scripts
- TencentCloud#2654 patrol_server.py: add explicit encoding=utf-8 to JSONL file reader
- TencentCloud#2656 Guard Path.iterdir() with OSError in browser_media and skill_package_store
- TencentCloud#2661 fix(octop): guard glob/rglob with OSError in expert catalog discovery
- TencentCloud#2664 fix(octop): guard slim_control endpoint read with OSError and JSON parse errors
- TencentCloud#2170 docs: complete the ~/.octop layout tree and note the wizard password location
- TencentCloud#2369 docs: fix wizard password path in .env.example (post-TencentCloud#2174)
- TencentCloud#2480 fix(thread-artifacts): prefer tool result path over request args (TencentCloud#2464)
- TencentCloud#2484 Fix active-turn flag leak with TTL-based stale eviction (issue TencentCloud#2456)
- TencentCloud#2490 fix(cli): handle EXDEV when backup output crosses filesystems
- TencentCloud#2494 fix(grep_safety): search only target file path, not parent dir siblings (TencentCloud#2352)
- TencentCloud#2509 fix(sso): only treat same-origin opener as Octop SSO popup (TencentCloud#2347)
- TencentCloud#2519 fix(provider-probe): skip disabled models when picking default probe target (TencentCloud#2312)
- TencentCloud#2522 fix(search-probe): make Kimi probe model configurable via env (TencentCloud#1926)
- TencentCloud#2526 fix(knowledge): emit citations only for hits that made it into context (TencentCloud#1490)
- TencentCloud#2527 fix(hitl): isolate pending approvals by thread, not just session (TencentCloud#1897)
- TencentCloud#2539 fix(message_keys): group-chat sessions key by room not by sender (TencentCloud#2067)
- TencentCloud#2542 return 409 PROVIDER_NAME_TAKEN on duplicate provider creation (TencentCloud#1967)
- TencentCloud#2548 fix(skillhub): retry HTTP request on transient CDN IPv4 failure (TencentCloud#1819)
- TencentCloud#2562 fix(cron): preserve interval next_run_time across rebuilds (TencentCloud#2292)
- TencentCloud#2581 fix(gateway): use time.monotonic() for bot scan-poll deadline loop
- TencentCloud#2592 fix(core): make cleanup and datetime parsing resilient
- TencentCloud#2631 fix(users): add ignore_errors=True to checkpoint rmtree
- TencentCloud#2699 fix(media): serialize inbound attachment path allocation and upload
- TencentCloud#2714 fix(gateway): parse creator events from a text-mode subprocess pipe
- TencentCloud#2720 fix(preferences): keep bookmarks with upper-case URL scheme
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants