Skip to content

feat(model-runtime): expose incremental LLM result accumulation - #355

Merged
laipz8200 merged 1 commit into
mainfrom
fix/preserve-model-replay-results
Oct 23, 2026
Merged

laipz8200 merged 1 commit into
mainfrom
fix/preserve-model-replay-results

Conversation

@WH-2099

@WH-2099 WH-2099 commented Oct 23, 2026

Copy link
Copy Markdown
Member

Important

  1. Make sure you have read our contribution guidelines
  2. Search existing issues and pull requests to confirm this change is not a duplicate
  3. Open or identify the issue this pull request resolves or advances
  4. Use a Conventional Commits title for this pull request, and mark breaking changes with !
  5. Remember that the pull request title will become the squash merge commit message
  6. If CLA Assistant prompts you, sign CLA.md in the pull request conversation

Related Issue

Closes #354

Summary

Host agent loops need to retain complete model responses while displaying streamed chunks.
Expose the existing _LLMChunkAccumulator as LLMResultAccumulator so callers can reuse its consume() and to_result() methods for replay and JSON persistence.
Non-stream normalization continues to use the same implementation.

Document one accumulator per invocation and add a public API regression covering mixed content order, an empty content block carrying provider state, exact tool-call arguments, and false-valued assistant state through JSON serialization.
The downstream consumers are in langgenius/dify#44502.

Validation

  • The regression fails on main 4389ea1 because the public class is absent; the existing private implementation satisfies its behavior assertions.
  • PYTEST_ADDOPTS='-n 4' just test: 995 passed on Python 3.12.
  • just check: passed, including lockfile, formatting, lint, types, and import boundaries.
  • uv run --python 3.12 pytest -n 0 tests/test_repository_docs.py: 29 passed.
  • Separate fresh test, knowledge, and final naming reviews passed.

Checklist

  • This pull request links the issue it resolves or advances
  • This pull request title follows Conventional Commits, and any breaking change is marked with !
  • If CLA Assistant prompted me, I signed CLA.md in the pull request conversation

Copilot AI balanced review requested due to automatic review settings October 23, 2026 02:17
@WH-2099 WH-2099 self-assigned this Oct 23, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The narrow API exposure preserves existing behavior, includes focused documentation and regression coverage, and has no unresolved findings.

No open issues 🎉

What changed in this PR

Exposes the existing model-response accumulator as LLMResultAccumulator, allowing hosts to display streamed chunks while retaining a complete response for replay and persistence.

Changes:

  • Publishes the accumulator without changing aggregation behavior or non-stream normalization.
  • Documents incremental usage and one accumulator per invocation.
  • Adds a public API regression covering content order, tool-call arguments, and provider-state preservation through JSON serialization.
File Description
tests/​model_runtime/​test_llm_result_accumulator.py Adds public API and JSON round-trip regression coverage.
src/​graphon/​model_runtime/​README.md Documents incremental accumulation and invocation lifetime.
src/​graphon/​model_runtime/​model_providers/​base/​large_language_model.py Publishes the accumulator and updates its internal use.

Review effort: Balanced

@laipz8200 laipz8200 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@laipz8200
laipz8200 merged commit 9f4a90d into main Oct 23, 2026
9 checks passed
@laipz8200
laipz8200 deleted the fix/preserve-model-replay-results branch October 23, 2026 03:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(model-runtime): expose incremental LLM result accumulation

3 participants