Skip to content

[Fusion] Fix previous schemas not being garbage collected - #10654

Merged
michaelstaib merged 10 commits into
ChilliCream:mainfrom
mnebes:fix-executor-swap-memory-leak
Oct 21, 2026
Merged

michaelstaib merged 10 commits into
ChilliCream:mainfrom
mnebes:fix-executor-swap-memory-leak

Conversation

@mnebes

@mnebes mnebes commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

We noticed that our gateway instances accumulated gen2 memory significantly over time:

memory-gen2

(yellow line is gen2)


We also noticed that the memory grows whenever we update the composed schema, which lead us to believe that somehow the caches are not released whenever a switchover happens. Our hypothesis is that selections left in pooled object[] tails can reference their declaring operation and schema. A replacement schema's cached operation can therefore retain an old schema, its disposed service provider, and that provider's cached operations. This can be then chained ever deeper. A subsequent memory dump analysis seems to support this hypothesis:

dotMemory

This PR is a proposed fix that:

  • clears rented operation-element arrays before use so unused capacity cannot retain selections from another operation/schema.
  • clears private compiler arrays when returning them to the shared pool.

The replaced published arrays are therefore left to GC to clean up; unpublished intermediate arrays remain pooled.

As always, let me know if you need anything from me 🙂

Copilot AI balanced review requested due to automatic review settings October 21, 2026 11:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused ownership and clearing changes address schema retention without invalidating published storage, with regression coverage and no identified blocking issues.

No open issues 🎉

What changed in this PR

Fixes Fusion retaining replaced schemas through pooled operation storage.

Changes:

  • Clears rented arrays and private arrays returned to the pool.
  • Keeps published arrays out of the pool while readers may still reference them.
  • Adds regression tests for storage reuse, growth, and schema collection.
File Description
src/​HotChocolate/​Fusion/​test/​Fusion.Execution.Tests/​Execution/​OperationCompilerTests.cs Tests array clearing, published storage preservation, and schema release.
src/​HotChocolate/​Fusion/​src/​Fusion.Execution/​Execution/​Nodes/​OperationCompiler.cs Clears pooled storage and tracks whether arrays can safely be returned.

Review effort: Balanced

@tobias-tengler
tobias-tengler force-pushed the fix-executor-swap-memory-leak branch from 7275361 to 5eb705c Compare October 21, 2026 12:11
@tobias-tengler

Copy link
Copy Markdown
Member

Thanks @mnebes :)
I made some changes and removed the pooling entirely. If Michael signs off on it, we can get this in!

@tobias-tengler
tobias-tengler force-pushed the fix-executor-swap-memory-leak branch from 4c661f5 to 926ab92 Compare October 21, 2026 12:24
@mnebes

mnebes commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @mnebes :) I made some changes and removed the pooling entirely. If Michael signs off on it, we can get this in!

Yeah, that works too :)

@tobias-tengler tobias-tengler changed the title [Fusion] Fix retaining replaced schemas through pooled storage [Fusion] Fix previous schemas not being garbage collected Oct 21, 2026
michaelstaib and others added 5 commits October 22, 2026 07:23
The composite selector returned its rented Selection buffer without clearing it, so the shared pool kept selections, and with them the previous schema, alive after an executor swap.
A lazy compile that failed after allocating ids left the operation's last id ahead of its storage, so the next lazy compile threw an IndexOutOfRangeException. The tests now also cover the previous storage staying intact when it grows, dispose the Fusion schemas, and wait up to 30 seconds for the previous schema to be collected.
@michaelstaib
michaelstaib merged commit a75cf5f into ChilliCream:main Oct 21, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants