You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Background API ignores the Context setting from Settings > Models (server is capped at 16K) #55
Using the Background API from a coding agent (OpenCode, openai-compatible, http://127.0.0.1:8080/v1), requests fail with effective prompt exceeds the configured context (context_length_exceeded) when the prompt is only about 16K tokens.
In OpenCode's session database, the failing request followed a prompt of ~16.2K tokens, and another response ended with finish: length at exactly 16384 total tokens (prompt + generation). Nothing near 64K was ever reached.
Turning Auto on shows the profiles Speed 8K / Balanced 16K / Context 200K. Context 200K makes the server very slow (the hint says it trades weight residency for context), and 64K is not available as a middle option for the server.
What I think is happening (from reading the source, so it may differ from the shipped build) RoutedServerModelProvider.inferencePlan(for:) uses automaticPlan(for:) (hard-coded .balanced profile) or otherwise grows the model's runtime defaults to 16K/8K. It does not read the per-model memory profile that the app's own chat uses. ServerEngineRuntime then loads the session with installed.inferencePlan(for: descriptor).contextTokens.
Expected
The Background API / tuff serve should honor the context size chosen in Settings (or expose its own setting), as long as it fits the memory budget.
Suggested fix
A small opt-in override, e.g. an environment variable TUFF_SERVER_CONTEXT_TOKENS, used only when the working set still fits device.safeAppMemoryBudgetBytes (otherwise fall back to the computed plan). I wrote an untested patch (could not build it on Linux) that wraps inferencePlan this way; happy to turn it into a PR, or better, wire it to the Settings profile / the server settings file.
Environment
MacBook with M2 Pro, 16 GB RAM, macOS, TUFF 9.4.0), Gemma engine 1.0.20, OpenCode as the client.
Hey, this is fixed in 9.4.1. The Background API and tuff serve now use each model's memory settings from Settings > Models: your Auto profile if Auto is on, or the Context you picked if it's off (as long as it fits in memory, otherwise it falls back to Balanced). /v1/models shows the new limit. I tested it with Ornith 1.5 9B set to 64K and a 31K-token prompt. Thanks for the detailed report and the patch!
Activity
betamuh commentedon Oct 17, 2026
tuff-server-context.patch
Keep the loaded plan after answers; serve each model's memory setting…
rexmhall09 commentedon Oct 17, 2026
Hey, this is fixed in 9.4.1. The Background API and
tuff servenow use each model's memory settings from Settings > Models: your Auto profile if Auto is on, or the Context you picked if it's off (as long as it fits in memory, otherwise it falls back to Balanced)./v1/modelsshows the new limit. I tested it with Ornith 1.5 9B set to 64K and a 31K-token prompt. Thanks for the detailed report and the patch!