Skip to content

Background API ignores the Context setting from Settings > Models (server is capped at 16K) #55

Closed

Description

@betamuh

Hi, and thanks for TUFF. It's the best-performing local runtime I've tried (I've downloaded hundreds of models).

What I see

  • Settings > Models > Memory (Auto off): Context = 64K, Expert cache = 16 slots (Gemma 4 26B-A4B, Ornith 1.5 9B / 35B-A3B).
  • Using the Background API from a coding agent (OpenCode, openai-compatible, http://127.0.0.1:8080/v1), requests fail with effective prompt exceeds the configured context (context_length_exceeded) when the prompt is only about 16K tokens.
  • In OpenCode's session database, the failing request followed a prompt of ~16.2K tokens, and another response ended with finish: length at exactly 16384 total tokens (prompt + generation). Nothing near 64K was ever reached.
  • Turning Auto on shows the profiles Speed 8K / Balanced 16K / Context 200K. Context 200K makes the server very slow (the hint says it trades weight residency for context), and 64K is not available as a middle option for the server.

What I think is happening (from reading the source, so it may differ from the shipped build)
RoutedServerModelProvider.inferencePlan(for:) uses automaticPlan(for:) (hard-coded .balanced profile) or otherwise grows the model's runtime defaults to 16K/8K. It does not read the per-model memory profile that the app's own chat uses. ServerEngineRuntime then loads the session with installed.inferencePlan(for: descriptor).contextTokens.

Expected
The Background API / tuff serve should honor the context size chosen in Settings (or expose its own setting), as long as it fits the memory budget.

Suggested fix
A small opt-in override, e.g. an environment variable TUFF_SERVER_CONTEXT_TOKENS, used only when the working set still fits device.safeAppMemoryBudgetBytes (otherwise fall back to the computed plan). I wrote an untested patch (could not build it on Linux) that wraps inferencePlan this way; happy to turn it into a PR, or better, wire it to the Settings profile / the server settings file.

Environment
MacBook with M2 Pro, 16 GB RAM, macOS, TUFF 9.4.0), Gemma engine 1.0.20, OpenCode as the client.

Thanks!

Activity

  1. betamuh commented on Oct 17, 2026

    @betamuh
    Author
  2. rexmhall09 commented on Oct 17, 2026

    @rexmhall09
    Collaborator

    Hey, this is fixed in 9.4.1. The Background API and tuff serve now use each model's memory settings from Settings > Models: your Auto profile if Auto is on, or the Context you picked if it's off (as long as it fits in memory, otherwise it falls back to Balanced). /v1/models shows the new limit. I tested it with Ornith 1.5 9B set to 64K and a 31K-token prompt. Thanks for the detailed report and the patch!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions