Show per-job inference stats (tok/s, tokens, time to first token) in the Jobs list

[cguldogan]

Description

The Jobs list says what ran, where and when, but nothing about how it performed. Anyone comparing two nodes, two engines or two quantizations of a model has to reach for a separate benchmark tool to learn the one number they care about: tokens per second.

This makes each finished job carry its own statistics, measured by the proxy that forwarded it, and shows them on the job card:

• Decode throughput in tokens per second, tokens generated, and time to first token. The desktop card gains one line (33.5 tok/s · 60 tokens · 1.8 s to first token); the terminal interface gains a TOK/S column.
• Token counts are the engine's own: the OpenAI usage object (LM Studio, vLLM, SGLang, Ollama's OpenAI route) or Ollama's native eval_count, whose eval_duration is also used as the decode time since it is more precise than a wall clock. A stream that carries no usage report (a client that did not ask for stream_options.include_usage) is counted from its chunks and shown with a ~ prefix, and the wire object says estimated: true.
• Time to first token is measured to the first response body byte, not the headers, because a streaming engine sends its headers before prefill finishes.

The point of the design is that it costs nothing noticeable on the inference path. The proxies already wrap the client-side response writer to enforce the idle write deadline; the new tap adds, per body write, one timestamp, one byte count and a copy into a fixed 4 KiB rolling tail. The body is never buffered and never JSON-decoded. Every engine emits its usage report at the end of the response, so the tail is all that needs inspecting once the stream ends, and it is searched with a byte scan for the last occurrence of the relevant key. The tail is discarded at the terminal event; prompts and generated text are still never retained or logged.

Screenshot of the Jobs list after the change (the top card carries the new line; the cards below it finished before the change and have none):

Related: #42 asks for prompt tokens, completion tokens and tokens per second per request, alongside the node that served it. This delivers those figures on the job record itself (the node was already there as scheduledOn); it does not add the prompt and response logging that issue also asks for, which the workload contract deliberately excludes.

Scope

Included: a shared nvpair-shared/inferstats package (the tap and the extraction) used by both proxies; an additive, omitempty stats object on the workload's terminal workload:completed / workload:errored event; parsing and display in the desktop (WorkloadStats type, Electron bridge, job card) and a column in nvpair-tui; README updates for the workload shape and both proxies; tests at every layer.

Excluded, deliberately: live throughput for a job that is still running (would need periodic upserts), any change to the request the client sent (the proxy does not inject stream_options.include_usage, so a client's stream is byte-for-byte what the engine produced), per-node or per-model aggregates, and token counts for compressed bodies (a client that asked for gzip and an engine that obliged gets timing only). Embeddings show prompt tokens only, since nothing is generated.

Validation

Environment: macOS 15 arm64, Go 1.27, Node 23; one DGX Spark (GB10) node running SGLang as the remote engine under test.

export PATH=/opt/homebrew/bin:$PATH
for m in shared ollama-proxy lmstudio-proxy nvpair-tui; do
  (cd services/$m && gofmt -l . && go vet ./... && go test ./... -count=1)
done
cd desktop && npm run typecheck && npm run lint && npm run test:unit && npm run service-contracts:check && npm run dead-code:check
node scripts/spdx-headers.mjs                                  # 877 checked, 0 missing
git merge-tree --write-tree upstream/main pr/job-stats         # clean

Results: vet clean and gofmt clean in every touched file (shared/splitlisten/splitlisten_test.go is flagged by gofmt on the unmodified base too and is not touched here). shared, lmstudio-proxy and nvpair-tui fully green, including the new inferstats unit tests (OpenAI non-streaming, SSE with and without usage, Ollama NDJSON, a body longer than the tail, generated text that mentions a usage key, compressed body, TTFT) and the proxy-level tests in both proxies (SSE with usage, SSE without usage, Ollama /api/chat, error body not measured). The one Go failure is ollama-proxy TestAliasSelfTargetMatchesBoundLoopbackAddressNotPortAlone, which binds 127.0.0.2 and fails identically on the unmodified base on macOS; unrelated. Desktop: typecheck clean; lint 0 errors (one pre-existing prettier warning in node-info-poller.ts, untouched); 212 unit tests in 38 files pass, 4 of them new; service contracts and dead-code checks pass; the commit is signed off.

End to end, from a macOS desktop node through its OpenAI-compatible proxy to a remote SGLang serving a Qwen3-8B NVFP4 model, one non-streaming chat completion:

• The engine's reply reported "usage":{"prompt_tokens":71,"completion_tokens":60,...}.
• The broker relayed workloads:upsert with "stats":{"promptTokens":71,"completionTokens":60,"tokensPerSecond":33.5,"ttftMs":1793} — counts identical to the engine's, no estimated flag.
• The Jobs card shows 33.5 tok/s · 60 tokens · 1.8 s to first token (screenshot above). workload:started carried no stats, and jobs that finished before the change render exactly as before.

Risk

• Wire compatibility. stats is one additive, optional field on workloadInfo. Every hop already tolerates it: the broker stamps the origin through a generic JSON map, the workload-manager forwards params as raw JSON, the broker store keeps the raw workloadInfo, and the scheduler and TUI decode only the fields they use. An older peer simply drops it; a newer peer receiving from an older one sees no stats and shows no line.
• Accuracy. Counts are the engine's when it reports them. The chunk-count fallback is labelled as an estimate everywhere it is shown, because an engine may batch several tokens per chunk. Throughput uses first-to-last body byte for a stream and the whole request for a non-streaming reply, so a non-streaming figure is a lower bound that includes prefill.
• Mis-parse. The tail is searched for the last "completion_tokens": (or "eval_count":), so generated text that happens to contain the same key cannot shadow the engine's report, which always comes later; a test covers it. Keys are matched with their quotes and colon so completion_tokens_details and prompt_eval_count never match. Any residual mis-parse affects a display figure only.
• Privacy. No new content leaves the proxy: only integers and one float ride the wire, the rolling tail is 4 KiB and dropped at the terminal event, and nothing new is logged.
• Versions: ollama-proxy 0.27.0, lmstudio-proxy 0.17.0 and nvpair-tui 0.8.0 (MINOR, additive IPC), product 0.92.0. No packaging, migration or data changes.

Checklist

• I have read the Contributing Guidelines.
• Every commit is signed off (git commit -s), certifying the Developer Certificate of Origin.
• New or existing tests cover the change.
• Relevant documentation is updated.
• I checked the diff, changed filenames, and commit messages for credentials, private data, internal URLs, internal issue identifiers, and generated artifacts.
• I recorded the validation commands and results above.
• I bumped any affected component in services/versions.json, and described user-visible changes above so they reach the release notes.

🤖 Generated with Claude Code

[NV-sschneider]

Thanks @cguldogan! This is an awesome PR. I'd love to get something like this in PAIR to look at inference stats. @ckelseynv, check this one out.