Full-message tokenization for droppedTokens telemetry blocks OpenCode for 46–77 seconds

[null-axiom]

Summary

Magic Context synchronously serializes and tokenizes the entire transformed message array twice to calculate droppedTokens telemetry. In a running OpenCode Desktop session, direct timing around these two calls measured approximately 77 seconds of tokenization in one transform and 46 seconds in another. The sidecar event-loop diagnostics recorded matching stalls.

This path remains present in the published 0.42.3 npm bundle. The runtime measurements below are from 0.42.0, with the previously reported SQL-reader/marker corrections applied locally and a temporary timing probe. This is separate from those SQL issues.

Expected behavior

Calculating a diagnostic metric should not synchronously block the shared OpenCode backend for tens of seconds. Exact token-budget enforcement should remain separate from inexpensive telemetry estimation.

Direct measurements

Captured September 14, 2026. Times below are EDT (UTC−4); each timestamp is the end of the measured call.

Time	Call	Message count	Serialized characters	JSON serialization	Tokenization
12:51:18.834	Before reductions	69	10,874,531	17.81 ms	33,945.69 ms
12:52:02.258	After reductions	62	10,773,432	17.68 ms	43,112.15 ms
12:53:08.829	Before reductions	162	5,500,050	9.86 ms	22,815.35 ms
12:53:31.837	After reductions	140	5,138,547	13.61 ms	22,864.78 ms

The first pair spent 77,057.84 ms inside the tokenizer, corresponding to a 77,443.63 ms sidecar event-loop stall reported at 12:52:02.268. The second pair spent 45,680.13 ms inside the tokenizer, corresponding to a 45,868.91 ms stall reported at 12:53:31.841. Small intervening cleanup operations account for much of the remaining interval.

The probe calls JSON.stringify once and the original estimateTokens once, measuring each with performance.now(). It preserves the original result and exceptions. It logs only slow calls (at least 250 ms), at most 20 records, with durations/counts only and no message content. These measurements localize the time to the synchronous estimateTokens call, including any GC occurring inside that interval; they do not distinguish individual tokenizer internals.

Verified published code

Package: @cortexkit/opencode-magic-context@0.42.3, downloaded from npm on September 14 without installing or executing it.

dist/index.js:33755–33756:

let droppedTokens = 0;
const tokensBeforeReductions = isCacheBustingPass || shouldApplyPendingOps
  ? estimateTokens(JSON.stringify(args.messages))
  : 0;

dist/index.js:33967–33968:

if (isCacheBustingPass) {
  droppedTokens = Math.max(0,
    tokensBeforeReductions - estimateTokens(JSON.stringify(args.messages)));
}

dist/index-wde5446v.js:6319–6332 still implements estimateTokens using:

activeTokenizer.encode(text, "all").length

The existing cheap fallback is estimateTokensHeuristically(text), which returns Math.ceil(text.length / 3.5). It is used only when the tokenizer is unavailable or fails, not to bound telemetry work.

The before/after delta is returned as droppedTokens, passed to recordPendingTransformDecision, and persisted as transform_decisions.dropped_tokens. These particular counts do not drive context-budget or scheduling decisions. Other similarly named values, such as undroppedTokens, are distinct and should not be changed as part of this fix.

Proposed narrow correction

Use the existing character-based estimate for both telemetry-only before/after counts, rather than invoking the exact tokenizer on the full serialized message array. Preserve the nonnegative delta calculation and leave estimateTokens and its real token-budget callers unchanged.

This deliberately changes the accuracy of dropped_tokens telemetry; it is not an exact-count-preserving optimization. Document that field as estimated. Serialization can remain for this narrow correction: it measured only 10–18 ms here, versus 23–43 seconds per tokenization call.

Suggested regression coverage:

• Exercise the real transform path with a multi-megabyte message array and pending operations/cache-busting enabled.
• Assert that these telemetry sites do not invoke the exact tokenizer.
• Verify that the emitted delta is nonnegative and uses the inexpensive estimate consistently before and after reductions.
• Verify unchanged transformed messages, applied operations, and actual token-budget decisions.
• Measure event-loop responsiveness outside the transform as well as elapsed telemetry work.

[magic-alfonso]

Confirmed at the two sites you cited and fixed on master (1d6ff3d). Rather than swapping in the character heuristic, dropped_tokens is now the sum of the persisted per-tag token counts for the tags actually reduced during the pass (Magic Context already computes those once at tag insert), so the number stays close to exact and the whole message array is never serialized or tokenized for it. A skeletonized tool call counts its removed output; a full removal counts output, input and reasoning; a partial text rewrite counts only its removed characters at the tokenizer's own fallback rate. The column is documented as an estimate and drives no budget or scheduling decision, as you noted.

Regressions: a tokenizer-seam spy on an 11.7 MB array asserts no whole-array input reaches the exact tokenizer, a fixture with three dropped tags pins the exact sum, the served array is sha256-identical on the cache-busting pass and the following defer pass, and the longest synchronous stall across a 10 MB pending-operation pass is bounded relative to the pass. Ships in v0.42.4. Thanks for the direct measurements and for scoping the fix to the telemetry path.