[isHeSatoshi]

Fixes #1198.

What was wrong

ModelResponse.truncated_tokens_count is documented as "How many tokens truncated" (src/lighteval/models/model_output.py), but on the transformers backend every generated sample reported a wrong number. A prompt that fits entirely in the model's context window still reported a non-zero truncation count, and a genuinely truncated prompt reported how many characters were dropped instead of how many tokens.

That is what #1198 is about: a 5-shot GSM8K run saved "padded_tokens_count": 178, "truncated_tokens_count": 2047 in its details file even though the prompt fits in the context window. 2047 is len(prompt) - 178, i.e. the character count of the prompt minus its token count.

Root cause

TransformersModel._padded_greedy_until filled Batch.truncated with

truncated=[max(len(c) - tokenized["input_ids"].shape[1], 0) for c in contexts],

contexts is a list of prompt strings, so len(c) counts characters, while tokenized["input_ids"].shape[1] is the padded token length. The greedy-until path lets the tokenizer truncate (truncation="longest_first"), so the untruncated token count is not recoverable from tokenized afterwards.

_loglikelihood_tokens (_build_model_inputs) computes the same field in tokens (max(len(orig_tokens) - effective_max_length, 0)), which is why this only shows up in generative evaluations.

What changed

• src/lighteval/models/transformers/transformers_model.py: _padded_greedy_until now measures the untruncated token count of each context before building the Batch, and Batch.truncated is context_token_count - padded_token_length. The extra tokenization passes verbose=False so it does not add a spurious "Token indices sequence length is longer than the specified maximum" warning. Nothing else changes: padded_tokens_count and input_tokens are untouched.
• tests/unit/models/test_transformers_model.py: two regression tests in a new TestTransformersModelTruncatedTokensCount class.

Verification

Environment: Windows 11, Python 3.11.9, uv venv + uv pip install -e ".[dev]", CPU torch, model gpt2 (context length 1024). Generation is stubbed out so Batch.truncated can be read directly; expected is max(context_token_count - 1024, 0).

On main:

gpt2 model max_length = 1024
WRONG | short prompt (fits comfortably)     | chars=    24 tokens=     5 | expected=    0 reported=   19
WRONG | medium prompt (fits, ~600 tokens)   | chars=  2400 tokens=   401 | expected=    0 reported= 1999
WRONG | over-long prompt (~1400 tokens)     | chars=  8400 tokens=  1401 | expected=  377 reported= 7376

With this change:

gpt2 model max_length = 1024
OK    | short prompt (fits comfortably)     | chars=    24 tokens=     5 | expected=    0 reported=    0
OK    | medium prompt (fits, ~600 tokens)   | chars=  2400 tokens=   401 | expected=    0 reported=    0
OK    | over-long prompt (~1400 tokens)     | chars=  8400 tokens=  1401 | expected=  377 reported=  377

The two new tests fail on main and pass here:

python -m pytest tests/unit/models/test_transformers_model.py::TestTransformersModelTruncatedTokensCount -q
# main:       2 failed in 128.93s   (AssertionError: Lists differ: [7376] != [377])
# this branch: 2 passed in 106.33s

python -m pytest tests/unit/models/test_transformers_model.py -q
# this branch: 17 passed, 16 subtests passed in 334.51s

ruff check / ruff format --check on both changed files
# All checks passed! / 2 files already formatted

Uncertainty a reviewer might raise

• _padded_greedy_until now tokenizes each batch twice: once to count the untruncated tokens and once (as before) to build the padded tensors. Tokenization is not the bottleneck of this path, but the count could instead be derived from a single untruncated tokenization plus a manual truncate/pad step.
• truncated_tokens_count is only consumed for the saved per-sample details, so this affects the reported metadata rather than the scores. SummaryTask.truncated / non_truncated in src/lighteval/logging/info_loggers.py are declared but never incremented by anything; I left those alone because that is a separate change.

---

AI assistance (disclosure): this change was prepared with an AI coding agent (Factory Droid) working in my checkout. Every result quoted above comes from actually running the commands shown, not from the agent's description. Happy to answer questions or rework anything.