5080deploy 3/9: merge upstream #391, OpenAI developer role

[gdevenyi]

Merges upstream FlashML-org#391 (fix(tokenizer): accept the OpenAI developer role, head 965e1780; closed by its author on 2026-09-14). Stacked on #10.

Changes

• Clean merge of fix(tokenizer): accept the OpenAI developer role (map to system for templates without it) FlashML-org/FreeToken#391; no local changes.
• TokenizeManager._render maps a developer message to system before apply_chat_template, unless the chat template itself mentions developer.

Testing

This merge and the next one in the stack (FlashML-org#475, both small tokenizer/tokenize.py changes) were tested together at 805e347, not separately.

Unit tests: pytest tests/tokenizer tests/server -m "not slow" at 805e347: 852 passed, 4 skipped.

Live (reasoning_effort: low, "Answer tersely." + "2+2?"):

• Before (on main e0886cc): the developer role returns HTTP 400 could not encode request: Unexpected message role.
• After: developer and system both give prompt_tokens 49 and the answer "4". Checked again on the final stack (2d56708): the developer role answers normally.

Server: same command as the previous PR in the stack. Startup to ready 60.2 s.

---

Environment:

• RTX 5080 16 GB (sm_120), AMD Ryzen 9 9950X (16 cores), 123 GB RAM
• Ubuntu 24.04.4, kernel 7.1.9-x64v3-xanmod1
• Driver 610.57.04, CUDA 13.3, torch 2.11.0+cu130, triton 3.6.0, FreeToken 0.1.2 (editable install)
• Checkpoint: nvidia/Qwen3.8-Flash-Next-NVFP4

Stack: the 5080deploy branch is main e0886cc plus upstream PRs merged one at a time for a single-user, full-context (262,144-token) deployment on this card. Each PR in the stack is based on the previous one, so its diff is exactly one merge step. Merge them in order, or merge the top of the stack alone.

How tests were run: with the server stopped, since a running server holds almost all VRAM and GPU tests fail spuriously. Benchmarks are streamed chat completions with 256 tokens generated with ignore_eos, one request at a time. Prompts are salted so the radix cache cannot hit.

• Prefill = prompt tokens / TTFT.
• Decode = (completion - 1) / time after the first token.
• "94K recall": a 94,526-token prompt of repository source with a codeword planted in the first line.
• "248K recall": a 248,029-token prompt with facts planted at token ~0, ~124K and ~240K.

---

Rebased 2026-09-17 onto upstream main cac247a (v0.1.3). The whole stack was rebuilt merge by merge on the new base (previous base e0886cc); the merge resolutions were replayed unchanged via git rerere, and the rebuilt tip differs from the old one by exactly the e0886cc..cac247a file set. This PR's head is now 925a217.

Rebased 2026-09-19 onto upstream main cc1f5c2 (4 commits past cac247a: FlashML-org#471 greedy sampling in mixed batches, FlashML-org#518 WeightLoadError, FlashML-org#521 tvm-ffi jit arch, FlashML-org#524 install index). Same replay as before via git rerere; the replayed stack differs from the previous tip 8adde91 by exactly the cac247a..cc1f5c2 file set. This PR's head is now d4a2701.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Bu6LgoxLR4wETqb7RPR2vt

[Copilot]

🟡 Changes recommended

Template capability detection may mishandle dictionary templates and incidental developer text.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds OpenAI developer role support by mapping unsupported roles to system before chat-template rendering.

Changes:

• Adds developer-role normalization.
• Preserves input immutability.
• Adds tokenizer regression coverage.

File summaries

File	Summary
tests/tokenizer/test_tokenize.py	Tests role mapping and input preservation.
python/freetoken/tokenizer/tokenize.py	Implements developer-role normalization before rendering.

Review details

Suppressed comments (1)

python/freetoken/tokenizer/tokenize.py:60

• The bare substring check does not prove that the template handles this role: a Jinja comment or error/help string such as developer is unsupported also makes this branch preserve the role, after which apply_chat_template still raises Unexpected message role. Detect an actual role handler or use a more reliable capability check instead of any occurrence in the template source.

    if "developer" in (chat_template or "") or not any(
        m.get("role") == "developer" for m in messages
    ):

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.