v1.0.0

Issues

• v1.0.0-rc.2: canonicalize_file rejects ByteLevel add_prefix_space=true supported by 0.23.2

  huggingface/tokenizers#2460 · Filter by author michaelfeil opened on Sep 26, 2026 · v1.0.0

• feat(wasm): publish pipeline

  Unread
  huggingface/tokenizers#2497·SBrandeis opened Oct 8, 2026 · Updated Oct 8, 2026
  1

• Support ByteLevel add_prefix_space during canonicalization

  Unread
  huggingface/tokenizers#2462·ducanhnguyen223 opened Sep 27, 2026 · Updated Oct 8, 2026
  4

• feat(python): Support encoding pairs of text

  Unread
  huggingface/tokenizers#2453·SBrandeis opened Sep 24, 2026 · Updated Oct 8, 2026
  1

• [1.0.0rc2] Possible regressions vs. 0.23.x: Byte atom not found in the vocabulary

  huggingface/tokenizers#2447 · Filter by author xenova opened on Sep 23, 2026 · v1.0.0
  4

  

• feat(python): role_to_token

  Unread
  huggingface/tokenizers#2454·SBrandeis opened Sep 24, 2026 · Updated Oct 8, 2026
  1

• feat(python): composition

  Unread
  huggingface/tokenizers#2414·SBrandeis opened Sep 16, 2026 · Updated Oct 8, 2026·Draft

• feat: experimental C bindings

  Unread
  huggingface/tokenizers#2375·SBrandeis opened Sep 1, 2026 · Updated Oct 8, 2026
  5

• Atomnorm

  target
  Unread
  huggingface/tokenizers#2209·ArthurZucker opened Jul 16, 2026 · Updated Oct 8, 2026
  1

• Draft, manual implem of bitsplit :)

  Unread
  huggingface/tokenizers#2340·ArthurZucker opened Aug 18, 2026 · Updated Oct 8, 2026·Draft
  1

• Don't emit control tokens in untrusted text segments

  huggingface/tokenizers#2498 · Filter by author SBrandeis opened on Oct 8, 2026 · v1.0.0
  2