All pull requests

Pull requests

• Restore TriviaQA prediction normalization

  #1446·mottopanikeiku opened Oct 22, 2026·

  ·

• fix: exclude compromised LiteLLM 1.107.0

  #1445·jagwar opened Oct 22, 2026·

  ·
  1

• Use actual gold labels for TruthfulQA MC2

  #1444·salma2vec opened Oct 20, 2026·

  ·
  1

• Count repeated tokens in F1 scores

  #1443·salma2vec opened Oct 20, 2026·

  ·
  1

• Keep explicit model parallel on one multi-GPU machine

  #1439·chetana1410 opened Oct 20, 2026·

  ·
  1

• Fix invalid type annotations (callable, np.array) in the package

  #1438·Michael-Duck opened Oct 20, 2026·

  ·
  1

• docs: point task guides at paths that exist

  #1436·xlb3055 opened Oct 20, 2026·

  ·

• Report truncated_tokens_count in tokens on the transformers backend

  #1435·isHeSatoshi opened Oct 19, 2026·

  ·
  1

• Fix sample cache path for local model directories

  #1434·Tialo opened Oct 18, 2026·

  ·

• fix: correct bootstrap standard errors and retain all resamples

  #1433·ningyixi opened Oct 18, 2026·

  ·

• [RFC] Add offline paired comparison of evaluation runs

  #1432·surgan12 opened Oct 16, 2026·

  ·

• Fix mathematical equivalence in the Inspect math scorer

  #1431·surgan12 opened Oct 16, 2026·

  ·
  1

• Fix multiclass F1 averaging for SuperGLUE CB

  #1429·March-7 opened Oct 15, 2026·

  ·

• Fix DummyModel rolling likelihood for choice-free documents

  #1428·AaravMehta-07 opened Oct 13, 2026·

  ·
  1

• Fix JudgeLLM crash with huggingface_hub 2.x for transformers/vllm judges

  #1427·li12345678990 opened Oct 12, 2026·

  ·
  2

• Use rouge-score-rs for ROUGE when it is installed

  #1426·kholdrex opened Oct 11, 2026·

  ·
  1

• Add KMMLU (Korean-native MMLU, 45 subjects)

  #1425·CY-HYUN opened Oct 10, 2026·

  ·
  1

• Add the TauBench retail benchmark with a tool-calling agent and an LLM user simulator

  #1423·zhongerdeajie opened Oct 10, 2026·

  ·
  1

• Fix Metrics.drop corpus aggregation to use mean instead of max

  #1422·salignatmoandal opened Oct 9, 2026·2 of 3 tasks·

  ·
  1

• Fix broken code examples in custom model docs

  #1421·singularity-14 opened Oct 9, 2026·

  ·

• Reject a missing details file when reloading an eval

  #1420·b423016 opened Oct 8, 2026·3 tasks done·

  ·

• Show scored sample counts in the results table

  #1419·b423016 opened Oct 8, 2026·2 tasks done·

  ·
  1

• fix: show a single vLLM data-parallel progress bar on the driver

  #1418·Steeve-Crypto opened Oct 7, 2026·2 tasks done·

  ·
  1

• fix(metrics): ignore boundary whitespace in word-unit counts

  #1416·March-7 opened Oct 4, 2026·

  ·

• Don't change global torch matmul precision when importing swiss_legal metrics

  #1415·jayzuccarelli opened Oct 2, 2026·

  ·
  1

  3