Reproducible comparisons of TypeSafe typed classifications and SOL structured classifications. The benchmark covers binary, multiclass, ordinal, ambiguous, short-context, long-context, and adversarial cases. A separate DiffusionGemma suite tests temporal, policy, relational, constraint, evidence, causal, and counterfactual reasoning with typed answers.
The benchmark requires these environment variables:
TYPESAFE_API_KEYTYPESAFE_API_URLTYPESAFE_MODELINFERENCE_API_KEYorNVIDIA_API_KEYSOL_API_URLSOL_MODELDIFFUSIONGEMMA_URLfor the DiffusionGemma comparison
Run:
python3 compare_typesafe_sol.py
python3 compare_jev_small_router.py
python3 compare_typesafe_needle.py
python3 compare_typesafe_diffusiongemma.py
python3 compare_typesafe_diffusiongemma_reasoning.pySet COMPACT_RESULTS=1 to print aggregate and adversarial summaries without the
full per-case rows.
Set RESULT_PATH when running the DiffusionGemma comparison to checkpoint each
completed case and resume an interrupted run.
The DiffusionGemma container guide includes the GHCR image, launch command, API example, and build recipe for the decision service and explorer.
The L40S FP8 deployment guide covers pulling the container, launching the service, and using the UI/API.
Install the reusable structured-decision skill with:
npx skills add pst2154/Typesafe_Testing --skill diffusiongemma-structuredThe skill documents Choice, Noul, and Score design and includes a standard-library
client that translates keyed TypeSafe-style questions to the DiffusionGemma wire
format. Set DIFFUSIONGEMMA_URL at runtime; no service location or credential is
embedded in the skill.
Routing comparisons:
- Jev vs. Llama 3.2 1B
- Jev vs. Qwen3.8-27B
- Detailed Jev vs. SOL and Qwen benchmark report
- TypeSafe vs. Needle 3 classification benchmark
- TypeSafe vs. DiffusionGemma typed-decision and bounded-reasoning benchmark
- NanoJev vs. DiffusionGemma matched decision accuracy
The repository contains no credentials, endpoint values, internal paths, or raw production data.