Repository navigation
[Eval] Qualify served OneJev 1.6 (default --model onejev) on browser fronts and pin Decision Index 0.3 evidence #114
Description
Motivation
#88 / PR #101 landed the served OneJev HTTP client (--model onejev, ONEJEV_URL, TypeSafe /v1/systemone + media). docs/capabilities.md already defaults the request id to OmniJev/OneJev1.6-4B, but the matrix still says live GPU / GGUF hardware validation is not recorded in this checkout.
Upstream OneJev 1.6 is a real step past the 1.5 numbers that #88 cited:
- Model card: OmniJev/OneJev1.6-4B (Apache-2.0; full fine-tune of Qwen3.5-4B;
flide serve --model OmniJev/OneJev1.6-4B). - Published public Decision Index 0.3 runs: OmniJev/onejev-decision-index-results (kit apolinario/decision-index
f5a6e4f): 1.6-4B public index 61.95 (vs 1.5-4B 54.89), 1.6-9B 65.30, 1.6-27B 69.55; every one of 140,178 scoreable requests answered. - Overview / serve path: OneJev site, OmniJev/OneJev.
This is not #103 (Open-Jev 9B/27B text through the generic jev HTTP adapter), not in-process omnijev, and not a new adapter — it is the missing qualification evidence for the path #88 already shipped.
Proposed work
- Pin exact Hugging Face revisions for at least
OmniJev/OneJev1.6-4B(and optionally 0.8B / 9B if hardware allows); refuse to claim a size that did not load. Record the revision the server reports, not only the env string. - Serve with
flide(or the documentedqevpath) and point--model onejevat it. Run WebVoyager Allrecipes task 0 and the deterministic local search / Flights shaping path already used for other multimodal backends; count successes, failures, timeouts andBLOCKED, with decision p50/p95 and hardware. - Optionally reproduce (or cite with pinned SHAs) the public Decision Index 0.3 4B row from the OmniJev results dataset so the capabilities matrix can separate "adapter works" from "board-reported quality". Do not redistribute suite inputs.
- Update
docs/capabilities.md/ adocs/results/onejev-1.6/note: wire-compatible vs qualified, Agents SHA, server commit, checkpoint revision, device.
Acceptance criteria
- Every claimed run names Agents SHA, OneJev/flide (or qev) revision, checkpoint revision, hardware and
S1A_DECISION_TIMEOUT_S. - At least one browser task comparison sits next to hosted Jev / OmniJev on the same budget, with full denominators.
- Capabilities matrix marks OneJev 1.6 as validated (or explicitly failed) for the recorded combination; no quality claim without the artifact.
- Distinct from [Eval] Qualify Open-Jev 9B/27B on the text reference path and one browser suite #103 / closed [Feat] Add a served OneJev multimodal decision backend (System One + media) #88; links [Tracking] Roadmap: applications, benchmarks, model backends and System1-Omni #54 for the matrix tracker.
Activity