Skip to content

research(jev-task-fit): assess fast-role/model-choice extension without promoting unproved decisions #517

Closed
Closed
research(jev-task-fit): assess fast-role/model-choice extension without promoting unproved decisions#517

Description

@tommy-ca
tommy-ca
opened on Oct 17, 2026

SDK/agent comparison and role-fit boundary update — 2026-10-17

Native JEV is now supported in Vercel AI SDK 7 TS (experimental_decide, ai >=7.0.128), and Vercel AI SDK for Python (ai.ops.experimental.evaluate via AI Gateway). PydanticAI documents a native OpenAIDecisionsModel for OpenAI's Decisions API, NOT proof of free OpenCode / TypeSafe System One compatibility. The existing PStack decision adapter and resolver are Python; adding Bun/Node runtime or Python generative-agent framework just for decision I/O is avoidable. Choose ONE typesafe-sdk Python transport as B in the code/LOC/dependency trade study, subject to actual provider fixtures. Alternative C AI SDK Python if another host need already adopts AI Gateway; D PydanticAI only if used as the execution-agent stack independently (not a prerequisite). All model-choice / fast-role / leaf-skill routing hypotheses remain shadow research and must bind to host #429 available models, explicit poteto-mode triggers, QStack thin overlay, and quant risk exclusions. Compare A deterministic baseline vs B TypeSafe classifier vs C conditional SDK alternative on real task success, latency and cost; no mandatory new generic framework. Draft PR #522.

Why new scope

Completed PStack Jev program #193 phase 1 explicitly excluded model-tier selection and had no grounded skill-suggestion taxonomy. references/decision-routing.md says skill suggestions are not exposed; references/decision-routing.json only permits playbook semantic candidates. User now wants best-fit fast worker roles/models and leaf skill suggestions for QStack. Do not treat completed phase 1 as proof of new authority.

Research-only work

  1. Reuse model-role contract [Program] Harness-neutral pstack role/model contracts and pstack-claude parity #429 and real host /setup-pstack model availability/effort mapping. Candidate model identifiers must be from an actual host catalog; no hard-coded or invented slugs. Require model-family independence for arena/interrogate.
  2. Compare (A) existing role/model policy (baseline), (B) purely deterministic task features and budget routing, (C) opt-in Jev closed-set suggestion for low-risk fast workers. Keep architecture and multi-agent verification on configured high-capability independent models regardless of Jev.
  3. Characterize task properties (complexity, code vs docs, context, tool support, latency/cost, privacy, reasoning, uncertainty). Benchmark route accuracy, task quality and total elapsed time/cost including Jev overhead. Use holdouts incl. QStack-specific replay/OMS/research tasks from https://github.com/tommy-ca/qstack/issues/1791.
  4. Before skills hints are even shadow-exposed, ground eligible skill ids and host-owned discovery metadata; do not expand the existing route registry with a QStack duplicate taxonomy.

Acceptance

New provider availability evidence — 2026-10-17

For any role-fit evaluation, use host-resolved available worker models, not the Jev decision model as a coding worker. Source-confirmed PStack Jev provider candidates include OpenCode Console/Zen jev-1.13-free (limited-time, no Go subscription needed) and paid jev-1.13, plus OpenRouter paid typesafe/jev-1.14 and typesafe/jev-1.13. OpenCode Go subscriptions are for coding worker models and its current official list has no Jev; keep separate model-role vs decision-provider capabilities. Integration implementation owned by #519/#520, provider selection & cross-provider acceptance by #521. Compare transport- and version-specific task-fit on the same labeled fixture before asserting superiority.

Activity

  1. tommy-ca commented on Oct 20, 2026

    @tommy-ca
    tommy-ca
    on Oct 20, 2026
    OwnerAuthor

    JEV Boundary & Resilience Update (Post-Epic #657)

    In accordance with strict boundary containment discipline (keeping JEV integration lean and clean within pstack with zero bloat to qstack):

    1. Defect Leaves Resolved & Shipped:

    2. Containment Discipline:

      • Zero duplicated routers, catalogs, or registries introduced into qstack.
      • Clear architectural separation between worker model role policy (host-catalog bound) and decision provider transport (confined to pstack).
      • Rollback equivalence, blinded candidate evaluation, and 6-host harness doctor checks verified green.

  2. tommy-ca commented on Oct 20, 2026

    @tommy-ca
    tommy-ca
    on Oct 20, 2026
    OwnerAuthor

    Thermo-Nuclear Code Quality Review Summary

    Audit Scope: JEV Resilience Implementation (dea54d5..804df41, PRs #662, #665, #666, #667) & Issue #517
    Base Revision: dea54d5cd723e06aa6a3afb78f9c3a79e82abb32
    Head Revision: 804df416c1ec0ce9f6888fbf02a43fe9714c776a
    Verdict: CONDITIONAL PASS WITH REQUIRED REMEDIATIONS
    Audit Artifacts: docs/audit/thermo-nuclear-517/ (summary.md + detail reports 01_ through 05_)

    Audit Findings & Code-Judo Remediation Plan

    1. Route Resolver Procedural Sprawl (Rule 0 & Rule 2):

      • resolve_skill_order_with_shadow in scripts/route_resolver.py has grown into a 180-line procedural conditional ladder across 13 inline checks with 14 return statements.
      • Remediation: Decouple into three cohesive stages (_is_egress_bypassed, _load_semantic_candidates, _evaluate_advisory_promotion), cutting cognitive complexity and dropping function length to ~35 lines while preserving 100% of behavior.
      • Report: docs/audit/thermo-nuclear-517/01_route_resolver_pipeline_and_bypass_chain.md.

    2. Decision Adapter Validation & Transport Boilerplate (Rule 4 & Rule 6):

      • scripts/decision_adapter.py repeats finite unit-interval validation blocks across confidence and probabilities, and constructs ProviderObservation(status="invalid") verbatim 8 times.
      • Remediation: Extract pure _parse_unit_interval(val) and _invalid_obs factory helper, cutting 35 lines of duplicate boilerplate.
      • Report: docs/audit/thermo-nuclear-517/02_decision_adapter_validation_and_transport.md.

    3. Receipt Lifecycle Cohesion & Serialization Redundancy (Rule 3 & Rule 5):

      • DecisionReceipt, CalibratedEvidenceReceipt, and CheckEvidenceReceipt in scripts/evidence_receipts.py duplicate to_dict and to_json, while deserialization is fragmented between manual 13-field parsing and fragile **kwargs unpacking.
      • Remediation: Unify under a ReceiptBase mixin with standardized to_dict, to_json, and schema-safe from_dict/load class methods.
      • Report: docs/audit/thermo-nuclear-517/03_evidence_receipts_and_lifecycle_cohesion.md.

    4. 1,000-Line File Threshold Near-Breach in Model Role Tests (Rule 1):

      • tests/test_model_role_policy.py has reached 995 lines (5 lines from the 1,000-line ceiling). Adding any new test for Issue research(jev-task-fit): assess fast-role/model-choice extension without promoting unproved decisions #517 would breach this strict limit.
      • Remediation: Decompose into three cohesive test suites: test_model_role_schema.py (~250 lines), test_model_role_constraints.py (~380 lines), and test_model_role_profiles.py (~415 lines).
      • Report: docs/audit/thermo-nuclear-517/04_model_role_policy_test_decomposition.md.

    5. Strict PStack/QStack Architecture Boundary & Role-Fit Discipline (Rule 6):

      • Zero duplicated routers, catalogs, or registries in qstack. All decision transport, receipt validation, and host catalog bindings remain strictly in pstack.
      • Candidate worker models must be sourced exclusively from host catalogs via /setup-pstack (Contract [Program] Harness-neutral pstack role/model contracts and pstack-claude parity #429).
      • High-capability roles (architect, independent-verifier, arena-cross-judge-pool) remain strictly pinned to configured frontier models; fast-worker role-fit is restricted exclusively to low-risk execution roles.
      • Report: docs/audit/thermo-nuclear-517/05_qstack_boundary_and_task_fit_architecture.md.

    Phased Remediation Work Breakdown

    • Wave 1 (Refactor & Judo): Decouple route resolver into 3-stage pipeline, extract adapter validation helper, and unify receipts under ReceiptBase.
    • Wave 2 (Test Decomposition): Decompose test_model_role_policy.py to keep all test files <500 lines.
    • Wave 3 (Issue research(jev-task-fit): assess fast-role/model-choice extension without promoting unproved decisions #517 Holdout Benchmarks): Benchmark fast-worker role suggestions on holdout replay tasks (qstack#1791) and generate calibrated receipts in references/calibrated-evidence.json before any advisory promotion.

  3. tommy-ca commented on Oct 20, 2026

    @tommy-ca
    tommy-ca
    on Oct 20, 2026
    OwnerAuthor

    Follow-Up Remediation Issues Filed (Issue Graph Update)

    The recursive planning breakdown from the Thermo-Nuclear Code Quality Review has been converted into tracking issues in the issue graph:

  4. tommy-ca
    added 2 commits that reference this issue on Oct 20, 2026

    docs(audit): add thermo-nuclear code quality review for JEV resilienc…

    b21540a

    feat(jev-task-fit): benchmark fast-role fit against holdout tasks bef…

    a737926

  5. tommy-ca commented on Oct 21, 2026

    @tommy-ca
    tommy-ca
    on Oct 21, 2026
    OwnerAuthor

    Wave 1–3 Remediation & Benchmarking Complete

    All three follow-up remediation waves established by the Thermo-Nuclear Code Quality Review (docs/audit/thermo-nuclear-517/) have been implemented, independently verified, tested, and merged into main:

    Summary of Completed Waves

    1. Wave 1 (Refactor & Judo, refactor(jev-judo): decouple route resolver pipeline and unify decision adapter validation #668 / PR refactor(jev-judo): decouple route resolver pipeline and unify decision adapter validation (#668) #671 — 9e73423):

      • Decoupled resolve_skill_order_with_shadow in scripts/route_resolver.py into a 3-stage pure pipeline (_is_egress_bypassed, _load_semantic_candidates, _evaluate_advisory_promotion).
      • Extracted shared input bounds validation (_parse_unit_interval, _invalid_obs) in scripts/decision_adapter.py.
      • Unified serialization and safe deserialization under ReceiptBase in scripts/evidence_receipts.py.

    2. Wave 2 (Test Decomposition & Rule 1 Compliance, test(model-role): decompose test_model_role_policy.py to respect 1k-line ceiling #669 / PR test(model-role): decompose test_model_role_policy.py to respect 1k-line ceiling (#669) #672 — 4b4fb98):

      • Decomposed monolithic 995-line tests/test_model_role_policy.py into 3 modular suites, all strictly under the 500-line ceiling:
        • tests/model_role_helpers.py (86 lines)
        • tests/test_model_role_schema.py (195 lines)
        • tests/test_model_role_constraints.py (244 lines)
        • tests/test_model_role_profiles.py (497 lines)
      • Updated check selection in scripts/select_checks.py and CI workflow.

    3. Wave 3 (Shadow Benchmarking & Task-Fit, research(jev-task-fit): benchmark fast-role fit against holdout tasks before advisory promotion #670 / PR feat(jev-task-fit): benchmark fast-role fit against holdout tasks before advisory promotion (#670, #517) #673 — a737926):

      • Formulated OpenSpec artifacts in openspec/changes/2026-10-21-fast-role-task-fit/ (proposal.md, design.md, adr.md, tasks.md).
      • Implemented scripts/eval_fast_role_task_fit.py benchmarking 12 holdout tasks (including 4 Quant OMS replay tasks from qstack#1791) across Baseline (A), Deterministic (B), and Semantic JEV (C).
      • Key Findings:
        • Deterministic task-feature routing yields 21.3% cost reduction with 0.0ms decision latency and zero external calls.
        • Semantic JEV yields 16.1% cost reduction gated behind references/calibrated-evidence.json.
        • Zero demotions of cognitive frontier roles (architect-runners, independent-verifier, arena-cross-judge-pool, judgment-and-prose, feature).
        • Zero unmapped model slugs; strict Contract [Program] Harness-neutral pstack role/model contracts and pstack-claude parity #429 catalog confinement.
        • Zero bloat to external repositories (qstack): all decision logic remains 100% in pstack.
      • Implemented tests/test_fast_role_task_fit.py (214 lines, Rule 1 compliant). All offline contracts pass in CI.

    Verification Summary

  6. tommy-ca
    added a commit that references this issue on Oct 21, 2026

    docs(audit): add post-implementation thermo-nuclear code quality revi…

    6664808

  7. tommy-ca commented on Oct 21, 2026

    @tommy-ca
    tommy-ca
    on Oct 21, 2026
    OwnerAuthor

    Final Thermo-Nuclear Review & Issue Closure Report

    The comprehensive post-implementation Thermo-Nuclear Code Quality Review has been completed and published under docs/audit/thermo-nuclear-517-final/ (commit 6664808).

    Audit Verdict: UNCONDITIONAL PASS

    All 8 non-negotiable Thermo-Nuclear standards have been audited and met:

    1. Rule 0 (Code Judo & Structural Simplification):

    2. Rule 1 (1,000-Line Ceiling Strictly Respected):

    3. Rule 2 & Rule 5 (Spaghetti Control & Type Boundaries):

      • Context egress guarded by SafeContext and EgressClass enums.
      • Replaced boolean-integer ambiguity with explicit finite float parsing [0.0, 1.0].
      • Wrapped provider interactions in strict fail-open boundaries.

    4. Rule 4 & Rule 7 (Empirical Superiority of Deterministic Routing):

      • Benchmarking across 12 holdout tasks (including 4 Quant OMS replay tasks from qstack#1791) proves that Deterministic Routing (B) achieves 21.3% cost reduction with 0.0ms decision latency and zero external network calls.
      • Semantic JEV yields 16.1% cost savings and remains in shadow mode behind references/calibrated-evidence.json.

    5. Rule 6 (Architectural Containment & Zero External Bloat):

      • Mechanically proven by tests/test_context_overlays.py: 0 external coupling strings or duplicate routers in qstack.
      • Contract [Program] Harness-neutral pstack role/model contracts and pstack-claude parity #429 catalog confinement verified (0 unmapped model slugs).
      • Cognitive frontier roles (architect-runners, independent-verifier, arena-cross-judge-pool, judgment-and-prose, feature) immutably protected against demotion.

    Delivered Artifacts & PR Graph

    All research-only and implementation criteria for Issue #517 are fulfilled. Closing issue as resolved.

  8. tommy-ca
    closed this as completedon Oct 21, 2026
  9. tommy-ca
    added 3 commits that reference this issue on Oct 21, 2026

    docs(openspec): record task-fit ship evidence - PR 673 merged, issues…

    a83eaa2

    fix(jev): gate semantic promotion on validated receipt, not file pres…

    3cecc5a

    fix(jev): pin expected provider/model in benchmark gate (#693)

    20164ba
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions