Repository navigation
Conversation
evals: fail before scoring when real mode has no non-blank Anthropic key and share model selection across every case shape. tests: exercise configuration failures, offline replay, intercepted real routing, repeated sessions and concurrent CLI jobs. docs: state the real-mode precondition and scripted evidence boundary. CHANGELOG: record before/after behavior and retained repository gate failures. .qa: append the limited read-only entry diagnostic with unverified journeys explicitly skipped. Verification: eight mode scenarios pass (five fail on the old runner); unchanged golden cases pass 6/6. Both typechecks and 2,787 tests pass in the normal local floor; its one failure is the unchanged expired marketed proof. Independent code and simplification review approve the exact patch; merge readiness remains blocked.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Scaffold Handoff — For Your Coding Agent
Immutability CheckMode: advisory ✅ No immutable files were modified in this branch. Changed Files
Needs Adversarial Review — Do NOT Apply YetThese proposals passed the reject check but have not been approved by
Safety Boundary
Immutable files (never modify):
Scaffold files (safe to modify):
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
npm run eval:realwithout an Anthropic key silently executed scripted golden plans, printedanthropic (real)andSCORE: 6/6 passed, and exited successfully. Reject missing, empty or whitespace-only keys before cases, the banner or scoring, and use the same model selector for single-turn, multi-turn and property cases. Deliberatenpm run evalremains scripted even when credentials are configured.Eight scenario tests cover actual CLI rejection, offline replay, locally intercepted real-model routing, constructor failure, repeated runs and concurrent jobs. The old runner fails five tests; the repair passes all eight. Existing golden cases and scorers are unchanged and still pass 6/6. No provider request or genuine model-quality claim.
The normal local full floor passes both typechecks and 2,787 tests, with one failure: the existing marketed live-product receipt is 45.6 days old against its 30-day freshness window. The same failure reproduces on unchanged main. Two earlier worker-only Excel COM failures pass in the normal parent environment. Raw failures are retained; this PR does not change the receipt timestamp, freshness policy or test.
This stays draft until the repository gate and exact-head CI are green. The additional QA-memory line records only a limited read-only entry-page diagnostic, with authenticated/provider journeys explicitly skipped and no whole-app score. No visual-source or model-refresh changes, production deployment, paid evaluations or automatic promotion.
Independent gpt-6-astra/xhigh code and simplification review approved the bounded five-file patch, SHA-256
db5e653776eeb89b8e75122c4091f1f217acd51ffa6c5bb0bf45a0ce4c8651fd, treef34c73edac0fceef8191a444d2fccb8f10ae01c9. The reviewer reran eight scenarios and actual-CLI Unicode-whitespace/configured-offline probes, inspected all 45 implementation evidence files, and separately marked repository readiness blocked by the stale-proof gate. Its alternate test-pool run is supplementary evidence, not a replacement for the canonical floor.Current CI also stops at the earlier production dependency audit: run38571197209 reports12 affected packages (11low,1high) from unchanged main dependency files. Existing PR256 already contains the Undici repair, so this eval-mode PR does not duplicate it. The audit and expired live-product receipt are separate merge blockers; all later unreached CI stages remain uncertified.