test(bench): add live in-memory FinBench TCR10 validation - #1085
Conversation
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Team Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Warning Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Co-authored-by: David Spencer <DecisionNerd@users.noreply.github.com>
Co-authored-by: David Spencer <DecisionNerd@users.noreply.github.com>
Co-authored-by: David Spencer <DecisionNerd@users.noreply.github.com>
Co-authored-by: David Spencer <DecisionNerd@users.noreply.github.com>
Co-authored-by: David Spencer <DecisionNerd@users.noreply.github.com>
Move live TCR10 onto the trusted Rust runner so a static JSON envelope cannot claim live execution. The query uses official startTime/endTime open-window Jaccard similarity rounded to 0.667, binds closed identities, and keeps TCR1/TW1 typed unsupported with distinct failure lanes. Co-authored-by: David Spencer <DecisionNerd@users.noreply.github.com>
Co-authored-by: David Spencer <DecisionNerd@users.noreply.github.com>
Replace obsolete company-ID rows with the official single-column jaccardSimilarity schema. Compatible and semantic-incompat static replay expect independently derived 0.667; reference-mismatch uses a deliberately wrong 0.500. Normalized and exact comparison stay truthful. Acquisition and identity checksums follow the updated validation reference. Co-authored-by: David Spencer <DecisionNerd@users.noreply.github.com>
6300049 to
b1af3d3
Compare
The FinBench adapter previously validated replayed files without proving public GraphForge execution. Add a bounded TCR10 lane whose Rust runner loads a pinned synthetic seed, executes the parameterized query through the public facade, and validates Jaccard similarity against an independent set-based reference.
The query preserves the open investment time window and three-decimal Jaccard result. Identity evidence declares the integer-timestamp representation, synthetic engineering fixture, pinned upstream specification/query/Datagen, parameters, reference, and actual runner executable. Caller-provided result envelopes cannot claim live execution. Static TCR10 fixtures now use the same Jaccard schema.
Preserve public API resource-limit errors as structured resource evidence instead of invalid-document errors. Initialization, load, and query failures now retain their phase and resource/harness classification. Regression tests cover real API and project resource-limit error variants; they do not claim an actual out-of-memory qualification.
TCR10 is the bounded live proof. Other mapped reads retain explicitly labeled static/reference replay coverage; unsupported transaction and query guarantees remain visible. No official dataset scale factor, full-suite live execution, or audited certification is claimed.
Validation on the integrated tree:
make -C benchmarks smoke: passed, including 386 Python tests, all Rust runner tests (20 FinBench tests), benchmark workspace Clippy, formatting, and the real Cargo dependency-boundary check.make pre-push-fastand final formatting/diff checks: passed.Closes #964. #952 remains the broader M11 completion gate. Exact-head CI run 33947835889 passed at
b1af3d373aedab46668f785dbc6eb60b51a89b78, with successful CI Gate, CLEAN merge state, and no unresolved review threads. Squash merged as412231a4018b1c7b0e75ab5e344d13b7a2283ec3; issue #964 is closed.