Skip to content

test(bench): add live in-memory FinBench TCR10 validation - #1085

Merged
DecisionNerd merged 10 commits into
mainfrom
cursor/964-live-finbench-5258
Sep 5, 2026
Merged

test(bench): add live in-memory FinBench TCR10 validation#1085
DecisionNerd merged 10 commits into
mainfrom
cursor/964-live-finbench-5258

Conversation

@DecisionNerd

@DecisionNerd DecisionNerd commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

The FinBench adapter previously validated replayed files without proving public GraphForge execution. Add a bounded TCR10 lane whose Rust runner loads a pinned synthetic seed, executes the parameterized query through the public facade, and validates Jaccard similarity against an independent set-based reference.

The query preserves the open investment time window and three-decimal Jaccard result. Identity evidence declares the integer-timestamp representation, synthetic engineering fixture, pinned upstream specification/query/Datagen, parameters, reference, and actual runner executable. Caller-provided result envelopes cannot claim live execution. Static TCR10 fixtures now use the same Jaccard schema.

Preserve public API resource-limit errors as structured resource evidence instead of invalid-document errors. Initialization, load, and query failures now retain their phase and resource/harness classification. Regression tests cover real API and project resource-limit error variants; they do not claim an actual out-of-memory qualification.

TCR10 is the bounded live proof. Other mapped reads retain explicitly labeled static/reference replay coverage; unsupported transaction and query guarantees remain visible. No official dataset scale factor, full-suite live execution, or audited certification is claimed.

Validation on the integrated tree:

  • make -C benchmarks smoke: passed, including 386 Python tests, all Rust runner tests (20 FinBench tests), benchmark workspace Clippy, formatting, and the real Cargo dependency-boundary check.
  • Real public-facade TCR10 execution, independent reference validation, provenance/parameter mutations, caller-envelope rejection, static Jaccard mismatch, and resource classification passed.
  • make pre-push-fast and final formatting/diff checks: passed.
  • Current dependency versions preserved; the lockfile adds only the runner's Arrow, facade, and SHA-2 edges.

Closes #964. #952 remains the broader M11 completion gate. Exact-head CI run 33947835889 passed at b1af3d373aedab46668f785dbc6eb60b51a89b78, with successful CI Gate, CLEAN merge state, and no unresolved review threads. Squash merged as 412231a4018b1c7b0e75ab5e344d13b7a2283ec3; issue #964 is closed.

@coderabbitai

coderabbitai Bot commented Sep 2, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Team

Run ID: e60c9be5-fde6-4b9b-b30c-affc20e733b8

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Warning

Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added documentation Improvements or additions to documentation release:none No release note or version impact labels Sep 2, 2026
cursoragent and others added 10 commits September 5, 2026 05:34
Co-authored-by: David Spencer <DecisionNerd@users.noreply.github.com>
Co-authored-by: David Spencer <DecisionNerd@users.noreply.github.com>
Co-authored-by: David Spencer <DecisionNerd@users.noreply.github.com>
Co-authored-by: David Spencer <DecisionNerd@users.noreply.github.com>
Co-authored-by: David Spencer <DecisionNerd@users.noreply.github.com>
Move live TCR10 onto the trusted Rust runner so a static JSON envelope
cannot claim live execution. The query uses official startTime/endTime
open-window Jaccard similarity rounded to 0.667, binds closed identities,
and keeps TCR1/TW1 typed unsupported with distinct failure lanes.

Co-authored-by: David Spencer <DecisionNerd@users.noreply.github.com>
Co-authored-by: David Spencer <DecisionNerd@users.noreply.github.com>
Replace obsolete company-ID rows with the official single-column
jaccardSimilarity schema. Compatible and semantic-incompat static replay
expect independently derived 0.667; reference-mismatch uses a deliberately
wrong 0.500. Normalized and exact comparison stay truthful. Acquisition
and identity checksums follow the updated validation reference.

Co-authored-by: David Spencer <DecisionNerd@users.noreply.github.com>
@DecisionNerd
DecisionNerd force-pushed the cursor/964-live-finbench-5258 branch from 6300049 to b1af3d3 Compare September 5, 2026 05:40
@DecisionNerd
DecisionNerd marked this pull request as ready for review September 5, 2026 05:40
@DecisionNerd
DecisionNerd merged commit 412231a into main Sep 5, 2026
21 checks passed
@DecisionNerd
DecisionNerd deleted the cursor/964-live-finbench-5258 branch September 5, 2026 05:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation release:none No release note or version impact

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(bench): add a separate GDC FinBench Transaction suite

3 participants