Skip to content

feat(eval): support mean-CI plots for toy targets - #1004

Open
nehal-a2z wants to merge 1 commit into
goodfire-ai:mainfrom
nehal-a2z:nehal/toy-mean-ci-metric
Open

nehal-a2z wants to merge 1 commit into
goodfire-ai:mainfrom
nehal-a2z:nehal/toy-mean-ci-metric

Conversation

@nehal-a2z

@nehal-a2z nehal-a2z commented Sep 16, 2026 •

Copy link
Copy Markdown

Description

Configuring CIMeanPerComponent for a toy run currently parses but fails when the evaluator is built. Bind the existing metric through the current StandaloneOperationPlan API to held-out toy evaluation batches so TMS and ResidMLP can emit its linear/log plots on the configured slow cadence. Reuse the shared lower-CI reducer and renderer, pooling sums and example counts across all evaluation batches.

Related Issue

Related to #242: provides an existing diagnostic for toy investigations. This PR does not resolve the scientific question in that issue.

Motivation and Context

Toy runs already support authored UV figures but reject the mean-CI plots needed to inspect highly active components. This adds the missing binding while preserving the existing figure keys, schema, and W&B requirement. It does not enable another evaluation by default.

How Has This Been Tested?

  • 37 focused CPU tests pass across toy evaluation, toy UV figures, ResidMLP config, and metric cadence.
  • New coverage exercises the real TMS forward/CI path, independently computed means, preparation and pass sampling indices, reset between passes, slow cadence, transport/unsupported-metric refusal, and PNG decoding. The plan compiles before execution at the fixed eval-batch shape, matching the current fast toy evaluators.
  • Four new cases fail on unmodified current main; they pass with the binding. PNG signatures, dimensions and decoding are checked in the test.
  • Scoped Ruff lint/format, basedpyright (0 errors/warnings), and git diff --check pass.

Rebased onto main 017a17af; uses the current decomposition/CI lifecycle and prepares the compiled reduction before running evaluation.

The intentional all-zero case retains the existing renderer's log-axis warning. Full training, the full suite, and live W&B uploads were not run.

Does this PR introduce a breaking change?

No. This binds one existing, explicitly requested metric; existing configurations and other evaluators are unchanged.

Sent from Autobox

@nehal-a2z
nehal-a2z marked this pull request as ready for review September 16, 2026 12:18
@nehal-a2z
nehal-a2z force-pushed the nehal/toy-mean-ci-metric branch from 5c60333 to 210be76 Compare October 5, 2026 02:46

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant