Skip to content

Train IC objectives on complete dates without NaN gradients - #82

Merged
initial-d merged 2 commits into
mainfrom
codex/correlation-gradient-20261027
Oct 28, 2026
Merged

initial-d merged 2 commits into
mainfrom
codex/correlation-gradient-20261027

Conversation

@initial-d

@initial-d initial-d commented Oct 27, 2026 •

Copy link
Copy Markdown
Owner

IC/RankIC training previously shuffled stock-days from different dates into the same correlation batch. Between-date level shifts can make the pooled correlation positive even when every daily cross-section has reversed stock ordering. The time split could also put stocks from the same date on both sides, and a training set smaller than batch_size silently performed no optimizer steps.

The trainer now defaults to one complete observed date per batch for ICLoss/RankICLoss, shuffles dates, and averages daily losses equally. Time validation holds out complete latest dates; val_split is a date fraction in this mode. Random daily splits also keep dates intact. Ordinary regression retains sample batching and includes the final partial batch. batch_mode="date" supports custom daily objectives; explicit sample batching is rejected for the built-in IC losses. Validation runs without autograd.

The PR also guards zero-variance operands before Pearson square roots: constant/singleton batches contribute zero loss and zero gradients instead of poisoning optimizer state with NaNs. Ordinary finite nonconstant batches retain the previous correlation formula/gradients. Correlation inputs must be aligned, nonempty and finite; gamma/temperature must be finite positive values.

Behavior and limits:

  • Date mode ignores batch_size; memory depends on the largest cross-section. Soft RankIC still uses quadratic pairwise rank memory.
  • Constant/singleton dates are retained with zero loss/gradients, so daily loss is an optimization diagnostic, not a mean of only defined IC values.
  • Random splits remain reproducibility options, not chronological validation evidence. Longer label horizons still require a suitable purge gap.
  • Validation boundaries and retained partial batches can change historical model diagnostics. No archived market result is revalidated and no alpha/investment claim is made.
  • Based on main plus the prior gradient fix in this PR; excludes pending Separate label validity and reject unusable return endpoints #80/Add streamed daily factor screening and stable rolling moments #81.

Local verification for head c955c1a8b937cdf78a33b1ed7cedb3ddf16e0a33, Python 3.12.14 / PyTorch 2.14.1, macOS 26.6.2, Apple M5 CPU, deterministic unit/synthetic fixtures only (no external market data, no trading-cost assumption applicable):

  • /tmp/projects-push-20261028-venv/bin/python -m pytest tests/test_cross_section_training.py tests/test_trainer_split.py tests/test_losses.py — 69 passed.
  • /tmp/projects-push-20261028-venv/bin/python -m pytest — 209 passed.
  • /tmp/projects-push-20261028-venv/bin/python -m ruff check src tests — passed.
  • git diff --check — passed before commit.

Regression tests cover the pooled-correlation counterexample, ragged and singleton dates, date-disjoint time/random splits, equal-date validation weighting, actual updates on undersized regression datasets, four-dtype degenerate gradients, normal gradient parity and Adam recovery.

GitHub CI passed on the exact head above: Python 3.9/3.10/3.11 lint, full tests and CLI smoke checks. The maintainer explicitly requested direct merge on 2026-10-28 after these checks passed. Merged using the repository's permitted administrator override; no independent review is claimed and branch protection settings were not changed.

@initial-d initial-d changed the title Keep IC loss gradients finite on constant batches Train IC objectives on complete dates without NaN gradients Oct 28, 2026
@initial-d
initial-d merged commit a365f90 into main Oct 28, 2026
3 checks passed
@initial-d
initial-d deleted the codex/correlation-gradient-20261027 branch October 28, 2026 03:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant