Skip to content

Expose skipped MLP training and empty portfolios in validation reports - #79

Merged
initial-d merged 2 commits into
mainfrom
codex/validation-diagnostics-20261018
Oct 18, 2026
Merged

initial-d merged 2 commits into
mainfrom
codex/validation-diagnostics-20261018

Conversation

@initial-d

Copy link
Copy Markdown
Owner

Summary

The full AkShare pipeline could silently skip every MLP training split, hold cash throughout, and still generate finite performance metrics that passed the report audit. Reports now expose joint factor coverage, split sample counts/skip reasons, prediction counts and portfolio cash/P&L exposure dates. Empty model portfolios and differing exposure windows have explicit warnings; full-panel metrics remain unchanged.

Raw synthetic replay and the October 18 public-data access failure are preserved separately in docs/audits/validation-coverage-20261018. This adds reporting evidence, not market-performance or proprietary-paper evidence. No issue is closed.

Type of change

  • Bug fix
  • Feature
  • Documentation
  • Test or benchmark
  • Refactor

Reproducibility

  • I added or updated tests where appropriate.
  • I updated docs or examples where behavior changed.
  • I did not commit private data, credentials, or large generated artifacts.

Collaboration

  • This PR is a solo contribution.
  • This PR includes real paired work and the commit message includes accurate Co-authored-by: trailers.

Validation

  • Full suite: 146 passed; Ruff and git diff --check passed.
  • Tests cover empty-factor skipped training and normal MLP predictions/holdings, alongside existing signal-date causality regressions.
  • Same-input replay with all 213 factors (120 synthetic dates x 20 stocks, seed 7): all weights, metrics and 0/7/15/30 bps cost-sensitivity results identical to base main. All 3 MLP splits lack sufficient training samples, all MLP portfolios hold cash, and the updated audit reports 8 coverage warnings instead of a pass.
  • CPU: Apple M5, macOS 26.6.2, Python 3.12.14, PyTorch 2.14.1; environment/commands/input hash/checksums in the evidence bundle.
  • Two fixed AkShare single-attempt probes again failed with upstream disconnects. No panel or new public-market result exists.

Coverage diagnostics do not fix differing evaluation windows or add realistic exchange execution. Archived July results remain unrevalidated.

@initial-d
initial-d merged commit f9efd00 into main Oct 18, 2026
3 checks passed
@initial-d
initial-d deleted the codex/validation-diagnostics-20261018 branch October 18, 2026 01:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant