diff --git a/CHANGELOG.md b/CHANGELOG.md index d751964..2abf2ab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,18 @@ ## Unreleased +### Public validation: signal-date availability + +- The generic and full AkShare validation scripts now select portfolios and + predict scores using signal-date availability. Previously they also required + next-day availability, so a known future halt or missing row changed today's + holdings. Historical training labels and trailing IC estimates still require + observations on both dates. +- New validation metadata records `availability_protocol`. Reports made with + v0.4.1 or earlier need revalidation; changing the selection mask can change + weights, turnover, and net performance. This is a causality fix, not alpha + evidence or a halt-aware execution model. + ### Markowitz optimiser: DPP form, reliability, and caching - The risk term is now `sum_squares(F.T @ w)` with `Sigma = F F^T` instead of diff --git a/README.md b/README.md index 3d6e328..f5cee1d 100644 --- a/README.md +++ b/README.md @@ -154,7 +154,9 @@ Archived public-data snapshot (2026-07-29, before v0.4.1): **The numbers below predate the v0.4.1 mask-first and halt-safe return fixes.** They remain historical records and have not been revalidated with the corrected -protocol. Re-run with v0.4.1 or later before drawing conclusions. The detailed +protocol. The [October 17 audit](docs/audits/public-revalidation-20261017/README.md) +also found next-day availability entering portfolio selection through v0.4.1; +revalidation requires that correction too. The detailed dashboard includes the protocol note, cost-sensitivity charts, and reproduction commands. diff --git a/README.zh-CN.md b/README.zh-CN.md index b981a6d..fdc1972 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -79,7 +79,9 @@ Demo 跑通后，可以继续[查看 benchmark](docs/benchmark_board.md)或 ## 归档公开验证（2026-07-29） **以下数字生成于 v0.4.1 修复前，尚未用修正后的 mask-first 与停牌收益口径重新验证。** -保留原始报告便于审计；不能把这些数字当作当前版本的验证结果。请使用 v0.4.1 或更新版本重跑。 +保留原始报告便于审计；不能把这些数字当作当前版本的验证结果。 +[10 月 17 日审计](docs/audits/public-revalidation-20261017/README.md)还发现 v0.4.1 及之前的 +公开验证用次日可用性决定当天选股，重跑也需要包含该修复。 归档结果采用 AkShare 的当前沪深 300 公开数据，在日频 213 因子近似流程中显式计入换手和交易成本。缓冲因子组合在 7 bps 有效成本假设下得到 22.20% 年化收益、0.919 Sharpe；同区间等权基线为 17.75% 和 0.882。完整口径、成本敏感性图和复现命令见[验证看板](docs/validation_dashboard.md)。 diff --git a/docs/audits/public-revalidation-20261017/README.md b/docs/audits/public-revalidation-20261017/README.md new file mode 100644 index 0000000..babf5b3 --- /dev/null +++ b/docs/audits/public-revalidation-20261017/README.md @@ -0,0 +1,121 @@ +# Signal-Date Causality Audit and Public-Data Blocker — 2026-10-17 + +**The public validation scripts used tomorrow's data availability to decide +today's holdings.** This audit fixes that lookahead in both validation paths. +The planned AkShare revalidation did not reach a backtest: both predeclared +single-symbol probes failed with upstream disconnects. No new public-market +performance result is claimed. + +This bundle distinguishes synthetic engineering evidence from the public-data +failure. Neither supports investment or deployable-alpha claims. The original +proprietary paper experiment was not run or evaluated here. + +## Scope and provenance + +- Base main: `ccdcf16a7c0768dbb31fa22f91ae02b692f0fd9b`. +- Corrected code: `3443987fcabdf1c46d198bfa7a878bd4e938a0a0`. +- Scope was recorded before data requests in [plan.md](plan.md). +- Exact commands: [commands.txt](commands.txt). +- Environment: [environment.json](environment.json), + [pip-freeze.txt](pip-freeze.txt). CPU: Apple M5, 10 logical CPUs, 32 GiB RAM; + macOS 26.6.2 arm64, Python 3.12.14, PyTorch 2.14.1, AkShare 1.19.1. CPU runs; + CUDA unavailable, MPS available but unused. +- Provider probes used the base plus the candidate patch; the fix was committed + before final verification. Evidence scripts were untracked during the runs. +- Bundle integrity: [checksums.txt](checksums.txt). Traceback repository paths + are redacted to `` and trailing whitespace is stripped; error messages + and test failures remain intact. + +## The causality issue + +`_valid_forward_mask` correctly requires observations at `t` and `t+1` to build +a historical training label. Both `run_validation` and `run_full_pipeline` also +used it for portfolio eligibility and out-of-sample inference. Consequently, a +stock absent tomorrow was excluded today even though it was observed today. + +The correction uses signal-date availability after the limit mask and factor +warm-up checks for selection and inference. Historical training labels and +trailing IC estimates retain their separate forward-label mask. New report +metadata records `availability_protocol`. No change to the proprietary paper +results is asserted; this finding concerns the public Python validation paths. + +## Synthetic regression evidence + +Two tests were run with the base script sources and failed because the stock's +weight was zero immediately before its future halt. The same tests pass with +the correction. [old-code-regressions.txt](old-code-regressions.txt) preserves +both assertion failures. Additional tests verify that MLP inference still scores +an observed pre-halt row without training on missing labels, and that trailing +IC ignores unavailable labels while scoring today's universe. + +The [same-input comparison](same-input-comparison.json), generated by +[compare_signal_masks.py](compare_signal_masks.py), runs both script versions +on one fixed synthetic panel: 90 dates, 12 stocks, seed 3, with a zero-filled +halt at row 31 for stock 0. Its field-byte fingerprint is in the JSON. + +| Equal-weight position | Before correction | After correction | +|---|---:|---:| +| Row 30, immediately before the halt | 0 | 1/12 | +| Row 31, the observed halt | 0 | 0 | + +The desired property is that tomorrow's missing observation does not determine +today's portfolio. The same-input example changes net returns and turnover in +both directions across the simple strategies; improvement in return is not the +acceptance criterion. The generated numbers use 5 bps costs plus 2 bps slippage, +daily cadence, and 100 moving-block bootstrap samples with 20-day blocks. They +are synthetic engineering diagnostics, not predictive evidence. + +Verification at the corrected code commit: + +- Full suite: **143 passed**, [pytest.txt](pytest.txt). +- Ruff: passed, [ruff.txt](ruff.txt). +- `mlquant demo`: completed on CPU using synthetic data; 200 held-out dates at + 5 bps costs. Its artifacts stay under `artifacts/small/`; no performance + claim is drawn from them. +- The generated corrected synthetic report passed the existing report audit: + [report-audit.json](report-audit.json). This quality check is not a profitability + test. +- `git diff --check`: passed. + +## Public-data failure, kept separately + +The predeclared first step was one AkShare request each for `600519` and +`000001`, forward-adjusted (`qfq`), daily, 2021-01-01 through 2025-01-01, +`retries=1`. [provider_probe.py](provider_probe.py) preserves the probe source; +[provider-probe.json](provider-probe.json) records timestamps and wall times, +and [provider-probe.txt](provider-probe.txt) preserves loader warnings and errors. + +| Symbol | Start, Asia/Shanghai | Outcome | +|---|---|---| +| `600519` | 2026-10-17 11:16:12 | Remote end closed the connection; no data returned | +| `000001` | 2026-10-17 11:16:14 | Remote end closed the connection; no data returned | + +The loader surfaced `ValueError: No data returned`, following +`RemoteDisconnected('Remote end closed connection without response')` warnings. +The cause beyond the observed disconnect is unknown; no HTTP 429 is asserted. +No panel was generated, so there is no public-data fingerprint, coverage audit, +portfolio, or new Sharpe estimate. Cost assumptions are not applicable to these +download-only probes. The planned expanded run's 7 bps assumption was not used. +No alternative provider or synthetic fallback was relabeled as public data. + +## What remains unvalidated + +- July 2026 A-share reports predate the mask-first/halt-safe fixes and this + availability correction. They are preserved with explicit protocol notes. +- Their normalized input panel is unavailable in this workspace. A fresh + provider download cannot isolate code changes from provider revisions, + constituent changes, or environment differences. The predeclared 20-name + diagnostic would not be the July 300-name reproduction. +- The engine still simplifies execution across halts and limits. A missing-day + forced exit or an infeasible trade is not modeled as a real exchange fill; + this correction removes selection lookahead, not execution risk. +- All-factor joint-mask coverage, MLP portfolio coverage, and equal evaluation + windows need measurement on real input before interpreting comparative returns. +- Public data remains subject to adjustments, survivorship bias, ST/IPO limit + metadata gaps, revisions, and access failures. A tradability proxy does not + replace licensed point-in-time data. + +The next data run should use the fixed scope in [plan.md](plan.md), preserve +normalized-input hashes, coverage, warnings and cost assumptions, and apply +the corrected code. The [validation dashboard](../../validation_dashboard.md) +continues to show archived evidence while that rerun is blocked by data access. diff --git a/docs/audits/public-revalidation-20261017/checksums.txt b/docs/audits/public-revalidation-20261017/checksums.txt new file mode 100644 index 0000000..f05ba5c --- /dev/null +++ b/docs/audits/public-revalidation-20261017/checksums.txt @@ -0,0 +1,14 @@ +add6f432fb0c4c1aeb362589835ca95f569d580770cd9ad4c739776f5ec5cf56 README.md +176b19059aca9b51e6f66120d1a15eeb36f560ca57e701d7088e93f003fb007e commands.txt +6b9d8ef1b33afe1dd2bccc1414e5af8f05d1ac02a7d878ef7e330d61a9dd686a compare_signal_masks.py +eb8aa9ae8580c0ae7e2a2122d553daf5f9a5108e3f311b92dea4c1fa045f51f5 environment.json +72bc96838ebef7d2f8313a97f8102e928dc8d2825a3c6eb53bfe48b94a0b6f86 old-code-regressions.txt +94a93d9356993469a3998a826f4071135a077b3551160f18bb940a6a2923102a pip-freeze.txt +50303cc1338ca2e1aa8f5f81c76f23b44851d732506558b7e39e6ca96e558c8e plan.md +8ae9b46dd96cece8f0554e24f8a5f50b4e178f2b79bae984a54233c7ceb25051 provider-probe.json +204a0c0e4d4309441a326134f2b4ba7736b31450ac2dffd8da47d61b8a6ff28f provider-probe.txt +0f603a2ddbb5c85d57444971425b29c39d39f8844752094c7f9ee9d396d15af7 provider_probe.py +765605818743ce3b523153f5fd32b74a3a1188f805bff25debee1ada8042abcf pytest.txt +f555ab552d0ccd7e557dc81d9c34f6797d26fb4fa8d8b5361e0c17ce51dd3be6 report-audit.json +82b3e6a6c090a57601d22943bd23fca9218d1031dbe5a7b754092f9a156b4f18 ruff.txt +7fad8c060e1f2a28eb864b163ec90c9af1ff63dbc10f4ce87701bb1f9fe54f98 same-input-comparison.json diff --git a/docs/audits/public-revalidation-20261017/commands.txt b/docs/audits/public-revalidation-20261017/commands.txt new file mode 100644 index 0000000..29f6b8c --- /dev/null +++ b/docs/audits/public-revalidation-20261017/commands.txt @@ -0,0 +1,19 @@ +Install: +/Users/duyimin/.cache/codex-runtimes/codex-primary-runtime/dependencies/python/bin/python3 -m venv .venv +.venv/bin/python -m pip install -e '.[dev]' +.venv/bin/python -m pip freeze + +Provider probe (source saved in provider_probe.py; original execution used the same code on stdin): +.venv/bin/python -u docs/audits/public-revalidation-20261017/provider_probe.py + +Old-code regressions (public validation script sources temporarily loaded from base ccdcf16; corrected sources restored before validation): +.venv/bin/python -m pytest tests/test_public_data_validation.py::test_signal_weights_do_not_use_tomorrows_availability tests/test_akshare_csi300_full_pipeline.py::test_full_pipeline_separates_signal_and_forward_label_masks -q + +Corrected verification at 3443987: +OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 .venv/bin/python -m pytest +.venv/bin/python -m ruff check src tests scripts/public_data_validation.py scripts/akshare_csi300_full_pipeline.py +OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 .venv/bin/mlquant demo +OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 .venv/bin/python docs/audits/public-revalidation-20261017/compare_signal_masks.py +.venv/bin/python scripts/audit_validation_report.py artifacts/revalidation-20261017/corrected/summary.json --output-md artifacts/revalidation-20261017/report-audit.md --output-json artifacts/revalidation-20261017/report-audit.json + +git diff --check diff --git a/docs/audits/public-revalidation-20261017/compare_signal_masks.py b/docs/audits/public-revalidation-20261017/compare_signal_masks.py new file mode 100644 index 0000000..e9214b3 --- /dev/null +++ b/docs/audits/public-revalidation-20261017/compare_signal_masks.py @@ -0,0 +1,95 @@ +"""Offline same-input comparison; synthetic engineering evidence only. + +Run from the repository root: + .venv/bin/python docs/audits/public-revalidation-20261017/compare_signal_masks.py +""" +from dataclasses import replace +import hashlib +import json +from pathlib import Path +import subprocess +import sys +import types + +import torch + +ROOT = Path(__file__).resolve().parents[3] +sys.path.insert(0, str(ROOT)) +sys.path.insert(0, str(ROOT / 'src')) +from mlquant.data.synthetic import SyntheticConfig, make_synthetic_panel +import scripts.public_data_validation as corrected + +BASE = 'ccdcf16a7c0768dbb31fa22f91ae02b692f0fd9b' +OUT = ROOT / 'artifacts/revalidation-20261017' +OUT.mkdir(parents=True, exist_ok=True) +torch.set_num_threads(1) +torch.set_num_interop_threads(1) +source = subprocess.check_output( + ['git', 'show', f'{BASE}:scripts/public_data_validation.py'], cwd=ROOT, text=True, +) +old = types.ModuleType('archived_validation') +old.__file__ = str(ROOT / 'scripts/public_data_validation.py') +sys.modules[old.__name__] = old +exec(compile(source, old.__file__, 'exec'), old.__dict__) + +# A stock is observed today but is halted tomorrow. Zero-filled data on the +# halt day mirrors loader behavior; both versions receive this identical panel. +panel = make_synthetic_panel(SyntheticConfig(n_dates=90, n_stocks=12, halt_prob=0, seed=3)) +mask = panel.mask.clone() +mask[31, 0] = False +fields = {name: getattr(panel, name).masked_fill(~mask, 0) + for name in ('open', 'high', 'low', 'close', 'volume', 'vwap', 'amount')} +panel = replace(panel, mask=mask, **fields) +panel.assert_consistent() +raw_mask = panel.mask.clone() +hash_state = hashlib.sha256() +for name in ('open', 'high', 'low', 'close', 'volume', 'vwap', 'amount', 'mask'): + hash_state.update(name.encode()) + hash_state.update(getattr(panel, name).numpy().tobytes()) + +result = { + 'category': 'synthetic engineering comparison; no predictive or investment evidence', + 'base_commit': BASE, + 'corrected_commit': subprocess.check_output(['git', 'rev-parse', 'HEAD'], cwd=ROOT, text=True).strip(), + 'synthetic_config': {'n_dates':90, 'n_stocks':12, 'seed':3, 'halt_prob':0}, + 'perturbation': 'zero-filled halt at date index 31, stock index 0', + 'panel_hash_method': 'SHA-256 over field names and C-order tensor bytes for open/high/low/close/volume/vwap/amount/mask', + 'panel_sha256': hash_state.hexdigest(), + 'threads': {'torch':1,'interop':1}, + 'costs_bps':5, 'slippage_bps':2, + 'versions': {}, +} +for label, module in (('old', old), ('corrected', corrected)): + module.load_validation_panel = lambda cfg: panel + captured = {} + summarise = module._summarise_strategy + + def capture(name, weights, returns, _captured=captured, _summarise=summarise, **kwargs): + _captured[name] = weights.copy() + return _summarise(name, weights, returns, **kwargs) + + module._summarise_strategy = capture + cfg = module.ValidationConfig( + source='synthetic', preset='us-large-100', tickers=(), start='2021-01-01', + end='2022-01-01', max_tickers=12, device='cpu', costs_bps=5, slippage_bps=2, + cost_grid_bps=(0,7,15,30), bootstrap_samples=100, bootstrap_block_size=20, + train_window=40, test_window=20, step=20, top_quantile=0.25, seed=3, + epochs=1, batch_size=128, hidden=16, + models=('equal_weight','momentum_20','alpha101_mean'), + output_dir=OUT / label, synthetic_dates=90, synthetic_stocks=12, + command=(sys.executable, str(Path(__file__).relative_to(ROOT))), + ) + rows = module.run_validation(cfg) + result['versions'][label] = { + 'equal_weight_before_halt': float(captured['equal_weight'][30,0]), + 'equal_weight_on_halt': float(captured['equal_weight'][31,0]), + 'results': rows, + } + module._summarise_strategy = summarise + assert torch.equal(panel.mask,raw_mask) +assert result['versions']['old']['equal_weight_before_halt'] == 0 +assert result['versions']['corrected']['equal_weight_before_halt'] > 0 +assert result['versions']['corrected']['equal_weight_on_halt'] == 0 +(OUT / 'same-input-comparison.json').write_text(json.dumps(result, indent=2) + '\n') +print(json.dumps({version:{key:row[key] for key in ('equal_weight_before_halt','equal_weight_on_halt')} + for version,row in result['versions'].items()},indent=2)) diff --git a/docs/audits/public-revalidation-20261017/environment.json b/docs/audits/public-revalidation-20261017/environment.json new file mode 100644 index 0000000..70f7e0c --- /dev/null +++ b/docs/audits/public-revalidation-20261017/environment.json @@ -0,0 +1,38 @@ +{ + "base_commit": "ccdcf16a7c0768dbb31fa22f91ae02b692f0fd9b", + "working_tree_dirty": true, + "python": "3.12.14 (main, Aug 25 2026, 13:50:33) [Clang 22.1.3 ]", + "pytorch": "2.14.1", + "akshare": "1.19.1", + "platform": "macOS-26.6.2-arm64-arm-64bit", + "machine": "arm64", + "cpu": "Apple M5", + "logical_cpus": 10, + "memory_bytes": 34359738368, + "device": "cpu", + "cuda_available": false, + "mps_available": true, + "torch_threads_default": 4, + "torch_interop_threads_default": 10, + "data_source": "AkShare public A-share OHLCV", + "cost_assumptions": "5 bps costs + 2 bps slippage for planned run; not applicable to provider probe", + "limitations": [ + "Public provider responses may change or fail.", + "No normalized July panel is available for exact comparison.", + "Candidate patch is uncommitted at probe time; base commit and patch are preserved." + ], + "corrected_code_commit": "3443987fcabdf1c46d198bfa7a878bd4e938a0a0", + "probe_working_tree": "base commit plus candidate patch; fixed code committed afterwards", + "verification_working_tree": "corrected code commit; untracked evidence scripts only", + "verification_threads": { + "OMP_NUM_THREADS": 1, + "OPENBLAS_NUM_THREADS": 1, + "same_input_torch": 1, + "same_input_interop": 1, + "pytest_torch": null, + "pytest_interop": null + }, + "pytest_threads_note": "OMP/BLAS environment fixed; test suite sets or changes PyTorch pools in individual tests; no test timing comparison claimed", + "demo_costs_bps": 5, + "same_input_effective_costs_bps": 7 +} diff --git a/docs/audits/public-revalidation-20261017/old-code-regressions.txt b/docs/audits/public-revalidation-20261017/old-code-regressions.txt new file mode 100644 index 0000000..f3f1f6d --- /dev/null +++ b/docs/audits/public-revalidation-20261017/old-code-regressions.txt @@ -0,0 +1,103 @@ +FF [100%] +=================================== FAILURES =================================== +____________ test_signal_weights_do_not_use_tomorrows_availability _____________ + +tmp_path = PosixPath('/private/var/folders/ld/dfm826ld4_d6myjvpl4f8_c80000gn/T/pytest-of-duyimin/pytest-8/test_signal_weights_do_not_use0') + + def test_signal_weights_do_not_use_tomorrows_availability(tmp_path): + """A halt discovered tomorrow must not change today's equal-weight book.""" + from dataclasses import replace + import scripts.public_data_validation as pdv + from mlquant.data.synthetic import SyntheticConfig, make_synthetic_panel + + panel = make_synthetic_panel(SyntheticConfig(n_dates=60, n_stocks=8, halt_prob=0, seed=3)) + mask = panel.mask.clone() + mask[31, 0] = False + halted = replace(panel, mask=mask) + cfg = ValidationConfig( + source="synthetic", preset="us-large-100", tickers=(), start="2021-01-01", + end="2022-01-01", max_tickers=8, device="cpu", costs_bps=5, slippage_bps=0, + cost_grid_bps=(), bootstrap_samples=0, bootstrap_block_size=10, train_window=30, + test_window=10, step=10, top_quantile=0.25, seed=3, epochs=1, batch_size=64, + hidden=8, models=("equal_weight",), output_dir=tmp_path, synthetic_dates=60, + synthetic_stocks=8, + ) + books = [] + real = pdv._equal_weight + + def capture(valid): + weights = real(valid) + books.append(weights.copy()) + return weights + + for data in (panel, halted): + with mock.patch.object(pdv, "load_validation_panel", return_value=data), \ + mock.patch.object(pdv, "_equal_weight", side_effect=capture): + run_validation(cfg) + import numpy as np +> assert np.allclose(books[0][:31], books[1][:31]) +E AssertionError: assert False +E + where False = (array([[0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ],\n [0.125, 0.125, 0.125, 0.125, 0.125, 0.125, 0.1...25, 0.125, 0.125, 0.125, 0.125],\n [0.125, 0.125, 0.125, 0.125, 0.125, 0.125, 0.125, 0.125]],\n dtype=float32), array([[0. , 0. , 0. , 0. , 0. ,\n 0. , 0. , 0. ],\n .... , 0.14285715, 0.14285715, 0.14285715, 0.14285715,\n 0.14285715, 0.14285715, 0.14285715]], dtype=float32)) +E + where = /.venv/lib/python3.12/site-packages/numpy/__init__.py'>.allclose + +tests/test_public_data_validation.py:323: AssertionError +_________ test_full_pipeline_separates_signal_and_forward_label_masks __________ + +tmp_path = PosixPath('/private/var/folders/ld/dfm826ld4_d6myjvpl4f8_c80000gn/T/pytest-of-duyimin/pytest-8/test_full_pipeline_separates_s0') +monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x112defc20> + + def test_full_pipeline_separates_signal_and_forward_label_masks(tmp_path, monkeypatch): + from dataclasses import replace + import scripts.akshare_csi300_full_pipeline as full + from mlquant.data.synthetic import SyntheticConfig, make_synthetic_panel + + panel = make_synthetic_panel(SyntheticConfig(n_dates=60, n_stocks=8, halt_prob=0, seed=3)) + mask = panel.mask.clone() + mask[31, 0] = False + panel = replace(panel, mask=mask) + cfg = full.FullPipelineConfig( + source="synthetic", preset="custom", tickers=(), start="2021-01-01", + end="2022-01-01", max_tickers=8, device="cpu", costs_bps=5, slippage_bps=0, + cost_grid_bps=(), train_window=30, test_window=10, step=10, top_quantile=0.25, + seed=3, epochs=1, batch_size=64, hidden=8, factor_set="alpha101", + covariance_window=10, optimizer_candidates=5, optimizer_weight_cap=0.25, + optimizer_risk_aversion=1, rebalance_step=1, factor_ic_window=20, + min_tradable_ratio=0.9, bootstrap_samples=0, bootstrap_block_size=10, + output_dir=tmp_path, + ) + seen = {} + + def predict(model, features, target, labels, validation_cfg, *, prediction_valid=None): + if prediction_valid is None: + prediction_valid = labels + seen["labels"] = labels.copy() + seen["signals"] = prediction_valid.copy() + return np.where(prediction_valid, features[:, :, 0], np.nan) + + def write(config, data, names, rows, sensitivity, strategies, returns): + seen["equal_weight"] = strategies["equal_weight_daily"] + + monkeypatch.setattr(full, "load_validation_panel", lambda cfg: panel) + monkeypatch.setattr(full, "_alpha101_features", lambda panel: ( + np.ones((60, 8, 3), dtype=np.float32), mask.numpy().copy(), ["a", "b", "c"], + )) + monkeypatch.setattr(full, "_fit_predict_model", predict) + monkeypatch.setattr(full, "_write_outputs", write) + full.run_full_pipeline(cfg) + +> assert seen["equal_weight"][30, 0] > 0 +E assert np.float32(0.0) > 0 + +tests/test_akshare_csi300_full_pipeline.py:228: AssertionError +----------------------------- Captured stdout call ----------------------------- +stage: loading AkShare panel +stage: panel loaded (60 dates x 8 stocks) +stage: computing factor set alpha101 +stage: computed 3 factors +stage: training walk-forward MLP +stage: building scores +stage: building daily portfolios +stage: writing reports +=========================== short test summary info ============================ +FAILED tests/test_public_data_validation.py::test_signal_weights_do_not_use_tomorrows_availability +FAILED tests/test_akshare_csi300_full_pipeline.py::test_full_pipeline_separates_signal_and_forward_label_masks diff --git a/docs/audits/public-revalidation-20261017/pip-freeze.txt b/docs/audits/public-revalidation-20261017/pip-freeze.txt new file mode 100644 index 0000000..16471fd --- /dev/null +++ b/docs/audits/public-revalidation-20261017/pip-freeze.txt @@ -0,0 +1,78 @@ +akshare==1.19.1 +ast_serialize==0.13.1 +baostock==0.9.4 +beautifulsoup4==4.15.0 +certifi==2026.7.22 +cffi==2.1.1 +charset-normalizer==3.5.2 +clarabel==0.11.1 +click==8.5.0 +cloudpickle==3.1.2 +coverage==7.16.2 +curl_cffi==0.16.3 +cvxpy==1.9.3 +decorator==5.3.1 +et_xmlfile==2.0.0 +filelock==4.2.0 +fsspec==2026.9.0 +highspy==1.15.1 +html5lib==1.1 +hypothesis==6.170.1 +idna==3.20 +iniconfig==2.3.1 +Jinja2==3.1.6 +joblib==1.6.0 +jsonpath==0.82.2 +librt==0.16.0 +lxml==6.1.3 +markdown-it-py==4.2.0 +MarkupSafe==3.0.4 +mdurl==0.1.2 +mini-racer==0.14.1 +-e git+https://github.com/initial-d/ml-quant-trading.git@ccdcf16a7c0768dbb31fa22f91ae02b692f0fd9b#egg=mlquantx&subdirectory=../../../../../../../../.codex/worktrees/visibility-refresh/ml-quant-trading +mpmath==1.3.0 +multitasking==0.0.13 +mypy==2.4.0 +mypy_extensions==1.1.0 +narwhals==2.27.1 +networkx==3.7 +numpy==2.5.4 +openpyxl==3.1.5 +osqp==1.1.3 +packaging==26.3 +pandas==3.0.6 +pathspec==1.1.1 +peewee==4.5.3 +platformdirs==4.13.3 +pluggy==1.7.0 +protobuf==7.36.2 +pycparser==3.11 +Pygments==2.21.0 +pytest==9.1.1 +pytest-cov==7.1.0 +python-dateutil==2.9.0.post0 +pytz==2026.5 +PyYAML==6.0.3 +qdldl==0.1.9.post1 +requests==2.34.2 +rich==15.0.0 +ruff==0.17.1 +scikit-learn==1.9.1 +scipy==1.18.1 +scs==3.3.1 +setuptools==84.0.0 +six==1.17.0 +sortedcontainers==2.4.0 +soupsieve==3.0.3 +sparsediffpy==0.6.1 +sympy==1.14.0 +tabulate==0.10.0 +threadpoolctl==3.7.0 +torch==2.14.1 +tqdm==4.70.1 +typing_extensions==4.16.0 +urllib3==2.8.0 +webencodings==0.6.1 +websockets==17.2 +xlrd==2.0.2 +yfinance==1.7.0 diff --git a/docs/audits/public-revalidation-20261017/plan.md b/docs/audits/public-revalidation-20261017/plan.md new file mode 100644 index 0000000..73aa8e1 --- /dev/null +++ b/docs/audits/public-revalidation-20261017/plan.md @@ -0,0 +1,35 @@ +# Predeclared revalidation scope — 2026-10-17 + +Recorded before external data requests or new backtest outcomes. +Base commit: `ccdcf16a7c0768dbb31fa22f91ae02b692f0fd9b`. + +Primary question: does changing tomorrow's availability alter today's portfolio? +The two public validation scripts currently use the forward-label mask for +portfolio selection and MLP inference. Fix this causality issue first, retaining +forward-label filtering for historical training and IC estimates. + +Public-data check: AkShare, CPU, forward-adjusted (`qfq`) daily OHLCV, +2021-01-01 to 2025-01-01. Probe `600519` and `000001` once each (loader retries=1). +If both succeed, expand to this fixed 20-name diagnostic universe: +`600519,000001,600036,601318,600900,601166,600030,000333,000651,000858,002415,002475,600276,601888,600887,601012,600309,601398,601288,601939`. +It is a deliberately chosen current large-cap sample, not point-in-time CSI 300 +membership and not an exact rerun of the July 300-name panel. Do not cherry-pick +replacement symbols after seeing outcomes. If the probe fails, preserve the +failure and do not hammer upstream or silently switch providers. + +Fixed run choices: 213 registered factors, daily cadence, top 20%, buffer exit +40%, walk-forward train/test/step=504/63/63, 1 epoch, batch 4096, hidden 32, +seed 7, covariance window 126, IC window 252, optimizer candidates 20, +weight cap 0.05, risk aversion 1, min raw coverage 90%; 5 bps costs + 2 bps +slippage, sensitivity 0/7/15/30 bps; 100 moving-block bootstrap samples with +20-day blocks. Costs are a research simplification, not execution estimates. + +Keep the normalized public panel local and hash it; publish metadata, commands, +logs and derived summaries only. Document provider version, retrieval timestamp, +OS/Python/PyTorch/hardware, dependency freeze, CPU thread counts and warnings. +The original July panel is absent here, so a change versus July cannot be +attributed solely to a code fix. Preserve the old report and its numbers. + +If public data is unavailable, deliver the causality fix and deterministic +regression evidence, plus the public-data blocker. Synthetic engineering +results do not replace public-market evidence or establish alpha. diff --git a/docs/audits/public-revalidation-20261017/provider-probe.json b/docs/audits/public-revalidation-20261017/provider-probe.json new file mode 100644 index 0000000..c682d39 --- /dev/null +++ b/docs/audits/public-revalidation-20261017/provider-probe.json @@ -0,0 +1,26 @@ +[ + { + "ticker": "600519", + "started_at": "2026-10-17T11:16:12.370361+08:00", + "retries": 1, + "start": "2021-01-01", + "end": "2025-01-01", + "adjust": "qfq", + "status": "failed", + "error_type": "ValueError", + "error": "No data returned for tickers ['600519'] from 2021-01-01 to 2025-01-01", + "wall_seconds": 1.8582072090357542 + }, + { + "ticker": "000001", + "started_at": "2026-10-17T11:16:14.228948+08:00", + "retries": 1, + "start": "2021-01-01", + "end": "2025-01-01", + "adjust": "qfq", + "status": "failed", + "error_type": "ValueError", + "error": "No data returned for tickers ['000001'] from 2021-01-01 to 2025-01-01", + "wall_seconds": 0.12058529094792902 + } +] diff --git a/docs/audits/public-revalidation-20261017/provider-probe.txt b/docs/audits/public-revalidation-20261017/provider-probe.txt new file mode 100644 index 0000000..2663e00 --- /dev/null +++ b/docs/audits/public-revalidation-20261017/provider-probe.txt @@ -0,0 +1,18 @@ +probe 600519 2026-10-17T11:16:12.370361+08:00 +AkShare download progress: 1/1 (600519) +Warning: Failed to fetch 600519: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) +Traceback (most recent call last): + File "", line 11, in + File "/src/mlquant/data/akshare_loader.py", line 109, in load_akshare_panel + raise ValueError( +ValueError: No data returned for tickers ['600519'] from 2021-01-01 to 2025-01-01 +{'ticker': '600519', 'started_at': '2026-10-17T11:16:12.370361+08:00', 'retries': 1, 'start': '2021-01-01', 'end': '2025-01-01', 'adjust': 'qfq', 'status': 'failed', 'error_type': 'ValueError', 'error': "No data returned for tickers ['600519'] from 2021-01-01 to 2025-01-01", 'wall_seconds': 1.8582072090357542} +probe 000001 2026-10-17T11:16:14.228948+08:00 +AkShare download progress: 1/1 (000001) +Warning: Failed to fetch 000001: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) +Traceback (most recent call last): + File "", line 11, in + File "/src/mlquant/data/akshare_loader.py", line 109, in load_akshare_panel + raise ValueError( +ValueError: No data returned for tickers ['000001'] from 2021-01-01 to 2025-01-01 +{'ticker': '000001', 'started_at': '2026-10-17T11:16:14.228948+08:00', 'retries': 1, 'start': '2021-01-01', 'end': '2025-01-01', 'adjust': 'qfq', 'status': 'failed', 'error_type': 'ValueError', 'error': "No data returned for tickers ['000001'] from 2021-01-01 to 2025-01-01", 'wall_seconds': 0.12058529094792902} diff --git a/docs/audits/public-revalidation-20261017/provider_probe.py b/docs/audits/public-revalidation-20261017/provider_probe.py new file mode 100644 index 0000000..5d2c1ec --- /dev/null +++ b/docs/audits/public-revalidation-20261017/provider_probe.py @@ -0,0 +1,37 @@ +"""Two single-attempt AkShare probes; only run during explicit data validation. + +This is the source of the October 17 stdin probe. It does not fall back to +synthetic or another provider. Downloaded prices, if any, stay in artifacts/. +""" +import datetime +import json +from pathlib import Path +import time +import traceback + +import torch +from mlquant.data.akshare_loader import load_akshare_panel + +base = Path('artifacts/revalidation-20261017') +base.mkdir(parents=True, exist_ok=True) +rows = [] +for ticker in ('600519', '000001'): + started = datetime.datetime.now(datetime.timezone(datetime.timedelta(hours=8))).isoformat() + print('probe', ticker, started, flush=True) + timer = time.perf_counter() + row = {'ticker': ticker, 'started_at': started, 'retries': 1, + 'start': '2021-01-01', 'end': '2025-01-01', 'adjust': 'qfq'} + try: + panel = load_akshare_panel((ticker,), '2021-01-01', '2025-01-01', + device='cpu', retries=1) + panel.assert_consistent() + row.update(status='success', dates=panel.n_dates, + tradable_ratio=float(panel.mask.float().mean())) + torch.save(panel, base / f'probe-{ticker}.pt') + except Exception as exc: + row.update(status='failed', error_type=type(exc).__name__, error=str(exc)) + traceback.print_exc() + row['wall_seconds'] = time.perf_counter() - timer + rows.append(row) + (base / 'provider-probe.json').write_text(json.dumps(rows, ensure_ascii=False, indent=2) + '\n') + print(row, flush=True) diff --git a/docs/audits/public-revalidation-20261017/pytest.txt b/docs/audits/public-revalidation-20261017/pytest.txt new file mode 100644 index 0000000..82d025c --- /dev/null +++ b/docs/audits/public-revalidation-20261017/pytest.txt @@ -0,0 +1,3 @@ +........................................................................ [ 50%] +....................................................................... [100%] +143 passed in 1.68s diff --git a/docs/audits/public-revalidation-20261017/report-audit.json b/docs/audits/public-revalidation-20261017/report-audit.json new file mode 100644 index 0000000..9261e09 --- /dev/null +++ b/docs/audits/public-revalidation-20261017/report-audit.json @@ -0,0 +1,7 @@ +[ + { + "check": "report", + "message": "report passed all audit checks", + "severity": "pass" + } +] \ No newline at end of file diff --git a/docs/audits/public-revalidation-20261017/ruff.txt b/docs/audits/public-revalidation-20261017/ruff.txt new file mode 100644 index 0000000..1f5f344 --- /dev/null +++ b/docs/audits/public-revalidation-20261017/ruff.txt @@ -0,0 +1 @@ +All checks passed! diff --git a/docs/audits/public-revalidation-20261017/same-input-comparison.json b/docs/audits/public-revalidation-20261017/same-input-comparison.json new file mode 100644 index 0000000..f4d30ae --- /dev/null +++ b/docs/audits/public-revalidation-20261017/same-input-comparison.json @@ -0,0 +1,198 @@ +{ + "category": "synthetic engineering comparison; no predictive or investment evidence", + "base_commit": "ccdcf16a7c0768dbb31fa22f91ae02b692f0fd9b", + "corrected_commit": "3443987fcabdf1c46d198bfa7a878bd4e938a0a0", + "synthetic_config": { + "n_dates": 90, + "n_stocks": 12, + "seed": 3, + "halt_prob": 0 + }, + "perturbation": "zero-filled halt at date index 31, stock index 0", + "panel_hash_method": "SHA-256 over field names and C-order tensor bytes for open/high/low/close/volume/vwap/amount/mask", + "panel_sha256": "32f9a95d5114915bfd809421aa42ee1c6ff096f415c29c8e6905e610ee17cf0f", + "threads": { + "torch": 1, + "interop": 1 + }, + "costs_bps": 5, + "slippage_bps": 2, + "versions": { + "old": { + "equal_weight_before_halt": 0.0, + "equal_weight_on_halt": 0.0, + "results": [ + { + "strategy": "equal_weight", + "ann_return": 0.044287800788879395, + "ann_vol": 0.2000580368758094, + "sharpe": 0.3154204091132801, + "sortino": 0.49605755454775885, + "calmar": 0.41335142354736987, + "max_dd": 0.10714321583509445, + "turnover": 0.013108613900840282, + "cost_drag_cumulative": 0.0016333331586793065, + "cost_drag": 0.0016333331586793065, + "n_periods": 90, + "effective_costs_bps": 7, + "gross_ann_return": 0.04910576343536377, + "gross_sharpe": 0.3385479432044191, + "final_equity": 1.0155972242355347, + "bootstrap_method": "moving-block bootstrap over portfolio returns", + "bootstrap_confidence_level": 0.95, + "bootstrap_samples": 100, + "bootstrap_block_size": 20, + "bootstrap_seed": 3, + "ann_return_ci_low": -0.3830034390091896, + "ann_return_ci_high": 0.7428882211446749, + "sharpe_ci_low": -2.2082388452294492, + "sharpe_ci_high": 2.8163394303886125 + }, + { + "strategy": "momentum_20", + "ann_return": 0.02814769744873047, + "ann_vol": 0.24300782624633022, + "sharpe": 0.23418531744298599, + "sortino": 0.41518732430958727, + "calmar": 0.20326174759951832, + "max_dd": 0.1384800523519516, + "turnover": 0.15730337798595428, + "cost_drag_cumulative": 0.019600000232458115, + "cost_drag": 0.019600000232458115, + "n_periods": 90, + "info_ratio": -0.05019038310349105, + "alpha_ann": -0.013632714748382568, + "effective_costs_bps": 7, + "gross_ann_return": 0.08628427982330322, + "gross_sharpe": 0.461052619535243, + "final_equity": 1.0099631547927856, + "bootstrap_method": "moving-block bootstrap over portfolio returns", + "bootstrap_confidence_level": 0.95, + "bootstrap_samples": 100, + "bootstrap_block_size": 20, + "bootstrap_seed": 4, + "ann_return_ci_low": -0.467574043571949, + "ann_return_ci_high": 1.2360792100429523, + "sharpe_ci_low": -2.306987016567595, + "sharpe_ci_high": 3.5398160928198 + }, + { + "strategy": "alpha101_mean", + "ann_return": 0.2780005931854248, + "ann_vol": 0.21164356639263318, + "sharpe": 1.2642373075338953, + "sortino": 1.6971297623632962, + "calmar": 2.6495642655500125, + "max_dd": 0.10492313653230667, + "turnover": 0.39700376987457275, + "cost_drag_cumulative": 0.04946666583418846, + "cost_drag": 0.04946666583418846, + "n_periods": 90, + "info_ratio": 1.6055234782015948, + "alpha_ann": 0.2168973684310913, + "effective_costs_bps": 7, + "gross_ann_return": 0.4676865339279175, + "gross_sharpe": 1.920316087328909, + "final_equity": 1.0915579795837402, + "bootstrap_method": "moving-block bootstrap over portfolio returns", + "bootstrap_confidence_level": 0.95, + "bootstrap_samples": 100, + "bootstrap_block_size": 20, + "bootstrap_seed": 5, + "ann_return_ci_low": -0.3666465938091278, + "ann_return_ci_high": 1.3573405742645246, + "sharpe_ci_low": -2.053203573644286, + "sharpe_ci_high": 4.345399803502202 + } + ] + }, + "corrected": { + "equal_weight_before_halt": 0.0833333358168602, + "equal_weight_on_halt": 0.0, + "results": [ + { + "strategy": "equal_weight", + "ann_return": 0.046443939208984375, + "ann_vol": 0.19989577932009217, + "sharpe": 0.3258344908228319, + "sortino": 0.5128407960339227, + "calmar": 0.43347428989691844, + "max_dd": 0.10714346915483475, + "turnover": 0.0074906363151967525, + "cost_drag_cumulative": 0.0009333332418464124, + "cost_drag": 0.0009333332418464124, + "n_periods": 90, + "effective_costs_bps": 7, + "gross_ann_return": 0.04918050765991211, + "gross_sharpe": 0.33890119897082527, + "final_equity": 1.0163456201553345, + "bootstrap_method": "moving-block bootstrap over portfolio returns", + "bootstrap_confidence_level": 0.95, + "bootstrap_samples": 100, + "bootstrap_block_size": 20, + "bootstrap_seed": 3, + "ann_return_ci_low": -0.3829801887273788, + "ann_return_ci_high": 0.7432099372148501, + "sharpe_ci_low": -2.208111800889787, + "sharpe_ci_high": 2.823489348390047 + }, + { + "strategy": "momentum_20", + "ann_return": 0.032007575035095215, + "ann_vol": 0.2429136059818577, + "sharpe": 0.24960534412823698, + "sortino": 0.446568482603939, + "calmar": 0.2311348645752327, + "max_dd": 0.138480082154274, + "turnover": 0.15543070435523987, + "cost_drag_cumulative": 0.019366666674613953, + "cost_drag": 0.019366666674613953, + "n_periods": 90, + "info_ratio": -0.036418871007895835, + "alpha_ann": -0.011981368064880371, + "effective_costs_bps": 7, + "gross_ann_return": 0.08962297439575195, + "gross_sharpe": 0.47371477043907734, + "final_equity": 1.0113157033920288, + "bootstrap_method": "moving-block bootstrap over portfolio returns", + "bootstrap_confidence_level": 0.95, + "bootstrap_samples": 100, + "bootstrap_block_size": 20, + "bootstrap_seed": 4, + "ann_return_ci_low": -0.4667390674352646, + "ann_return_ci_high": 1.2530412435531604, + "sharpe_ci_low": -2.300664232144129, + "sharpe_ci_high": 3.5740676915040166 + }, + { + "strategy": "alpha101_mean", + "ann_return": 0.26635825634002686, + "ann_vol": 0.2113034095049847, + "sharpe": 1.2225802710498868, + "sortino": 1.6370434609559177, + "calmar": 2.5386063441725315, + "max_dd": 0.10492302477359772, + "turnover": 0.4026217460632324, + "cost_drag_cumulative": 0.05016666650772095, + "cost_drag": 0.05016666650772095, + "n_periods": 90, + "info_ratio": 1.5232065758593647, + "alpha_ann": 0.2033555507659912, + "effective_costs_bps": 7, + "gross_ann_return": 0.457183837890625, + "gross_sharpe": 1.8892848139834049, + "final_equity": 1.0879961252212524, + "bootstrap_method": "moving-block bootstrap over portfolio returns", + "bootstrap_confidence_level": 0.95, + "bootstrap_samples": 100, + "bootstrap_block_size": 20, + "bootstrap_seed": 5, + "ann_return_ci_low": -0.36923874616622926, + "ann_return_ci_high": 1.315951323509215, + "sharpe_ci_low": -2.075380070000468, + "sharpe_ci_high": 4.293933344927756 + } + ] + } + } +} diff --git a/docs/public_data_validation.md b/docs/public_data_validation.md index 3d7a391..ee808c5 100644 --- a/docs/public_data_validation.md +++ b/docs/public_data_validation.md @@ -462,6 +462,22 @@ python scripts/public_data_validation.py \ A synthetic run cannot replace a public-data report, but it confirms that the validation harness itself is working on your platform. +## Signal-Date Availability + +Portfolio selection and out-of-sample prediction use only the availability mask +on signal date `t`, after the mask-first limit rule and factor warm-up checks. +Tomorrow's availability must not decide whether a stock can be selected today. +Forward-label eligibility (`t` and `t+1` both observed) is used separately for +historical model training and trailing IC estimation. + +Reports produced through v0.4.1 used the forward-label mask for portfolio +selection and prediction too. That excluded a stock immediately before a missing +or halted next-day observation, introducing lookahead into the universe. Re-run +such reports with the corrected signal-date protocol; new metadata records it in +`availability_protocol`. Holding and execution across halts remain simplified: +this change removes tomorrow's availability from selection but does not add a +halt-aware execution simulator. + ## Auditing Reports Before sharing a validation run, audit the generated `summary.json`: diff --git a/docs/validation_akshare_csi300_20260729.md b/docs/validation_akshare_csi300_20260729.md index a965562..8787c0e 100644 --- a/docs/validation_akshare_csi300_20260729.md +++ b/docs/validation_akshare_csi300_20260729.md @@ -8,6 +8,13 @@ > The numbers are kept as recorded; re-run with v0.4.1 or later for mask-first > results. +> **Availability note (2026-10-17).** The public validation scripts through +> v0.4.1 also used next-day availability for today's portfolio selection and +> prediction. The corrected implementation separates signal-date eligibility from +> historical label eligibility. These archived numbers have not been revalidated +> under that correction. See [the protocol](public_data_validation.md#signal-date-availability) +> and [the regression and data-blocker report](audits/public-revalidation-20261017/README.md). + This report records the first full AkShare-backed public A-share validation run for `v0.2.2`. It is a reproducibility and engineering diagnostic, not a trading diff --git a/docs/validation_akshare_csi300_full_pipeline_20260729.md b/docs/validation_akshare_csi300_full_pipeline_20260729.md index 9f00d5f..97fea0a 100644 --- a/docs/validation_akshare_csi300_full_pipeline_20260729.md +++ b/docs/validation_akshare_csi300_full_pipeline_20260729.md @@ -8,6 +8,13 @@ > The numbers are kept as recorded; re-run with v0.4.1 or later for mask-first > results. +> **Availability note (2026-10-17).** The public validation scripts through +> v0.4.1 also used next-day availability for today's portfolio selection and +> prediction. The corrected implementation separates signal-date eligibility from +> historical label eligibility. These archived numbers have not been revalidated +> under that correction. See [the protocol](public_data_validation.md#signal-date-availability) +> and [the regression and data-blocker report](audits/public-revalidation-20261017/README.md). + This report records a paper-style public-data approximation on the AkShare CSI 300 universe. Unlike the lighter `v0.2.2` baseline report, this run uses the diff --git a/docs/validation_dashboard.md b/docs/validation_dashboard.md index 8342b95..07579cd 100644 --- a/docs/validation_dashboard.md +++ b/docs/validation_dashboard.md @@ -6,6 +6,13 @@ > The numbers are kept as recorded; re-run with v0.4.1 or later for mask-first > results. +> **Availability note (2026-10-17).** The public validation scripts through +> v0.4.1 also used next-day availability for today's portfolio selection and +> prediction. The corrected implementation separates signal-date eligibility from +> historical label eligibility. These archived numbers have not been revalidated +> under that correction. See [the protocol](public_data_validation.md#signal-date-availability) +> and [the regression and data-blocker report](audits/public-revalidation-20261017/README.md). + This page tracks the maintained public-data validation snapshot for `ml-quant-trading`. It is designed as a reproducibility dashboard, not a live trading scoreboard. diff --git a/scripts/akshare_csi300_full_pipeline.py b/scripts/akshare_csi300_full_pipeline.py index 1643ad8..edbb0cd 100644 --- a/scripts/akshare_csi300_full_pipeline.py +++ b/scripts/akshare_csi300_full_pipeline.py @@ -240,6 +240,7 @@ def _rolling_ic_weighted_scores( *, lookback: int, rebalance_step: int, + label_valid: np.ndarray | None = None, ) -> np.ndarray: """Build factor-composite scores from trailing IC estimates only.""" if features.shape[:2] != target.shape or target.shape != valid.shape: @@ -248,12 +249,16 @@ def _rolling_ic_weighted_scores( scores = np.full((n_dates, n_stocks), np.nan, dtype=np.float32) lookback = max(20, lookback) rebalance_step = max(1, rebalance_step) + if label_valid is None: + label_valid = valid + if label_valid.shape != valid.shape: + raise ValueError("label and signal masks must have the same shape") for t in range(n_dates): if t < 20 or t % rebalance_step != 0: continue start = max(0, t - lookback) - hist_valid = valid[start:t] + hist_valid = label_valid[start:t] if int(hist_valid.sum()) < max(100, n_features * 30): continue @@ -606,9 +611,8 @@ def run_full_pipeline(cfg: FullPipelineConfig) -> list[dict[str, float | int | s returns = _to_numpy(panel.returns) target = _forward_returns(panel) factor_panel = _limit_masked_panel(panel) - # Limit days leave the universe only on the signal date t; checking t+1 - # against the limit mask would peek at tomorrow's limit move. - valid_panel = _valid_forward_mask(panel) & _to_numpy(factor_panel.mask).astype(bool) + valid_panel = _to_numpy(factor_panel.mask).astype(bool, copy=True) + label_valid = _valid_forward_mask(panel) & valid_panel click.echo(f"stage: computing factor set {cfg.factor_set}") if cfg.factor_set == "alpha101": features, factor_valid, factor_names = _alpha101_features(factor_panel) @@ -624,7 +628,10 @@ def run_full_pipeline(cfg: FullPipelineConfig) -> list[dict[str, float | int | s valid_alpha = valid_panel & factor_valid click.echo("stage: training walk-forward MLP") - raw_preds = _fit_predict_model("mlp_alpha101", features, target, valid_alpha, validation_cfg) + raw_preds = _fit_predict_model( + "mlp_alpha101", features, target, label_valid & factor_valid, validation_cfg, + prediction_valid=valid_alpha, + ) click.echo("stage: building scores") neutralized_preds = _neutralize_score_matrix(raw_preds, valid_alpha) alpha_scores = np.nanmean(features, axis=2) @@ -635,6 +642,7 @@ def run_full_pipeline(cfg: FullPipelineConfig) -> list[dict[str, float | int | s valid_alpha, lookback=cfg.factor_ic_window, rebalance_step=cfg.rebalance_step, + label_valid=label_valid & factor_valid, ) neutralized_ic_weighted = _neutralize_score_matrix(ic_weighted_scores, valid_alpha) diff --git a/scripts/public_data_validation.py b/scripts/public_data_validation.py index 25eb300..9c12ae6 100644 --- a/scripts/public_data_validation.py +++ b/scripts/public_data_validation.py @@ -23,7 +23,7 @@ from dataclasses import dataclass from datetime import datetime, timezone from pathlib import Path -from typing import Iterable, Sequence +from typing import Sequence import click import numpy as np @@ -321,7 +321,8 @@ def _forward_returns(panel: Panel) -> np.ndarray: def _valid_forward_mask(panel: Panel, factor_mask: torch.Tensor | None = None) -> np.ndarray: - valid = _to_numpy(panel.mask).astype(bool) + # This is label eligibility, not the signal-date portfolio universe. + valid = _to_numpy(panel.mask).astype(bool, copy=True) valid[:-1] &= valid[1:] valid[-1] = False if factor_mask is not None: @@ -397,7 +398,15 @@ def _fit_predict_model( target: np.ndarray, valid: np.ndarray, cfg: ValidationConfig, + *, + prediction_valid: np.ndarray | None = None, ) -> np.ndarray: + # Training may require a known forward label. Inference must use only + # information available on the signal date, including on pre-halt days. + if prediction_valid is None: + prediction_valid = valid + if prediction_valid.shape != valid.shape: + raise ValueError("prediction and label masks must have the same shape") _seed_everything(cfg.seed) preds = np.full(target.shape, np.nan, dtype=np.float32) splits = _walk_forward_splits( @@ -409,7 +418,7 @@ def _fit_predict_model( in_dim = features.shape[-1] for split_id, (train_start, train_end, test_start, test_end) in enumerate(splits): train_valid = valid[train_start:train_end] - test_valid = valid[test_start:test_end] + test_valid = prediction_valid[test_start:test_end] x_train = features[train_start:train_end][train_valid] y_train = target[train_start:train_end][train_valid] x_test = features[test_start:test_end][test_valid] @@ -597,6 +606,10 @@ def _metadata(cfg: ValidationConfig, panel: Panel) -> dict[str, object]: "cuda_device": torch.cuda.get_device_name(0) if torch.cuda.is_available() else "none", "panel": _panel_coverage(panel), "limit_mask": LIMIT_MASK_PROTOCOL, + "availability_protocol": ( + "signal-date availability for portfolio selection and prediction; " + "next-date availability is used only for historical training labels" + ), "limitations": notes, } @@ -622,9 +635,9 @@ def run_validation(cfg: ValidationConfig) -> list[dict[str, float | int | str]]: returns = _to_numpy(panel.returns) target = _forward_returns(panel) factor_panel = _limit_masked_panel(panel) - # Limit days leave the universe only on the signal date t; checking t+1 - # against the limit mask would peek at tomorrow's limit move. - valid_panel = _valid_forward_mask(panel) & _to_numpy(factor_panel.mask).astype(bool) + # Portfolio eligibility cannot depend on tomorrow's halt or data coverage. + valid_panel = _to_numpy(factor_panel.mask).astype(bool, copy=True) + label_valid = _valid_forward_mask(panel) & valid_panel alpha_features: np.ndarray | None = None alpha_valid: np.ndarray | None = None @@ -656,7 +669,10 @@ def run_validation(cfg: ValidationConfig) -> list[dict[str, float | int | str]]: for model_name in ("mlp_alpha101", "transformer_alpha101"): if model_name in cfg.models: assert alpha_features is not None - preds = _fit_predict_model(model_name, alpha_features, target, valid_alpha, cfg) + preds = _fit_predict_model( + model_name, alpha_features, target, label_valid & alpha_valid, cfg, + prediction_valid=valid_alpha, + ) strategies[model_name] = _long_only_top_quantile_weights( preds, valid_alpha & np.isfinite(preds), diff --git a/tests/test_akshare_csi300_full_pipeline.py b/tests/test_akshare_csi300_full_pipeline.py index 1c2a4cf..bad8d80 100644 --- a/tests/test_akshare_csi300_full_pipeline.py +++ b/tests/test_akshare_csi300_full_pipeline.py @@ -184,3 +184,64 @@ def test_write_equity_curves_exports_daily_series(tmp_path): assert rows[1]["date"] == "2021-01-04" assert float(rows[-1]["equal_weight_daily_equity"]) > 1.10 assert "equal_weight_daily_return" in rows[0] + + +def test_full_pipeline_separates_signal_and_forward_label_masks(tmp_path, monkeypatch): + from dataclasses import replace + import scripts.akshare_csi300_full_pipeline as full + from mlquant.data.synthetic import SyntheticConfig, make_synthetic_panel + + panel = make_synthetic_panel(SyntheticConfig(n_dates=60, n_stocks=8, halt_prob=0, seed=3)) + mask = panel.mask.clone() + mask[31, 0] = False + panel = replace(panel, mask=mask) + cfg = full.FullPipelineConfig( + source="synthetic", preset="custom", tickers=(), start="2021-01-01", + end="2022-01-01", max_tickers=8, device="cpu", costs_bps=5, slippage_bps=0, + cost_grid_bps=(), train_window=30, test_window=10, step=10, top_quantile=0.25, + seed=3, epochs=1, batch_size=64, hidden=8, factor_set="alpha101", + covariance_window=10, optimizer_candidates=5, optimizer_weight_cap=0.25, + optimizer_risk_aversion=1, rebalance_step=1, factor_ic_window=20, + min_tradable_ratio=0.9, bootstrap_samples=0, bootstrap_block_size=10, + output_dir=tmp_path, + ) + seen = {} + + def predict(model, features, target, labels, validation_cfg, *, prediction_valid=None): + if prediction_valid is None: + prediction_valid = labels + seen["labels"] = labels.copy() + seen["signals"] = prediction_valid.copy() + return np.where(prediction_valid, features[:, :, 0], np.nan) + + def write(config, data, names, rows, sensitivity, strategies, returns): + seen["equal_weight"] = strategies["equal_weight_daily"] + + monkeypatch.setattr(full, "load_validation_panel", lambda cfg: panel) + monkeypatch.setattr(full, "_alpha101_features", lambda panel: ( + np.ones((60, 8, 3), dtype=np.float32), mask.numpy().copy(), ["a", "b", "c"], + )) + monkeypatch.setattr(full, "_fit_predict_model", predict) + monkeypatch.setattr(full, "_write_outputs", write) + full.run_full_pipeline(cfg) + + assert seen["equal_weight"][30, 0] > 0 + assert seen["equal_weight"][31, 0] == 0 + assert seen["signals"][30, 0] + assert not seen["labels"][30, 0] + + +def test_rolling_ic_ignores_missing_labels_but_scores_current_universe(): + rng = np.random.default_rng(12) + features = rng.normal(size=(80, 8, 2)).astype(np.float32) + target = features[:, :, 0] * 0.01 + signal_valid = np.ones((80, 8), dtype=bool) + label_valid = signal_valid.copy() + label_valid[20:30, 0] = False + changed = target.copy() + changed[~label_valid] = 1e8 + kwargs = dict(lookback=30, rebalance_step=1, label_valid=label_valid) + clean = _rolling_ic_weighted_scores(features, target, signal_valid, **kwargs) + poisoned = _rolling_ic_weighted_scores(features, changed, signal_valid, **kwargs) + assert np.allclose(clean, poisoned, equal_nan=True) + assert np.isfinite(clean[25, 0]) diff --git a/tests/test_public_data_validation.py b/tests/test_public_data_validation.py index 1b826ef..afad017 100644 --- a/tests/test_public_data_validation.py +++ b/tests/test_public_data_validation.py @@ -287,3 +287,70 @@ def spy(panel): assert (seen["mask"] == _limit_masked_panel(panel).mask).all() metadata = json.loads((tmp_path / "metadata.json").read_text(encoding="utf-8")) assert metadata["limit_mask"].startswith("mask-first") + + +def test_signal_weights_do_not_use_tomorrows_availability(tmp_path): + """A halt discovered tomorrow must not change today's equal-weight book.""" + from dataclasses import replace + import scripts.public_data_validation as pdv + from mlquant.data.synthetic import SyntheticConfig, make_synthetic_panel + + panel = make_synthetic_panel(SyntheticConfig(n_dates=60, n_stocks=8, halt_prob=0, seed=3)) + mask = panel.mask.clone() + mask[31, 0] = False + halted = replace(panel, mask=mask) + cfg = ValidationConfig( + source="synthetic", preset="us-large-100", tickers=(), start="2021-01-01", + end="2022-01-01", max_tickers=8, device="cpu", costs_bps=5, slippage_bps=0, + cost_grid_bps=(), bootstrap_samples=0, bootstrap_block_size=10, train_window=30, + test_window=10, step=10, top_quantile=0.25, seed=3, epochs=1, batch_size=64, + hidden=8, models=("equal_weight",), output_dir=tmp_path, synthetic_dates=60, + synthetic_stocks=8, + ) + books = [] + real = pdv._equal_weight + + def capture(valid): + weights = real(valid) + books.append(weights.copy()) + return weights + + for data in (panel, halted): + with mock.patch.object(pdv, "load_validation_panel", return_value=data), \ + mock.patch.object(pdv, "_equal_weight", side_effect=capture): + run_validation(cfg) + import numpy as np + assert np.allclose(books[0][:31], books[1][:31]) + assert books[1][30, 0] > 0 + assert books[1][31, 0] == 0 + assert bool(panel.mask[31, 0]) + assert not bool(halted.mask[31, 0]) + + +def test_model_predicts_before_halt_without_training_on_missing_label(tmp_path): + import numpy as np + import scripts.public_data_validation as pdv + + cfg = ValidationConfig( + source="synthetic", preset="us-large-100", tickers=(), start="2021-01-01", + end="2022-01-01", max_tickers=8, device="cpu", costs_bps=5, slippage_bps=0, + cost_grid_bps=(), bootstrap_samples=0, bootstrap_block_size=10, train_window=30, + test_window=10, step=10, top_quantile=0.25, seed=3, epochs=1, batch_size=64, + hidden=8, models=("mlp_alpha101",), output_dir=tmp_path, synthetic_dates=60, + synthetic_stocks=8, + ) + features = np.random.default_rng(3).normal(size=(60, 8, 3)).astype(np.float32) + target = features[:, :, 0] * 0.01 + signal_valid = np.ones((60, 8), dtype=bool) + signal_valid[31, 0] = False + label_valid = signal_valid.copy() + label_valid[:-1] &= signal_valid[1:] + label_valid[-1] = False + target[~label_valid] = np.nan + preds = pdv._fit_predict_model( + "mlp_alpha101", features, target, label_valid, cfg, + prediction_valid=signal_valid, + ) + assert np.isfinite(preds[30, 0]) + assert np.isnan(preds[31, 0]) + assert np.isfinite(preds[40, 0]) # no NaN label poisoned later training