diff --git a/CHANGELOG.md b/CHANGELOG.md index 2abf2ab..1c3eaee 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,16 @@ ## Unreleased +### Full-pipeline validation: execution coverage + +- Reports now show joint factor-mask coverage, per-split training/prediction + sample counts and skip reasons, finite predictions, and portfolio cash and + P&L exposure dates. An untrained or all-cash MLP is explicitly marked as + not evidence of model performance. +- Report audits surface coverage warnings. Full-panel metrics are preserved; + differing warm-up/exposure dates are flagged instead of presenting them as + matched-window comparisons. Synthetic reports identify their evidence class. + ### Public validation: signal-date availability - The generic and full AkShare validation scripts now select portfolios and diff --git a/docs/audits/validation-coverage-20261018/README.md b/docs/audits/validation-coverage-20261018/README.md new file mode 100644 index 0000000..06db24f --- /dev/null +++ b/docs/audits/validation-coverage-20261018/README.md @@ -0,0 +1,88 @@ +# Validation Execution Coverage — 2026-10-18 + +**A finite zero-return row can come from an MLP that never trained or invested.** +The full-pipeline report now displays factor coverage, training/prediction +samples and skip reasons, and portfolio cash/P&L exposure dates. The report +audit surfaces these warnings while preserving the failed run. This is evidence +about reporting quality, not investment performance or a paper-result update. + +The signal-date causality correction was merged in +[PR #78](https://github.com/initial-d/ml-quant-trading/pull/78), main commit +`a0b33491266d578d63c300ad3d79832d237c40db`. The new coverage implementation and +replay script are committed at `fed2db53109cc0a419b6d02d0e4db6de308b5bdd`. +See [the protocol](../../public_data_validation.md#full-pipeline-execution-coverage). + +## Synthetic engineering evidence + +[compare_coverage.py](compare_coverage.py) runs the base and current full +pipeline on the same synthetic panel: 120 dates, 20 stocks, seed 7, no random +halts, all 213 registered factors. Actual synthetic dates are 2020-01-02 through +2020-06-17; config start/end are unused by the injected synthetic loader. +It compares the actual factor-library output, model training, portfolios, +summary metrics and cost sensitivity, without network access. + +- Weights, metrics and cost sensitivity are identical before and after the + reporting change: assertions and the input fingerprint are in + [comparison.json](comparison.json). +- Joint factor-valid signal cells: **1,040 / 2,400**; forward-label cells: + **1,020**. This short panel is deliberately inadequate for full-factor MLP + training, not an estimate of real-data coverage. +- All three MLP splits are skipped. Training sample counts are **0, 240, 640**, + below the unchanged minimum of **852** (`4 * 213`). Prediction samples existed, + but no split trained, no finite prediction was produced, and all MLP portfolios + stayed cash for all 120 dates. +- The equal-weight baseline has 118 P&L exposure dates; factor-mean variants + have 51; MLP variants have zero. Full-panel metrics retain warm-up cash, and + the report explicitly warns that these are not matched-window comparisons. +- The base report's existing audit passes. The new report has eight coverage + warnings. Neither an audit pass nor these synthetic metrics establish alpha. + +The generated [synthetic report](synthetic-summary.md) and +[raw report JSON](synthetic-summary.json) keep metrics beside the coverage +warnings. [coverage-run.txt](coverage-run.txt) preserves execution output. +Do not compare these results with the archived July market backtests. + +## Verification and environment + +- Full test suite: **146 passed**, [pytest.txt](pytest.txt). Tests cover empty + factor masks and a normally trained MLP, with the existing causality tests. +- Ruff: passed, [ruff.txt](ruff.txt); `git diff --check`: passed. +- A regression test against the base fails because coverage is absent: + [old-diagnostics-test.txt](old-diagnostics-test.txt). +- Exact commands: [commands.txt](commands.txt). Environment: + [environment.json](environment.json), [pip-freeze.txt](pip-freeze.txt). + Python 3.12.14, PyTorch 2.14.1, AkShare 1.19.1, macOS 26.6.2 arm64, + Apple M5, 10 logical CPUs, 32 GiB RAM. CPU only; CUDA unavailable; MPS unused. + Replay torch/interop/OMP/BLAS threads: 1. Costs: 5 bps plus 2 bps slippage; + sensitivity: 0/7/15/30 effective bps. No bootstrap inference. +- Integrity: [checksums.txt](checksums.txt). Workspace paths in copied logs + and reports are redacted to ``; original local artifacts remain under + `artifacts/revalidation-20261018/`. Input market prices are not published. + +## Public-data access failure, kept separately + +The fixed probe from [the October 17 plan](../public-revalidation-20261017/plan.md) +was retried once per symbol on October 18 after merging the causality fix. +It again failed for both symbols, so the planned expanded run did not start. + +| Symbol | Started, Asia/Shanghai | Outcome | +|---|---|---| +| `600519` | 2026-10-18 09:03:10 | Upstream disconnect; no data returned | +| `000001` | 2026-10-18 09:03:11 | Upstream disconnect; no data returned | + +[provider-probe.json](provider-probe.json) records exact timestamps, durations, +source dates 2021-01-01 through 2025-01-01, daily `qfq`, and `retries=1`. +[provider-probe.txt](provider-probe.txt) preserves warnings and tracebacks; +[provider_probe.py](provider_probe.py) preserves the executed probe source, +copied from the October 17 probe with only its output directory changed. +Probe code commit: `a0b33491266d578d63c300ad3d79832d237c40db`. + +The observed warning is `RemoteDisconnected('Remote end closed connection +without response')`; the loader raises `ValueError: No data returned`. +The underlying cause is unknown. No panel, coverage audit, market backtest, +new Sharpe result or proprietary paper experiment exists for this attempt. +Costs are not applicable to download-only probes. Synthetic evidence above +does not replace the blocked public-data run. + +Coverage diagnostics reveal execution gaps; they do not repair evaluation +windows, add exchange execution modeling, or revalidate archived results. diff --git a/docs/audits/validation-coverage-20261018/checksums.txt b/docs/audits/validation-coverage-20261018/checksums.txt new file mode 100644 index 0000000..b4f18c5 --- /dev/null +++ b/docs/audits/validation-coverage-20261018/checksums.txt @@ -0,0 +1,15 @@ +ed4a9a8c1570c1ccc774af58e86aaf90e4c01350cd6487bb4d5df141dfbe568e README.md +51a2df710fce70f402c91f005194dcc2e9a0da514052af63f160c09fc43db610 commands.txt +608810bccca2aabb610af178ec37624e21dc0813e25544b5e1ac4e1a014d0eb4 compare_coverage.py +3ab5e3d788b0956c1aa34929c2f3ebcd0a34708b170d643cfdd6ff4a954e7fab comparison.json +e01ddb90e0c5eaa5449f14adcd09a5ffd0907cc6c33ca19d24e5d13dc3c88dc7 coverage-run.txt +5f096df220beae8aa7281ada240170262d8ee9afb139704f318718d0253842e6 environment.json +c4d7fcde962eab8f50ee80fbd51b92843f99a02f53185c0a8bd5880b0bc158cb old-diagnostics-test.txt +d0aa7915427de1673fe56a597d3dbbb9406367cdcfe59e3de98d8f64d67878e8 pip-freeze.txt +a063f25fb7b6a28496aaf173e847ef170ece6eb1a2d6aa26a5d599f8888fa644 provider-probe.json +a11f5232bbfb873fbefbbe9b12e40d7cfe4ab04c30e9b232daee81ee1e64d369 provider-probe.txt +a8ecb4221d14b7cf80fbcfedcc4ab39167e244ae630bf3b1f22906c3fced95a9 provider_probe.py +14f6b19eb6bccf83ac43c98eb02812c3608a99f61e490dd31423543313ac06cd pytest.txt +82b3e6a6c090a57601d22943bd23fca9218d1031dbe5a7b754092f9a156b4f18 ruff.txt +4bbabeda17e90941097d69b797b832c7e8756cdbefd7b8b201be8c732472bc96 synthetic-summary.json +bd69e1a2c625815b53c4ef4d39dfb0e4ddb9060230af4520dd126f0c6c2f1153 synthetic-summary.md diff --git a/docs/audits/validation-coverage-20261018/commands.txt b/docs/audits/validation-coverage-20261018/commands.txt new file mode 100644 index 0000000..8b82489 --- /dev/null +++ b/docs/audits/validation-coverage-20261018/commands.txt @@ -0,0 +1,20 @@ +# Run from repository root with the editable .venv described in environment.json. +# Old-code regression, before the implementation patch (base main plus new test): +OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 .venv/bin/python -m pytest tests/test_akshare_csi300_full_pipeline.py::test_full_pipeline_reports_skipped_training_and_empty_portfolios -q + +# Candidate code; byte-identical implementation to fed2db53109cc0a419b6d02d0e4db6de308b5bdd: +OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 .venv/bin/python -m pytest tests/test_akshare_csi300_full_pipeline.py tests/test_public_data_validation.py tests/test_audit_validation_report.py -q +OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 .venv/bin/python -m pytest +.venv/bin/python -m ruff check src tests scripts/public_data_validation.py scripts/akshare_csi300_full_pipeline.py scripts/audit_validation_report.py docs/audits/validation-coverage-20261018/compare_coverage.py +git diff --check + +# Clean recorded code commit; no network, synthetic only: +OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 .venv/bin/python docs/audits/validation-coverage-20261018/compare_coverage.py + +# Download-only probe source is preserved as provider_probe.py. Executed before +# the diagnostics patch at a0b33491266d578d63c300ad3d79832d237c40db: +PYTHONPATH=src .venv/bin/python artifacts/revalidation-20261018/provider_probe.py + +# Replay loads the base runner/writer and model helper with: +git show a0b33491266d578d63c300ad3d79832d237c40db:scripts/public_data_validation.py +git show a0b33491266d578d63c300ad3d79832d237c40db:scripts/akshare_csi300_full_pipeline.py diff --git a/docs/audits/validation-coverage-20261018/compare_coverage.py b/docs/audits/validation-coverage-20261018/compare_coverage.py new file mode 100644 index 0000000..9ab063b --- /dev/null +++ b/docs/audits/validation-coverage-20261018/compare_coverage.py @@ -0,0 +1,102 @@ +"""Offline report comparison on one real factor-library synthetic input. + +Run from the repository root with OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1. +No external data is requested. Synthetic outputs are engineering evidence only. +""" +from dataclasses import replace +import hashlib +import json +from pathlib import Path +import platform +import subprocess +import sys +import types + +import numpy as np +import torch + +ROOT = Path(__file__).resolve().parents[3] +sys.path.insert(0, str(ROOT)) +sys.path.insert(0, str(ROOT / "src")) +from mlquant.data.synthetic import SyntheticConfig, make_synthetic_panel +import scripts.akshare_csi300_full_pipeline as current +from scripts.audit_validation_report import audit_report, findings_to_dicts + +BASE = "a0b33491266d578d63c300ad3d79832d237c40db" +OUT = ROOT / "artifacts/revalidation-20261018/coverage" +OUT.mkdir(parents=True, exist_ok=True) +torch.set_num_threads(1) +torch.set_num_interop_threads(1) + + +def archived_module(name, path): + source = subprocess.check_output(["git", "show", f"{BASE}:{path}"], cwd=ROOT, text=True) + module = types.ModuleType(name) + module.__file__ = str(ROOT / path) + sys.modules[name] = module + exec(compile(source, module.__file__, "exec"), module.__dict__) + return module + + +old_helpers = archived_module("coverage_base_helpers", "scripts/public_data_validation.py") +old = archived_module("coverage_base_full", "scripts/akshare_csi300_full_pipeline.py") +old._fit_predict_model = old_helpers._fit_predict_model +panel = make_synthetic_panel(SyntheticConfig(n_dates=120, n_stocks=20, halt_prob=0, seed=7)) +panel.assert_consistent() +fingerprint = hashlib.sha256() +for name in ("open", "high", "low", "close", "volume", "vwap", "amount", "mask"): + fingerprint.update(name.encode()) + fingerprint.update(getattr(panel, name).numpy().tobytes()) + +cfg = current.FullPipelineConfig( + source="synthetic", preset="custom", tickers=(), start="2021-01-01", end="2022-01-01", + max_tickers=20, device="cpu", costs_bps=5, slippage_bps=2, cost_grid_bps=(0, 7, 15, 30), + train_window=60, test_window=20, step=20, top_quantile=0.2, seed=7, epochs=1, + batch_size=128, hidden=16, factor_set="all", covariance_window=20, + optimizer_candidates=10, optimizer_weight_cap=0.1, optimizer_risk_aversion=1, + rebalance_step=1, factor_ic_window=30, min_tradable_ratio=0.9, + bootstrap_samples=0, bootstrap_block_size=20, output_dir=OUT, + command=(sys.executable, str(Path(__file__).relative_to(ROOT))), +) +books = {} +reports = {} +for label, module in (("base", old), ("current", current)): + module.load_validation_panel = lambda config: panel + writer = module._write_outputs + + def capture(config, data, names, rows, sensitivity, strategies, returns, + _label=label, _writer=writer, **kwargs): + books[_label] = {name: weights.copy() for name, weights in strategies.items()} + _writer(config, data, names, rows, sensitivity, strategies, returns, **kwargs) + + module._write_outputs = capture + module.run_full_pipeline(replace(cfg, output_dir=OUT / label)) + reports[label] = json.loads((OUT / label / "summary.json").read_text()) + +assert reports["base"]["results"] == reports["current"]["results"] +assert reports["base"]["cost_sensitivity"] == reports["current"]["cost_sensitivity"] +assert all(np.array_equal(books["base"][name], weights) for name, weights in books["current"].items()) +diagnostics = reports["current"]["diagnostics"] +assert diagnostics["model"]["trained_splits"] == 0 +assert diagnostics["strategies"]["mlp_top20_raw_daily"]["invested_dates"] == 0 +comparison = { + "category": "synthetic engineering evidence; no investment or predictive evidence", + "base_commit": BASE, + "current_commit": subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=ROOT, text=True).strip(), + "python": platform.python_version(), "pytorch": torch.__version__, "platform": platform.platform(), + "device": "cpu", "torch_threads": 1, "torch_interop_threads": 1, + "synthetic_config": {"n_dates": 120, "n_stocks": 20, "halt_prob": 0, "seed": 7}, + "panel_sha256": fingerprint.hexdigest(), + "panel_hash_method": "SHA-256 of field names and C-order bytes: open/high/low/close/volume/vwap/amount/mask", + "weights_identical": True, "metrics_identical": True, "cost_sensitivity_identical": True, + "costs_bps": 5, "slippage_bps": 2, + "diagnostics": diagnostics, + "audit": {label: findings_to_dicts(audit_report(report)) for label, report in reports.items()}, + "limitations": [ + "Short synthetic panel deliberately lacks enough joint-factor training samples; not a market experiment.", + "Base runner/writer and model helper loaded from recorded Git commit; other unchanged helpers are shared.", + "No bootstrap inference; full-panel metrics retain warm-up cash and differing exposure windows.", + ], +} +(OUT / "comparison.json").write_text(json.dumps(comparison, indent=2) + "\n") +print(json.dumps({"identical_weights_and_metrics": True, "diagnostics": diagnostics}, indent=2)) diff --git a/docs/audits/validation-coverage-20261018/comparison.json b/docs/audits/validation-coverage-20261018/comparison.json new file mode 100644 index 0000000..8949238 --- /dev/null +++ b/docs/audits/validation-coverage-20261018/comparison.json @@ -0,0 +1,201 @@ +{ + "category": "synthetic engineering evidence; no investment or predictive evidence", + "base_commit": "a0b33491266d578d63c300ad3d79832d237c40db", + "current_commit": "fed2db53109cc0a419b6d02d0e4db6de308b5bdd", + "python": "3.12.14", + "pytorch": "2.14.1", + "platform": "macOS-26.6.2-arm64-arm-64bit", + "device": "cpu", + "torch_threads": 1, + "torch_interop_threads": 1, + "synthetic_config": { + "n_dates": 120, + "n_stocks": 20, + "halt_prob": 0, + "seed": 7 + }, + "panel_sha256": "6c09274ae546aa1cfeae4397392d33f19ae02025d62eb64ffb224a1f60f2bc6f", + "panel_hash_method": "SHA-256 of field names and C-order bytes: open/high/low/close/volume/vwap/amount/mask", + "weights_identical": true, + "metrics_identical": true, + "cost_sensitivity_identical": true, + "costs_bps": 5, + "slippage_bps": 2, + "diagnostics": { + "factor_label_cells": 1020, + "factor_signal_cells": 1040, + "factor_signal_ratio": 0.43333333333333335, + "limit_eligible_cells": 2380, + "metric_window": "full_panel_including_warmup_cash", + "model": { + "finite_prediction_cells": 0, + "min_train_samples": 852, + "prediction_dates": 0, + "skipped_splits": 3, + "splits": [ + { + "status": "insufficient_training_samples", + "test_end_exclusive": 80, + "test_samples": 240, + "test_start": 60, + "train_end_exclusive": 60, + "train_samples": 0, + "train_start": 0 + }, + { + "status": "insufficient_training_samples", + "test_end_exclusive": 100, + "test_samples": 400, + "test_start": 80, + "train_end_exclusive": 80, + "train_samples": 240, + "train_start": 20 + }, + { + "status": "insufficient_training_samples", + "test_end_exclusive": 119, + "test_samples": 380, + "test_start": 100, + "train_end_exclusive": 100, + "train_samples": 640, + "train_start": 40 + } + ], + "total_splits": 3, + "trained_splits": 0 + }, + "panel_cells": 2400, + "raw_tradable_cells": 2400, + "strategies": { + "equal_weight_daily": { + "cash_dates": 1, + "first_pnl_exposure_date": "2020-01-06T00:00:00.000000", + "invested_dates": 119, + "last_pnl_exposure_date": "2020-06-17T00:00:00.000000", + "pnl_exposure_dates": 118 + }, + "factor_ic_optimized_daily": { + "cash_dates": 120, + "first_pnl_exposure_date": null, + "invested_dates": 0, + "last_pnl_exposure_date": null, + "pnl_exposure_dates": 0 + }, + "factor_ic_weighted_daily": { + "cash_dates": 120, + "first_pnl_exposure_date": null, + "invested_dates": 0, + "last_pnl_exposure_date": null, + "pnl_exposure_dates": 0 + }, + "factor_mean_buffered_daily": { + "cash_dates": 68, + "first_pnl_exposure_date": "2020-04-08T00:00:00.000000", + "invested_dates": 52, + "last_pnl_exposure_date": "2020-06-17T00:00:00.000000", + "pnl_exposure_dates": 51 + }, + "factor_mean_daily": { + "cash_dates": 68, + "first_pnl_exposure_date": "2020-04-08T00:00:00.000000", + "invested_dates": 52, + "last_pnl_exposure_date": "2020-06-17T00:00:00.000000", + "pnl_exposure_dates": 51 + }, + "factor_mean_optimized_daily": { + "cash_dates": 68, + "first_pnl_exposure_date": "2020-04-08T00:00:00.000000", + "invested_dates": 52, + "last_pnl_exposure_date": "2020-06-17T00:00:00.000000", + "pnl_exposure_dates": 51 + }, + "mlp_optimized_daily": { + "cash_dates": 120, + "first_pnl_exposure_date": null, + "invested_dates": 0, + "last_pnl_exposure_date": null, + "pnl_exposure_dates": 0 + }, + "mlp_top20_neutralized_daily": { + "cash_dates": 120, + "first_pnl_exposure_date": null, + "invested_dates": 0, + "last_pnl_exposure_date": null, + "pnl_exposure_dates": 0 + }, + "mlp_top20_raw_daily": { + "cash_dates": 120, + "first_pnl_exposure_date": null, + "invested_dates": 0, + "last_pnl_exposure_date": null, + "pnl_exposure_dates": 0 + } + }, + "warnings": [ + "No walk-forward split was trained; zero-return MLP rows are not evidence of model performance.", + "The model produced no finite predictions.", + "factor_ic_weighted_daily: no invested dates; an all-cash row is not evidence of model performance.", + "factor_ic_optimized_daily: no invested dates; an all-cash row is not evidence of model performance.", + "mlp_top20_raw_daily: no invested dates; an all-cash row is not evidence of model performance.", + "mlp_top20_neutralized_daily: no invested dates; an all-cash row is not evidence of model performance.", + "mlp_optimized_daily: no invested dates; an all-cash row is not evidence of model performance.", + "Strategies have different P&L exposure dates. Full-panel metrics include warm-up cash and are not matched-window comparisons." + ] + }, + "audit": { + "base": [ + { + "severity": "pass", + "check": "report", + "message": "report passed all audit checks" + } + ], + "current": [ + { + "severity": "warning", + "check": "execution_coverage", + "message": "No walk-forward split was trained; zero-return MLP rows are not evidence of model performance." + }, + { + "severity": "warning", + "check": "execution_coverage", + "message": "The model produced no finite predictions." + }, + { + "severity": "warning", + "check": "execution_coverage", + "message": "factor_ic_weighted_daily: no invested dates; an all-cash row is not evidence of model performance." + }, + { + "severity": "warning", + "check": "execution_coverage", + "message": "factor_ic_optimized_daily: no invested dates; an all-cash row is not evidence of model performance." + }, + { + "severity": "warning", + "check": "execution_coverage", + "message": "mlp_top20_raw_daily: no invested dates; an all-cash row is not evidence of model performance." + }, + { + "severity": "warning", + "check": "execution_coverage", + "message": "mlp_top20_neutralized_daily: no invested dates; an all-cash row is not evidence of model performance." + }, + { + "severity": "warning", + "check": "execution_coverage", + "message": "mlp_optimized_daily: no invested dates; an all-cash row is not evidence of model performance." + }, + { + "severity": "warning", + "check": "execution_coverage", + "message": "Strategies have different P&L exposure dates. Full-panel metrics include warm-up cash and are not matched-window comparisons." + } + ] + }, + "limitations": [ + "Short synthetic panel deliberately lacks enough joint-factor training samples; not a market experiment.", + "Base runner/writer and model helper loaded from recorded Git commit; other unchanged helpers are shared.", + "No bootstrap inference; full-panel metrics retain warm-up cash and differing exposure windows." + ] +} diff --git a/docs/audits/validation-coverage-20261018/coverage-run.txt b/docs/audits/validation-coverage-20261018/coverage-run.txt new file mode 100644 index 0000000..5fa724f --- /dev/null +++ b/docs/audits/validation-coverage-20261018/coverage-run.txt @@ -0,0 +1,140 @@ +stage: loading AkShare panel +stage: panel loaded (120 dates x 20 stocks) +stage: computing factor set all +stage: computed 213 factors +stage: training walk-forward MLP +stage: building scores +stage: building daily portfolios +stage: writing reports +stage: loading AkShare panel +stage: panel loaded (120 dates x 20 stocks) +stage: computing factor set all +stage: computed 213 factors +stage: training walk-forward MLP +stage: building scores +stage: building daily portfolios +stage: writing reports +{ + "identical_weights_and_metrics": true, + "diagnostics": { + "factor_label_cells": 1020, + "factor_signal_cells": 1040, + "factor_signal_ratio": 0.43333333333333335, + "limit_eligible_cells": 2380, + "metric_window": "full_panel_including_warmup_cash", + "model": { + "finite_prediction_cells": 0, + "min_train_samples": 852, + "prediction_dates": 0, + "skipped_splits": 3, + "splits": [ + { + "status": "insufficient_training_samples", + "test_end_exclusive": 80, + "test_samples": 240, + "test_start": 60, + "train_end_exclusive": 60, + "train_samples": 0, + "train_start": 0 + }, + { + "status": "insufficient_training_samples", + "test_end_exclusive": 100, + "test_samples": 400, + "test_start": 80, + "train_end_exclusive": 80, + "train_samples": 240, + "train_start": 20 + }, + { + "status": "insufficient_training_samples", + "test_end_exclusive": 119, + "test_samples": 380, + "test_start": 100, + "train_end_exclusive": 100, + "train_samples": 640, + "train_start": 40 + } + ], + "total_splits": 3, + "trained_splits": 0 + }, + "panel_cells": 2400, + "raw_tradable_cells": 2400, + "strategies": { + "equal_weight_daily": { + "cash_dates": 1, + "first_pnl_exposure_date": "2020-01-06T00:00:00.000000", + "invested_dates": 119, + "last_pnl_exposure_date": "2020-06-17T00:00:00.000000", + "pnl_exposure_dates": 118 + }, + "factor_ic_optimized_daily": { + "cash_dates": 120, + "first_pnl_exposure_date": null, + "invested_dates": 0, + "last_pnl_exposure_date": null, + "pnl_exposure_dates": 0 + }, + "factor_ic_weighted_daily": { + "cash_dates": 120, + "first_pnl_exposure_date": null, + "invested_dates": 0, + "last_pnl_exposure_date": null, + "pnl_exposure_dates": 0 + }, + "factor_mean_buffered_daily": { + "cash_dates": 68, + "first_pnl_exposure_date": "2020-04-08T00:00:00.000000", + "invested_dates": 52, + "last_pnl_exposure_date": "2020-06-17T00:00:00.000000", + "pnl_exposure_dates": 51 + }, + "factor_mean_daily": { + "cash_dates": 68, + "first_pnl_exposure_date": "2020-04-08T00:00:00.000000", + "invested_dates": 52, + "last_pnl_exposure_date": "2020-06-17T00:00:00.000000", + "pnl_exposure_dates": 51 + }, + "factor_mean_optimized_daily": { + "cash_dates": 68, + "first_pnl_exposure_date": "2020-04-08T00:00:00.000000", + "invested_dates": 52, + "last_pnl_exposure_date": "2020-06-17T00:00:00.000000", + "pnl_exposure_dates": 51 + }, + "mlp_optimized_daily": { + "cash_dates": 120, + "first_pnl_exposure_date": null, + "invested_dates": 0, + "last_pnl_exposure_date": null, + "pnl_exposure_dates": 0 + }, + "mlp_top20_neutralized_daily": { + "cash_dates": 120, + "first_pnl_exposure_date": null, + "invested_dates": 0, + "last_pnl_exposure_date": null, + "pnl_exposure_dates": 0 + }, + "mlp_top20_raw_daily": { + "cash_dates": 120, + "first_pnl_exposure_date": null, + "invested_dates": 0, + "last_pnl_exposure_date": null, + "pnl_exposure_dates": 0 + } + }, + "warnings": [ + "No walk-forward split was trained; zero-return MLP rows are not evidence of model performance.", + "The model produced no finite predictions.", + "factor_ic_weighted_daily: no invested dates; an all-cash row is not evidence of model performance.", + "factor_ic_optimized_daily: no invested dates; an all-cash row is not evidence of model performance.", + "mlp_top20_raw_daily: no invested dates; an all-cash row is not evidence of model performance.", + "mlp_top20_neutralized_daily: no invested dates; an all-cash row is not evidence of model performance.", + "mlp_optimized_daily: no invested dates; an all-cash row is not evidence of model performance.", + "Strategies have different P&L exposure dates. Full-panel metrics include warm-up cash and are not matched-window comparisons." + ] + } +} diff --git a/docs/audits/validation-coverage-20261018/environment.json b/docs/audits/validation-coverage-20261018/environment.json new file mode 100644 index 0000000..c452026 --- /dev/null +++ b/docs/audits/validation-coverage-20261018/environment.json @@ -0,0 +1,35 @@ +{ + "code_commit": "fed2db53109cc0a419b6d02d0e4db6de308b5bdd", + "provider_probe_commit": "a0b33491266d578d63c300ad3d79832d237c40db", + "python": "3.12.14 (main, Aug 25 2026, 13:50:33) [Clang 22.1.3 ]", + "pytorch": "2.14.1", + "akshare": "1.19.1", + "platform": "macOS-26.6.2-arm64-arm-64bit", + "machine": "arm64", + "cpu": "Apple M5", + "logical_cpus": 10, + "memory_bytes": 34359738368, + "device": "cpu", + "cuda_available": false, + "mps_available": true, + "comparison_threads": { + "OMP_NUM_THREADS": 1, + "OPENBLAS_NUM_THREADS": 1, + "torch": 1, + "interop": 1 + }, + "pytest_threads_note": "OMP/BLAS fixed at 1; individual tests may change torch pools; no timing comparison claimed.", + "evidence_classes": { + "comparison": "synthetic engineering", + "provider_probes": "public-data access failure; no panel" + }, + "costs_bps": 5, + "slippage_bps": 2, + "provider_probe_costs": "not applicable; download only", + "limitations": [ + "No public-data panel or new market performance result.", + "No proprietary paper experiment.", + "Comparison preserves full-panel warm-up cash and differing exposure windows." + ], + "working_tree_note": "Tests/lint passed on candidate code identical to recorded code commit; replay used clean committed tree." +} diff --git a/docs/audits/validation-coverage-20261018/old-diagnostics-test.txt b/docs/audits/validation-coverage-20261018/old-diagnostics-test.txt new file mode 100644 index 0000000..5005269 --- /dev/null +++ b/docs/audits/validation-coverage-20261018/old-diagnostics-test.txt @@ -0,0 +1,46 @@ +F [100%] +=================================== FAILURES =================================== +_______ test_full_pipeline_reports_skipped_training_and_empty_portfolios _______ + +tmp_path = PosixPath('/private/var/folders/ld/dfm826ld4_d6myjvpl4f8_c80000gn/T/pytest-of-duyimin/pytest-11/test_full_pipeline_reports_ski0') +monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x11307f590> + + def test_full_pipeline_reports_skipped_training_and_empty_portfolios(tmp_path, monkeypatch): + """A no-label model must be reported as skipped, not as zero-return skill.""" + import json + import scripts.akshare_csi300_full_pipeline as full + from mlquant.data.synthetic import SyntheticConfig, make_synthetic_panel + + panel = make_synthetic_panel(SyntheticConfig(n_dates=60, n_stocks=8, halt_prob=0, seed=3)) + cfg = full.FullPipelineConfig( + source="synthetic", preset="custom", tickers=(), start="2021-01-01", + end="2022-01-01", max_tickers=8, device="cpu", costs_bps=5, slippage_bps=2, + cost_grid_bps=(), train_window=30, test_window=10, step=10, top_quantile=0.25, + seed=3, epochs=1, batch_size=64, hidden=8, factor_set="alpha101", + covariance_window=10, optimizer_candidates=5, optimizer_weight_cap=0.25, + optimizer_risk_aversion=1, rebalance_step=1, factor_ic_window=20, + min_tradable_ratio=0.9, bootstrap_samples=0, bootstrap_block_size=10, + output_dir=tmp_path, + ) + monkeypatch.setattr(full, "load_validation_panel", lambda cfg: panel) + monkeypatch.setattr(full, "_alpha101_features", lambda panel: ( + np.ones((60, 8, 3), dtype=np.float32), np.zeros((60, 8), dtype=bool), ["a", "b", "c"], + )) + full.run_full_pipeline(cfg) + report = json.loads((tmp_path / "summary.json").read_text()) +> diagnostics = report["diagnostics"] + ^^^^^^^^^^^^^^^^^^^^^ +E KeyError: 'diagnostics' + +tests/test_akshare_csi300_full_pipeline.py:273: KeyError +----------------------------- Captured stdout call ----------------------------- +stage: loading AkShare panel +stage: panel loaded (60 dates x 8 stocks) +stage: computing factor set alpha101 +stage: computed 3 factors +stage: training walk-forward MLP +stage: building scores +stage: building daily portfolios +stage: writing reports +=========================== short test summary info ============================ +FAILED tests/test_akshare_csi300_full_pipeline.py::test_full_pipeline_reports_skipped_training_and_empty_portfolios diff --git a/docs/audits/validation-coverage-20261018/pip-freeze.txt b/docs/audits/validation-coverage-20261018/pip-freeze.txt new file mode 100644 index 0000000..737b49b --- /dev/null +++ b/docs/audits/validation-coverage-20261018/pip-freeze.txt @@ -0,0 +1,78 @@ +akshare==1.19.1 +ast_serialize==0.13.1 +baostock==0.9.4 +beautifulsoup4==4.15.0 +certifi==2026.7.22 +cffi==2.1.1 +charset-normalizer==3.5.2 +clarabel==0.11.1 +click==8.5.0 +cloudpickle==3.1.2 +coverage==7.16.2 +curl_cffi==0.16.3 +cvxpy==1.9.3 +decorator==5.3.1 +et_xmlfile==2.0.0 +filelock==4.2.0 +fsspec==2026.9.0 +highspy==1.15.1 +html5lib==1.1 +hypothesis==6.170.1 +idna==3.20 +iniconfig==2.3.1 +Jinja2==3.1.6 +joblib==1.6.0 +jsonpath==0.82.2 +librt==0.16.0 +lxml==6.1.3 +markdown-it-py==4.2.0 +MarkupSafe==3.0.4 +mdurl==0.1.2 +mini-racer==0.14.1 +-e git+https://github.com/initial-d/ml-quant-trading.git@fed2db53109cc0a419b6d02d0e4db6de308b5bdd#egg=mlquantx&subdirectory=../../../../../../../../.codex/worktrees/visibility-refresh/ml-quant-trading +mpmath==1.3.0 +multitasking==0.0.13 +mypy==2.4.0 +mypy_extensions==1.1.0 +narwhals==2.27.1 +networkx==3.7 +numpy==2.5.4 +openpyxl==3.1.5 +osqp==1.1.3 +packaging==26.3 +pandas==3.0.6 +pathspec==1.1.1 +peewee==4.5.3 +platformdirs==4.13.3 +pluggy==1.7.0 +protobuf==7.36.2 +pycparser==3.11 +Pygments==2.21.0 +pytest==9.1.1 +pytest-cov==7.1.0 +python-dateutil==2.9.0.post0 +pytz==2026.5 +PyYAML==6.0.3 +qdldl==0.1.9.post1 +requests==2.34.2 +rich==15.0.0 +ruff==0.17.1 +scikit-learn==1.9.1 +scipy==1.18.1 +scs==3.3.1 +setuptools==84.0.0 +six==1.17.0 +sortedcontainers==2.4.0 +soupsieve==3.0.3 +sparsediffpy==0.6.1 +sympy==1.14.0 +tabulate==0.10.0 +threadpoolctl==3.7.0 +torch==2.14.1 +tqdm==4.70.1 +typing_extensions==4.16.0 +urllib3==2.8.0 +webencodings==0.6.1 +websockets==17.2 +xlrd==2.0.2 +yfinance==1.7.0 diff --git a/docs/audits/validation-coverage-20261018/provider-probe.json b/docs/audits/validation-coverage-20261018/provider-probe.json new file mode 100644 index 0000000..5a7996b --- /dev/null +++ b/docs/audits/validation-coverage-20261018/provider-probe.json @@ -0,0 +1,26 @@ +[ + { + "ticker": "600519", + "started_at": "2026-10-18T09:03:10.963270+08:00", + "retries": 1, + "start": "2021-01-01", + "end": "2025-01-01", + "adjust": "qfq", + "status": "failed", + "error_type": "ValueError", + "error": "No data returned for tickers ['600519'] from 2021-01-01 to 2025-01-01", + "wall_seconds": 0.3465902078896761 + }, + { + "ticker": "000001", + "started_at": "2026-10-18T09:03:11.310246+08:00", + "retries": 1, + "start": "2021-01-01", + "end": "2025-01-01", + "adjust": "qfq", + "status": "failed", + "error_type": "ValueError", + "error": "No data returned for tickers ['000001'] from 2021-01-01 to 2025-01-01", + "wall_seconds": 0.1371590420603752 + } +] diff --git a/docs/audits/validation-coverage-20261018/provider-probe.txt b/docs/audits/validation-coverage-20261018/provider-probe.txt new file mode 100644 index 0000000..d3d8820 --- /dev/null +++ b/docs/audits/validation-coverage-20261018/provider-probe.txt @@ -0,0 +1,22 @@ +probe 600519 2026-10-18T09:03:10.963270+08:00 +AkShare download progress: 1/1 (600519) +Warning: Failed to fetch 600519: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) +Traceback (most recent call last): + File "/artifacts/revalidation-20261018/provider_probe.py", line 25, in + panel = load_akshare_panel((ticker,), '2021-01-01', '2025-01-01', + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/src/mlquant/data/akshare_loader.py", line 109, in load_akshare_panel + raise ValueError( +ValueError: No data returned for tickers ['600519'] from 2021-01-01 to 2025-01-01 +{'ticker': '600519', 'started_at': '2026-10-18T09:03:10.963270+08:00', 'retries': 1, 'start': '2021-01-01', 'end': '2025-01-01', 'adjust': 'qfq', 'status': 'failed', 'error_type': 'ValueError', 'error': "No data returned for tickers ['600519'] from 2021-01-01 to 2025-01-01", 'wall_seconds': 0.3465902078896761} +probe 000001 2026-10-18T09:03:11.310246+08:00 +AkShare download progress: 1/1 (000001) +Warning: Failed to fetch 000001: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) +Traceback (most recent call last): + File "/artifacts/revalidation-20261018/provider_probe.py", line 25, in + panel = load_akshare_panel((ticker,), '2021-01-01', '2025-01-01', + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/src/mlquant/data/akshare_loader.py", line 109, in load_akshare_panel + raise ValueError( +ValueError: No data returned for tickers ['000001'] from 2021-01-01 to 2025-01-01 +{'ticker': '000001', 'started_at': '2026-10-18T09:03:11.310246+08:00', 'retries': 1, 'start': '2021-01-01', 'end': '2025-01-01', 'adjust': 'qfq', 'status': 'failed', 'error_type': 'ValueError', 'error': "No data returned for tickers ['000001'] from 2021-01-01 to 2025-01-01", 'wall_seconds': 0.1371590420603752} diff --git a/docs/audits/validation-coverage-20261018/provider_probe.py b/docs/audits/validation-coverage-20261018/provider_probe.py new file mode 100644 index 0000000..78dae40 --- /dev/null +++ b/docs/audits/validation-coverage-20261018/provider_probe.py @@ -0,0 +1,37 @@ +"""Two single-attempt AkShare probes; only run during explicit data validation. + +This is the source of the October 17 stdin probe. It does not fall back to +synthetic or another provider. Downloaded prices, if any, stay in artifacts/. +""" +import datetime +import json +from pathlib import Path +import time +import traceback + +import torch +from mlquant.data.akshare_loader import load_akshare_panel + +base = Path('artifacts/revalidation-20261018') +base.mkdir(parents=True, exist_ok=True) +rows = [] +for ticker in ('600519', '000001'): + started = datetime.datetime.now(datetime.timezone(datetime.timedelta(hours=8))).isoformat() + print('probe', ticker, started, flush=True) + timer = time.perf_counter() + row = {'ticker': ticker, 'started_at': started, 'retries': 1, + 'start': '2021-01-01', 'end': '2025-01-01', 'adjust': 'qfq'} + try: + panel = load_akshare_panel((ticker,), '2021-01-01', '2025-01-01', + device='cpu', retries=1) + panel.assert_consistent() + row.update(status='success', dates=panel.n_dates, + tradable_ratio=float(panel.mask.float().mean())) + torch.save(panel, base / f'probe-{ticker}.pt') + except Exception as exc: + row.update(status='failed', error_type=type(exc).__name__, error=str(exc)) + traceback.print_exc() + row['wall_seconds'] = time.perf_counter() - timer + rows.append(row) + (base / 'provider-probe.json').write_text(json.dumps(rows, ensure_ascii=False, indent=2) + '\n') + print(row, flush=True) diff --git a/docs/audits/validation-coverage-20261018/pytest.txt b/docs/audits/validation-coverage-20261018/pytest.txt new file mode 100644 index 0000000..64ef5f0 --- /dev/null +++ b/docs/audits/validation-coverage-20261018/pytest.txt @@ -0,0 +1,4 @@ +........................................................................ [ 49%] +........................................................................ [ 98%] +.. [100%] +146 passed in 2.09s diff --git a/docs/audits/validation-coverage-20261018/ruff.txt b/docs/audits/validation-coverage-20261018/ruff.txt new file mode 100644 index 0000000..1f5f344 --- /dev/null +++ b/docs/audits/validation-coverage-20261018/ruff.txt @@ -0,0 +1 @@ +All checks passed! diff --git a/docs/audits/validation-coverage-20261018/synthetic-summary.json b/docs/audits/validation-coverage-20261018/synthetic-summary.json new file mode 100644 index 0000000..28a07d4 --- /dev/null +++ b/docs/audits/validation-coverage-20261018/synthetic-summary.json @@ -0,0 +1,1095 @@ +{ + "cost_sensitivity": [ + { + "ann_return": -0.3677975535392761, + "ann_vol": 0.17149189563420442, + "calmar": -1.7537800542736213, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "cost_scenario": "0.00_bps", + "effective_costs_bps": 0, + "final_equity": 0.8038399815559387, + "gross_ann_return": -0.3677975535392761, + "gross_sharpe": -2.5862080858194, + "max_dd": 0.2097170352935791, + "n_periods": 120, + "sharpe": -2.5862080858194, + "sortino": -4.068807664432775, + "strategy": "equal_weight_daily", + "turnover": 0.004201680887490511 + }, + { + "alpha_ann": 0.8402873277664185, + "ann_return": 0.18931734561920166, + "ann_vol": 0.10196878349203697, + "calmar": 4.431076019441128, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "cost_scenario": "0.00_bps", + "effective_costs_bps": 0, + "final_equity": 1.086065649986267, + "gross_ann_return": 0.18931734561920166, + "gross_sharpe": 1.7512184922770142, + "info_ratio": 4.093098840227367, + "max_dd": 0.042724914848804474, + "n_periods": 120, + "sharpe": 1.7512184922770142, + "sortino": 2.3824957426716633, + "strategy": "factor_mean_daily", + "turnover": 0.2142857164144516 + }, + { + "alpha_ann": 0.6428650617599487, + "ann_return": 0.0606304407119751, + "ann_vol": 0.11185735030894661, + "calmar": 1.4033032899426896, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "cost_scenario": "0.00_bps", + "effective_costs_bps": 0, + "final_equity": 1.0284267663955688, + "gross_ann_return": 0.0606304407119751, + "gross_sharpe": 0.5815132202712182, + "info_ratio": 3.3168830581819853, + "max_dd": 0.043205514550209045, + "n_periods": 120, + "sharpe": 0.5815132202712182, + "sortino": 0.8303638265206976, + "strategy": "factor_mean_buffered_daily", + "turnover": 0.10504201799631119 + }, + { + "alpha_ann": 0.5350992679595947, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "cost_scenario": "0.00_bps", + "effective_costs_bps": 0, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.5862080858194, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "factor_ic_weighted_daily", + "turnover": 0.0 + }, + { + "alpha_ann": 0.5139510631561279, + "ann_return": -0.02314436435699463, + "ann_vol": 0.10030827884518878, + "calmar": -0.4169373202968613, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "cost_scenario": "0.00_bps", + "effective_costs_bps": 0, + "final_equity": 0.9889112710952759, + "gross_ann_return": -0.02314436435699463, + "gross_sharpe": -0.18373875590112781, + "info_ratio": 2.9853100461517146, + "max_dd": 0.05551041662693024, + "n_periods": 120, + "sharpe": -0.18373875590112781, + "sortino": -0.2003691426384479, + "strategy": "factor_mean_optimized_daily", + "turnover": 0.14705882966518402 + }, + { + "alpha_ann": 0.5350992679595947, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "cost_scenario": "0.00_bps", + "effective_costs_bps": 0, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.5862080858194, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "factor_ic_optimized_daily", + "turnover": 0.0 + }, + { + "alpha_ann": 0.5350992679595947, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "cost_scenario": "0.00_bps", + "effective_costs_bps": 0, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.5862080858194, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "mlp_top20_raw_daily", + "turnover": 0.0 + }, + { + "alpha_ann": 0.5350992679595947, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "cost_scenario": "0.00_bps", + "effective_costs_bps": 0, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.5862080858194, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "mlp_top20_neutralized_daily", + "turnover": 0.0 + }, + { + "alpha_ann": 0.5350992679595947, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "cost_scenario": "0.00_bps", + "effective_costs_bps": 0, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.5862080858194, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "mlp_optimized_daily", + "turnover": 0.0 + }, + { + "ann_return": -0.36872631311416626, + "ann_vol": 0.1714796985969865, + "calmar": -1.7535845834705788, + "cost_drag": 0.000699999975040555, + "cost_drag_cumulative": 0.000699999975040555, + "cost_scenario": "7.00_bps", + "effective_costs_bps": 7, + "final_equity": 0.8032774329185486, + "gross_ann_return": -0.3677975535392761, + "gross_sharpe": -2.5862080858194, + "max_dd": 0.21027004718780518, + "n_periods": 120, + "sharpe": -2.594964405367442, + "sortino": -4.065047960974997, + "strategy": "equal_weight_daily", + "turnover": 0.004201680887490511 + }, + { + "alpha_ann": 0.7099497318267822, + "ann_return": 0.10343921184539795, + "ann_vol": 0.10208844012321669, + "calmar": 2.227343295650437, + "cost_drag": 0.0356999970972538, + "cost_drag_cumulative": 0.0356999970972538, + "cost_scenario": "7.00_bps", + "effective_costs_bps": 7, + "final_equity": 1.0479881763458252, + "gross_ann_return": 0.18931734561920166, + "gross_sharpe": 1.7512184922770142, + "info_ratio": 3.5897342994073815, + "max_dd": 0.046440623700618744, + "n_periods": 120, + "sharpe": 1.014802603652409, + "sortino": 1.3435274522572755, + "strategy": "factor_mean_daily", + "turnover": 0.2142857164144516 + }, + { + "alpha_ann": 0.585809588432312, + "ann_return": 0.02229022979736328, + "ann_vol": 0.11249457527821735, + "calmar": 0.45888473491770504, + "cost_drag": 0.017500000074505806, + "cost_drag_cumulative": 0.017500000074505806, + "cost_scenario": "7.00_bps", + "effective_costs_bps": 7, + "final_equity": 1.0105531215667725, + "gross_ann_return": 0.0606304407119751, + "gross_sharpe": 0.5815132202712182, + "info_ratio": 3.06896806078499, + "max_dd": 0.048574790358543396, + "n_periods": 120, + "sharpe": 0.25153679302062976, + "sortino": 0.3550296739305559, + "strategy": "factor_mean_buffered_daily", + "turnover": 0.10504201799631119 + }, + { + "alpha_ann": 0.5373574495315552, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "cost_scenario": "7.00_bps", + "effective_costs_bps": 7, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.594964405367442, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "factor_ic_weighted_daily", + "turnover": 0.0 + }, + { + "alpha_ann": 0.4401404857635498, + "ann_return": -0.07218611240386963, + "ann_vol": 0.10075428383945753, + "calmar": -1.0902054686620908, + "cost_drag": 0.02449999935925007, + "cost_drag_cumulative": 0.02449999935925007, + "cost_scenario": "7.00_bps", + "effective_costs_bps": 7, + "final_equity": 0.9649507999420166, + "gross_ann_return": -0.02314436435699463, + "gross_sharpe": -0.18373875590112781, + "info_ratio": 2.623477973682566, + "max_dd": 0.06621330976486206, + "n_periods": 120, + "sharpe": -0.6935738619259407, + "sortino": -0.7586057440630053, + "strategy": "factor_mean_optimized_daily", + "turnover": 0.14705882966518402 + }, + { + "alpha_ann": 0.5373574495315552, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "cost_scenario": "7.00_bps", + "effective_costs_bps": 7, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.594964405367442, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "factor_ic_optimized_daily", + "turnover": 0.0 + }, + { + "alpha_ann": 0.5373574495315552, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "cost_scenario": "7.00_bps", + "effective_costs_bps": 7, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.594964405367442, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "mlp_top20_raw_daily", + "turnover": 0.0 + }, + { + "alpha_ann": 0.5373574495315552, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "cost_scenario": "7.00_bps", + "effective_costs_bps": 7, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.594964405367442, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "mlp_top20_neutralized_daily", + "turnover": 0.0 + }, + { + "alpha_ann": 0.5373574495315552, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "cost_scenario": "7.00_bps", + "effective_costs_bps": 7, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.594964405367442, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "mlp_optimized_daily", + "turnover": 0.0 + }, + { + "ann_return": -0.3697879910469055, + "ann_vol": 0.17147309002045752, + "calmar": -1.7533573444435588, + "cost_drag": 0.001500000013038516, + "cost_drag_cumulative": 0.001500000013038516, + "cost_scenario": "15.00_bps", + "effective_costs_bps": 15, + "final_equity": 0.8026338219642639, + "gross_ann_return": -0.3677975535392761, + "gross_sharpe": -2.5862080858194, + "max_dd": 0.21090281009674072, + "n_periods": 120, + "sharpe": -2.6048620719133306, + "sortino": -4.088767162253362, + "strategy": "equal_weight_daily", + "turnover": 0.004201680887490511 + }, + { + "alpha_ann": 0.5721330642700195, + "ann_return": 0.012786030769348145, + "ann_vol": 0.10269407846216488, + "calmar": 0.2082441209307298, + "cost_drag": 0.07649999856948853, + "cost_drag_cumulative": 0.07649999856948853, + "cost_scenario": "15.00_bps", + "effective_costs_bps": 15, + "final_equity": 1.0060683488845825, + "gross_ann_return": 0.18931734561920166, + "gross_sharpe": 1.7512184922770142, + "info_ratio": 3.0148568587542157, + "max_dd": 0.06139924004673958, + "n_periods": 120, + "sharpe": 0.1744951166672956, + "sortino": 0.22132341603024094, + "strategy": "factor_mean_daily", + "turnover": 0.2142857164144516 + }, + { + "alpha_ann": 0.5229805707931519, + "ann_return": -0.01985412836074829, + "ann_vol": 0.11336750614368853, + "calmar": -0.3631147673339024, + "cost_drag": 0.03750000149011612, + "cost_drag_cumulative": 0.03750000149011612, + "cost_scenario": "15.00_bps", + "effective_costs_bps": 15, + "final_equity": 0.9904959797859192, + "gross_ann_return": 0.0606304407119751, + "gross_sharpe": 0.5815132202712182, + "info_ratio": 2.7870704914272286, + "max_dd": 0.05467728152871132, + "n_periods": 120, + "sharpe": -0.12087629794079335, + "sortino": -0.16565159939136148, + "strategy": "factor_mean_buffered_daily", + "turnover": 0.10504201799631119 + }, + { + "alpha_ann": 0.5399385690689087, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "cost_scenario": "15.00_bps", + "effective_costs_bps": 15, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.6048620719133306, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "factor_ic_weighted_daily", + "turnover": 0.0 + }, + { + "alpha_ann": 0.3601268529891968, + "ann_return": -0.12525629997253418, + "ann_vol": 0.10146126107732727, + "calmar": -1.4714880855058088, + "cost_drag": 0.05250000208616257, + "cost_drag_cumulative": 0.05250000208616257, + "cost_scenario": "15.00_bps", + "effective_costs_bps": 15, + "final_equity": 0.9382621645927429, + "gross_ann_return": -0.02314436435699463, + "gross_sharpe": -0.18373875590112781, + "info_ratio": 2.2112444908856963, + "max_dd": 0.08512219786643982, + "n_periods": 120, + "sharpe": -1.268272590111361, + "sortino": -1.389917826239091, + "strategy": "factor_mean_optimized_daily", + "turnover": 0.14705882966518402 + }, + { + "alpha_ann": 0.5399385690689087, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "cost_scenario": "15.00_bps", + "effective_costs_bps": 15, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.6048620719133306, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "factor_ic_optimized_daily", + "turnover": 0.0 + }, + { + "alpha_ann": 0.5399385690689087, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "cost_scenario": "15.00_bps", + "effective_costs_bps": 15, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.6048620719133306, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "mlp_top20_raw_daily", + "turnover": 0.0 + }, + { + "alpha_ann": 0.5399385690689087, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "cost_scenario": "15.00_bps", + "effective_costs_bps": 15, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.6048620719133306, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "mlp_top20_neutralized_daily", + "turnover": 0.0 + }, + { + "alpha_ann": 0.5399385690689087, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "cost_scenario": "15.00_bps", + "effective_costs_bps": 15, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.6048620719133306, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "mlp_optimized_daily", + "turnover": 0.0 + }, + { + "ann_return": -0.37177467346191406, + "ann_vol": 0.17148179796581672, + "calmar": -1.7529236265447192, + "cost_drag": 0.003000000026077032, + "cost_drag_cumulative": 0.003000000026077032, + "cost_scenario": "30.00_bps", + "effective_costs_bps": 30, + "final_equity": 0.801427960395813, + "gross_ann_return": -0.3677975535392761, + "gross_sharpe": -2.5862080858194, + "max_dd": 0.21208834648132324, + "n_periods": 120, + "sharpe": -2.623099259373021, + "sortino": -4.131320836207907, + "strategy": "equal_weight_daily", + "turnover": 0.004201680887490511 + }, + { + "alpha_ann": 0.3426694869995117, + "ann_return": -0.13778698444366455, + "ann_vol": 0.10514257824258488, + "calmar": -1.3309281787891998, + "cost_drag": 0.15299999713897705, + "cost_drag_cumulative": 0.15299999713897705, + "cost_scenario": "30.00_bps", + "effective_costs_bps": 30, + "final_equity": 0.9318376779556274, + "gross_ann_return": 0.18931734561920166, + "gross_sharpe": 1.7512184922770142, + "info_ratio": 1.9511071455705136, + "max_dd": 0.10352698713541031, + "n_periods": 120, + "sharpe": -1.357493754733313, + "sortino": -1.7298763678859788, + "strategy": "factor_mean_daily", + "turnover": 0.2142857164144516 + }, + { + "alpha_ann": 0.411712646484375, + "ann_return": -0.09431183338165283, + "ann_vol": 0.11540787088236548, + "calmar": -1.2092760920180625, + "cost_drag": 0.07500000298023224, + "cost_drag_cumulative": 0.07500000298023224, + "cost_scenario": "30.00_bps", + "effective_costs_bps": 30, + "final_equity": 0.953923761844635, + "gross_ann_return": 0.0606304407119751, + "gross_sharpe": 0.5815132202712182, + "info_ratio": 2.2648440382933943, + "max_dd": 0.07799032330513, + "n_periods": 120, + "sharpe": -0.801101877570457, + "sortino": -1.086566126727915, + "strategy": "factor_mean_buffered_daily", + "turnover": 0.10504201799631119 + }, + { + "alpha_ann": 0.5447870492935181, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "cost_scenario": "30.00_bps", + "effective_costs_bps": 30, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.623099259373021, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "factor_ic_weighted_daily", + "turnover": 0.0 + }, + { + "alpha_ann": 0.22177624702453613, + "ann_return": -0.2167910933494568, + "ann_vol": 0.10333829639944056, + "calmar": -1.6881959182119977, + "cost_drag": 0.10500000417232513, + "cost_drag_cumulative": 0.10500000417232513, + "cost_scenario": "30.00_bps", + "effective_costs_bps": 30, + "final_equity": 0.8901547789573669, + "gross_ann_return": -0.02314436435699463, + "gross_sharpe": -0.18373875590112781, + "info_ratio": 1.4466498684901026, + "max_dd": 0.12841583788394928, + "n_periods": 120, + "sharpe": -2.3121199319615187, + "sortino": -2.5871642152629852, + "strategy": "factor_mean_optimized_daily", + "turnover": 0.14705882966518402 + }, + { + "alpha_ann": 0.5447870492935181, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "cost_scenario": "30.00_bps", + "effective_costs_bps": 30, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.623099259373021, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "factor_ic_optimized_daily", + "turnover": 0.0 + }, + { + "alpha_ann": 0.5447870492935181, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "cost_scenario": "30.00_bps", + "effective_costs_bps": 30, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.623099259373021, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "mlp_top20_raw_daily", + "turnover": 0.0 + }, + { + "alpha_ann": 0.5447870492935181, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "cost_scenario": "30.00_bps", + "effective_costs_bps": 30, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.623099259373021, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "mlp_top20_neutralized_daily", + "turnover": 0.0 + }, + { + "alpha_ann": 0.5447870492935181, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "cost_scenario": "30.00_bps", + "effective_costs_bps": 30, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.623099259373021, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "mlp_optimized_daily", + "turnover": 0.0 + } + ], + "diagnostics": { + "factor_label_cells": 1020, + "factor_signal_cells": 1040, + "factor_signal_ratio": 0.43333333333333335, + "limit_eligible_cells": 2380, + "metric_window": "full_panel_including_warmup_cash", + "model": { + "finite_prediction_cells": 0, + "min_train_samples": 852, + "prediction_dates": 0, + "skipped_splits": 3, + "splits": [ + { + "status": "insufficient_training_samples", + "test_end_exclusive": 80, + "test_samples": 240, + "test_start": 60, + "train_end_exclusive": 60, + "train_samples": 0, + "train_start": 0 + }, + { + "status": "insufficient_training_samples", + "test_end_exclusive": 100, + "test_samples": 400, + "test_start": 80, + "train_end_exclusive": 80, + "train_samples": 240, + "train_start": 20 + }, + { + "status": "insufficient_training_samples", + "test_end_exclusive": 119, + "test_samples": 380, + "test_start": 100, + "train_end_exclusive": 100, + "train_samples": 640, + "train_start": 40 + } + ], + "total_splits": 3, + "trained_splits": 0 + }, + "panel_cells": 2400, + "raw_tradable_cells": 2400, + "strategies": { + "equal_weight_daily": { + "cash_dates": 1, + "first_pnl_exposure_date": "2020-01-06T00:00:00.000000", + "invested_dates": 119, + "last_pnl_exposure_date": "2020-06-17T00:00:00.000000", + "pnl_exposure_dates": 118 + }, + "factor_ic_optimized_daily": { + "cash_dates": 120, + "first_pnl_exposure_date": null, + "invested_dates": 0, + "last_pnl_exposure_date": null, + "pnl_exposure_dates": 0 + }, + "factor_ic_weighted_daily": { + "cash_dates": 120, + "first_pnl_exposure_date": null, + "invested_dates": 0, + "last_pnl_exposure_date": null, + "pnl_exposure_dates": 0 + }, + "factor_mean_buffered_daily": { + "cash_dates": 68, + "first_pnl_exposure_date": "2020-04-08T00:00:00.000000", + "invested_dates": 52, + "last_pnl_exposure_date": "2020-06-17T00:00:00.000000", + "pnl_exposure_dates": 51 + }, + "factor_mean_daily": { + "cash_dates": 68, + "first_pnl_exposure_date": "2020-04-08T00:00:00.000000", + "invested_dates": 52, + "last_pnl_exposure_date": "2020-06-17T00:00:00.000000", + "pnl_exposure_dates": 51 + }, + "factor_mean_optimized_daily": { + "cash_dates": 68, + "first_pnl_exposure_date": "2020-04-08T00:00:00.000000", + "invested_dates": 52, + "last_pnl_exposure_date": "2020-06-17T00:00:00.000000", + "pnl_exposure_dates": 51 + }, + "mlp_optimized_daily": { + "cash_dates": 120, + "first_pnl_exposure_date": null, + "invested_dates": 0, + "last_pnl_exposure_date": null, + "pnl_exposure_dates": 0 + }, + "mlp_top20_neutralized_daily": { + "cash_dates": 120, + "first_pnl_exposure_date": null, + "invested_dates": 0, + "last_pnl_exposure_date": null, + "pnl_exposure_dates": 0 + }, + "mlp_top20_raw_daily": { + "cash_dates": 120, + "first_pnl_exposure_date": null, + "invested_dates": 0, + "last_pnl_exposure_date": null, + "pnl_exposure_dates": 0 + } + }, + "warnings": [ + "No walk-forward split was trained; zero-return MLP rows are not evidence of model performance.", + "The model produced no finite predictions.", + "factor_ic_weighted_daily: no invested dates; an all-cash row is not evidence of model performance.", + "factor_ic_optimized_daily: no invested dates; an all-cash row is not evidence of model performance.", + "mlp_top20_raw_daily: no invested dates; an all-cash row is not evidence of model performance.", + "mlp_top20_neutralized_daily: no invested dates; an all-cash row is not evidence of model performance.", + "mlp_optimized_daily: no invested dates; an all-cash row is not evidence of model performance.", + "Strategies have different P&L exposure dates. Full-panel metrics include warm-up cash and are not matched-window comparisons." + ] + }, + "metadata": { + "availability_protocol": "signal-date availability for portfolio selection and prediction; next-date availability is used only for historical training labels", + "batch_size": 128, + "bootstrap_block_size": 20, + "bootstrap_confidence_level": null, + "bootstrap_method": "disabled", + "bootstrap_samples": 0, + "bootstrap_seed_base": null, + "command": "/.venv/bin/python docs/audits/validation-coverage-20261018/compare_coverage.py", + "cost_grid_bps": [ + 0, + 7, + 15, + 30 + ], + "costs_bps": 5, + "cuda_available": false, + "cuda_device": "none", + "cuda_version": "unavailable", + "effective_costs_bps": 7, + "end": "2022-01-01", + "epochs": 1, + "generated_at_utc": "2026-10-18T01:09:29+00:00", + "hidden": 16, + "limit_mask": "mask-first: A-share limit-move days (exchange limit prices when present, otherwise per-board return proxy) are removed from the panel mask before factors are computed and from the tradable universe; P&L uses unmasked returns", + "limitations": [ + "This full-pipeline run uses cross-sectional score neutralization only; it does not claim industry or size neutrality." + ], + "max_tickers": 20, + "models": [ + "mlp_alpha101" + ], + "panel": { + "date_end": "2020-06-17T00:00:00.000000", + "date_start": "2020-01-02T00:00:00.000000", + "n_dates": 120, + "n_stocks": 20, + "stocks_with_any_data": 20, + "stocks_with_no_data": [], + "stocks_with_partial_data_count": 0, + "stocks_with_partial_data_sample": [], + "total_cells": 2400, + "tradable_cells": 2400, + "tradable_ratio": 1.0 + }, + "pipeline": { + "covariance_window": 20, + "factor_ic_window": 30, + "factor_set": "all", + "feature_count": 213, + "min_tradable_ratio": 0.9, + "model": "mlp_alpha101 implementation trained on selected factor matrix", + "neutralization": "cross_sectional_zscore_on_model_scores", + "neutralization_limit": "no industry or size neutralization because public AkShare run does not provide those exposures", + "optimizer_candidates": 10, + "optimizer_risk_aversion": 1, + "optimizer_weight_cap": 0.1, + "portfolio_optimizer": "LedoitWolf diagonal risk-scaled long-only mean-variance weights on top scored candidates", + "rebalance_step": 1, + "stages": [ + "akshare_current_csi300_universe", + "ohlcv_panel", + "mask_first_limit_move_mask", + "selected_factor_set_features", + "walk_forward_mlp_training", + "cross_sectional_score_neutralization", + "daily_or_configured_top_quantile_selection", + "daily_or_configured_optimized_candidate_weights", + "cost_aware_backtest" + ] + }, + "platform": "macOS-26.6.2-arm64-arm-64bit", + "preset": "custom", + "python": "3.12.14", + "pytorch": "2.14.1", + "seed": 7, + "slippage_bps": 2, + "source": "synthetic", + "start": "2021-01-01", + "step": 20, + "synthetic_dates": 260, + "synthetic_stocks": 80, + "test_window": 20, + "tickers": [], + "top_quantile": 0.2, + "train_window": 60 + }, + "results": [ + { + "ann_return": -0.36872631311416626, + "ann_vol": 0.1714796985969865, + "calmar": -1.7535845834705788, + "cost_drag": 0.000699999975040555, + "cost_drag_cumulative": 0.000699999975040555, + "effective_costs_bps": 7, + "final_equity": 0.8032774329185486, + "gross_ann_return": -0.3677975535392761, + "gross_sharpe": -2.5862080858194, + "max_dd": 0.21027004718780518, + "n_periods": 120, + "sharpe": -2.594964405367442, + "sortino": -4.065047960974997, + "strategy": "equal_weight_daily", + "turnover": 0.004201680887490511 + }, + { + "alpha_ann": 0.7099497318267822, + "ann_return": 0.10343921184539795, + "ann_vol": 0.10208844012321669, + "calmar": 2.227343295650437, + "cost_drag": 0.0356999970972538, + "cost_drag_cumulative": 0.0356999970972538, + "effective_costs_bps": 7, + "final_equity": 1.0479881763458252, + "gross_ann_return": 0.18931734561920166, + "gross_sharpe": 1.7512184922770142, + "info_ratio": 3.5897342994073815, + "max_dd": 0.046440623700618744, + "n_periods": 120, + "sharpe": 1.014802603652409, + "sortino": 1.3435274522572755, + "strategy": "factor_mean_daily", + "turnover": 0.2142857164144516 + }, + { + "alpha_ann": 0.585809588432312, + "ann_return": 0.02229022979736328, + "ann_vol": 0.11249457527821735, + "calmar": 0.45888473491770504, + "cost_drag": 0.017500000074505806, + "cost_drag_cumulative": 0.017500000074505806, + "effective_costs_bps": 7, + "final_equity": 1.0105531215667725, + "gross_ann_return": 0.0606304407119751, + "gross_sharpe": 0.5815132202712182, + "info_ratio": 3.06896806078499, + "max_dd": 0.048574790358543396, + "n_periods": 120, + "sharpe": 0.25153679302062976, + "sortino": 0.3550296739305559, + "strategy": "factor_mean_buffered_daily", + "turnover": 0.10504201799631119 + }, + { + "alpha_ann": 0.5373574495315552, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "effective_costs_bps": 7, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.594964405367442, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "factor_ic_weighted_daily", + "turnover": 0.0 + }, + { + "alpha_ann": 0.4401404857635498, + "ann_return": -0.07218611240386963, + "ann_vol": 0.10075428383945753, + "calmar": -1.0902054686620908, + "cost_drag": 0.02449999935925007, + "cost_drag_cumulative": 0.02449999935925007, + "effective_costs_bps": 7, + "final_equity": 0.9649507999420166, + "gross_ann_return": -0.02314436435699463, + "gross_sharpe": -0.18373875590112781, + "info_ratio": 2.623477973682566, + "max_dd": 0.06621330976486206, + "n_periods": 120, + "sharpe": -0.6935738619259407, + "sortino": -0.7586057440630053, + "strategy": "factor_mean_optimized_daily", + "turnover": 0.14705882966518402 + }, + { + "alpha_ann": 0.5373574495315552, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "effective_costs_bps": 7, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.594964405367442, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "factor_ic_optimized_daily", + "turnover": 0.0 + }, + { + "alpha_ann": 0.5373574495315552, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "effective_costs_bps": 7, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.594964405367442, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "mlp_top20_raw_daily", + "turnover": 0.0 + }, + { + "alpha_ann": 0.5373574495315552, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "effective_costs_bps": 7, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.594964405367442, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "mlp_top20_neutralized_daily", + "turnover": 0.0 + }, + { + "alpha_ann": 0.5373574495315552, + "ann_return": 0.0, + "ann_vol": 0.0, + "calmar": null, + "cost_drag": 0.0, + "cost_drag_cumulative": 0.0, + "effective_costs_bps": 7, + "final_equity": 1.0, + "gross_ann_return": 0.0, + "gross_sharpe": 0.0, + "info_ratio": 2.594964405367442, + "max_dd": 0.0, + "n_periods": 120, + "sharpe": 0.0, + "sortino": null, + "strategy": "mlp_optimized_daily", + "turnover": 0.0 + } + ] +} \ No newline at end of file diff --git a/docs/audits/validation-coverage-20261018/synthetic-summary.md b/docs/audits/validation-coverage-20261018/synthetic-summary.md new file mode 100644 index 0000000..72356e7 --- /dev/null +++ b/docs/audits/validation-coverage-20261018/synthetic-summary.md @@ -0,0 +1,111 @@ +# AkShare CSI 300 Full-Pipeline Validation Summary + +This report is a synthetic engineering diagnostic, not a trading claim. + +| Field | Value | +|---|---| +| Source | synthetic | +| Preset | custom | +| Dates x stocks | 120 x 20 | +| Date range | 2020-01-02T00:00:00.000000 to 2020-06-17T00:00:00.000000 | +| Tradable ratio | 1.0000 | +| Model | mlp_alpha101 implementation trained on selected factor matrix | +| Factor set | all | +| Features | 213 factors | +| Neutralization | cross_sectional_zscore_on_model_scores | +| Factor IC window | 30 trading days | +| Portfolio optimizer | LedoitWolf diagonal risk-scaled long-only mean-variance weights on top scored candidates | +| Optimizer candidates / cap | 10 / 0.1000 | +| Rebalance | daily | +| Costs + slippage | 5.00 + 2.00 bps | +| Python / PyTorch | 3.12.14 / 2.14.1 | + +## Execution Coverage + +Factor-valid signal cells: 1040 / 2400; historical label cells: 1020. + +MLP splits trained / planned: 0 / 3; finite prediction cells: 0. Per-split sample counts and skip reasons are in `summary.json`. + +| Strategy | Invested dates | Cash dates | P&L exposure dates | First / last P&L exposure | +|---|---:|---:|---:|---| +| equal_weight_daily | 119 | 1 | 118 | 2020-01-06T00:00:00.000000 / 2020-06-17T00:00:00.000000 | +| factor_mean_daily | 52 | 68 | 51 | 2020-04-08T00:00:00.000000 / 2020-06-17T00:00:00.000000 | +| factor_mean_buffered_daily | 52 | 68 | 51 | 2020-04-08T00:00:00.000000 / 2020-06-17T00:00:00.000000 | +| factor_ic_weighted_daily | 0 | 120 | 0 | None / None | +| factor_mean_optimized_daily | 52 | 68 | 51 | 2020-04-08T00:00:00.000000 / 2020-06-17T00:00:00.000000 | +| factor_ic_optimized_daily | 0 | 120 | 0 | None / None | +| mlp_top20_raw_daily | 0 | 120 | 0 | None / None | +| mlp_top20_neutralized_daily | 0 | 120 | 0 | None / None | +| mlp_optimized_daily | 0 | 120 | 0 | None / None | + +- **Coverage warning:** No walk-forward split was trained; zero-return MLP rows are not evidence of model performance. +- **Coverage warning:** The model produced no finite predictions. +- **Coverage warning:** factor_ic_weighted_daily: no invested dates; an all-cash row is not evidence of model performance. +- **Coverage warning:** factor_ic_optimized_daily: no invested dates; an all-cash row is not evidence of model performance. +- **Coverage warning:** mlp_top20_raw_daily: no invested dates; an all-cash row is not evidence of model performance. +- **Coverage warning:** mlp_top20_neutralized_daily: no invested dates; an all-cash row is not evidence of model performance. +- **Coverage warning:** mlp_optimized_daily: no invested dates; an all-cash row is not evidence of model performance. +- **Coverage warning:** Strategies have different P&L exposure dates. Full-panel metrics include warm-up cash and are not matched-window comparisons. + +Full-panel metrics include warm-up cash. P&L exposure uses the previous date's weights; cash dates can still incur exit costs. + +| strategy | ann_return | ann_vol | sharpe | max_dd | turnover | cost_drag_cumulative | gross_ann_return | gross_sharpe | info_ratio | alpha_ann | final_equity | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| equal_weight_daily | -0.3687 | 0.1715 | -2.5950 | 0.2103 | 0.0042 | 0.0007 | -0.3678 | -2.5862 | | | 0.8033 | +| factor_mean_daily | 0.1034 | 0.1021 | 1.0148 | 0.0464 | 0.2143 | 0.0357 | 0.1893 | 1.7512 | 3.5897 | 0.7099 | 1.0480 | +| factor_mean_buffered_daily | 0.0223 | 0.1125 | 0.2515 | 0.0486 | 0.1050 | 0.0175 | 0.0606 | 0.5815 | 3.0690 | 0.5858 | 1.0106 | +| factor_ic_weighted_daily | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 2.5950 | 0.5374 | 1.0000 | +| factor_mean_optimized_daily | -0.0722 | 0.1008 | -0.6936 | 0.0662 | 0.1471 | 0.0245 | -0.0231 | -0.1837 | 2.6235 | 0.4401 | 0.9650 | +| factor_ic_optimized_daily | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 2.5950 | 0.5374 | 1.0000 | +| mlp_top20_raw_daily | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 2.5950 | 0.5374 | 1.0000 | +| mlp_top20_neutralized_daily | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 2.5950 | 0.5374 | 1.0000 | +| mlp_optimized_daily | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 2.5950 | 0.5374 | 1.0000 | + +## Cost Sensitivity + +| cost_scenario | strategy | effective_costs_bps | ann_return | sharpe | max_dd | turnover | cost_drag_cumulative | final_equity | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 0.00_bps | equal_weight_daily | 0 | -0.3678 | -2.5862 | 0.2097 | 0.0042 | 0.0000 | 0.8038 | +| 0.00_bps | factor_mean_daily | 0 | 0.1893 | 1.7512 | 0.0427 | 0.2143 | 0.0000 | 1.0861 | +| 0.00_bps | factor_mean_buffered_daily | 0 | 0.0606 | 0.5815 | 0.0432 | 0.1050 | 0.0000 | 1.0284 | +| 0.00_bps | factor_ic_weighted_daily | 0 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | +| 0.00_bps | factor_mean_optimized_daily | 0 | -0.0231 | -0.1837 | 0.0555 | 0.1471 | 0.0000 | 0.9889 | +| 0.00_bps | factor_ic_optimized_daily | 0 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | +| 0.00_bps | mlp_top20_raw_daily | 0 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | +| 0.00_bps | mlp_top20_neutralized_daily | 0 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | +| 0.00_bps | mlp_optimized_daily | 0 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | +| 7.00_bps | equal_weight_daily | 7 | -0.3687 | -2.5950 | 0.2103 | 0.0042 | 0.0007 | 0.8033 | +| 7.00_bps | factor_mean_daily | 7 | 0.1034 | 1.0148 | 0.0464 | 0.2143 | 0.0357 | 1.0480 | +| 7.00_bps | factor_mean_buffered_daily | 7 | 0.0223 | 0.2515 | 0.0486 | 0.1050 | 0.0175 | 1.0106 | +| 7.00_bps | factor_ic_weighted_daily | 7 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | +| 7.00_bps | factor_mean_optimized_daily | 7 | -0.0722 | -0.6936 | 0.0662 | 0.1471 | 0.0245 | 0.9650 | +| 7.00_bps | factor_ic_optimized_daily | 7 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | +| 7.00_bps | mlp_top20_raw_daily | 7 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | +| 7.00_bps | mlp_top20_neutralized_daily | 7 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | +| 7.00_bps | mlp_optimized_daily | 7 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | +| 15.00_bps | equal_weight_daily | 15 | -0.3698 | -2.6049 | 0.2109 | 0.0042 | 0.0015 | 0.8026 | +| 15.00_bps | factor_mean_daily | 15 | 0.0128 | 0.1745 | 0.0614 | 0.2143 | 0.0765 | 1.0061 | +| 15.00_bps | factor_mean_buffered_daily | 15 | -0.0199 | -0.1209 | 0.0547 | 0.1050 | 0.0375 | 0.9905 | +| 15.00_bps | factor_ic_weighted_daily | 15 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | +| 15.00_bps | factor_mean_optimized_daily | 15 | -0.1253 | -1.2683 | 0.0851 | 0.1471 | 0.0525 | 0.9383 | +| 15.00_bps | factor_ic_optimized_daily | 15 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | +| 15.00_bps | mlp_top20_raw_daily | 15 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | +| 15.00_bps | mlp_top20_neutralized_daily | 15 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | +| 15.00_bps | mlp_optimized_daily | 15 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | +| 30.00_bps | equal_weight_daily | 30 | -0.3718 | -2.6231 | 0.2121 | 0.0042 | 0.0030 | 0.8014 | +| 30.00_bps | factor_mean_daily | 30 | -0.1378 | -1.3575 | 0.1035 | 0.2143 | 0.1530 | 0.9318 | +| 30.00_bps | factor_mean_buffered_daily | 30 | -0.0943 | -0.8011 | 0.0780 | 0.1050 | 0.0750 | 0.9539 | +| 30.00_bps | factor_ic_weighted_daily | 30 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | +| 30.00_bps | factor_mean_optimized_daily | 30 | -0.2168 | -2.3121 | 0.1284 | 0.1471 | 0.1050 | 0.8902 | +| 30.00_bps | factor_ic_optimized_daily | 30 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | +| 30.00_bps | mlp_top20_raw_daily | 30 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | +| 30.00_bps | mlp_top20_neutralized_daily | 30 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | +| 30.00_bps | mlp_optimized_daily | 30 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 1.0000 | + +## Interpretation Notes + +- The model attempts walk-forward training; inspect execution coverage before interpreting its metrics. +- Score neutralization is cross-sectional only, not industry or size neutralization. +- Optimized portfolios are constrained to top-scored candidates to keep the public run tractable. +- Read net return together with gross return, turnover, and cost drag. +- This is not investment advice and not a full paper reproduction. diff --git a/docs/public_data_validation.md b/docs/public_data_validation.md index ee808c5..6965bf1 100644 --- a/docs/public_data_validation.md +++ b/docs/public_data_validation.md @@ -478,6 +478,35 @@ such reports with the corrected signal-date protocol; new metadata records it in this change removes tomorrow's availability from selection but does not add a halt-aware execution simulator. +## Full-Pipeline Execution Coverage + +The full-pipeline report includes an `Execution Coverage` section and a +`diagnostics` object in `summary.json`. Inspect these before interpreting the +result table: + +- Raw tradable, limit-eligible, joint factor-valid signal, and historical label + cell counts show how much input survives each mask. The factor signal ratio + uses all panel cells as its denominator. +- The MLP records each planned split's training/prediction samples and status. + Training needs at least `max(32, 4 * feature_count)` samples; insufficient + training samples or no prediction samples cause a recorded skip. Finite + prediction cells count the final prediction matrix, including any overlap + overwritten by later splits. +- Each portfolio records invested and cash dates, plus first/last P&L exposure + dates. The engine earns a return on date `t` from the weights on `t-1`; + last-row positions have no later return in the panel. Cash dates can still + incur exit costs. + +If no split trains, no finite prediction appears, or a strategy stays all-cash, +the report displays a coverage warning. An all-cash MLP's finite zero-return +metrics are **not evidence of model performance**. The run remains recorded +as useful failure evidence. + +Metrics still use the full panel, including warm-up cash. Strategies can have +different exposure dates, so their full-panel metrics are not matched-window +comparisons. Coverage reporting does not repair evaluation-window differences, +prove alpha, or revalidate archived reports. + ## Auditing Reports Before sharing a validation run, audit the generated `summary.json`: @@ -495,6 +524,12 @@ cost settings, unusual turnover/drawdown, and non-positive final equity. It is a quality gate for reproducibility reports, not a judgement about whether a strategy is good. +For new full-pipeline reports, the audit also surfaces execution-coverage +warnings, including skipped training, empty portfolios, and differing P&L +exposure dates. These are warnings rather than errors so failed runs can be +preserved. Archived reports without `diagnostics` retain the existing checks; +an audit pass alone does not establish that a model trained or held positions. + For the default artifact path, the Make target is: ```bash diff --git a/docs/validation_dashboard.md b/docs/validation_dashboard.md index 07579cd..a0a6651 100644 --- a/docs/validation_dashboard.md +++ b/docs/validation_dashboard.md @@ -17,6 +17,21 @@ This page tracks the maintained public-data validation snapshot for `ml-quant-trading`. It is designed as a reproducibility dashboard, not a live trading scoreboard. +## Current Validation Status — 2026-10-18 + +The signal-date availability fix is merged in +[PR #78](https://github.com/initial-d/ml-quant-trading/pull/78). Public-data +revalidation remains blocked: the two fixed AkShare probes again returned +upstream disconnects and no panel on October 18. There is **no newer market +performance snapshot**. + +New full-pipeline reports display factor, training, prediction and portfolio +coverage. The [coverage audit and raw evidence](audits/validation-coverage-20261018/README.md) +show a synthetic case where an all-cash, untrained MLP previously passed the +report audit. This is engineering evidence only; archived July returns below +remain unrevalidated. Full-panel metrics retain warm-up cash and can have +different exposure dates across strategies. + ## Latest Snapshot | Field | Value | diff --git a/scripts/akshare_csi300_full_pipeline.py b/scripts/akshare_csi300_full_pipeline.py index edbb0cd..b8b40c5 100644 --- a/scripts/akshare_csi300_full_pipeline.py +++ b/scripts/akshare_csi300_full_pipeline.py @@ -525,6 +525,49 @@ def _full_pipeline_metadata( return metadata +def _coverage_diagnostics(panel, valid_panel, valid_alpha, label_valid, model, strategies) -> dict: + """Report execution coverage without changing the backtest or its window.""" + warnings = [] + if not valid_alpha.any(): + warnings.append("No factor-valid signal cells remain after the joint factor mask.") + if model.get("trained_splits") == 0: + warnings.append("No walk-forward split was trained; zero-return MLP rows are not evidence of model performance.") + elif model.get("skipped_splits", 0): + warnings.append(f"{model['skipped_splits']} walk-forward splits were skipped; inspect sample counts and reasons.") + if model.get("finite_prediction_cells") == 0: + warnings.append("The model produced no finite predictions.") + coverage = {} + exposure_masks = [] + for name, weights in strategies.items(): + invested = np.abs(weights).sum(axis=1) > 0 + # The engine earns returns[t] using weights[t-1]. Last-row weights + # can incur entry costs but have no subsequent return in this panel. + exposed = np.concatenate(([False], invested[:-1])) + dates = np.flatnonzero(exposed) + coverage[name] = { + "invested_dates": int(invested.sum()), + "cash_dates": int((~invested).sum()), + "pnl_exposure_dates": int(exposed.sum()), + "first_pnl_exposure_date": str(panel.dates[dates[0]]) if dates.size else None, + "last_pnl_exposure_date": str(panel.dates[dates[-1]]) if dates.size else None, + } + exposure_masks.append(exposed) + if not invested.any(): + warnings.append(f"{name}: no invested dates; an all-cash row is not evidence of model performance.") + if any(not np.array_equal(exposure_masks[0], mask) for mask in exposure_masks[1:]): + warnings.append("Strategies have different P&L exposure dates. Full-panel metrics include warm-up cash and are not matched-window comparisons.") + return { + "panel_cells": int(valid_panel.size), + "raw_tradable_cells": int(_to_numpy(panel.mask).astype(bool).sum()), + "limit_eligible_cells": int(valid_panel.sum()), + "factor_signal_cells": int(valid_alpha.sum()), + "factor_signal_ratio": float(valid_alpha.mean()), + "factor_label_cells": int(label_valid.sum()), + "model": model, "strategies": coverage, "warnings": warnings, + "metric_window": "full_panel_including_warmup_cash", + } + + def _write_outputs( cfg: FullPipelineConfig, panel, @@ -533,6 +576,8 @@ def _write_outputs( sensitivity_rows: Sequence[dict[str, float | int | str]], strategies: dict[str, np.ndarray], returns: np.ndarray, + *, + diagnostics: dict | None = None, ) -> None: cfg.output_dir.mkdir(parents=True, exist_ok=True) metadata = _full_pipeline_metadata(cfg, panel, factor_names) @@ -553,6 +598,7 @@ def _write_outputs( "metadata": metadata, "results": list(rows), "cost_sensitivity": list(sensitivity_rows), + "diagnostics": diagnostics, } ), f, @@ -563,7 +609,8 @@ def _write_outputs( panel_meta = metadata["panel"] pipeline = metadata["pipeline"] f.write("# AkShare CSI 300 Full-Pipeline Validation Summary\n\n") - f.write("This report is a public-data research diagnostic, not a trading claim.\n\n") + evidence = "synthetic engineering" if cfg.source == "synthetic" else "public-data research" + f.write(f"This report is a {evidence} diagnostic, not a trading claim.\n\n") f.write("| Field | Value |\n|---|---|\n") f.write(f"| Source | {cfg.source} |\n") f.write(f"| Preset | {cfg.preset} |\n") @@ -581,12 +628,31 @@ def _write_outputs( f.write(f"| Rebalance | {rebalance_label} |\n") f.write(f"| Costs + slippage | {cfg.costs_bps:.2f} + {cfg.slippage_bps:.2f} bps |\n") f.write(f"| Python / PyTorch | {platform.python_version()} / {torch.__version__} |\n\n") + if diagnostics is not None: + f.write("## Execution Coverage\n\n") + f.write(f"Factor-valid signal cells: {diagnostics['factor_signal_cells']} / {diagnostics['panel_cells']}; " + f"historical label cells: {diagnostics['factor_label_cells']}.\n\n") + model = diagnostics["model"] + f.write(f"MLP splits trained / planned: {model.get('trained_splits', 'unknown')} / " + f"{model.get('total_splits', 'unknown')}; finite prediction cells: " + f"{model.get('finite_prediction_cells', 'unknown')}. " + "Per-split sample counts and skip reasons are in `summary.json`.\n\n") + f.write("| Strategy | Invested dates | Cash dates | P&L exposure dates | First / last P&L exposure |\n|---|---:|---:|---:|---|\n") + for name, coverage in diagnostics["strategies"].items(): + f.write(f"| {name} | {coverage['invested_dates']} | {coverage['cash_dates']} | " + f"{coverage['pnl_exposure_dates']} | {coverage['first_pnl_exposure_date']} / " + f"{coverage['last_pnl_exposure_date']} |\n") + f.write("\n") + for warning in diagnostics["warnings"]: + f.write(f"- **Coverage warning:** {warning}\n") + f.write("\nFull-panel metrics include warm-up cash. P&L exposure uses the previous date's weights; " + "cash dates can still incur exit costs.\n\n") f.write(_markdown_table(rows)) if sensitivity_rows: f.write("\n\n## Cost Sensitivity\n\n") f.write(_cost_sensitivity_table(sensitivity_rows)) f.write("\n\n## Interpretation Notes\n\n") - f.write("- The model is trained walk-forward on the selected public-data factor set.\n") + f.write("- The model attempts walk-forward training; inspect execution coverage before interpreting its metrics.\n") f.write("- Score neutralization is cross-sectional only, not industry or size neutralization.\n") f.write("- Optimized portfolios are constrained to top-scored candidates to keep the public run tractable.\n") f.write("- Read net return together with gross return, turnover, and cost drag.\n") @@ -628,9 +694,11 @@ def run_full_pipeline(cfg: FullPipelineConfig) -> list[dict[str, float | int | s valid_alpha = valid_panel & factor_valid click.echo("stage: training walk-forward MLP") + model_diagnostics = {} raw_preds = _fit_predict_model( "mlp_alpha101", features, target, label_valid & factor_valid, validation_cfg, prediction_valid=valid_alpha, + diagnostics=model_diagnostics, ) click.echo("stage: building scores") neutralized_preds = _neutralize_score_matrix(raw_preds, valid_alpha) @@ -717,7 +785,11 @@ def run_full_pipeline(cfg: FullPipelineConfig) -> list[dict[str, float | int | s rows, sensitivity_rows = _strategy_rows(strategies, returns, cfg) click.echo("stage: writing reports") - _write_outputs(cfg, panel, factor_names, rows, sensitivity_rows, strategies, returns) + diagnostics = _coverage_diagnostics( + panel, valid_panel, valid_alpha, label_valid & factor_valid, model_diagnostics, strategies, + ) + _write_outputs(cfg, panel, factor_names, rows, sensitivity_rows, strategies, returns, + diagnostics=diagnostics) return rows diff --git a/scripts/audit_validation_report.py b/scripts/audit_validation_report.py index 737544b..4994ae0 100644 --- a/scripts/audit_validation_report.py +++ b/scripts/audit_validation_report.py @@ -125,6 +125,11 @@ def audit_report( if final_equity is not None and final_equity <= 0: findings.append(_finding("error", f"{strategy}.final_equity", "final equity must be positive")) + diagnostics = report.get("diagnostics") + if isinstance(diagnostics, dict): + for warning in diagnostics.get("warnings", []): + findings.append(_finding("warning", "execution_coverage", str(warning))) + if not findings: findings.append(_finding("pass", "report", "report passed all audit checks")) return findings diff --git a/scripts/public_data_validation.py b/scripts/public_data_validation.py index 9c12ae6..358db69 100644 --- a/scripts/public_data_validation.py +++ b/scripts/public_data_validation.py @@ -400,6 +400,7 @@ def _fit_predict_model( cfg: ValidationConfig, *, prediction_valid: np.ndarray | None = None, + diagnostics: dict | None = None, ) -> np.ndarray: # Training may require a known forward label. Inference must use only # information available on the signal date, including on pre-halt days. @@ -416,13 +417,25 @@ def _fit_predict_model( step=cfg.step, ) in_dim = features.shape[-1] + split_records = [] + min_train_samples = max(32, in_dim * 4) for split_id, (train_start, train_end, test_start, test_end) in enumerate(splits): train_valid = valid[train_start:train_end] test_valid = prediction_valid[test_start:test_end] x_train = features[train_start:train_end][train_valid] y_train = target[train_start:train_end][train_valid] x_test = features[test_start:test_end][test_valid] - if x_train.shape[0] < max(32, in_dim * 4) or x_test.shape[0] == 0: + record = { + "train_start": train_start, "train_end_exclusive": train_end, + "test_start": test_start, "test_end_exclusive": test_end, + "train_samples": int(x_train.shape[0]), "test_samples": int(x_test.shape[0]), + } + split_records.append(record) + if x_train.shape[0] < min_train_samples: + record["status"] = "insufficient_training_samples" + continue + if x_test.shape[0] == 0: + record["status"] = "no_prediction_samples" continue mu = x_train.mean(axis=0, keepdims=True) @@ -465,7 +478,18 @@ def _fit_predict_model( split_preds = np.full(test_valid.shape, np.nan, dtype=np.float32) split_preds[test_valid] = pred.astype(np.float32) preds[test_start:test_end] = split_preds + record["status"] = "trained" + record["finite_predictions"] = int(np.isfinite(pred).sum()) click.echo(f"trained {model_name} split {split_id + 1}/{len(splits)}") + if diagnostics is not None: + trained = sum(record["status"] == "trained" for record in split_records) + diagnostics.update({ + "total_splits": len(splits), "trained_splits": trained, + "skipped_splits": len(splits) - trained, + "min_train_samples": min_train_samples, "splits": split_records, + "finite_prediction_cells": int(np.isfinite(preds).sum()), + "prediction_dates": int(np.isfinite(preds).any(axis=1).sum()), + }) return preds diff --git a/tests/test_akshare_csi300_full_pipeline.py b/tests/test_akshare_csi300_full_pipeline.py index bad8d80..48e183d 100644 --- a/tests/test_akshare_csi300_full_pipeline.py +++ b/tests/test_akshare_csi300_full_pipeline.py @@ -3,6 +3,7 @@ import csv import numpy as np +import pytest from scripts.akshare_csi300_full_pipeline import ( _buffered_top_quantile_weights, @@ -207,14 +208,14 @@ def test_full_pipeline_separates_signal_and_forward_label_masks(tmp_path, monkey ) seen = {} - def predict(model, features, target, labels, validation_cfg, *, prediction_valid=None): + def predict(model, features, target, labels, validation_cfg, *, prediction_valid=None, diagnostics=None): if prediction_valid is None: prediction_valid = labels seen["labels"] = labels.copy() seen["signals"] = prediction_valid.copy() return np.where(prediction_valid, features[:, :, 0], np.nan) - def write(config, data, names, rows, sensitivity, strategies, returns): + def write(config, data, names, rows, sensitivity, strategies, returns, *, diagnostics=None): seen["equal_weight"] = strategies["equal_weight_daily"] monkeypatch.setattr(full, "load_validation_panel", lambda cfg: panel) @@ -245,3 +246,47 @@ def test_rolling_ic_ignores_missing_labels_but_scores_current_universe(): poisoned = _rolling_ic_weighted_scores(features, changed, signal_valid, **kwargs) assert np.allclose(clean, poisoned, equal_nan=True) assert np.isfinite(clean[25, 0]) + + +@pytest.mark.parametrize("factor_available", [False, True]) +def test_full_pipeline_reports_skipped_training_and_empty_portfolios(tmp_path, monkeypatch, factor_available): + """A no-label model must be reported as skipped, not as zero-return skill.""" + import json + import scripts.akshare_csi300_full_pipeline as full + from mlquant.data.synthetic import SyntheticConfig, make_synthetic_panel + + panel = make_synthetic_panel(SyntheticConfig(n_dates=60, n_stocks=8, halt_prob=0, seed=3)) + cfg = full.FullPipelineConfig( + source="synthetic", preset="custom", tickers=(), start="2021-01-01", + end="2022-01-01", max_tickers=8, device="cpu", costs_bps=5, slippage_bps=2, + cost_grid_bps=(), train_window=30, test_window=10, step=10, top_quantile=0.25, + seed=3, epochs=1, batch_size=64, hidden=8, factor_set="alpha101", + covariance_window=10, optimizer_candidates=5, optimizer_weight_cap=0.25, + optimizer_risk_aversion=1, rebalance_step=1, factor_ic_window=20, + min_tradable_ratio=0.9, bootstrap_samples=0, bootstrap_block_size=10, + output_dir=tmp_path, + ) + monkeypatch.setattr(full, "load_validation_panel", lambda cfg: panel) + monkeypatch.setattr(full, "_alpha101_features", lambda panel: ( + np.ones((60, 8, 3), dtype=np.float32), np.full((60, 8), factor_available, dtype=bool), ["a", "b", "c"], + )) + full.run_full_pipeline(cfg) + report = json.loads((tmp_path / "summary.json").read_text()) + diagnostics = report["diagnostics"] + model = diagnostics["model"] + portfolio = diagnostics["strategies"]["mlp_top20_raw_daily"] + if factor_available: + assert diagnostics["factor_signal_cells"] == 472 # synthetic row 0 is unavailable + assert model["trained_splits"] == 3 + assert model["finite_prediction_cells"] == 232 + assert portfolio["invested_dates"] == portfolio["pnl_exposure_dates"] == 29 + assert portfolio["first_pnl_exposure_date"] == str(panel.dates[31]) + assert any("not matched-window" in warning for warning in diagnostics["warnings"]) + else: + assert diagnostics["factor_signal_cells"] == 0 + assert model["trained_splits"] == 0 + assert model["skipped_splits"] == 3 + assert all(split["status"] == "insufficient_training_samples" for split in model["splits"]) + assert portfolio["invested_dates"] == 0 + assert any("No walk-forward split" in warning for warning in diagnostics["warnings"]) + assert "not evidence of model performance" in (tmp_path / "summary.md").read_text() diff --git a/tests/test_audit_validation_report.py b/tests/test_audit_validation_report.py index ddbf2a5..be083d2 100644 --- a/tests/test_audit_validation_report.py +++ b/tests/test_audit_validation_report.py @@ -62,6 +62,18 @@ def test_audit_validation_report_flags_errors_and_warnings(): assert any(item.check == "equal_weight.final_equity" for item in findings) +def test_audit_reports_empty_model_as_warning_even_when_metrics_are_finite(): + report = _valid_report() + report["results"][0].update(ann_return=0, ann_vol=0, sharpe=0, turnover=0, final_equity=1) + report["diagnostics"] = { + "warnings": ["No walk-forward split was trained; zero-return MLP rows are not evidence of model performance."] + } + findings = audit_report(report) + assert not has_errors(findings) # preserve failed runs as useful evidence + assert any(item.severity == "warning" and item.check == "execution_coverage" for item in findings) + assert not any(item.severity == "pass" for item in findings) + + def test_audit_validation_report_fixture_is_json_serializable(): encoded = json.dumps(_valid_report()) assert "equal_weight" in encoded