Expose skipped MLP training and empty portfolios in validation reports

CHANGELOG.md

@@ -2,6 +2,16 @@
 
 ## Unreleased
 
+### Full-pipeline validation: execution coverage
+
+- Reports now show joint factor-mask coverage, per-split training/prediction
+  sample counts and skip reasons, finite predictions, and portfolio cash and
+  P&L exposure dates. An untrained or all-cash MLP is explicitly marked as
+  not evidence of model performance.
+- Report audits surface coverage warnings. Full-panel metrics are preserved;
+  differing warm-up/exposure dates are flagged instead of presenting them as
+  matched-window comparisons. Synthetic reports identify their evidence class.
+
 ### Public validation: signal-date availability
 
 - The generic and full AkShare validation scripts now select portfolios and

docs/audits/validation-coverage-20261018/README.md

@@ -0,0 +1,88 @@
+# Validation Execution Coverage — 2026-10-18
+
+**A finite zero-return row can come from an MLP that never trained or invested.**
+The full-pipeline report now displays factor coverage, training/prediction
+samples and skip reasons, and portfolio cash/P&L exposure dates. The report
+audit surfaces these warnings while preserving the failed run. This is evidence
+about reporting quality, not investment performance or a paper-result update.
+
+The signal-date causality correction was merged in
+[PR #78](https://github.com/initial-d/ml-quant-trading/pull/78), main commit
+`a0b33491266d578d63c300ad3d79832d237c40db`. The new coverage implementation and
+replay script are committed at `fed2db53109cc0a419b6d02d0e4db6de308b5bdd`.
+See [the protocol](../../public_data_validation.md#full-pipeline-execution-coverage).
+
+## Synthetic engineering evidence
+
+[compare_coverage.py](compare_coverage.py) runs the base and current full
+pipeline on the same synthetic panel: 120 dates, 20 stocks, seed 7, no random
+halts, all 213 registered factors. Actual synthetic dates are 2020-01-02 through
+2020-06-17; config start/end are unused by the injected synthetic loader.
+It compares the actual factor-library output, model training, portfolios,
+summary metrics and cost sensitivity, without network access.
+
+- Weights, metrics and cost sensitivity are identical before and after the
+  reporting change: assertions and the input fingerprint are in
+  [comparison.json](comparison.json).
+- Joint factor-valid signal cells: **1,040 / 2,400**; forward-label cells:
+  **1,020**. This short panel is deliberately inadequate for full-factor MLP
+  training, not an estimate of real-data coverage.
+- All three MLP splits are skipped. Training sample counts are **0, 240, 640**,
+  below the unchanged minimum of **852** (`4 * 213`). Prediction samples existed,
+  but no split trained, no finite prediction was produced, and all MLP portfolios
+  stayed cash for all 120 dates.
+- The equal-weight baseline has 118 P&L exposure dates; factor-mean variants
+  have 51; MLP variants have zero. Full-panel metrics retain warm-up cash, and
+  the report explicitly warns that these are not matched-window comparisons.
+- The base report's existing audit passes. The new report has eight coverage
+  warnings. Neither an audit pass nor these synthetic metrics establish alpha.
+
+The generated [synthetic report](synthetic-summary.md) and
+[raw report JSON](synthetic-summary.json) keep metrics beside the coverage
+warnings. [coverage-run.txt](coverage-run.txt) preserves execution output.
+Do not compare these results with the archived July market backtests.
+
+## Verification and environment
+
+- Full test suite: **146 passed**, [pytest.txt](pytest.txt). Tests cover empty
+  factor masks and a normally trained MLP, with the existing causality tests.
+- Ruff: passed, [ruff.txt](ruff.txt); `git diff --check`: passed.
+- A regression test against the base fails because coverage is absent:
+  [old-diagnostics-test.txt](old-diagnostics-test.txt).
+- Exact commands: [commands.txt](commands.txt). Environment:
+  [environment.json](environment.json), [pip-freeze.txt](pip-freeze.txt).
+  Python 3.12.14, PyTorch 2.14.1, AkShare 1.19.1, macOS 26.6.2 arm64,
+  Apple M5, 10 logical CPUs, 32 GiB RAM. CPU only; CUDA unavailable; MPS unused.
+  Replay torch/interop/OMP/BLAS threads: 1. Costs: 5 bps plus 2 bps slippage;
+  sensitivity: 0/7/15/30 effective bps. No bootstrap inference.
+- Integrity: [checksums.txt](checksums.txt). Workspace paths in copied logs
+  and reports are redacted to `<repo>`; original local artifacts remain under
+  `artifacts/revalidation-20261018/`. Input market prices are not published.
+
+## Public-data access failure, kept separately
+
+The fixed probe from [the October 17 plan](../public-revalidation-20261017/plan.md)
+was retried once per symbol on October 18 after merging the causality fix.
+It again failed for both symbols, so the planned expanded run did not start.
+
+| Symbol | Started, Asia/Shanghai | Outcome |
+|---|---|---|
+| `600519` | 2026-10-18 09:03:10 | Upstream disconnect; no data returned |
+| `000001` | 2026-10-18 09:03:11 | Upstream disconnect; no data returned |
+
+[provider-probe.json](provider-probe.json) records exact timestamps, durations,
+source dates 2021-01-01 through 2025-01-01, daily `qfq`, and `retries=1`.
+[provider-probe.txt](provider-probe.txt) preserves warnings and tracebacks;
+[provider_probe.py](provider_probe.py) preserves the executed probe source,
+copied from the October 17 probe with only its output directory changed.
+Probe code commit: `a0b33491266d578d63c300ad3d79832d237c40db`.
+
+The observed warning is `RemoteDisconnected('Remote end closed connection
+without response')`; the loader raises `ValueError: No data returned`.
+The underlying cause is unknown. No panel, coverage audit, market backtest,
+new Sharpe result or proprietary paper experiment exists for this attempt.
+Costs are not applicable to download-only probes. Synthetic evidence above
+does not replace the blocked public-data run.
+
+Coverage diagnostics reveal execution gaps; they do not repair evaluation
+windows, add exchange execution modeling, or revalidate archived results.

docs/audits/validation-coverage-20261018/checksums.txt

@@ -0,0 +1,15 @@
+ed4a9a8c1570c1ccc774af58e86aaf90e4c01350cd6487bb4d5df141dfbe568e  README.md
+51a2df710fce70f402c91f005194dcc2e9a0da514052af63f160c09fc43db610  commands.txt
+608810bccca2aabb610af178ec37624e21dc0813e25544b5e1ac4e1a014d0eb4  compare_coverage.py
+3ab5e3d788b0956c1aa34929c2f3ebcd0a34708b170d643cfdd6ff4a954e7fab  comparison.json
+e01ddb90e0c5eaa5449f14adcd09a5ffd0907cc6c33ca19d24e5d13dc3c88dc7  coverage-run.txt
+5f096df220beae8aa7281ada240170262d8ee9afb139704f318718d0253842e6  environment.json
+c4d7fcde962eab8f50ee80fbd51b92843f99a02f53185c0a8bd5880b0bc158cb  old-diagnostics-test.txt
+d0aa7915427de1673fe56a597d3dbbb9406367cdcfe59e3de98d8f64d67878e8  pip-freeze.txt
+a063f25fb7b6a28496aaf173e847ef170ece6eb1a2d6aa26a5d599f8888fa644  provider-probe.json
+a11f5232bbfb873fbefbbe9b12e40d7cfe4ab04c30e9b232daee81ee1e64d369  provider-probe.txt
+a8ecb4221d14b7cf80fbcfedcc4ab39167e244ae630bf3b1f22906c3fced95a9  provider_probe.py
+14f6b19eb6bccf83ac43c98eb02812c3608a99f61e490dd31423543313ac06cd  pytest.txt
+82b3e6a6c090a57601d22943bd23fca9218d1031dbe5a7b754092f9a156b4f18  ruff.txt
+4bbabeda17e90941097d69b797b832c7e8756cdbefd7b8b201be8c732472bc96  synthetic-summary.json
+bd69e1a2c625815b53c4ef4d39dfb0e4ddb9060230af4520dd126f0c6c2f1153  synthetic-summary.md

docs/audits/validation-coverage-20261018/commands.txt

@@ -0,0 +1,20 @@
+# Run from repository root with the editable .venv described in environment.json.
+# Old-code regression, before the implementation patch (base main plus new test):
+OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 .venv/bin/python -m pytest tests/test_akshare_csi300_full_pipeline.py::test_full_pipeline_reports_skipped_training_and_empty_portfolios -q
+
+# Candidate code; byte-identical implementation to fed2db53109cc0a419b6d02d0e4db6de308b5bdd:
+OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 .venv/bin/python -m pytest tests/test_akshare_csi300_full_pipeline.py tests/test_public_data_validation.py tests/test_audit_validation_report.py -q
+OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 .venv/bin/python -m pytest
+.venv/bin/python -m ruff check src tests scripts/public_data_validation.py scripts/akshare_csi300_full_pipeline.py scripts/audit_validation_report.py docs/audits/validation-coverage-20261018/compare_coverage.py
+git diff --check
+
+# Clean recorded code commit; no network, synthetic only:
+OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 .venv/bin/python docs/audits/validation-coverage-20261018/compare_coverage.py
+
+# Download-only probe source is preserved as provider_probe.py. Executed before
+# the diagnostics patch at a0b33491266d578d63c300ad3d79832d237c40db:
+PYTHONPATH=src .venv/bin/python artifacts/revalidation-20261018/provider_probe.py
+
+# Replay loads the base runner/writer and model helper with:
+git show a0b33491266d578d63c300ad3d79832d237c40db:scripts/public_data_validation.py
+git show a0b33491266d578d63c300ad3d79832d237c40db:scripts/akshare_csi300_full_pipeline.py

docs/audits/validation-coverage-20261018/compare_coverage.py

@@ -0,0 +1,102 @@
+"""Offline report comparison on one real factor-library synthetic input.
+
+Run from the repository root with OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1.
+No external data is requested. Synthetic outputs are engineering evidence only.
+"""
+from dataclasses import replace
+import hashlib
+import json
+from pathlib import Path
+import platform
+import subprocess
+import sys
+import types
+
+import numpy as np
+import torch
+
+ROOT = Path(__file__).resolve().parents[3]
+sys.path.insert(0, str(ROOT))
+sys.path.insert(0, str(ROOT / "src"))
+from mlquant.data.synthetic import SyntheticConfig, make_synthetic_panel
+import scripts.akshare_csi300_full_pipeline as current
+from scripts.audit_validation_report import audit_report, findings_to_dicts
+
+BASE = "a0b33491266d578d63c300ad3d79832d237c40db"
+OUT = ROOT / "artifacts/revalidation-20261018/coverage"
+OUT.mkdir(parents=True, exist_ok=True)
+torch.set_num_threads(1)
+torch.set_num_interop_threads(1)
+
+
+def archived_module(name, path):
+    source = subprocess.check_output(["git", "show", f"{BASE}:{path}"], cwd=ROOT, text=True)
+    module = types.ModuleType(name)
+    module.__file__ = str(ROOT / path)
+    sys.modules[name] = module
+    exec(compile(source, module.__file__, "exec"), module.__dict__)
+    return module
+
+
+old_helpers = archived_module("coverage_base_helpers", "scripts/public_data_validation.py")
+old = archived_module("coverage_base_full", "scripts/akshare_csi300_full_pipeline.py")
+old._fit_predict_model = old_helpers._fit_predict_model
+panel = make_synthetic_panel(SyntheticConfig(n_dates=120, n_stocks=20, halt_prob=0, seed=7))
+panel.assert_consistent()
+fingerprint = hashlib.sha256()
+for name in ("open", "high", "low", "close", "volume", "vwap", "amount", "mask"):
+    fingerprint.update(name.encode())
+    fingerprint.update(getattr(panel, name).numpy().tobytes())
+
+cfg = current.FullPipelineConfig(
+    source="synthetic", preset="custom", tickers=(), start="2021-01-01", end="2022-01-01",
+    max_tickers=20, device="cpu", costs_bps=5, slippage_bps=2, cost_grid_bps=(0, 7, 15, 30),
+    train_window=60, test_window=20, step=20, top_quantile=0.2, seed=7, epochs=1,
+    batch_size=128, hidden=16, factor_set="all", covariance_window=20,
+    optimizer_candidates=10, optimizer_weight_cap=0.1, optimizer_risk_aversion=1,
+    rebalance_step=1, factor_ic_window=30, min_tradable_ratio=0.9,
+    bootstrap_samples=0, bootstrap_block_size=20, output_dir=OUT,
+    command=(sys.executable, str(Path(__file__).relative_to(ROOT))),
+)
+books = {}
+reports = {}
+for label, module in (("base", old), ("current", current)):
+    module.load_validation_panel = lambda config: panel
+    writer = module._write_outputs
+
+    def capture(config, data, names, rows, sensitivity, strategies, returns,
+                _label=label, _writer=writer, **kwargs):
+        books[_label] = {name: weights.copy() for name, weights in strategies.items()}
+        _writer(config, data, names, rows, sensitivity, strategies, returns, **kwargs)
+
+    module._write_outputs = capture
+    module.run_full_pipeline(replace(cfg, output_dir=OUT / label))
+    reports[label] = json.loads((OUT / label / "summary.json").read_text())
+
+assert reports["base"]["results"] == reports["current"]["results"]
+assert reports["base"]["cost_sensitivity"] == reports["current"]["cost_sensitivity"]
+assert all(np.array_equal(books["base"][name], weights) for name, weights in books["current"].items())
+diagnostics = reports["current"]["diagnostics"]
+assert diagnostics["model"]["trained_splits"] == 0
+assert diagnostics["strategies"]["mlp_top20_raw_daily"]["invested_dates"] == 0
+comparison = {
+    "category": "synthetic engineering evidence; no investment or predictive evidence",
+    "base_commit": BASE,
+    "current_commit": subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=ROOT, text=True).strip(),
+    "python": platform.python_version(), "pytorch": torch.__version__, "platform": platform.platform(),
+    "device": "cpu", "torch_threads": 1, "torch_interop_threads": 1,
+    "synthetic_config": {"n_dates": 120, "n_stocks": 20, "halt_prob": 0, "seed": 7},
+    "panel_sha256": fingerprint.hexdigest(),
+    "panel_hash_method": "SHA-256 of field names and C-order bytes: open/high/low/close/volume/vwap/amount/mask",
+    "weights_identical": True, "metrics_identical": True, "cost_sensitivity_identical": True,
+    "costs_bps": 5, "slippage_bps": 2,
+    "diagnostics": diagnostics,
+    "audit": {label: findings_to_dicts(audit_report(report)) for label, report in reports.items()},
+    "limitations": [
+        "Short synthetic panel deliberately lacks enough joint-factor training samples; not a market experiment.",
+        "Base runner/writer and model helper loaded from recorded Git commit; other unchanged helpers are shared.",
+        "No bootstrap inference; full-panel metrics retain warm-up cash and differing exposure windows.",
+    ],
+}
+(OUT / "comparison.json").write_text(json.dumps(comparison, indent=2) + "\n")
+print(json.dumps({"identical_weights_and_metrics": True, "diagnostics": diagnostics}, indent=2))