feat: add dated, interpreted run analyses

docs/operations/observability.md

@@ -171,9 +171,10 @@ while observing after runs. Clear `active` when the observation period ends. Use
 new `change_id` for a revised prediction, scope, or cutover. Code validates each
 declaration and preserves each admitted run's snapshot; operators maintain stability
 across independent runs. Recording the prediction beforehand distinguishes intended
-effects from explanations developed after seeing outcomes. A pre-cutover marked
-row supplies that evidence; a current configuration or first after row alone does
-not establish preregistration.
+effects from explanations developed after seeing outcomes. A pre-cutover marked row
+is a baseline sample, but its logical start time does not prove when the prediction
+was registered. Keep contemporaneous evidence if preregistration matters; neither
+the current configuration nor the first after row establishes it.
 
 Summary and index records store the same nested `rollout` object with normalized
 UTC `effective_at`. Capture happens immediately before the first admitted launch;
@@ -240,6 +241,94 @@ Compare parsed instants, including fractional seconds: equality is after cutover
 Do not manufacture cohorts from content hashes, repository revisions, file mtimes,
 or guessed dates. Existing aggregate trends do not consume rollout markers.
 
+## Dated run analysis
+
+The packaged deck files under `src/autoskillit/assets/deck/` are the rendering
+template. A dated analysis preserves a separate snapshot of the report-index rows,
+prepared measurements, and exact template bundle used for that run. Coverage is
+limited to retained report-index sources; it does not reconstruct pruned session
+history.
+
+Use a fresh directory whose name starts with a UTC timestamp in
+`YYYY-MM-DD_HHMMSS` form. The token makes repeated runs distinct:
+
+```bash
+RUN_DIR=".autoskillit/temp/analyze-run/2026-10-11_162824_demo"
+autoskillit sessions analysis-prepare "$RUN_DIR" --change-id prompt-cost-study
+```
+
+Preparation refreshes and reads the report index, then freezes the five populations
+(`sessions`, `requests`, `tools`, `subagents`, and `turns`) in `measurements.json`.
+Its `payload.generated_at_ms` is the latest integer `time_ms` across those
+populations, ignoring booleans and defaulting to zero when no valid time exists.
+This fixed corpus anchor drives the report’s time windows; it is not the time the
+analysis was created. The `analysis-finish` command records the actual finish time
+in UTC milliseconds.
+
+Ask the analysis skill to interpret the frozen packet:
+
+```text
+/autoskillit:analyze-run --change-id prompt-cost-study
+```
+
+The skill chooses a new dated directory under
+`.autoskillit/temp/analyze-run/`, runs both preparation and finish, and returns a
+`report_path` only after completion. Omit `--change-id` for an analysis with no
+rollout cohort. To drive the two CLI steps directly, write a candidate to the
+`.candidate-findings.json` path returned by preparation, then finish the same
+directory:
+
+```bash
+autoskillit sessions analysis-finish "$RUN_DIR"
+```
+
+A completed directory contains `measurements.json`, `findings.json`, `report.html`,
+and `analysis.json`; the temporary candidate is removed. The frozen packet retains
+the HTML template, CSS, script bytes and order, so rendering uses the same CSP
+script hashes. Verification rerenders from those frozen inputs and compares the
+HTML bytes exactly. It does not compare an HTML hash that would be sensitive to the
+finish timestamp.
+
+An unchanged input fingerprint can reuse a verified completed interpretation and
+its provenance. Existing completions with that fingerprint must verify and agree;
+inconsistent candidates fail closed. Publication serializes under a finite
+five-second lease and checks again inside the lease, so concurrent finishes
+converge on the same verified candidate. Changed inputs produce a different
+fingerprint and require a new interpretation; prior findings are not carried
+forward.
+
+With a `change_id`, the cohort includes only session rows bearing that exact id;
+missing markers and other ids are excluded. The comparison parses each selected
+row's original aware `timestamp`, including fractional seconds. A start before
+`effective_at` is a baseline; a start exactly at cutover is after. `scope` describes
+the intended change but does not filter rows. Malformed matched markers, conflicting
+declarations, and selected rows without aware start times fail closed. If the id is
+not present in retained runs, preparation errors rather than returning an empty
+cohort. A pre-cutover row remains a baseline, not independent proof that the
+prediction was preregistered.
+
+Read sample counts per field: each token measure has its own `observed_sessions`,
+`eligible_sessions`, and `state_counts`; failure share has its own sample and
+unknown counts. `measured_zero` is an observed zero, while `unknown` is missing or
+invalid evidence, not zero. The candidate at `candidate_path` must contain the
+exact `snapshot_digest` returned by preparation, an `interpreter` object with
+`role`, `model`, and `prompt`, and a nonempty `findings` list. Each finding has
+`view_id`, nonblank plain-text `text`, and `evidence_ids`. Copy ids from the
+packet's `evidence` map: each resolves to a specific table row, prepared measure,
+or rollout group-and-metric comparison, and the referenced view must equal the
+finding's `view_id`. An empty corpus still needs an explicit no-observations
+finding; only then may its `evidence_ids` be empty. Numeric measures are computed
+by Python and should be reported from the referenced record. Structural checks
+validate the digest and evidence links, not whether the prose is true or whether
+an observed difference is causal. The analyst must check each claim against its
+referenced measurement.
+
+Related rationale: [Megagon Labs’ discussion of auditable source-grounded
+tables](https://megagon.ai/data-provenance-source-grounded-ai/) explains why
+item-level evidence and separate scope checks help make structured claims
+independently reviewable. This workflow applies that distinction to measurements
+and interpretation prose.
+
 ## Native headless OTLP capture
 
 Every headless executor invocation starts one best-effort OTLP/HTTP-JSON sink

docs/skills/catalog.md

@@ -53,7 +53,8 @@ runs:
 `prepare-pr`, `compose-pr`, `open-integration-pr`, `merge-pr`, `analyze-prs`, `review-pr`,
 `resolve-review`, `implement-worktree-no-merge`, `resolve-failures`,
 `retry-worktree`, `resolve-merge-conflicts`, `audit-impl`, `smoke-task`,
-`report-bug`, `pipeline-summary`, `diagnose-ci`, `analyze-pipeline-health`, `file-audit-issues`, `verify-diag`,
+`report-bug`, `pipeline-summary`, `diagnose-ci`, `analyze-pipeline-health`,
+`analyze-run`, `file-audit-issues`, `verify-diag`,
 `resolve-claims-review`, `resolve-design-review`, `resolve-research-review`,
 `compose-research-pr`, `prepare-research-pr`, `review-research-pr`,
 `promote-to-main`, `classify-experiment-type`, `apply-review-dimensions`

docs/skills/visibility.md

@@ -45,11 +45,12 @@ simultaneously or reclassify session-role skills between tiers. See
 ### Tier 3 — Pipeline-Only (Automation Skills)
 
 - **Location**: `src/autoskillit/skills_extended/` (same directory as Tier 2)
-- **Default members** (32 total):
+- **Default members**:
   `prepare-pr`, `compose-pr`, `open-integration-pr`, `merge-pr`, `analyze-prs`,
   `review-pr`, `resolve-review`, `implement-worktree-no-merge`, `resolve-failures`,
   `retry-worktree`, `resolve-merge-conflicts`, `audit-impl`, `smoke-task`,
-  `report-bug`, `pipeline-summary`, `diagnose-ci`, `analyze-pipeline-health`, `verify-diag`,
+  `report-bug`, `pipeline-summary`, `diagnose-ci`, `analyze-pipeline-health`,
+  `analyze-run`, `verify-diag`,
   `compose-research-pr`, `prepare-research-pr`, `resolve-claims-review`,
   `resolve-design-review`, `resolve-research-review`, `review-research-pr`,
   `audit-claims`, `build-execution-map`, `promote-to-main`,

src/autoskillit/assets/deck/deck.css

@@ -148,6 +148,20 @@ svg { display: block; max-width: 100%; height: auto; }
 .view-review button:disabled, .view-review-signal button:disabled { cursor: not-allowed; opacity: .6; }
 .view-review__reason { color: var(--warn); font-size: 12px; }
 .view-actor { color: var(--ink-3); font: 11px var(--mono); }
+.run-analysis__populations { margin: 0; padding-left: 20px; color: var(--ink-2); }
+.run-analysis__finding { border-top: 1px solid var(--rule); padding: 12px 0; }
+.run-analysis__comparison { border-left: 2px solid var(--rule-2); padding-left: 12px; }
+.run-analysis__measure { margin: 8px 0; }
+.run-analysis__measure p { margin: 3px 0; }
+.run-analysis__fields { margin: 6px 0; }
+.run-analysis__evidence { margin: 6px 0; }
+.run-analysis__evidence summary { color: var(--s1); cursor: pointer; }
+.run-analysis__evidence pre { color: var(--ink-2); font: 10px/1.5 var(--mono);
+  white-space: pre-wrap; overflow-wrap: anywhere; }
+.trend-rollout__measure { margin: 0; min-width: 180px; max-width: 420px;
+  font: 10px/1.4 var(--mono); white-space: pre-wrap; overflow-wrap: anywhere; }
+.trend-rollout__delta { display: block; min-width: 220px; max-width: 480px;
+  line-height: 1.45; white-space: normal; overflow-wrap: anywhere; }
 .foot { color: var(--ink-3); font: 10px/1.5 var(--mono); padding: 0 28px 24px; }
 @media screen and (max-width: 860px) {
   .shell { grid-template-columns: minmax(0,1fr); }

src/autoskillit/assets/deck/shell.js

@@ -243,6 +243,137 @@ globalThis.DeckShell = (() => {
       }, "Reset cohort"));
     }
 
+    function renderAnalysisCard(analysis, viewId) {
+      const display = value => value == null || value === "" ? "not recorded" :
+        typeof value === "object" ? JSON.stringify(value) : String(value);
+      const interpreter = analysis.interpreter ?? {};
+      const identity = el("dl", {class: "run-analysis__identity"}, [
+        el("dt", {}, "Input fingerprint"), el("dd", {}, display(analysis.input_fingerprint)),
+        el("dt", {}, "Interpreter role"), el("dd", {}, display(interpreter.role)),
+        el("dt", {}, "Model"), el("dd", {}, display(interpreter.model)),
+        el("dt", {}, "Prompt"), el("dd", {}, display(interpreter.prompt))
+      ]);
+      const populations = Object.entries(analysis.coverage?.populations ?? {}).map(
+        ([name, coverage]) => {
+          const states = Object.entries(coverage.measure_states ?? {}).map(([field, counts]) =>
+            field + ": " + Object.entries(counts ?? {}).map(([state, count]) =>
+              state + " " + display(count)).join(", "));
+          return el("li", {}, name + " · rows " + display(coverage.rows) + " · untimed " +
+            display(coverage.untimed_rows) + (states.length ? " · " + states.join("; ") : ""));
+        });
+      const cohort = analysis.cohort ?? {};
+      const hasCohort = cohort.change_id != null && String(cohort.change_id).length > 0;
+      const cohortContent = hasCohort ? el("dl", {class: "run-analysis__cohort"}, [
+        el("dt", {}, "Change ID"), el("dd", {}, display(cohort.change_id)),
+        el("dt", {}, "Effective at"), el("dd", {}, display(cohort.effective_at)),
+        el("dt", {}, "Scope"), el("dd", {}, display(cohort.scope)),
+        el("dt", {}, "Expected effect"), el("dd", {}, display(cohort.expected_effect)),
+        el("dt", {}, "Before / after / excluded"), el("dd", {},
+          [cohort.before_count, cohort.after_count, cohort.excluded_count]
+            .map(display).join(" / ")),
+        el("dt", {}, "Marked pre-cutover baseline"), el("dd", {},
+          cohort.marked_pre_cutover_observation ? "yes" : "no"),
+        el("dt", {}, "Preregistration"), el("dd", {}, display(cohort.preregistration))
+      ]) : el("div", {}, [
+        el("p", {}, "No rollout cohort applied."),
+        el("p", {}, display(cohort.reason)),
+        el("p", {}, "Baseline and preregistration assessment is unavailable " +
+          "without a rollout cohort.")
+      ]);
+
+      function evidenceMeasure(label, measure, fallbackUnit) {
+        if (!measure || typeof measure !== "object") {
+          return el("p", {}, label + " · " + display(measure));
+        }
+        const measured = DeckCore.measureValue(measure);
+        const value = measured == null ? "value unavailable" : display(measured);
+        const unit = measure.unit ?? fallbackUnit ?? "unit not reported";
+        const observed = measure.observed_sessions ?? measure.sample_size;
+        const eligible = measure.eligible_sessions ?? measure.eligible_count;
+        const unknown = measure.unknown_sessions ?? measure.unknown_runs ??
+          measure.state_counts?.unknown;
+        const sampleText = measure.earlier_samples != null || measure.later_samples != null ?
+          "Observed before/after samples " + DeckCore.countLabel(measure.earlier_samples) + "/" +
+            DeckCore.countLabel(measure.later_samples) :
+          "Observed samples " + DeckCore.countLabel(observed);
+        const states = Object.entries(measure.state_counts ?? {}).map(([state, count]) =>
+          state + " " + DeckCore.formatCount(count)).join(", ");
+        return el("div", {class: "run-analysis__measure"}, [
+          el("p", {}, [el("strong", {}, label), " · ", availabilityCell(measure), " · ", value,
+            " ", unit, measure.reason ? " · " + measure.reason : ""]),
+          el("p", {}, sampleText + " · eligible " + DeckCore.countLabel(eligible) + " · unknown " +
+            DeckCore.countLabel(unknown)),
+          states ? el("p", {}, "Availability counts · " + states) : null,
+          measure.guardrail ? el("p", {}, "Guardrail · " + measure.guardrail) : null
+        ]);
+      }
+
+      function evidenceSummary(measurement) {
+        if (!measurement || typeof measurement !== "object") {
+          return el("p", {}, display(measurement));
+        }
+        if (measurement.before && measurement.after && measurement.delta && measurement.metric) {
+          return el("div", {class: "run-analysis__comparison"}, [
+            el("p", {}, "Measure · " + display(measurement.metric)),
+            Array.isArray(measurement.group) ? el("p", {}, "Group · " +
+              measurement.group.map(display).join(" · ")) : null,
+            evidenceMeasure("Before", measurement.before, measurement.delta.unit),
+            evidenceMeasure("After", measurement.after, measurement.delta.unit),
+            evidenceMeasure("Delta", measurement.delta, measurement.delta.unit)
+          ]);
+        }
+        if ("state" in measurement || "value" in measurement) {
+          return evidenceMeasure("Measurement", measurement, measurement.unit);
+        }
+        return el("dl", {class: "run-analysis__fields"}, Object.entries(measurement).flatMap(
+          ([key, value]) => [el("dt", {}, key), el("dd", {}, display(value))]
+        ));
+      }
+
+      const children = [
+        el("h2", {}, "Run analysis"),
+        identity,
+        el("h3", {}, "Frozen coverage"),
+        el("p", {}, display(analysis.coverage?.limitation)),
+        el("p", {}, "This analysis covers the frozen retained report-index population. " +
+          "Current chart filters do not change its evidence."),
+        populations.length ? el("ul", {class: "run-analysis__populations"}, populations) :
+          el("p", {}, "No retained indexed populations."),
+        el("h3", {}, "Rollout marker"),
+        cohortContent
+      ];
+
+      if (viewId !== null) {
+        const findings = (analysis.findings ?? []).filter(finding => finding.view_id === viewId);
+        children.push(el("h3", {}, "Interpretation"));
+        children.push(el("p", {}, "Findings below apply to this built view and the frozen " +
+          "population shown above; interactive filters do not alter them."));
+        if (!findings.length) {
+          children.push(el("p", {class: "view-empty"},
+            "No interpretation was recorded for this view."));
+        }
+        findings.forEach(finding => {
+          const evidenceItems = (finding.evidence_ids ?? []).map(id => {
+            const evidence = analysis.evidence?.[id];
+            if (!evidence) return el("li", {}, "Missing frozen evidence: " + id);
+            return el("li", {}, [
+              evidenceSummary(evidence.measurement),
+              el("details", {class: "run-analysis__evidence"}, [
+                el("summary", {}, id + " · " + display(evidence.source)),
+                el("pre", {}, JSON.stringify(evidence.measurement, null, 2))
+              ])
+            ]);
+          });
+          children.push(el("div", {class: "run-analysis__finding"}, [
+            el("p", {}, finding.text),
+            evidenceItems.length ? el("ul", {}, evidenceItems) :
+              el("p", {}, "No evidence references in this finding.")
+          ]));
+        });
+      }
+      return el("section", {class: "card run-analysis"}, children);
+    }
+
     function render() {
       route = DeckCore.decodeRoute(location.hash, model.landing);
       renderNav();
@@ -313,15 +444,27 @@ globalThis.DeckShell = (() => {
         }
         content.appendChild(noticeCard);
       }
+      if (model.analysis) {
+        content.appendChild(renderAnalysisCard(
+          model.analysis, view && view.status === "built" ? view.id : null
+        ));
+      }
 
       const first = model.history.first_ms;
       const last = model.history.last_ms;
       const history = first === null || last === null ? "—" :
         DeckCore.formatDate(first) + " → " + DeckCore.formatDate(last);
-      const generated = new Date(model.generated_at_ms).toISOString().slice(0, 16);
-      document.getElementById("deck-foot").textContent = "report index v" +
-        model.index_schema_version + " · " + DeckCore.formatCount(model.tables.sessions.length) +
-        " session rows · history " + history + " · generated " + generated + " UTC";
+      const footer = document.getElementById("deck-foot");
+      const footerPrefix = "report index v" + model.index_schema_version + " · " +
+        DeckCore.formatCount(model.tables.sessions.length) + " session rows · history " + history;
+      if (model.analysis) {
+        footer.textContent = footerPrefix + " · corpus/window anchor " +
+          new Date(model.generated_at_ms).toISOString() + " · analysis created " +
+          new Date(model.analysis.generated_at_ms).toISOString();
+      } else {
+        const generated = new Date(model.generated_at_ms).toISOString().slice(0, 16);
+        footer.textContent = footerPrefix + " · generated " + generated + " UTC";
+      }
     }
 
     window.addEventListener("hashchange", render);