Source-data hygiene: FNS-sourced SNAP, loud bundle drift, label year keys (#77, #78, #79)

[MaxGhenis]

Summary

Three source-data hygiene fixes, one commit each. Closes #77, #78, #79.

#77 — Replace fabricated SNAP_DATA in db/etl_snap.py with parsed FNS facts

db/etl_snap.py hardcoded a SNAP_DATA dict of 2021–2023 national and
"top 10 states by participation" values that were suspiciously round with no
per-value citation. The loader is live, not dead: it is wired through
db/cli.py (ledger load snap / ledger load all, which CI runs on every
push at .github/workflows/ci.yml) and re-exported by
ledger/targets/loaders.py and ledger/jurisdictions/us/__init__.py.
Spot-checks against the real FNS FY24 workbook confirm the values were
fabricated (2023 CA households 2,891k vs parsed 3,129k; national benefits
112,848M vs 93,847M).

Per the issue's decision tree (live ⇒ replace with FNS-parsed figures +
provenance), load_snap_targets now sources household, participant, and
benefit values from the trustworthy packages/usda_snap/fy69_to_current
source package, which parses the FY24 workbook cell-by-cell with guard cells
and a checksum-locked artifact. Each target records per-value provenance
(source file, vintage, sha256, R2 URI, source record id) in its notes. The
years argument now filters by fiscal-year reference period, so a year the
package does not publish yields nothing rather than mislabeling FY2024 data.

#78 — Fail loudly on source-package alias/directory drift in build-bundle

build-bundle iterates the hand-maintained SOURCE_PACKAGE_ALIASES map, so a
packages/* directory added/renamed/consolidated without a matching alias
entry was silently dropped from the merged consumer_facts.jsonl — the
failure mode that produced an incomplete populace fiscal-refresh build.

New drift detection over the packages/* authoring surface makes a default
build-bundle fail loudly when the alias map and layout disagree:
discover_source_package_dirs(), find_alias_map_drift(), and
assert_alias_map_covers_packages() (raising SourcePackageAliasDriftError).
build_bundle() runs the assertion for the default (non-explicit) source list
before building, so the export is either complete or a clear error — never
silently partial. Explicit --sources builds are unaffected.

Note: the committed alias map is currently already in sync with packages/*
(63 ↔ 63), so the specific drift examples in the issue have since been
reconciled; this fixes the mechanism that allowed the silent drift.

#79 — Treat non-integer manifest year keys as labels in source_package.py

_render_string (source_package.py:1735) eagerly computed
filing_year = year + 1 before formatting, so any string value rendered under
a non-integer files:/column_by_year key (a release/vintage label such as
the SSA Table 7.B1 extracted_targets entry) crashed with
TypeError: can only concatenate str (not "int") to str — even for values
with no {year} placeholder, like source_table and sheet_name.

Non-integer keys are labels, not years: there is no year to interpolate.
_render_string now skips {year}/{filing_year} templating when year is
not an int and returns the value unchanged, so a label build resolves literal
record-set periods (e.g. SSA period 2024) instead of failing. Integer years
still template both {year} and {filing_year}, preserving the SOI
tax-year/filing-year headers.

Ledger Governance

This PR touches source-data surfaces (a jurisdiction ETL and the source-package
build/render path), so noting the governance context:

• Approved Ledger agent role: ledger-source-ingestor (source-data loader
  fidelity + bundle completeness)
• Deterministic checks run: full pytest -q suite (559 passed, 1 pre-existing
  skip) and CI-equivalent ruff check ledger policyengine_ledger db scripts tests (clean); the FNS source package still validates and builds
  (ledger validate-package / build-suite), and ledger load all succeeds.
• LLM judge verdicts: (deferred to reviewer)
  • ledger-source-fidelity: SNAP values now trace cell-by-cell to the FNS
    FY24 workbook instead of an uncited hardcoded dict.
  • ledger-target-profile: n/a — no target profile changes.
  • ledger-contract: n/a — no consumer-contract schema changes; build-bundle silently skips packages whose aliases drifted from the db/data layout #78 makes
    the bundle export complete-or-fail rather than silently partial.
  • ledger-boundary: preserved — every SNAP target value is a published FNS
    figure; no reconciliation, aging, or imputation added.

Boundary impact: #77 replaces fabricated numbers with published FNS values
(strictly improves provenance); #78 and #79 are build/render robustness fixes
that add no new facts and change no published values.

Tests

• Legacy etl_snap.py contains hardcoded state values that appear fabricated #77: tests/test_etl_snap.py rewritten to assert against the parsed source
  facts (via load_snap_facts) instead of any hardcoded table, so it can no
  longer bless fabricated numbers. 12 tests.
• build-bundle silently skips packages whose aliases drifted from the db/data layout #78: tests/test_source_package_alias_drift.py (new) plants an unmapped
  package directory and asserts build_bundle raises, plus unit-tests
  discovery and drift detection against temp package trees, and pins that the
  committed alias map covers every committed package directory. 7 tests.
  Existing tests/test_ledger_bundle.py (6 tests, incl. the 38,939-row
  exact-count fixture) still pass — the guard does not change in-sync output.
• String year keys in manifest files: crash the {year} templating in source_package.py #79: tests/test_ledger_source_package.py reproduces the year + 1 crash
  first (red), then asserts integer years template filing_year, label years
  pass strings through / leave stray {year} literal, and
  build_facts("extracted_targets") on the real SSA package builds facts.

Full suite: 559 passed, 1 skipped; ruff clean on the CI surface.

🤖 Generated with Claude Code