[catherinebirney]

cc:
Closes: #1116

What changed? Why?

• Integrate the recent changes made to the GHG FBS method for adding nowcast BEA, facility data, and modified transportation approach into a final version of the Cornerstone GHG FBS for the v1.0.0 release. Completed for the time series 2017-2024.
• Default so that the loader now returns GHG_national_Cornerstone_{year} - instead of any of the outdated GHG stems
• The GHG_national_Cornerstone_2024.yaml results match the GHG_national_Cornerstone_nowcast_facilities_transportation_2024.yaml results that we have most recently been using
• Delete all of the GHG FBS that we have recently created for testing purposes. There are no code/method changes in this PR, we are only renaming FBS method yamls and deleting outdated yamls
• First pass on cleaning up references - there are still references in comments and configs/ to FBS yaml methods that this PR removes. This is known and we are leaving these as is for now, will address in future PR
• Electricity G/T/D v3 runs on the nowcast path, where eGRID is already in GHG_national_Cornerstone_nowcast_{year} (use egrid elec data in ghg FBS #911), so the separate *_egrid overlays are unused.
• Adds pinned versions of the GHG FBS to configs and waterfalls

Testing

Compared GHG_national_Cornerstone_2024.yaml to GHG_national_Cornerstone_nowcast_facilities_transportation_2024.yaml on main - ensure the FBS results are the same

[bl-young]

I reviewed the consolidation claim for this draft: that GHG_national_Cornerstone_{year} matches the prior NFT stack (nowcast + facilities + transportation), not facilities-only.

Config equivalence. Restored the deleted NFT YAML chain from main and ran diff_methods (resolved config) for each year 2017-2024:

GHG_national_Cornerstone_nowcast_facilities_transportation_{year} vs GHG_national_Cornerstone_{year} -> no differences in every year.

This is as of f8872eb

[bl-young]

test_fbs_regeneration passes upon updating the GHG FBS pin name:
https://github.com/cornerstone-data/bedrock/actions/runs/39522590920/job/118969718561

But note see other integration test failures: https://github.com/cornerstone-data/bedrock/actions/runs/39522590920/job/118969718846

[catherinebirney]

marked ready too soon - I'm working on the failing tests

@bl-young tests now pass, ready for review

[bl-young]

Review (PR #1133)

Tip: 5cd421d8 on cb__ghg-fbs-yaml-cleanup. Default CI (format / lint / typecheck / test) is green through 801bd080; confirm CI + test_integration on 5cd421d8 after push.

What already looks good

• Resolved diff_methods: NFT vs GHG_national_Cornerstone_{year} matches for 2017-2024.
• FBS regen: cornerstone_ghg_fbs_2024_pin.json method is GHG_national_Cornerstone_2024 vs NFT golden parquet -- passed on integration run 39522590920.
• Nowcast balance job: passed on that run.
• Cross-stem pins: 801bd080 -- when cornerstone_ghg_fbs_filename is set, that exact object loads even if its stem differs from the selected method. Canonical v0_6, v0.4/v0.5 ladder, and waterfall G4/G5 can keep historical nowcast/facilities/NFT hashes.
• Electricity FBS path: 5cd421d8 -- drop _EGRID_FBS_METHOD_BY_SOURCE_YEAR / egrid_fbs_method_for_* / _load_egrid_fbs_for_electricity_disagg. All GHG loads use GHG_national_Cornerstone_{year} (year YAMLs embed eGRID). Supported (usa_detail_io_source, usa_ghg_data_year) pairs live on USAConfig as ELECTRICITY_DISAGG_SUPPORTED_IO_YEAR. Delete test_egrid_fbs_method.py; add allow-list unit tests in test_usa_config.py.

Integration run 39522590920 -- what failed and why (historical)

Job split: FBS regen / nowcast balance green; test_cornerstone_usa 9 failed. That run is pre-801bd080 and pre-5cd421d8.

A. Stem mismatch (fixed in code by 801bd080)

ValueError: cornerstone_ghg_fbs_filename '...' does not match the selected FBS stem 'GHG_national_Cornerstone_2024'

Test	Pin that mismatched
test_published_B_matches_snapshot	NFT ...82e24cd (canonical)
test_b_usa_non_finetuned_snapshot	same
test_domestic_lci_equals_e	same
waterfall v06_waterfall_g4_nowcast	...nowcast_2024_...0e1b0f0
waterfall v06_waterfall_g5_facilities	...facilities_2024_...a166975
waterfall 2025_usa_cornerstone_v0_6	NFT ...82e24cd

Load-time stem error is gone on tip. A re-dispatch may still surface content failures (snapshot / waterfall sheet drift).

B. Electricity path (fixed in code by 5cd421d8)

Test	Error on 39522590920
TestElectricityDisaggregationPipeline::test_pipeline_aq_and_diagnostics	FileNotFoundError ... base_name='GHG_national_Cornerstone_2023_egrid', then AttributeError: ... pop
test_reagg_internals_stay_407_published_are_405	same
test_reagg_identities_on_scaled_q	same

Mechanism (old): fixtures test_usa_config_waste_disagg_electricity_{disaggregation,reaggregation}.yaml use usa_ghg_data_year: 2023, no pin. Disagg selected GHG_national_Cornerstone_2023_egrid via the deleted method map; no YAML / GCS object; getFlowBySector fallback -> NoneType.pop.

Fix: load year stem; allow-list (bea_published, 2023|2024) and (nowcast, 2024) on USAConfig. Product v0_3 electricity configs that pin ..._2024_egrid_...672c37c keep working via cross-stem pin load.

What remains for merge-ready shape

1. Re-dispatch test_integration on tip including 5cd421d8 -- expect groups A and B load failures gone; watch for new content/snapshot failures.
2. Follow-ups (later PR OK):
   • Drop no-op use_facility_ghg_attribution (field + YAML keys + waterfall asserts).
   • Stale comments / analysis scripts still naming nowcast_* / *_egrid stems (PR body calls this intentional for now).
   • Optional: re-upload blessed FBS under GHG_national_Cornerstone_{year}_v…_<hash> and align pin filenames with the year stem.

Tests removed / adapted

Change	Verdict
Delete test_egrid_yaml_schema_shape.py	Appropriate
Adapt facilities + industrial distillate tests -> year YAML	Appropriate
Delete test_ghg_nests_point_at_energy_cornerstone	Reasonable; optional Energy-nest follow-up
test_usa_config.py asserts historical pin filenames	Fine with cross-stem pins
Delete test_egrid_fbs_method.py; add ELECTRICITY_DISAGG_SUPPORTED_IO_YEAR tests	Appropriate (5cd421d8)

Workflows

No workflow hardcodes of old FBS stems. No rename needed. Weekday fbs_regeneration is the right numerical guard for YAML vs NFT golden.

Suggested merge bar

• Config equivalence NFT vs year YAML (2017-2024)
• Cross-stem pin load (801bd080)
• FBS regen green (method pin update)
• Electricity: year-stem load + USAConfig IO/year allow-list (5cd421d8)
• test_integration re-run on tip (5cd421d8): groups A/B load failures gone
• (optional follow-up) retire use_facility_ghg_attribution + stale stem docs

[bl-young]

@jvendries could you do a review of this specifically how it relates to electricity work (see especially 5cd421d)

[jvendries]

Electricity disagg focused review (Project 29 electricity G/T/D v3)

cc @bl-young

Verdict: Approve from the v3 side. Project 29 does not use the *_egrid YAML files this PR deletes. 5cd421d is the electricity change that matters, and it is correct.

v3 runs on the nowcast path. Electric-power emissions already come from plant-level eGRID inside the nowcast GHG method (#911), not from a separate GHG_national_Cornerstone_{year}_egrid file. The year YAML on this branch still calls egrid_to_sector, so that stays in place.

Before this PR, electricity disaggregation used a special loader: published configs looked up those *_egrid YAMLs; nowcast used GHG_national_Cornerstone_nowcast_{year}. After this PR, every path loads GHG_national_Cornerstone_{year}, or a pinned parquet if the config names one. That matches how v3 actually runs.

The config check for which year/IO pairs can turn on disaggregation is the same as before: published 2023, published 2024, and nowcast 2024. Cross-stem pins (801bd08) still let older electricity configs load their historical parquets (v0.3 *_egrid and v0.4 nowcast).

#1002 is unchanged: there is still no check that EIA-923 fuel receipts and eGRID heat input tell the same story.

Follow-ups (not merge blockers)

• A nowcast + disaggregation run without a parquet pin now loads the full year method (nowcast + facilities + transportation), not nowcast-only. Existing configs that pin the nowcast parquet are unchanged. A new config with no pin follows canonical 2024.
• We still cannot turn on the G/T/D split on nowcast 2017–2023. That is fine for now under v3, which currently does G/T/D on nowcast 2024. The 2018/2022 MECS work retargets the manufacturing electricity Use row; it does not use this flag. Expanding the year list for disaggregation to work with other nowcasted years is follow up work under v3.
• Two electricity diagnostic scripts still talk about the old *_egrid method. decompose_d_n_step.py says the 3-way split reloads GHG_national_Cornerstone_2024_egrid. year_alignment_bly_e.py still describes a separate eGRID FBS branch. Update that copy so the next v3 run does not look for a method that is gone.

[bl-young]

Integration looks good on tip a8b4066: test_integration #652 green (test_cornerstone_usa, FBS regen, nowcast balance).

That clears the remaining merge-bar item from the earlier review (stem mismatches + eGRID path + the transient usa NameError). Config equivalence 2017-2024 and cross-stem pins still stand. Optional follow-ups (use_facility_ghg_attribution, stale stem docs) can wait.

Approving.