Add 2024 ATB Update Pipeline

[Yunzhi-Chen]

Summary

Adds a workflow that converts corrected ATB 2024 data into ReEDS-compatible inputs. It downloads, formats, plots, and validates ATB data, fills historical years from an explicitly configured source, optionally smooths future years, and produces before/after plots for review.

Structure

atb/
├── config.yaml            # all user choices; validated by config.schema.json
├── scripts/               # pipeline (13 modules)
├── historical/            # prepared history, committed — atb.csv, real.csv, manual.csv
├── manual_input/          # projection-year config (CSP ratios, offshore multipliers, coal overrides)
├── scraped_input/         # raw download cache (git-ignored)
├── output/                # generated ReEDS inputs (git-ignored)
└── comparison/            # validation and before/after plots (git-ignored)

How to run

cd atb
python scripts/future_atb_scraper.py     # cache the ATB flat file and workbook
python scripts/run_pipeline.py           # format, plot, compare

run_pipeline.py runs offline against the committed history and never rewrites it. Use --only format to generate CSVs without plots.

To rebuild history from original sources:

python scripts/historical_data_scraper.py    # --no-download to rebuild from cache

Verification

With smoothing disabled and all metrics set to manual, the workflow reproduces the inputs ReEDS uses today. Of 32 files: 21 exact, 8 differing only by floating-point noise (≤1e-9 relative), 3 offshore files differing by up to 0.018% in the CF multiplier, consistent with a different ATB 2024 release than the pinned v3.0.0.

Sanity-check plots — smoothing off, manual history, vs current ReEDS:

sanity_check_2024.zip

New ATB 2024 plots for future years:

New ATB 2024 plots and the components:

New ATB 2024 vs current ReEDS 2024:

plot_comparison_2024.zip

Scope and open questions

• Geothermal is not generated;

LLM usage

LLM tools assisted with implementation, documentation, debugging, and validation. Outputs were reviewed and tested against the current ReEDS inputs.