ci(labels): add a labeler config audit against the canonical label set

[ss-o]

Addresses the durable part of #527: making labeler drift detectable instead of silent.

The failure mode

A dangling labeler reference never fails. actions/labeler applies labels through the issues API, which creates a label that does not exist rather than erroring. So a labeler.yml naming a deleted legacy label silently recreates it on the next matching pull request, and an org-wide label cleanup partially undoes itself with nothing in any log to explain why.

This is not hypothetical. enhancement ✨ returned to z-shell/zi after #467 deleted it org-wide, because zi's labeler.yml maps that exact name to *.zsh and lib/**.

What this adds

scripts/labeler-config-audit.rb, a read-only audit reporting labeler.yml keys that are absent from lib/labels.yml, annotated with the canonical replacement where legacy_migrations records one.

A live run across all 93 repositories finds 162 non-canonical references in 23 repositories:

Referenced label	Repos	Canonical replacement
documentation 📝	23	type:docs
enhancement ✨	23	type:maintenance
maintenance 📈	23	type:maintenance
ci 🤖	23	area:ci
dependencies 📦	22	area:dependencies
i18n 🌐	22	none recorded
annex 🌀	11	area:annex
plugin ⚙️	10	area:plugin
package 📦	3	area:package
annex 💠	1	none recorded
plugin 🧿	1	none recorded

Of those 23 repositories, 21 are in label-sync scope and 2 are forks.

Note this counts something slightly wider than "references that would recreate a label". 162 is every reference to a non-canonical name; the subset naming a label that does not currently exist, and would therefore be created on the next matching pull request, was 139 across the 21 in-scope repositories. The wider definition is the right one for a gate, because a reference to a non-canonical label that happens to exist today is still drift.

Design notes

• Read-only, deliberately. Which labels land on a repository's pull requests is a per-repository editorial call rather than a uniform mechanical substitution, and three of the referenced names have no canonical replacement to substitute. The script reports; the owning repository edits.
• Both labeler schemas. v4 and v5 are both in use across the org. Only top-level keys are label names in either, so the audit reads keys and ignores values entirely.
• A missing config is absence, not a finding. Reporting every repository without a labeler config would make the audit noisy enough to ignore.
• Only a 404 means "no config". Any other API error surfaces rather than being silently reported as a clean repository.

Tests

11 unit tests against an injected fixture client, no live API calls, following the scripts/test-repo-settings-audit.rb pattern. They cover both schemas, missing configs, the .yaml extension, legacy keys resolving to a replacement, unmapped keys reporting a null replacement rather than a guess, invalid YAML, exit codes, and the 404-versus-other-error distinction.

$ ruby scripts/test-labeler-config-audit.rb
PASS test_canonical_only_config_is_clean
PASS test_clean_repo_exits_zero
PASS test_invalid_yaml_raises
PASS test_json_output_and_exit_code_on_drift
PASS test_legacy_key_reports_its_replacement
PASS test_missing_config_is_not_a_finding
PASS test_missing_target_argument_is_a_usage_error
PASS test_non_404_api_error_propagates
PASS test_unmapped_key_reports_nil_replacement
PASS test_v5_schema_reads_only_top_level_keys
PASS test_yaml_extension_is_also_read

The workflow also self-audits this repository. That step is a no-op today, since z-shell/.github has no labeler config, and exists so that adding one cannot introduce drift unnoticed.

Uses no non-default gems: base64 stopped being a default gem in Ruby 3.4, so decoding goes through core unpack1("m").

Not included

Rewriting the 23 labeler.yml files, which is per-repository work across separate repositories, and deciding the three unmapped names (i18n 🌐, annex 💠, plugin 🧿). Both remain open on #527.

Related: #467, #524, #527

[ss-o]

Correction to the framing in this PR description, recorded here rather than silently amended.

The description says a drifted labeler.yml recreates labels on the next matching pull request, and applies that to all 23 repositories. That only holds where a workflow actually runs actions/labeler. Checking every workflow file in the 23:

• 17 run actions/labeler, so the config is live and does recreate labels.
• 6 have no workflow consuming the config (F-Sy-H, fzf, z-a-meta-plugins, zsh, zsh-select, zsh-zoxide), so it is inert.

z-shell/zi, the confirmed instance cited in the description, is in the live group, so the motivating example stands. Full detail in #527.

This does not change the code or the tests. The audit reports config drift against the canonical label set, which is the right thing for it to report either way: an inert labeler.yml is still drift, and it is exactly what turns live the moment someone adds the workflow. It does change the urgency attached to 6 of the 23 rows in the table above.

[ss-o]

Second correction, and this one retracts the motivating claim in the description above.

The description says a drifted labeler.yml recreates deleted labels on the next matching pull request, and cites z-shell/zi as a confirmed instance. Both are wrong.

• Recreation needs the workflow enabled. Of the 17 repositories with a labeler workflow, 12 are active and 5 are disabled_manually, including z-shell/zi.
• It also needs the workflow to succeed. The shared template passes repo-token: "${{ secrets.GH_PAT }}", and that secret is not populated. Triggering it produced Error: Parameter token or opts.auth is required, so the action exits before applying any label.
• zi's labeler is disabled, so it applied nothing. The zi#373 timeline shows a human applied enhancement ✨ on 2026-08-15. The real cause is the one in chore(labels): finish recovery from the 2026-05-16 label strip #524: chore(labels): run the legacy label cleanup across the organization #467's delete pass did not converge in zi, leaving the label selectable.

Full detail and retraction in #527.

What this means for this PR

The code and tests are unaffected, and I am not proposing changes to them. The audit reports labeler.yml keys that are absent from lib/labels.yml. That is what it does, what it is tested for, and what its output says. It never claimed anything about recreation, and its findings are unchanged: 162 non-canonical references across 23 repositories.

What changes is the justification. This is configuration-drift detection and hygiene, not a guard against an active regression. I would rather you evaluate it on that basis than on the stronger claim I originally made for it.

The value that survives is still real: these configs name labels that no longer exist, they carry Docusaurus-template leftovers into repositories with no such paths, and nothing else in CI would notice. It becomes materially more important if GH_PAT is ever populated or the disabled workflows are re-enabled, since the drift would then start applying deleted labels for real.