From e261af37453109052cca265fd2395f40d272d294 Mon Sep 17 00:00:00 2001 From: Ricardo Dias Date: Tue, 13 Oct 2026 15:12:21 +0100 Subject: [PATCH 01/16] docs: design spec for the pre-release OBS project release flow Two-hop release: staging -> per-PR pre-release project (QA runs there, staging stays writable) -> release project on merge. Multi-major release PRs, source_commit record in release.yaml, no sync-main lock. Signed-off-by: Ricardo Dias --- ...-prerelease-project-release-flow-design.md | 352 ++++++++++++++++++ 1 file changed, 352 insertions(+) create mode 100644 docs/superpowers/specs/2026-10-13-prerelease-project-release-flow-design.md diff --git a/docs/superpowers/specs/2026-10-13-prerelease-project-release-flow-design.md b/docs/superpowers/specs/2026-10-13-prerelease-project-release-flow-design.md new file mode 100644 index 000000000..315c6c3f5 --- /dev/null +++ b/docs/superpowers/specs/2026-10-13-prerelease-project-release-flow-design.md @@ -0,0 +1,352 @@ +# Pre-release OBS project release flow + +Date: 2026-10-13 +Status: approved design, not yet implemented + +## Problem + +The release pipeline copies binaries from `ppg:staging:` into +`ppg:releases:` at merge time of the release PR. Because the binaries +being released live in staging until that moment, the whole window from +"release cut" to "release copied" has to keep staging immutable: + +- QA for the release runs against staging, so any staging change invalidates + it. +- `obs-release.yml` disables `sync-main.yml` for the duration of the copy (the + "release lock"), because merging the release PR raises the computed + `PPG_RELEASE` counter and a staging sync would retag the container images + the release is still copying. +- `sync release` refuses to run when the staging tree changed since the tag + (`git diff ..HEAD`). +- Releases dispatched close together displace each other in the + `sync-main-sync` concurrency group, so a PR cannot safely release more than + one PG major. + +Development on staging is blocked while a release is in flight, and a QA +failure found on the candidate forces a fix-and-redo cycle under that lock. + +## Goals + +1. Staging stays writable while a release candidate is under QA. +2. QA runs against the exact binaries that will be released, and a QA failure + is fixed in staging and re-tested without restarting the release. +3. One release PR can release several PG majors (and the `common` tools + release) at once. +4. The release record in git names the staging tree the binaries were built + from. +5. No sync-main lock, no counter race, no displaced dispatches. + +## Non-goals + +- Labelling QA pipelines as release / nightly / PR lanes. Until that exists a + release runs every lane declared in the staging project and its + subprojects. +- Changing `sync release-pr`; it stays a manual recovery command. +- Mirroring package sources into the release tier. Release projects remain + binary-only snapshots described by `project.yaml` mirrors. + +## Overview + +The release becomes two `osc release` hops with a frozen intermediate project: + +``` +ppg:staging:18 ──(hop 1, PR open)──▶ :pr-N:ppg:releases:18 ──(hop 2, merge)──▶ ppg:releases:18 + freeze staging QA runs here no staging involvement +``` + +- `percona-obs project release` cuts the release locally and records the + staging source commit. +- Opening the release PR makes `obs-pr-check.yml` populate a **pre-release + project** under the PR namespace (hop 1) and run every QA lane against it. +- Fixes go into staging through normal PRs; the release PR is refreshed and + repopulated until QA is green. +- Merging tags the release and dispatches `obs-release.yml` once, which copies + the pre-release into the release project (hop 2), publishes the GitHub + releases and deletes the pre-release. + +## 1. OBS layout and lifecycle + +### Naming + +For release PR number N and release mirror `ppg:releases:18` the pre-release +project is `:pr-N:ppg:releases:18`, with nested subprojects +`…:containers`, `…:extras`, `…:extras:containers`, `…:tarballs`, one per +mirror directory. The name is the ordinary PR-profile resolution (PR rootprj + +project id); there is no new naming rule. A multi-version PR holds one such +tree per `release.yaml` it changes, side by side. + +### Meta + +Each pre-release project's meta is produced from the mirror `project.yaml` in +the PR branch exactly as the production release project's would be: same +repositories, `build: false`, project-config copied, subproject and +repository paths rewritten to the PR root the way `sync push --branch-from` +rewrites them. The `qa:` blocks carried into the mirrors (section 2) resolve +`${OBS_ROOTPRJ}` to the PR root, so QA lands on the pre-release. + +**Publish is forced on** in the pre-release regardless of the mirror's +`publish:` flags, so Jenkins always has repositories to consume. The mirror's +own flags apply only on hop 2. + +### Instance slicing + +Both hops run once per `OBS_INSTANCES` entry with that instance's profile. +The slice is applied by the primitives both commands share: + +- subprojects: only mirror directories `project_in_slice` admits; +- repositories: pruned to the profile's include/exclude globs on meta apply, + and the copy loop only releases repositories that exist on the target; +- freeze scope: computed from the in-slice source tree. + +So on labs the pre-release tree is the container subprojects with the +`ubi*`/`UBI_*`/`images` repositories; on build.opensuse.org it is the top +project, `extras` and `tarballs` with everything else. QA routing is the +existing `resolve_qa_instance` by `qa_types`. + +### Lifecycle + +- Created or re-applied on every push to the release PR by `obs-pr-check.yml`, + and on demand through a `workflow_dispatch` entry of the populate job that + takes a PR number. +- Repopulated on every push and on every dispatch; repopulation is the + update-in-place `osc release` path, idempotent. +- Deleted by `obs-release.yml` after hop 2 succeeded on every instance, not by + `obs-pr-cleanup.yml`. A failed hop 2 leaves the pre-release in place for + re-dispatch. `obs-pr-cleanup.yml` still deletes it for a release PR closed + unmerged; `obs-stale-cleanup.yml` still covers abandoned PRs. + +### One in-flight release PR per major + +`obs-pr-check.yml`'s `resolve` job fails for a release PR when another open PR +(found through the GitHub search API) changes the same `release.yaml`. The +first one opened wins. Two PRs releasing the same major would race on the +release counter and on hop 2. + +## 2. Tool changes + +### `release.yaml` format + +Entries gain a mapping form; plain strings remain accepted for history: + +```yaml +project: ppg:staging:18 +releases: + - ppg/18.4-1 + - ppg/18.6-1 + - tag: ppg/18.7-1 + source_commit: 9e9d9738c0… # full SHA of main +``` + +The last entry is the current release, as today. `source_commit` is the SHA +of `main` whose staging tree the pre-release was built from. + +### `project release` (existing command, three changes) + +1. **Open-release refresh.** If the last entry's tag does not exist in git, + the release id is still open: the command keeps that entry, regenerates + the mirror tree, replaces the existing `CHANGELOG.md` section for that id + instead of prepending a new one, and updates `source_commit`. Only when + the last tag exists does it derive the next id and append. + `--release-id` still overrides. This is what makes the fix loop (section + 5) a re-run of the same command. +2. **Source commit.** Resolved as the head SHA of the last successful + `sync-main.yml` run on `main`, read through the GitHub API (`gh api`), + overridable with `--source-commit `. The command fails if the SHA is + not an ancestor of HEAD, so the branch always contains the tree it claims + to release. +3. **QA blocks in mirrors.** The mirror writer carries each staging + project's `qa:` block into the corresponding mirror. Release mirrors are + `standalone` and have no `macros.yaml`, so the writer renders `%!{…}` + macros with the staging chain at write time (e.g. + `OBS_PROJECT: ${OBS_ROOTPRJ}:ppg:releases:18`, `VERSION: ppg-18.6`); + `ppg:staging:` inside parameter values is rewritten to + `ppg:releases:` the same way subproject paths are. Only `${…}` env + tokens survive, resolved per profile at QA time. + +### `sync prerelease ` (new command) + +Runs with the PR profile. Steps: + +1. Read `release.yaml`; require a mapping entry with `source_commit`. +2. **Record check**: diff the freeze-scope paths between `source_commit` and + the head of the last successful sync-main run; any difference aborts with + "staging changed since the cut, re-run `project release`". Runs before + anything is frozen. +3. Create or re-apply the pre-release project tree from the mirror + `project.yaml` files, publish forced on, builds disabled. +4. Drain, assert green, freeze staging plus the aggregate sources, + `osc release` into the pre-release for the top project and every in-slice + subproject, verify binaries landed, restore. This is today's + `_freeze_and_run` path, moved into the shared module. +5. Write `source_commit` into each pre-release project's description so hop 2 + can re-check it without git. + +`--dry-run` keeps today's checks (mirrors complete, changelog section +present, tag unused, source reachable, staging green) and performs steps 1–2. + +### `sync release --from ` (changed) + +With `--from` the command loses the freeze and the git divergence check: + +1. Re-check that the pre-release description's `source_commit` equals the + tagged `release.yaml` entry's. +2. Apply the release project meta from the mirror (repo additions, publish + flags, prjconf), builds disabled. +3. `osc release --target-project --no-delay` from the + pre-release for the top project and every in-slice subproject. No + temporary `releasetarget` entries are needed on the pre-release. +4. Verify binaries landed (bounded by `--verify-timeout`). + +Without `--from` the command keeps its current staging-sourced behaviour as a +recovery path. + +### Shared module `percona_obs/release_copy.py` + +Holds what both hops use: subproject pair collection +(`_collect_release_subprojects`), meta apply, the copy loop, and verify. +`release_freeze.py` keeps the freeze primitives, `release_scope.py` the scope +computation. `cmd_sync_release` and the new `cmd_sync_prerelease` become thin +orchestration. + +## 3. Workflows + +### `obs-pr-check.yml` + +- `resolve` gains the one-in-flight guard for release PRs. +- The release path replaces today's `sync release --dry-run` with a + `populate` job, one leg per `OBS_INSTANCES` entry, serialised on that + instance's `sync-main-sync-` concurrency group (no + cancel-in-progress) so population never overlaps a staging upload. Each leg + runs `sync prerelease` with the PR profile for every `release.yaml` the PR + changes. The PR comment lists the pre-release projects, the source commit + and the per-leg outcome. +- QA runs automatically after `populate` through the existing + `detect-qa-matrix` and `qa` jobs, with no trigger label. The matrix is + discovered from the PR project's subprojects on each instance, so every + lane of every released major is included and each lane's `status_context` + carries its project name. +- `workflow_dispatch` with a `pr_number` input runs the same `populate` and + QA jobs on the PR head (repopulate without a push). + +### `obs-pr-cleanup.yml` + +- Merged release PR: create one tag per changed `release.yaml` on + `merge_commit_sha` (the last rebased commit), then dispatch + `obs-release.yml` exactly once with `pr_number` and the list of tags. No + lock, no per-tag dispatch loop. +- The `delete` job is skipped for a merged release PR. It still runs for a + release PR closed unmerged and for every other PR. + +### `obs-release.yml` + +- Inputs: `pr_number`, `tags` (space separated). Manual recovery: + `gh workflow run obs-release.yml -f pr_number=N -f tags="ppg/17.12-1 ppg/18.7-1"`. +- Per `OBS_INSTANCES` leg, checked out at the first tag (all tags share the + commit): `sync release --from :pr-N:` + for every tag in order. The `lock` job, the sync-main disable/enable and + the lock-state reporting are deleted. The per-instance concurrency group + stays (hop 2 writes release project meta). +- `finalize` after all legs: one GitHub release per tag from its `CHANGELOG.md` + section, then `sync delete --yes --recursive` of the PR project tree on + every instance. Runs only when every leg succeeded. + +### `sync-main.yml` + +The release-only push skip is removed. Merging a release PR raises +`PPG_RELEASE`; staging images are retagged on the next sync and nothing races +it, because the released images already sit in the pre-release. + +### `obs-stale-cleanup.yml` + +Unchanged. + +## 4. Error handling and recovery + +### Hop 1 (PR open) + +| Failure | Behaviour | Recovery | +|---|---|---| +| Staging not green / drain timeout | `sync prerelease` fails before copying; freeze restored in `finally`; PR comment names the packages | Fix staging, repopulate | +| Staging moved since the cut | Record check fails before any freeze, printing the staging-path diff | Re-run `project release`, push | +| Partial copy (OBS error mid `osc release`) | Freeze restored; pre-release holds a mix; verify fails loudly | Repopulate (update-in-place, idempotent) | +| Push during population | Second run queues behind the first in the per-instance group | None needed | + +Nothing writes to the PR branch, so there is no bot commit, no trigger loop +and no orphan head commit. + +### Hop 2 (after merge) + +| Failure | Behaviour | Recovery | +|---|---|---| +| A leg fails | `finalize` skips GitHub releases and the delete; tag exists; pre-release intact; staging untouched | Re-dispatch with the same `pr_number` and `tags` (idempotent) | +| Record mismatch (pre-release description ≠ tagged entry) | Leg refuses | The merged record is wrong: open a new release PR | +| Pre-release already deleted | Leg fails on missing source project | Repopulate via `workflow_dispatch` of obs-pr-check with the PR number (works while the PR head ref resolves), then re-dispatch obs-release | +| Dispatch failure in obs-pr-cleanup | Tags exist, nothing copied | Manual dispatch | + +### What disappears + +The sync-main lock, the "sync-main is DISABLED" annotations, the +pending-cancellation race, the release-only push skip in sync-main, and the +`git diff ..HEAD` divergence check. + +## 5. User process + +1. `project release ppg:staging:18` (and `ppg:staging:17`, `ppg:staging:common` + on the same branch for a combined release). Push the branch, open the PR. +2. `obs-pr-check.yml` populates the pre-release per instance and runs every + staging QA lane against it. +3. QA fails: fix the package in staging through a normal PR; once merged and + synced, rebase the release branch, re-run `project release` (refreshes the + open id, the changelog section and `source_commit`), push. When only + staging changed and the changelog is unaffected, + `gh workflow run obs-pr-check.yml -f pr_number=N` repopulates instead. +4. Everything green: merge. `obs-pr-cleanup.yml` tags and dispatches + `obs-release.yml` once; it copies the pre-release into the release + projects, creates the GitHub releases, deletes the pre-release. + +## 6. Testing + +Unit tests in `tests/`: + +- `release.yaml` parsing of string and mapping entries; last-entry selection. +- `project release` open-release refresh rule (tag absent → refresh in place; + tag present → next id appended); CHANGELOG section replacement. +- QA block rewrite in the mirror writer (macro rendering, project path + rewrite, env tokens preserved). +- Record check diff logic in `sync prerelease`. +- `sync release --from` argument plumbing and the description re-check. + +Workflow logic is validated by a rehearsal on labs with a throwaway release +id through the whole loop: populate, deliberate repopulate, merge, manual +re-dispatch of hop 2. + +## 7. Documentation + +- `docs/PERCONA_OBS_TOOL.md`: "Releasing packages" rewritten around the two + hops, the `release.yaml` mapping form, the refresh rule, recovery recipes. +- `.github/copilot-instructions.md`: workflow descriptions for + `obs-pr-check`, `obs-pr-cleanup`, `obs-release`, `sync-main`. +- `root/README.md`, `root/ppg/releases/README.md`: tier description mentions + the pre-release. + +## 8. Rollout + +1. **Tool PR**: `release_copy.py`, `sync prerelease`, `sync release --from`, + `project release` changes, tests. Mergeable on its own; `sync release` + without `--from` is unchanged so the old flow keeps working. +2. **Workflows PR**: obs-pr-check populate path, obs-pr-cleanup single + dispatch, obs-release rewrite, sync-main skip removal, docs. One PR, since + they only make sense together. +3. **Rehearsal** on labs before the next real release train. + +## Decisions log + +- Pre-release lives under the PR namespace (not a fixed `ppg:prerelease:` + nor a per-release-id project): reuses PR lifecycle, cleanup and QA fan-out. +- Release QA = all staging lanes, until pipeline labelling exists. +- Population and QA are fully automatic on a release PR, no labels. +- `source_commit` is recorded by `project release` on the developer's + machine, not committed back by CI: a workflow-token push would become an + untested PR head and block required checks. +- `osc release --target-project` is used for hop 2 so the pre-release meta + stays a pure mirror with no `releasetarget` entries. From 0508cad2d49b1a6759a3848f44aed1ce42b32257 Mon Sep 17 00:00:00 2001 From: Ricardo Dias Date: Tue, 13 Oct 2026 15:27:43 +0100 Subject: [PATCH 02/16] docs: implementation plan for the pre-release flow tool PR Signed-off-by: Ricardo Dias --- .../plans/2026-10-13-prerelease-flow-tool.md | 2248 +++++++++++++++++ ...6-10-13-prerelease-flow-tool.md.tasks.json | 96 + 2 files changed, 2344 insertions(+) create mode 100644 docs/superpowers/plans/2026-10-13-prerelease-flow-tool.md create mode 100644 docs/superpowers/plans/2026-10-13-prerelease-flow-tool.md.tasks.json diff --git a/docs/superpowers/plans/2026-10-13-prerelease-flow-tool.md b/docs/superpowers/plans/2026-10-13-prerelease-flow-tool.md new file mode 100644 index 000000000..63e809950 --- /dev/null +++ b/docs/superpowers/plans/2026-10-13-prerelease-flow-tool.md @@ -0,0 +1,2248 @@ +# Pre-release Flow — Tool PR Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:subagent-driven-development (recommended) or superpowers-extended-cc:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Land the `percona-obs` side of the two-hop release flow (rollout step 1 of the spec): `release.yaml` mapping entries, `sync prerelease`, `sync release --from`, the shared `release_copy` module, and the `project release` changes (source commit, open-release refresh, QA blocks in mirrors). + +**Architecture:** Two new small modules (`release_record.py` for `release.yaml` entries, `github_api.py` for the `gh api` lookup of the last synced commit) plus one shared orchestration module (`release_copy.py`) that both hops call. `cmd_sync_release` is refactored onto `release_copy` with unchanged behaviour, then gains `--from`; `cmd_sync_prerelease` is a new thin command. `project release` gains three features in `cmd_project.py`. Workflow changes are a separate plan (rollout step 2); only the one `release.yaml` reader in `obs-pr-cleanup.yml` is touched here so a mapping-form entry cannot break tagging if a release is cut before step 2 lands. + +**Tech Stack:** Python 3.11, `osc` library + `osc` CLI, PyYAML, pytest (monkeypatch style, no network), black, pyright, `gh` CLI. + +**Spec:** `docs/superpowers/specs/2026-10-13-prerelease-project-release-flow-design.md` + +## Global Constraints + +- Work in the worktree `.claude/worktrees/prerelease-flow` on branch `prerelease-flow` (off `percona/main`). Never touch the primary checkout. +- After every code change run `venv/bin/black percona_obs/` then `venv/bin/pyright` (must report `0 errors`), then `venv/bin/pytest -q`. All three must pass before a task's commit. +- Commits: `git commit -s`. No `Co-Authored-By` lines of any kind (CLAUDE.md rule overrides the session reminder). Never `git push`, never open a PR. +- No OBS or GitHub network access in tests: every `osc.core.*`, `subprocess.run` and `gh` call is monkeypatched. +- `release.yaml` string entries (`- ppg/18.4-1`) must keep working everywhere; the mapping form is `{tag: , source_commit: }`. +- The pre-release source project is addressed through a profile passed as `--branch-from` (same convention as `sync push`), never guessed from the PR root. +- `osc release` keeps the temporary `` mechanism (`_add_release_targets` / `_remove_release_targets` in a `finally`) for both hops, and additionally passes `--target-project `. Reason: `osc release` documents "It requires defined release targets set to trigger=manual"; `--target-project` only selects among them. The pre-release meta therefore carries release targets only transiently during a copy, which satisfies the spec's intent (a pure mirror at rest). +- The OBS-side copy of `source_commit` is a `source_commit: ` line appended to the pre-release project's ``; it is rewritten at the end of every population because `_apply_project_config` overwrites the description from the mirror first. +- Spec section numbering is referenced below as "spec §N". + +**User decisions (already made):** +- Pre-release lives under the PR namespace: `:pr-N::releases:`. +- Release QA runs every `qa:` lane declared in the staging project and its subprojects (no label filter yet). +- Population and QA are automatic on a release PR (workflow plan), no trigger labels. +- `source_commit` is recorded locally by `project release`; CI never commits back to the PR branch. +- `project release` refreshes an open release id in place when its tag does not exist yet. +- Approach A: two commands (`sync prerelease`, `sync release --from`) on a shared `release_copy.py`. +- Multi-major release PRs are supported; one in-flight release PR per major (enforced in the workflow plan). + +--- + +## File structure + +| File | Responsibility | +|---|---| +| `percona_obs/release_record.py` (new) | Parse/dump `release.yaml` `releases:` entries (string and mapping forms, legacy `revision:`). | +| `percona_obs/github_api.py` (new) | `gh api` wrapper: repo slug from a clone URL, head SHA of the last sync-main run whose sync jobs succeeded. | +| `percona_obs/release_copy.py` (new) | Shared hop mechanics: subproject pair collection, mirror meta apply, `osc release` copy of a project tree, verify, orphan report. | +| `percona_obs/obs_api.py` | `publish_override` on `_apply_project_config`; project description marker read/write. | +| `percona_obs/cmd_sync.py` | `cmd_sync_release` refactored onto `release_copy`, then `--from`; new `cmd_sync_prerelease`. | +| `percona_obs/cmd_project.py` | `project release`: QA blocks in mirrors, `source_commit`, open-release refresh. | +| `percona_obs/common.py` | `compute_ppg_release` reads entries through `release_record`. | +| `percona_obs/cli.py` | New `sync prerelease` parser, new flags on `sync release` and `project release`. | +| `.github/workflows/obs-pr-cleanup.yml` | The one-line `release.yaml` reader understands the mapping form. | +| `docs/PERCONA_OBS_TOOL.md`, `.github/copilot-instructions.md`, `CLAUDE.md` | Command reference and key-file tables. | +| `tests/test_release_record.py`, `tests/test_github_api.py`, `tests/test_release_copy.py`, `tests/test_sync_prerelease.py` (new); `tests/test_sync_release.py`, `tests/test_project_release.py`, `tests/test_computed_macros.py`, `tests/test_obs_api_markers.py` (new) | Unit tests. | + +--- + +### Task 0: Worktree environment and baseline + +**Goal:** A working `venv/` in the worktree and a green baseline so later failures are attributable. + +**Files:** +- Create: `venv/` (git-ignored) in `.claude/worktrees/prerelease-flow` + +**Acceptance Criteria:** +- [ ] `venv/bin/pytest -q` passes on the untouched branch +- [ ] `venv/bin/black --check percona_obs/` and `venv/bin/pyright` pass + +**Verify:** `venv/bin/pytest -q && venv/bin/black --check percona_obs/ && venv/bin/pyright` → all green, pyright prints `0 errors` + +**Steps:** + +- [ ] **Step 1: Create the venv** + +```bash +cd /home/rdias/Work/percona-obs-packaging/.claude/worktrees/prerelease-flow +python3 -m venv venv +venv/bin/pip install -q -r requirements.txt +venv/bin/pip install -q pytest +``` + +- [ ] **Step 2: Run the baseline** + +Run: `venv/bin/pytest -q && venv/bin/black --check percona_obs/ && venv/bin/pyright` +Expected: all tests pass, black reports nothing to change, pyright `0 errors`. + +No commit (nothing changed). + +--- + +### Task 1: `release_record.py` and mapping-form `release.yaml` entries + +**Goal:** One parser for `releases:` entries used by every reader (`compute_ppg_release`, `project release`, `sync release`, the cleanup workflow one-liner), accepting strings, `{tag, source_commit}` mappings and legacy `revision:`. + +**Files:** +- Create: `percona_obs/release_record.py` +- Modify: `percona_obs/common.py:319-346` (`compute_ppg_release`) +- Modify: `percona_obs/cmd_project.py:2293-2300` (existing release list) +- Modify: `percona_obs/cmd_sync.py:2628-2635` (release list in `cmd_sync_release`) +- Modify: `.github/workflows/obs-pr-cleanup.yml:138` +- Test: `tests/test_release_record.py`, `tests/test_computed_macros.py` + +**Acceptance Criteria:** +- [ ] `parse_release_entries` returns `ReleaseEntry(tag, source_commit)` for both forms and raises `SystemExit` naming an unrecognised entry +- [ ] `compute_ppg_release` counts a mapping entry's tag (`{tag: ppg/18.4-2}` counts for 18.4) +- [ ] `project release` and `sync release` still work with string-only files (existing tests pass) +- [ ] The cleanup one-liner prints the tag for a mapping last entry + +**Verify:** `venv/bin/pytest -q tests/test_release_record.py tests/test_computed_macros.py tests/test_project_release.py tests/test_sync_release.py` → PASS + +**Steps:** + +- [ ] **Step 1: Write the failing tests** + +`tests/test_release_record.py`: + +```python +"""Unit tests for percona_obs.release_record.""" + +import pytest + +from percona_obs.release_record import ( + ReleaseEntry, + current_release, + dump_release_entry, + parse_release_entries, + release_tags, +) + + +def test_parse_string_and_mapping_entries(): + data = { + "releases": [ + "ppg/18.4-1", + {"tag": "ppg/18.6-1"}, + {"tag": "ppg/18.7-1", "source_commit": "abc123"}, + ] + } + assert parse_release_entries(data) == [ + ReleaseEntry("ppg/18.4-1"), + ReleaseEntry("ppg/18.6-1"), + ReleaseEntry("ppg/18.7-1", "abc123"), + ] + assert release_tags(data) == ["ppg/18.4-1", "ppg/18.6-1", "ppg/18.7-1"] + assert current_release(data) == ReleaseEntry("ppg/18.7-1", "abc123") + + +def test_parse_legacy_revision_and_empty(): + assert parse_release_entries({"revision": "ppg/17.9-1"}) == [ + ReleaseEntry("ppg/17.9-1") + ] + assert parse_release_entries({}) == [] + assert parse_release_entries("not a dict") == [] + assert current_release({}) is None + + +def test_parse_rejects_garbage_entry(): + with pytest.raises(SystemExit, match="unrecognised releases entry"): + parse_release_entries({"releases": [{"source_commit": "x"}]}) + + +def test_dump_round_trip(): + assert dump_release_entry(ReleaseEntry("ppg/18.4-1")) == "ppg/18.4-1" + assert dump_release_entry(ReleaseEntry("ppg/18.7-1", "abc")) == { + "tag": "ppg/18.7-1", + "source_commit": "abc", + } +``` + +Append to `tests/test_computed_macros.py`: + +```python +MAPPING_RELEASE_YAML = """\ +project: ppg:staging:18 +releases: +- ppg/18.3-1 +- tag: ppg/18.4-1 + source_commit: 9e9d9738c0ffee +""" + + +def test_compute_counts_mapping_entries(): + assert compute_ppg_release("ppg", "18.4", MAPPING_RELEASE_YAML) == "2" + assert compute_ppg_release("ppg", "18.3", MAPPING_RELEASE_YAML) == "2" + + +def test_compute_tolerates_garbage_entries(): + assert compute_ppg_release("ppg", "18.4", "releases: [{x: 1}]\n") == "1" +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `venv/bin/pytest -q tests/test_release_record.py tests/test_computed_macros.py` +Expected: FAIL with `ModuleNotFoundError: percona_obs.release_record` and `assert '1' == '2'`. + +- [ ] **Step 3: Create `percona_obs/release_record.py`** + +```python +"""``release.yaml`` ``releases:`` entries. + +Two entry forms coexist: the plain tag string (``- ppg/18.4-1``, all +historical entries) and the mapping form written by ``project release`` +since the pre-release flow (``- {tag: ppg/18.7-1, source_commit: }``). +``source_commit`` is the full SHA of ``main`` whose staging tree the +release's binaries were built from. The legacy single ``revision:`` field +is still read as a one-element list. +""" + +from dataclasses import dataclass +from typing import Any + + +@dataclass(frozen=True) +class ReleaseEntry: + tag: str + source_commit: "str | None" = None + + +def parse_release_entries(data: Any) -> "list[ReleaseEntry]": + """Return the ``releases:`` list of a loaded release.yaml as entries. + + Raises SystemExit naming the offending item for anything that is neither + a string nor a mapping with a string ``tag``. + """ + if not isinstance(data, dict): + return [] + raw = data.get("releases") + if not raw and data.get("revision"): + return [ReleaseEntry(str(data["revision"]))] + if not isinstance(raw, list): + return [] + entries: list[ReleaseEntry] = [] + for item in raw: + if isinstance(item, str): + entries.append(ReleaseEntry(item)) + elif isinstance(item, dict) and isinstance(item.get("tag"), str): + sc = item.get("source_commit") + entries.append(ReleaseEntry(item["tag"], str(sc) if sc else None)) + else: + raise SystemExit( + f"error: release.yaml: unrecognised releases entry {item!r} " + "(expected a tag string or {tag: ..., source_commit: ...})" + ) + return entries + + +def release_tags(data: Any) -> "list[str]": + return [e.tag for e in parse_release_entries(data)] + + +def current_release(data: Any) -> "ReleaseEntry | None": + """The last entry is the release in flight (or just shipped).""" + entries = parse_release_entries(data) + return entries[-1] if entries else None + + +def dump_release_entry(entry: ReleaseEntry) -> "str | dict[str, str]": + """YAML-ready value: a bare tag when there is nothing else to record.""" + if entry.source_commit is None: + return entry.tag + return {"tag": entry.tag, "source_commit": entry.source_commit} +``` + +- [ ] **Step 4: Route the three readers through it** + +`percona_obs/common.py`, inside `compute_ppg_release`, replace the block + +```python + raw = data.get("releases") + tags = [str(t) for t in raw] if isinstance(raw, list) else [] + if not tags and data.get("revision"): + tags = [str(data["revision"])] +``` + +with + +```python + from .release_record import release_tags # local: release_record is leaf + + try: + tags = release_tags(data) + except SystemExit: + return "1" +``` + +`percona_obs/cmd_project.py` (around line 2293), replace + +```python + existing_releases: list[str] = [] + if not is_first_release: + existing_data = load_yaml(release_dir / "release.yaml") + raw = existing_data.get("releases") + if isinstance(raw, list): + existing_releases = [str(r) for r in raw] + elif existing_data.get("revision"): + existing_releases = [str(existing_data["revision"])] +``` + +with + +```python + existing_entries: list[ReleaseEntry] = [] + if not is_first_release: + existing_entries = parse_release_entries( + load_yaml(release_dir / "release.yaml") + ) + existing_releases: list[str] = [e.tag for e in existing_entries] +``` + +and add `from .release_record import ReleaseEntry, parse_release_entries` to the imports. Keep `new_releases_list = existing_releases + [tag]` for now (Task 8 rewrites it). + +`percona_obs/cmd_sync.py` in `cmd_sync_release`, replace + +```python + releases_list: list[str] = release_data.get("releases") or [] + if not releases_list and release_data.get("revision"): + releases_list = [str(release_data["revision"])] + tag = releases_list[-1] if releases_list else "" +``` + +with + +```python + releases_list = release_tags(release_data) + tag = releases_list[-1] if releases_list else "" +``` + +and import `from .release_record import release_tags`. + +`.github/workflows/obs-pr-cleanup.yml` line 138, replace the python one-liner string with: + +``` +"import sys,yaml; d=yaml.safe_load(sys.stdin); r=d.get('releases') or []; last=r[-1] if r else d.get('revision') or ''; print(last.get('tag','') if isinstance(last, dict) else last)" +``` + +- [ ] **Step 5: Run tests, format, type-check** + +Run: `venv/bin/black percona_obs/ && venv/bin/pyright && venv/bin/pytest -q` +Expected: PASS, `0 errors`. + +- [ ] **Step 6: Commit** + +```bash +git add percona_obs/release_record.py percona_obs/common.py percona_obs/cmd_project.py percona_obs/cmd_sync.py .github/workflows/obs-pr-cleanup.yml tests/test_release_record.py tests/test_computed_macros.py +git commit -s -m "release.yaml: accept {tag, source_commit} entries via release_record" +``` + +--- + +### Task 2: `github_api.py` — last synced head SHA + +**Goal:** Resolve "the commit of `main` whose staging tree OBS currently holds" as the head SHA of the newest completed `sync-main.yml` run on `main` whose sync jobs all concluded `success`, through `gh api`. + +**Files:** +- Create: `percona_obs/github_api.py` +- Test: `tests/test_github_api.py` + +**Acceptance Criteria:** +- [ ] `repo_slug_from_url` handles `https://github.com/percona/obs-packaging.git`, `https://github.com/percona/obs-packaging`, `git@github.com:percona/obs-packaging.git` +- [ ] `last_synced_head_sha` skips a run whose `percona-obs sync push (...)` job failed even if the run's other jobs passed, and returns the next older qualifying run's `head_sha` +- [ ] A `gh` failure raises `SystemExit` with the stderr text; no qualifying run raises `SystemExit` + +**Verify:** `venv/bin/pytest -q tests/test_github_api.py` → PASS + +**Steps:** + +- [ ] **Step 1: Write the failing tests** + +`tests/test_github_api.py`: + +```python +"""Unit tests for percona_obs.github_api (gh is monkeypatched).""" + +import pytest + +import percona_obs.github_api as gha + + +@pytest.mark.parametrize( + "url", + [ + "https://github.com/percona/obs-packaging.git", + "https://github.com/percona/obs-packaging", + "git@github.com:percona/obs-packaging.git", + ], +) +def test_repo_slug_from_url(url): + assert gha.repo_slug_from_url(url) == "percona/obs-packaging" + + +def test_repo_slug_rejects_unknown(): + with pytest.raises(SystemExit, match="cannot derive owner/repo"): + gha.repo_slug_from_url("file:///tmp/x") + + +def _fake_api(monkeypatch, runs, jobs_by_run): + def fake(path): + if "/workflows/sync-main.yml/runs" in path: + return {"workflow_runs": runs} + for run_id, jobs in jobs_by_run.items(): + if f"/runs/{run_id}/jobs" in path: + return {"jobs": jobs} + raise AssertionError(path) + + monkeypatch.setattr(gha, "_gh_api_json", fake) + + +def test_last_synced_skips_runs_with_failed_sync_job(monkeypatch): + runs = [ + {"id": 3, "head_sha": "c" * 40}, + {"id": 2, "head_sha": "b" * 40}, + {"id": 1, "head_sha": "a" * 40}, + ] + jobs = { + 3: [ + {"name": "percona-obs sync push (boo)", "conclusion": "success"}, + {"name": "percona-obs sync push (labs)", "conclusion": "failure"}, + {"name": "Wait for OBS builds (boo)", "conclusion": "success"}, + ], + 2: [ + {"name": "percona-obs sync push (boo)", "conclusion": "success"}, + {"name": "percona-obs sync push (labs)", "conclusion": "success"}, + {"name": "Wait for OBS builds (boo)", "conclusion": "failure"}, + ], + } + _fake_api(monkeypatch, runs, jobs) + assert gha.last_synced_head_sha("percona/obs-packaging") == "b" * 40 + + +def test_last_synced_no_qualifying_run(monkeypatch): + _fake_api(monkeypatch, [{"id": 1, "head_sha": "a" * 40}], {1: []}) + with pytest.raises(SystemExit, match="no completed sync-main.yml run"): + gha.last_synced_head_sha("percona/obs-packaging") + + +def test_gh_failure_is_reported(monkeypatch): + class R: + returncode = 1 + stdout = "" + stderr = "gh: not logged in" + + monkeypatch.setattr(gha.subprocess, "run", lambda *a, **k: R()) + with pytest.raises(SystemExit, match="not logged in"): + gha._gh_api_json("repos/x/y") +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `venv/bin/pytest -q tests/test_github_api.py` +Expected: FAIL with `ModuleNotFoundError`. + +- [ ] **Step 3: Create `percona_obs/github_api.py`** + +```python +"""GitHub REST lookups through the ``gh`` CLI. + +Only ``gh`` is used (no token handling in the tool): locally the operator is +logged in, in CI ``GH_TOKEN`` is set. Every call is a subprocess so tests +monkeypatch ``_gh_api_json``. +""" + +import json +import re +import subprocess +from typing import Any + +_SLUG_RE = re.compile(r"github\.com[/:]([^/\s]+/[^/\s]+?)(?:\.git)?/?$") + +# Job name prefix of sync-main.yml's upload legs ("percona-obs sync push ()"). +SYNC_JOB_PREFIX = "percona-obs sync push" + + +def repo_slug_from_url(url: str) -> str: + """``https://github.com/o/r.git`` / ``git@github.com:o/r.git`` → ``o/r``.""" + m = _SLUG_RE.search(url.strip()) + if not m: + raise SystemExit(f"error: cannot derive owner/repo from {url!r}") + return m.group(1) + + +def _gh_api_json(path: str) -> Any: + r = subprocess.run(["gh", "api", path], capture_output=True, text=True) + if r.returncode != 0: + raise SystemExit(f"error: gh api {path} failed: {r.stderr.strip()}") + return json.loads(r.stdout or "null") + + +def last_synced_head_sha( + repo_slug: str, + *, + workflow_file: str = "sync-main.yml", + branch: str = "main", + job_prefix: str = SYNC_JOB_PREFIX, + max_runs: int = 20, +) -> str: + """Head SHA of the newest completed *workflow_file* run on *branch* whose + sync jobs (name starting with *job_prefix*) all concluded ``success``. + + The run's own conclusion is ignored on purpose: the build poll job fails + the run after the upload already happened, and that upload is what OBS + holds. + """ + data = _gh_api_json( + f"repos/{repo_slug}/actions/workflows/{workflow_file}/runs" + f"?branch={branch}&status=completed&per_page={max_runs}" + ) + for run in data.get("workflow_runs") or []: + jobs = ( + _gh_api_json(f"repos/{repo_slug}/actions/runs/{run['id']}/jobs?per_page=100") + .get("jobs") + or [] + ) + sync_jobs = [j for j in jobs if str(j.get("name", "")).startswith(job_prefix)] + if sync_jobs and all(j.get("conclusion") == "success" for j in sync_jobs): + return str(run["head_sha"]) + raise SystemExit( + f"error: no completed {workflow_file} run on {branch} of {repo_slug} " + f"with successful sync jobs in the last {max_runs} runs; " + "pass --source-commit explicitly" + ) +``` + +- [ ] **Step 4: Run tests, format, type-check** + +Run: `venv/bin/black percona_obs/ && venv/bin/pyright && venv/bin/pytest -q tests/test_github_api.py` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add percona_obs/github_api.py tests/test_github_api.py +git commit -s -m "github_api: look up the last synced main commit via gh" +``` + +--- + +### Task 3: `release_copy.py` and `cmd_sync_release` refactor (behaviour unchanged) + +**Goal:** Move the copy mechanics out of `cmd_sync.py` into `percona_obs/release_copy.py` and make `cmd_sync_release` a thin orchestration with a single code path for first and update releases. Behaviour must be identical; the existing tests prove it after their monkeypatch targets move. + +**Files:** +- Create: `percona_obs/release_copy.py` +- Modify: `percona_obs/cmd_sync.py` (`_collect_release_subprojects`, `_filter_release_repo_names`, `_sync_release_subprojects` removed; `cmd_sync_release` body) +- Test: `tests/test_sync_release.py` (rewired), `tests/test_release_copy.py` (new) + +**Acceptance Criteria:** +- [ ] `release_copy.copy_project_binaries` adds release targets, runs `osc release --no-delay --target-project `, removes targets in `finally` even when osc fails +- [ ] `release_copy.release_tree` copies top then each pair, applies each pair's mirror meta before its copy, verifies when `verify_timeout > 0`, warns about orphans +- [ ] `cmd_sync_release` first-release and update paths both run the freeze sequence and the same `release_tree`; all pre-existing `test_sync_release.py` assertions hold + +**Verify:** `venv/bin/pytest -q tests/test_sync_release.py tests/test_release_copy.py` → PASS + +**Steps:** + +- [ ] **Step 1: Write the failing tests for the new module** + +`tests/test_release_copy.py`: + +```python +"""Unit tests for percona_obs.release_copy (OBS and osc monkeypatched).""" + +from pathlib import Path + +import pytest + +import percona_obs.release_copy as rc + + +def _stub_obs(monkeypatch, calls): + monkeypatch.setattr( + rc, "_read_project_release_source", lambda apiurl, prj: ([], None) + ) + monkeypatch.setattr( + rc, "filter_release_repo_names", lambda apiurl, names, target: names + ) + monkeypatch.setattr( + rc, "_add_release_targets", lambda *a, **k: calls.append(("add", a[1], a[2])) + ) + monkeypatch.setattr( + rc, + "_remove_release_targets", + lambda *a, **k: calls.append(("remove", a[1], a[2])), + ) + monkeypatch.setattr( + rc, "_apply_project_config", lambda *a, **k: calls.append(("meta", a[1])) or (False, True) + ) + monkeypatch.setattr(rc, "_disable_project_builds", lambda *a, **k: None) + monkeypatch.setattr( + rc, "_fetch_obs_subproject_names", lambda apiurl, prefix: set() + ) + + +def test_copy_project_binaries_uses_target_project_and_restores(monkeypatch): + calls = [] + _stub_obs(monkeypatch, calls) + seen = {} + monkeypatch.setattr( + rc.subprocess, "run", lambda cmd, **k: seen.setdefault("cmd", cmd) + ) + rc.copy_project_binaries("http://obs", "S", "T") + assert "--target-project" in seen["cmd"] and "T" in seen["cmd"] + assert seen["cmd"][seen["cmd"].index("release") + 1] == "S" + assert calls == [("add", "S", "T"), ("remove", "S", "T")] + + +def test_copy_project_binaries_restores_on_failure(monkeypatch): + calls = [] + _stub_obs(monkeypatch, calls) + + def boom(*a, **k): + raise RuntimeError("osc failed") + + monkeypatch.setattr(rc.subprocess, "run", boom) + with pytest.raises(RuntimeError): + rc.copy_project_binaries("http://obs", "S", "T") + assert calls[-1] == ("remove", "S", "T") + + +def test_release_tree_order_and_verify(monkeypatch, tmp_path): + calls = [] + _stub_obs(monkeypatch, calls) + monkeypatch.setattr( + rc.subprocess, "run", lambda cmd, **k: calls.append(("osc", cmd[4])) + ) + monkeypatch.setattr( + rc, + "verify_release_landed", + lambda apiurl, s, t, timeout_s=600: calls.append(("verify", s, t)), + ) + pairs = [("containers", tmp_path / "containers")] + rc.release_tree( + "http://obs", "S", "T", pairs, "home:Admin", {}, verify_timeout=5 + ) + assert [c for c in calls if c[0] in ("osc", "meta")] == [ + ("osc", "S"), + ("meta", "T:containers"), + ("osc", "S:containers"), + ] + assert ("verify", "S", "T") in calls + assert ("verify", "S:containers", "T:containers") in calls + + +def test_release_tree_publish_override_reaches_meta(monkeypatch, tmp_path): + seen = {} + calls = [] + _stub_obs(monkeypatch, calls) + monkeypatch.setattr( + rc, + "_apply_project_config", + lambda *a, **k: seen.setdefault("publish", k.get("publish_override")) + or (False, True), + ) + monkeypatch.setattr(rc.subprocess, "run", lambda *a, **k: None) + rc.release_tree( + "http://obs", + "S", + "T", + [("x", tmp_path)], + "home:Admin", + {}, + publish_override=True, + verify_timeout=0, + ) + assert seen["publish"] is True + + +def test_warn_orphans(monkeypatch, capsys): + monkeypatch.setattr( + rc, + "_fetch_obs_subproject_names", + lambda apiurl, prefix: {"T:containers", "T:containers:ubi9"}, + ) + rc.warn_orphans("http://obs", "T", [("containers", Path("/x"))], "home:Admin") + out = capsys.readouterr().out + assert "T:containers:ubi9" in out and "sync delete" in out +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `venv/bin/pytest -q tests/test_release_copy.py` +Expected: FAIL with `ModuleNotFoundError`. + +- [ ] **Step 3: Create `percona_obs/release_copy.py`** + +Move `_collect_release_subprojects` (cmd_sync.py 2440-2480) and `_filter_release_repo_names` (2482-2498) here verbatim under the names `collect_release_subprojects` and `filter_release_repo_names` (docstrings kept), and add the new functions: + +```python +"""Shared mechanics of the two release hops. + +Hop 1 (``sync prerelease``): staging → per-PR pre-release project. +Hop 2 (``sync release --from``): pre-release → release project. +Both apply mirror project.yaml meta to the target tree, copy binaries with +``osc release`` and verify the copy landed. The legacy single-hop +``sync release`` (no ``--from``) uses the same primitives. + +``osc release`` requires ```` entries on the +source, so a copy adds them, runs osc with ``--target-project`` and removes +them again in a ``finally`` — the source meta is unchanged at rest. +""" + +import os +import subprocess +import sys +from pathlib import Path + +from .common import ( + _REPO_DIR, + _print_ok, + _print_pending, + find_projects, + resolve_project_path, +) +from .obs_api import ( + _add_release_targets, + _apply_project_config, + _disable_project_builds, + _fetch_obs_project_repository_names, + _fetch_obs_subproject_names, + _read_project_release_source, + _remove_release_targets, +) +from .project_config import project_in_slice +from .release_freeze import verify_release_landed + +_OSC_BIN = os.path.join(os.path.dirname(sys.executable), "osc") + + +# --- moved verbatim from cmd_sync (renamed, no leading underscore) --- +# def collect_release_subprojects(source_project_id, release_path): ... +# def filter_release_repo_names(apiurl, source_repo_names, release_obs_project): ... + + +def apply_mirror_meta( + apiurl: str, + target_obs_project: str, + mirror_path: Path, + rootprj: str, + env_vars: "dict[str, str]", + *, + publish_override: "dict[str, bool] | bool | None" = None, +) -> None: + """Create or update *target_obs_project* from the mirror project.yaml and + make sure builds are disabled. Two passes when OBS stripped paths.""" + paths_stripped, _ = _apply_project_config( + apiurl, + target_obs_project, + mirror_path, + rootprj, + env_vars=env_vars, + publish_override=publish_override, + ) + if paths_stripped: + _apply_project_config( + apiurl, + target_obs_project, + mirror_path, + rootprj, + env_vars=env_vars, + publish_override=publish_override, + ) + _disable_project_builds(apiurl, target_obs_project) + + +def copy_project_binaries( + apiurl: str, source_obs_project: str, target_obs_project: str +) -> None: + """``osc release`` every repo of *source* that exists in *target*.""" + repo_elems, _ = _read_project_release_source(apiurl, source_obs_project) + repo_names = filter_release_repo_names( + apiurl, + [r.get("name", "") for r in repo_elems if r.get("name")], + target_obs_project, + ) + _add_release_targets(apiurl, source_obs_project, target_obs_project, repo_names) + try: + _print_pending(f"releasing {source_obs_project} → {target_obs_project}") + subprocess.run( + [ + _OSC_BIN, + "-A", + apiurl, + "release", + source_obs_project, + "--no-delay", + "--target-project", + target_obs_project, + ], + check=True, + ) + finally: + _remove_release_targets(apiurl, source_obs_project, target_obs_project) + _print_ok(f"released {source_obs_project} → {target_obs_project}") + + +def warn_orphans( + apiurl: str, + target_obs_project: str, + pairs: "list[tuple[str, Path]]", + rootprj: str, +) -> None: + """Report target-tier OBS subprojects with no local mirror (never deleted).""" + local_names = {name for name, _ in pairs} + for obs_sub in sorted(_fetch_obs_subproject_names(apiurl, target_obs_project)): + sub_name = obs_sub[len(target_obs_project) + 1 :] + if sub_name not in local_names: + print( + f"warning: orphaned release subproject on OBS: {obs_sub} " + "(no local mirror) — remove manually with " + f"'percona-obs sync delete {obs_sub[len(rootprj) + 1 :]}'", + flush=True, + ) + + +def release_tree( + apiurl: str, + source_obs_project: str, + target_obs_project: str, + pairs: "list[tuple[str, Path]]", + rootprj: str, + env_vars: "dict[str, str]", + *, + publish_override: "dict[str, bool] | bool | None" = None, + verify_timeout: int = 600, +) -> None: + """Copy the top project and every (subproject, mirror) pair, then verify. + + The caller has already applied the top project's meta (that happens + before any freeze); subproject meta is applied here, right before each + subproject's copy, exactly as the single-hop release always did. + """ + copy_project_binaries(apiurl, source_obs_project, target_obs_project) + for subproject_name, mirror_path in pairs: + target_sub = f"{target_obs_project}:{subproject_name}" + apply_mirror_meta( + apiurl, + target_sub, + mirror_path, + rootprj, + env_vars, + publish_override=publish_override, + ) + copy_project_binaries( + apiurl, f"{source_obs_project}:{subproject_name}", target_sub + ) + if verify_timeout > 0: + verify_release_landed( + apiurl, source_obs_project, target_obs_project, timeout_s=verify_timeout + ) + for subproject_name, _ in pairs: + verify_release_landed( + apiurl, + f"{source_obs_project}:{subproject_name}", + f"{target_obs_project}:{subproject_name}", + timeout_s=verify_timeout, + ) + warn_orphans(apiurl, target_obs_project, pairs, rootprj) +``` + +`_apply_project_config`'s `publish_override` keyword does not exist until Task 4; add it there before running pyright on this task, or do Task 4's Step 3 edit now (it is a 4-line change) — either order, both must land before this task's commit. Simplest: apply Task 4 Step 3 now and still complete Task 4's tests in Task 4. + +- [ ] **Step 4: Refactor `cmd_sync_release`** + +In `percona_obs/cmd_sync.py`: +1. Delete `_collect_release_subprojects`, `_filter_release_repo_names`, `_sync_release_subprojects`. +2. Add `from .release_copy import apply_mirror_meta, collect_release_subprojects, release_tree`. +3. Replace the call `_collect_release_subprojects(source_project_id, release_path)` with `collect_release_subprojects(...)`. +4. Replace everything from `_print_pending(f"checking release project {release_obs_project}")` to the end of the function with: + +```python + _print_pending(f"checking release project {release_obs_project}") + project_exists = _obs_project_exists(apiurl, release_obs_project) + + if not args.force: + _check_source_unchanged_since_tag(source_project_id, tag, args) + + if project_exists: + # Re-apply the local release project config so repo additions, + # publish flags and prjconf changes reach OBS on update releases too. + apply_mirror_meta( + apiurl, release_obs_project, release_path, args.rootprj, shared_env_vars + ) + else: + # First release: snapshot the source topology, builds disabled. + _create_release_project(apiurl, source_obs_project, release_obs_project) + _copy_project_conf(apiurl, source_obs_project, release_obs_project) + + _freeze_and_run( + lambda: release_tree( + apiurl, + source_obs_project, + release_obs_project, + subproject_pairs, + args.rootprj, + shared_env_vars, + verify_timeout=args.verify_timeout, + ) + ) + + suffix = f" (update: {tag})" if project_exists and tag else "" + _print_ok(f"release {source_obs_project} → {release_obs_project}{suffix}") +``` + +and extract the duplicated divergence check into a module-level helper placed right above `cmd_sync_release`: + +```python +def _check_source_unchanged_since_tag(source_project_id: str, tag: str, args) -> None: + """Abort when the source project's tree changed since *tag* (legacy + single-hop release only; ``--skip-tag-check`` skips, no tag = nothing).""" + if not tag: + return + if getattr(args, "skip_tag_check", False): + _print_same(f"skipping tag divergence check for {tag} (--skip-tag-check)") + return + source_path = resolve_project_path(source_project_id) + _print_pending(f"checking for changes in {source_project_id} since tag {tag}") + result = subprocess.run( + ["git", "diff", "--name-only", f"{tag}..HEAD", "--", str(source_path)], + capture_output=True, + text=True, + cwd=_REPO_DIR, + ) + if result.returncode != 0: + print(result.stderr, end="", file=sys.stderr) + raise SystemExit(f"error: git diff failed — is the tag '{tag}' present locally?") + changed = [l.strip() for l in result.stdout.splitlines() if l.strip()] + if changed: + print( + f"error: source project {source_project_id} has changed since tag {tag}:", + file=sys.stderr, + ) + for f in changed: + print(f" {f}", file=sys.stderr) + raise SystemExit( + "The source project has diverged from the release tag. " + "Use --force to skip this check." + ) +``` + +Remove the now-unused imports from cmd_sync (`_add_release_targets`, `_remove_release_targets`, `_read_project_release_source`, `_fetch_obs_project_repository_names`, `_disable_project_builds` if no other use, `verify_release_landed`) — pyright/black will not flag unused imports, so grep each name before removing: `grep -n "" percona_obs/cmd_sync.py`. + +- [ ] **Step 5: Rewire `tests/test_sync_release.py`** + +- Change the import line to `from percona_obs.release_copy import collect_release_subprojects` and `import percona_obs.release_copy as rc`; drop `_sync_release_subprojects`. +- `test_collect_pairs_and_missing`, `test_collect_release_subprojects_respects_slice`: call `collect_release_subprojects`, patch `rc.resolve_project_path` (not `cmd_sync.`). +- `test_missing_mirror_is_hard_error` and `test_orphan_reporting` tested `_sync_release_subprojects`; replace them with: + +```python +def test_missing_mirror_is_hard_error(tmp_path, monkeypatch): + src, rel = _mk_tree(tmp_path, monkeypatch) + _wire_release_update_path(monkeypatch, src, rel) + args = _Args(project="ppg:releases:17", rootprj="home:Admin") + with pytest.raises(SystemExit, match="tarballs"): + cmd_sync.cmd_sync_release(args) +``` + +(the orphan behaviour is now covered by `tests/test_release_copy.py::test_warn_orphans`). +- In `_wire_release_update_path` and `_wire_release_first_path`, keep the `cmd_sync.*` patches for `resolve_project_path`, `_REPO_DIR`, `osc.conf.config`, `_obs_project_exists`, `_create_release_project`, `_copy_project_conf`, and patch the moved touchpoints on `rc`: `rc._apply_project_config → (False, None)`, `rc._disable_project_builds`, `rc._read_project_release_source → ([], None)`, `rc.filter_release_repo_names → names`, `rc._add_release_targets`, `rc._remove_release_targets`, `rc._fetch_obs_subproject_names → set()`, `rc.resolve_project_path` (same lambda as cmd_sync's). +- Every `monkeypatch.setattr(cmd_sync.subprocess, "run", ...)` that intercepts `osc release` becomes `rc.subprocess`; `cmd_sync.subprocess.run` is still used by the git divergence check, which `_Args.skip_tag_check = True` bypasses. +- Every `monkeypatch.setattr(cmd_sync, "verify_release_landed", ...)` becomes `rc`. +- `test_filter_release_repo_names_warns_and_drops`: import/patch on `rc` (`rc._fetch_obs_project_repository_names`). + +- [ ] **Step 6: Run tests, format, type-check** + +Run: `venv/bin/black percona_obs/ && venv/bin/pyright && venv/bin/pytest -q` +Expected: PASS, `0 errors`. In particular `test_freeze_order_and_restore_on_failure` still sees `["drain", "green", "freeze", ..., "restore"]`. + +- [ ] **Step 7: Commit** + +```bash +git add percona_obs/release_copy.py percona_obs/cmd_sync.py percona_obs/obs_api.py tests/test_sync_release.py tests/test_release_copy.py +git commit -s -m "sync release: move copy mechanics into release_copy, one code path" +``` + +--- + +### Task 4: `obs_api` — publish override and description marker + +**Goal:** Two small OBS-side primitives the new hops need: force-publish on a project built from a mirror, and a `source_commit:` marker in a project description. + +**Files:** +- Modify: `percona_obs/obs_api.py:1215-1280` (`_apply_project_config`), new functions after `_disable_project_builds` +- Test: `tests/test_obs_api_markers.py` + +**Acceptance Criteria:** +- [ ] `_apply_project_config(..., publish_override=True)` emits `` regardless of the mirror's `publish:`; `None` keeps today's behaviour +- [ ] `set_project_description_marker` appends `source_commit: ` once (replacing an existing marker line) and pushes the meta +- [ ] `read_project_description_marker` returns the value or `None` + +**Verify:** `venv/bin/pytest -q tests/test_obs_api_markers.py` → PASS + +**Steps:** + +- [ ] **Step 1: Write the failing tests** + +`tests/test_obs_api_markers.py`: + +```python +"""Unit tests for the project description marker and publish override (percona_obs.obs_api).""" + +from pathlib import Path + +import percona_obs.obs_api as obs_api + +_META = ( + 'tMirror of X\n' + "source_commit: old" +) + + +def test_marker_round_trip(monkeypatch): + pushed = {} + monkeypatch.setattr(obs_api.osc.core, "show_project_meta", lambda a, p: _META) + monkeypatch.setattr( + obs_api, "_edit_project_meta", lambda a, p, meta, force: pushed.setdefault("meta", meta) or False + ) + obs_api.set_project_description_marker("http://obs", "P", "source_commit", "new") + assert pushed["meta"].count("source_commit:") == 1 + assert "source_commit: new" in pushed["meta"] + assert "Mirror of X" in pushed["meta"] + + monkeypatch.setattr(obs_api.osc.core, "show_project_meta", lambda a, p: pushed["meta"]) + assert obs_api.read_project_description_marker("http://obs", "P", "source_commit") == "new" + assert obs_api.read_project_description_marker("http://obs", "P", "other") is None + + +def test_publish_override_forces_enable(monkeypatch, tmp_path): + (tmp_path / "project.yaml").write_text( + "publish: {r: false}\nrepositories:\n - name: r\n archs: [x86_64]\n paths: []\n" + ) + seen = {} + monkeypatch.setattr( + obs_api, + "build_project_meta", + lambda *a, **k: seen.setdefault("publish", k.get("publish")) or "", + ) + import urllib.error + + def not_found(*a, **k): + raise urllib.error.HTTPError("u", 404, "nf", {}, None) + + monkeypatch.setattr(obs_api.osc.core, "show_project_meta", not_found) + monkeypatch.setattr(obs_api.osc.core, "show_project_conf", not_found) + monkeypatch.setattr(obs_api, "_fetch_root_project_managed_elements", lambda a, r: []) + monkeypatch.setattr(obs_api, "_inject_obs_managed_elements", lambda m, e: m) + obs_api._apply_project_config( + "http://obs", "root:P", tmp_path, "root", dry_run=True, publish_override=True + ) + assert seen["publish"] is True +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `venv/bin/pytest -q tests/test_obs_api_markers.py` +Expected: FAIL (`AttributeError: set_project_description_marker`, `TypeError: unexpected keyword publish_override`). + +- [ ] **Step 3: Implement** + +In `_apply_project_config`, add the keyword parameter `publish_override: "dict[str, bool] | bool | None" = None` after `existing_branch_projects`, extend the docstring with "``publish_override`` replaces the mirror's ``publish:`` flags (the pre-release forces publishing on so QA always has repositories)", and change the `build_project_meta(...)` call to pass + +```python + publish=( + publish_override + if publish_override is not None + else project_config.get("publish") + ), +``` + +Add after `_disable_project_builds`: + +```python +def set_project_description_marker( + apiurl: str, obs_project_name: str, key: str, value: str +) -> None: + """Append (or replace) a ``: `` line in the project description. + + Used to pin ``source_commit`` on a pre-release project so the final hop + can check it without git. ``_apply_project_config`` rewrites the + description from the mirror, so callers set the marker last. + """ + raw = _decode_obs_response(osc.core.show_project_meta(apiurl, obs_project_name)) + root = ET.fromstring(raw) + desc = root.find("description") + if desc is None: + desc = ET.SubElement(root, "description") + kept = [ + line + for line in (desc.text or "").splitlines() + if not line.startswith(f"{key}: ") + ] + desc.text = "\n".join([*kept, f"{key}: {value}"]).strip() + ET.indent(root, space=" ") + _edit_project_meta( + apiurl, obs_project_name, ET.tostring(root, encoding="unicode"), force=False + ) + _print_update(f"project meta {obs_project_name} ({key}: {value})") + + +def read_project_description_marker( + apiurl: str, obs_project_name: str, key: str +) -> "str | None": + raw = _decode_obs_response(osc.core.show_project_meta(apiurl, obs_project_name)) + desc = ET.fromstring(raw).findtext("description") or "" + for line in desc.splitlines(): + if line.startswith(f"{key}: "): + return line[len(key) + 2 :].strip() or None + return None +``` + +- [ ] **Step 4: Run tests, format, type-check** + +Run: `venv/bin/black percona_obs/ && venv/bin/pyright && venv/bin/pytest -q` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add percona_obs/obs_api.py tests/test_obs_api_markers.py +git commit -s -m "obs_api: publish override and project description markers" +``` + +--- + +### Task 5: `sync prerelease` (hop 1) + +**Goal:** New command that, under the PR profile, checks the `source_commit` record against the last synced `main`, creates/re-applies the pre-release tree from the mirrors (publish forced on), freezes staging, copies, verifies, restores, and pins `source_commit` in the pre-release description. + +**Files:** +- Modify: `percona_obs/cmd_sync.py` (new `cmd_sync_prerelease`, helpers `_scope_local_paths`, `_changed_files_between`) +- Modify: `percona_obs/cli.py` (new `sync prerelease` parser after `sync release`) +- Test: `tests/test_sync_prerelease.py` + +**Acceptance Criteria:** +- [ ] Without a mapping entry carrying `source_commit`, the command fails with "has no source_commit" +- [ ] A non-empty diff of scope paths between `source_commit` and the synced head aborts before `wait_for_quiesce` is called, listing the files +- [ ] Order of operations: record check → top meta (publish True) → drain → green → freeze → copy tree → restore → marker; marker is not written when the copy raised +- [ ] `--dry-run` runs the record check and the green check, writes nothing +- [ ] `--synced-commit` bypasses `gh`; `--force` bypasses the record check only + +**Verify:** `venv/bin/pytest -q tests/test_sync_prerelease.py` → PASS + +**Steps:** + +- [ ] **Step 1: Write the failing tests** + +`tests/test_sync_prerelease.py`: + +```python +"""Unit tests for `sync prerelease` (percona_obs.cmd_sync.cmd_sync_prerelease). + +OBS, osc, git and gh are all monkeypatched; the staging/release trees come +from tests.test_sync_release._mk_tree. +""" + +import pytest + +import percona_obs.cmd_sync as cmd_sync +import percona_obs.release_copy as rc +from tests.test_sync_release import _REPO_YAML, _mk_tree + +SHA = "a" * 40 + + +class _Args: + def __init__(self, **kw): + self.project = "ppg:releases:17" + self.rootprj = "home:Admin:PR:pr-5" + self.branch_from = "main" + self.synced_commit = SHA + self.source_commit = None + self.force = False + self.dry_run = False + self.no_freeze = False + self.freeze_timeout = 1 + self.verify_timeout = 0 + self.env_overrides = None + self.profile = None + self.__dict__.update(kw) + + +def _tree_with_record(tmp_path, monkeypatch, source_commit=SHA): + src, rel = _mk_tree(tmp_path, monkeypatch) + (rel / "tarballs").mkdir() + (rel / "tarballs" / "project.yaml").write_text( + "build: false\n" + _REPO_YAML.format(name="ssl3") + ) + entry = ( + f" - tag: ppg/17.11-1\n source_commit: {source_commit}\n" + if source_commit + else " - ppg/17.11-1\n" + ) + (rel / "release.yaml").write_text(f"project: ppg:staging:17\nreleases:\n{entry}") + return src, rel + + +def _wire(monkeypatch, src, rel, calls): + monkeypatch.setattr( + cmd_sync, "resolve_project_path", lambda pid: src if pid == "ppg:staging:17" else rel + ) + monkeypatch.setattr( + rc, "resolve_project_path", lambda pid: src if pid == "ppg:staging:17" else rel + ) + monkeypatch.setattr(cmd_sync, "_REPO_DIR", rel.parents[2]) + monkeypatch.setattr(cmd_sync.osc.conf, "config", {"apiurl": "http://obs"}) + monkeypatch.setattr( + cmd_sync, "_load_profile", lambda name: {"rootprj": "home:Admin", "apiurl": "http://obs"} + ) + monkeypatch.setattr(cmd_sync, "_obs_project_exists", lambda *a, **k: True) + monkeypatch.setattr(cmd_sync, "_changed_files_between", lambda *a, **k: []) + monkeypatch.setattr( + cmd_sync, + "apply_mirror_meta", + lambda apiurl, prj, path, rootprj, env, publish_override=None: calls.append( + ("meta", prj, publish_override) + ), + ) + monkeypatch.setattr(cmd_sync, "wait_for_quiesce", lambda *a, **k: calls.append("drain")) + monkeypatch.setattr(cmd_sync, "assert_all_green", lambda *a, **k: calls.append("green") or []) + monkeypatch.setattr(cmd_sync, "freeze_builds", lambda *a, **k: calls.append("freeze") or {"p": ""}) + monkeypatch.setattr(cmd_sync, "freeze_packages", lambda *a, **k: {}) + monkeypatch.setattr(cmd_sync, "restore_packages", lambda *a, **k: None) + monkeypatch.setattr(cmd_sync, "restore_builds", lambda *a, **k: calls.append("restore")) + monkeypatch.setattr( + cmd_sync, "release_tree", lambda *a, **k: calls.append(("copy", a[1], a[2], k.get("publish_override"))) + ) + monkeypatch.setattr( + cmd_sync, + "set_project_description_marker", + lambda apiurl, prj, key, value: calls.append(("marker", prj, value)), + ) + + +def test_requires_source_commit(tmp_path, monkeypatch): + src, rel = _tree_with_record(tmp_path, monkeypatch, source_commit=None) + _wire(monkeypatch, src, rel, []) + with pytest.raises(SystemExit, match="has no source_commit"): + cmd_sync.cmd_sync_prerelease(_Args()) + + +def test_record_mismatch_aborts_before_drain(tmp_path, monkeypatch): + src, rel = _tree_with_record(tmp_path, monkeypatch) + calls = [] + _wire(monkeypatch, src, rel, calls) + monkeypatch.setattr( + cmd_sync, "_changed_files_between", lambda *a, **k: ["root/ppg/staging/17/x/rpm/x.spec"] + ) + with pytest.raises(SystemExit, match="staging changed since the cut"): + cmd_sync.cmd_sync_prerelease(_Args()) + assert "drain" not in calls + + +def test_happy_path_order(tmp_path, monkeypatch): + src, rel = _tree_with_record(tmp_path, monkeypatch) + calls = [] + _wire(monkeypatch, src, rel, calls) + cmd_sync.cmd_sync_prerelease(_Args()) + assert calls[0] == ("meta", "home:Admin:PR:pr-5:ppg:releases:17", True) + assert calls[1:4] == ["drain", "green", "freeze"] + assert calls[4] == ("copy", "home:Admin:ppg:staging:17", "home:Admin:PR:pr-5:ppg:releases:17", True) + assert calls[5] == "restore" + assert calls[6] == ("marker", "home:Admin:PR:pr-5:ppg:releases:17", SHA) + + +def test_marker_not_written_when_copy_fails(tmp_path, monkeypatch): + src, rel = _tree_with_record(tmp_path, monkeypatch) + calls = [] + _wire(monkeypatch, src, rel, calls) + + def boom(*a, **k): + raise RuntimeError("osc") + + monkeypatch.setattr(cmd_sync, "release_tree", boom) + with pytest.raises(RuntimeError): + cmd_sync.cmd_sync_prerelease(_Args()) + assert "restore" in calls and not any(c[0] == "marker" for c in calls if isinstance(c, tuple)) + + +def test_dry_run_checks_only(tmp_path, monkeypatch): + src, rel = _tree_with_record(tmp_path, monkeypatch) + (rel / "CHANGELOG.md").write_text("# Changelog\n\n## [17.11-1] - 2026-01-01\n") + calls = [] + _wire(monkeypatch, src, rel, calls) + monkeypatch.setattr( + cmd_sync.subprocess, "run", lambda *a, **k: type("R", (), {"returncode": 1, "stdout": "", "stderr": ""})() + ) + cmd_sync.cmd_sync_prerelease(_Args(dry_run=True)) + assert calls == ["green"] + + +def test_scope_local_paths_maps_obs_names_back(tmp_path, monkeypatch): + src, rel = _tree_with_record(tmp_path, monkeypatch) + monkeypatch.setattr(cmd_sync, "resolve_project_path", lambda pid: src.parents[2] / pid.replace(":", "/")) + paths = cmd_sync._scope_local_paths( + src, + ["home:Admin:ppg:staging:17", "home:Admin:ppg:staging:17:containers"], + {"home:Admin:ppg:common:deps": {"geos"}}, + "home:Admin", + ) + assert src in paths + assert src.parents[2] / "ppg/common/deps/geos" in paths + assert any(p.name == "macros.yaml" for p in paths) +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `venv/bin/pytest -q tests/test_sync_prerelease.py` +Expected: FAIL with `AttributeError: cmd_sync_prerelease`. + +- [ ] **Step 3: Implement in `percona_obs/cmd_sync.py`** + +Add imports: `from .common import _macros_chain_files` (extend the existing `from .common import (...)` block), `from .obs_api import set_project_description_marker`, `from .release_record import current_release, release_tags` (merge with Task 1's import), `from .github_api import last_synced_head_sha, repo_slug_from_url`. + +Add the helpers and the command right after `cmd_sync_release`: + +```python +def _scope_local_paths( + source_path: Path, + freeze_scope: "list[str]", + freeze_packages_map: "dict[str, set[str]]", + rootprj: str, +) -> "list[Path]": + """Local git paths whose content decides the released binaries: the + source tree, its macros chain, every whole-frozen project and every + aggregate-frozen package.""" + paths: list[Path] = [source_path, *_macros_chain_files(source_path)] + for prj in freeze_scope: + p = resolve_project_path(prj[len(rootprj) + 1 :]) + if p.is_dir() and p not in paths: + paths.append(p) + for prj, pkgs in freeze_packages_map.items(): + base = resolve_project_path(prj[len(rootprj) + 1 :]) + paths.extend(base / pkg for pkg in sorted(pkgs)) + return paths + + +def _changed_files_between(a: str, b: str, paths: "list[Path]") -> "list[str]": + """``git diff --name-only a b -- paths`` (both commits must exist locally).""" + for sha in (a, b): + r = subprocess.run( + ["git", "cat-file", "-e", f"{sha}^{{commit}}"], cwd=_REPO_DIR, capture_output=True + ) + if r.returncode != 0: + raise SystemExit( + f"error: commit {sha} is not available locally; " + "fetch the repository (full history) and retry" + ) + rel = [str(p.relative_to(_REPO_DIR)) if p.is_absolute() else str(p) for p in paths] + r = subprocess.run( + ["git", "diff", "--name-only", a, b, "--", *rel], + cwd=_REPO_DIR, + capture_output=True, + text=True, + ) + if r.returncode != 0: + raise SystemExit(f"error: git diff {a} {b} failed: {r.stderr.strip()}") + return [l.strip() for l in r.stdout.splitlines() if l.strip()] + + +def cmd_sync_prerelease(args) -> None: + """Hop 1: copy a staging project tree into this PR's pre-release project. + + Runs with the PR profile (``-P pr-N``); ``--branch-from`` names the + profile of the production instance that holds staging. Steps (spec §2): + record check, pre-release meta from the mirrors with publish forced on, + drain/green/freeze staging, osc release, verify, restore, pin + ``source_commit`` in the pre-release description. + """ + release_path = resolve_project_path(args.project) + release_file = release_path / "release.yaml" + if not release_file.is_file(): + raise SystemExit( + f"error: release.yaml not found at {release_path.relative_to(_REPO_DIR)}" + ) + env_vars = parse_env_overrides(args.env_overrides) if args.env_overrides else {} + release_data = load_yaml_with_env(release_file, env_vars) + source_project_id = release_data.get("project") + if not source_project_id: + raise SystemExit(f"error: release.yaml missing 'project' field: {release_file}") + entry = current_release(release_data) + if entry is None: + raise SystemExit("error: release.yaml has no releases: entries") + source_commit = args.source_commit or entry.source_commit + if not source_commit: + raise SystemExit( + f"error: release {entry.tag} has no source_commit in release.yaml; " + "re-run 'percona-obs project release' (or pass --source-commit)" + ) + + branch_profile = _load_profile(args.branch_from) + source_rootprj = branch_profile.get("rootprj", "") + if not source_rootprj: + raise SystemExit(f"error: profile {args.branch_from!r} has no rootprj") + apiurl: str = osc.conf.config["apiurl"] + source_obs_project = f"{source_rootprj}:{source_project_id}" + prerelease_obs_project = f"{args.rootprj}:{args.project}" + source_path = resolve_project_path(source_project_id) + # Mirror paths are rewritten to the PR root through ${OBS_ROOTPRJ}. + shared_env_vars = {**env_vars, **auto_rootprj_env(args.rootprj)} + # The freeze scope lives on the production root. + source_env_vars = {**env_vars, **auto_rootprj_env(source_rootprj)} + + subproject_pairs, missing_mirrors = collect_release_subprojects( + source_project_id, release_path + ) + source_sub_obs_projects = [ + f"{source_obs_project}:{name}" for name, _ in subproject_pairs + ] + freeze_scope, freeze_packages_map = _build_freeze_scope( + source_obs_project, + source_sub_obs_projects, + source_path, + source_project_id, + source_rootprj, + source_env_vars, + ) + quiesce_scope = freeze_scope + sorted(freeze_packages_map) + + # --- record check (spec §2, step 2) --- + if not args.force: + synced = args.synced_commit or last_synced_head_sha( + repo_slug_from_url( + shared_env_vars.get("PERCONA_OBS_PACKAGING_REPO") + or _load_profile_env(args.branch_from).get("PERCONA_OBS_PACKAGING_REPO", "") + ) + ) + changed = _changed_files_between( + source_commit, + synced, + _scope_local_paths(source_path, freeze_scope, freeze_packages_map, source_rootprj), + ) + if changed: + listing = "\n".join(f" {f}" for f in changed[:50]) + raise SystemExit( + f"error: staging changed since the cut ({source_commit[:12]} → " + f"{synced[:12]}):\n{listing}\n" + "Re-run 'percona-obs project release' on a rebased branch." + ) + _print_same(f"source record {source_commit[:12]} matches synced main {synced[:12]}") + + if missing_mirrors: + listing = "\n".join(f" {source_project_id}:{n}" for n in sorted(missing_mirrors)) + raise SystemExit( + f"error: staging subprojects have no release mirror:\n{listing}\n" + "Re-run 'percona-obs project release' to regenerate the release tree." + ) + + if args.dry_run: + failures: list[str] = [] + release_id = entry.tag.split("/")[-1] + changelog_file = release_path / "CHANGELOG.md" + if not changelog_file.is_file(): + failures.append("CHANGELOG.md is missing") + elif f"## [{release_id}]" not in changelog_file.read_text("utf-8"): + failures.append(f"CHANGELOG.md has no '## [{release_id}]' section") + r = subprocess.run( + ["git", "rev-parse", "--verify", "--quiet", f"refs/tags/{entry.tag}"], + capture_output=True, + cwd=_REPO_DIR, + ) + if r.returncode == 0: + failures.append(f"tag {entry.tag} already exists") + if not _obs_project_exists(apiurl, source_obs_project): + failures.append(f"source project {source_obs_project} not found on OBS") + else: + failures.extend( + f"not green: {p}" + for p in assert_all_green(apiurl, quiesce_scope, packages=freeze_packages_map) + ) + if failures: + for f in failures: + print(f" ✗ {f}", file=sys.stderr) + raise SystemExit("error: prerelease dry-run failed") + _print_ok(f"prerelease dry-run passed for {entry.tag} (dry run)") + return + + # Top-level pre-release meta from the mirror, publish forced on (spec §1). + apply_mirror_meta( + apiurl, + prerelease_obs_project, + release_path, + args.rootprj, + shared_env_vars, + publish_override=True, + ) + + snapshots: dict[str, str] = {} + pkg_snapshots: dict[tuple[str, str], str] = {} + if not args.no_freeze: + _print_pending(f"draining scheduler for {len(quiesce_scope)} project(s)") + wait_for_quiesce( + apiurl, quiesce_scope, timeout_s=args.freeze_timeout, packages=freeze_packages_map + ) + problems = assert_all_green(apiurl, quiesce_scope, packages=freeze_packages_map) + if problems: + listing = "\n".join(f" {p}" for p in problems[:50]) + raise SystemExit(f"error: source projects are not fully green:\n{listing}") + snapshots = freeze_builds(apiurl, freeze_scope) + try: + pkg_snapshots = freeze_packages(apiurl, freeze_packages_map) + except Exception: + restore_builds(apiurl, snapshots) + raise + try: + release_tree( + apiurl, + source_obs_project, + prerelease_obs_project, + subproject_pairs, + args.rootprj, + shared_env_vars, + publish_override=True, + verify_timeout=args.verify_timeout, + ) + finally: + if pkg_snapshots: + restore_packages(apiurl, pkg_snapshots) + if snapshots: + restore_builds(apiurl, snapshots) + + set_project_description_marker( + apiurl, prerelease_obs_project, "source_commit", source_commit + ) + for name, _ in subproject_pairs: + set_project_description_marker( + apiurl, f"{prerelease_obs_project}:{name}", "source_commit", source_commit + ) + _print_ok(f"prerelease {source_obs_project} → {prerelease_obs_project} ({entry.tag})") +``` + +`_load_profile_env` is already imported in cmd_sync from `.cmd_profile` (check the import block at line 18; add it if absent). If the test's `_Args` has no `PERCONA_OBS_PACKAGING_REPO`, the `--synced-commit` path short-circuits before `gh`, so the lookup is never reached in tests. + +Note: `_freeze_and_run` is a closure inside `cmd_sync_release`; the freeze block above duplicates its ~20 lines. Extract it instead: move `_freeze_and_run`'s body to a module-level + +```python +def _run_frozen(apiurl, quiesce_scope, freeze_scope, freeze_packages_map, args, action) -> None: +``` + +used by both commands (same body, `args.no_freeze` / `args.freeze_timeout` read from `args`), and have `cmd_sync_release` call it. Then the prerelease command's freeze block becomes `_run_frozen(apiurl, quiesce_scope, freeze_scope, freeze_packages_map, args, lambda: release_tree(...))`. The tests in `tests/test_sync_release.py` patch `cmd_sync.wait_for_quiesce` etc., which still works at module level. + +- [ ] **Step 4: Wire the CLI** + +In `percona_obs/cli.py`, after `sync_release_parser.set_defaults(...)`: + +```python + sync_prerelease_parser = sync_subparsers.add_parser( + "prerelease", + help="Hop 1 of a release: copy a staging tree into this PR's pre-release project.", + ) + sync_prerelease_parser.add_argument( + "project", help="Release project identifier (e.g. ppg:releases:18)." + ) + sync_prerelease_parser.add_argument( + "--branch-from", + required=True, + metavar="PROFILE", + dest="branch_from", + help="Profile of the production instance holding the staging source.", + ) + sync_prerelease_parser.add_argument( + "--synced-commit", + metavar="SHA", + default=None, + dest="synced_commit", + help="Head SHA of the last successful sync-main run (default: looked up with gh).", + ) + sync_prerelease_parser.add_argument( + "--source-commit", + metavar="SHA", + default=None, + dest="source_commit", + help="Override the source_commit recorded in release.yaml.", + ) + sync_prerelease_parser.add_argument( + "--force", action="store_true", default=False, + help="Skip the source_commit record check.", + ) + sync_prerelease_parser.add_argument( + "--no-freeze", action="store_true", default=False, + help="Skip the drain/green-check/build-freeze sequence around osc release.", + ) + sync_prerelease_parser.add_argument( + "--freeze-timeout", type=int, default=3600, dest="freeze_timeout", + help="Seconds to wait for the source scheduler to drain (default: 3600).", + ) + sync_prerelease_parser.add_argument( + "--verify-timeout", type=int, default=600, dest="verify_timeout", + help="Seconds to wait for copied packages to land (default: 600; 0 disables).", + ) + sync_prerelease_parser.add_argument( + "--dry-run", action="store_true", default=False, dest="dry_run", + help="Run the record check, mirror/changelog/tag checks and the green check only.", + ) + sync_prerelease_parser.set_defaults(func=cmd_sync_prerelease) +``` + +and add `cmd_sync_prerelease` to the `from .cmd_sync import (...)` list. + +- [ ] **Step 5: Run tests, format, type-check** + +Run: `venv/bin/black percona_obs/ && venv/bin/pyright && venv/bin/pytest -q` +Expected: PASS. Also `venv/bin/python -m percona_obs sync prerelease --help` prints the new flags. + +- [ ] **Step 6: Commit** + +```bash +git add percona_obs/cmd_sync.py percona_obs/cli.py tests/test_sync_prerelease.py +git commit -s -m "sync prerelease: hop 1, staging to the PR pre-release project" +``` + +--- + +### Task 6: `sync release --from ` (hop 2) + +**Goal:** With `--from`, `sync release` skips the freeze and the git divergence check, verifies the pre-release's `source_commit` marker against the tagged `release.yaml`, applies the mirror meta to the release tree and copies from the pre-release. + +**Files:** +- Modify: `percona_obs/cmd_sync.py` (`cmd_sync_release`) +- Modify: `percona_obs/cli.py` (`--from` on `sync release`) +- Test: `tests/test_sync_release.py` + +**Acceptance Criteria:** +- [ ] `--from X`: no `wait_for_quiesce`/`freeze_builds` call, no git diff, `release_tree(apiurl, "X", , ...)` called once, meta applied from mirrors for a new release project too (not `_create_release_project`) +- [ ] Marker mismatch → `SystemExit("source_commit mismatch")` before any meta write; `--force` skips the check +- [ ] Without `--from` behaviour is Task 3's + +**Verify:** `venv/bin/pytest -q tests/test_sync_release.py -k from_prerelease` → PASS + +**Steps:** + +- [ ] **Step 1: Write the failing tests** (append to `tests/test_sync_release.py`) + +```python +def _args_from(rel, **kw): + a = _Args(project="ppg:releases:17", rootprj="home:Admin") + a.from_project = "home:Admin:PR:pr-5:ppg:releases:17" + a.__dict__.update(kw) + return a + + +def _mapping_release_yaml(rel, sha="b" * 40): + (rel / "release.yaml").write_text( + f"project: ppg:staging:17\nreleases:\n - tag: ppg/17.11-1\n source_commit: {sha}\n" + ) + + +def test_from_prerelease_skips_freeze_and_copies_from_it(tmp_path, monkeypatch): + src, rel = _mk_tree(tmp_path, monkeypatch) + (rel / "tarballs").mkdir() + (rel / "tarballs" / "project.yaml").write_text("build: false\n" + _REPO_YAML.format(name="ssl3")) + _mapping_release_yaml(rel) + _wire_release_update_path(monkeypatch, src, rel) + monkeypatch.setattr(cmd_sync, "_obs_project_exists", lambda *a, **k: False) + monkeypatch.setattr(cmd_sync, "read_project_description_marker", lambda a, p, k: "b" * 40) + fail = lambda *a, **k: pytest.fail("freeze must be skipped with --from") + monkeypatch.setattr(cmd_sync, "wait_for_quiesce", fail) + monkeypatch.setattr(cmd_sync, "freeze_builds", fail) + monkeypatch.setattr(cmd_sync, "_create_release_project", fail) + metas, copies = [], [] + monkeypatch.setattr(cmd_sync, "apply_mirror_meta", lambda a, prj, *r, **k: metas.append(prj)) + monkeypatch.setattr(cmd_sync, "release_tree", lambda a, s, t, *r, **k: copies.append((s, t))) + cmd_sync.cmd_sync_release(_args_from(rel)) + assert metas == ["home:Admin:ppg:releases:17"] + assert copies == [("home:Admin:PR:pr-5:ppg:releases:17", "home:Admin:ppg:releases:17")] + + +def test_from_prerelease_marker_mismatch(tmp_path, monkeypatch): + src, rel = _mk_tree(tmp_path, monkeypatch) + (rel / "tarballs").mkdir() + (rel / "tarballs" / "project.yaml").write_text("build: false\n" + _REPO_YAML.format(name="ssl3")) + _mapping_release_yaml(rel) + _wire_release_update_path(monkeypatch, src, rel) + monkeypatch.setattr(cmd_sync, "read_project_description_marker", lambda a, p, k: "c" * 40) + monkeypatch.setattr(cmd_sync, "apply_mirror_meta", lambda *a, **k: pytest.fail("no meta on mismatch")) + with pytest.raises(SystemExit, match="source_commit mismatch"): + cmd_sync.cmd_sync_release(_args_from(rel)) +``` + +Also add `self.from_project = None` to `_Args.__init__`. + +- [ ] **Step 2: Run to verify failure** + +Run: `venv/bin/pytest -q tests/test_sync_release.py -k from_prerelease` +Expected: FAIL (freeze called / no mismatch error). + +- [ ] **Step 3: Implement** + +Import `read_project_description_marker` from `.obs_api` and `current_release` from `.release_record` in cmd_sync. In `cmd_sync_release`, right after `project_exists = _obs_project_exists(...)` (Task 3's layout), insert the `--from` branch before the legacy path: + +```python + from_project: "str | None" = getattr(args, "from_project", None) + if from_project: + # Hop 2 (spec §2): no staging involvement, no freeze, no git check. + entry = current_release(release_data) + if entry is None: + raise SystemExit("error: release.yaml has no releases: entries") + if not args.force: + marker = read_project_description_marker(apiurl, from_project, "source_commit") + if marker != entry.source_commit: + raise SystemExit( + f"error: source_commit mismatch: {from_project} was populated from " + f"{marker or ''} but release.yaml {entry.tag} records " + f"{entry.source_commit or ''}; repopulate the pre-release " + "or cut a new release" + ) + apply_mirror_meta( + apiurl, release_obs_project, release_path, args.rootprj, shared_env_vars + ) + release_tree( + apiurl, + from_project, + release_obs_project, + subproject_pairs, + args.rootprj, + shared_env_vars, + verify_timeout=args.verify_timeout, + ) + _print_ok(f"release {from_project} → {release_obs_project} ({entry.tag})") + return +``` + +The `--dry-run` block earlier in the function stays as is for both modes (its green check reads staging, which is still a sensible pre-merge signal; with `--from` it is informational). + +- [ ] **Step 4: CLI flag** + +In `cli.py`, on `sync_release_parser`: + +```python + sync_release_parser.add_argument( + "--from", + metavar="OBS_PROJECT", + default=None, + dest="from_project", + help="Hop 2: copy from this pre-release OBS project (full name, e.g. " + "isv:percona:PR:pr-12:ppg:releases:18) instead of from staging; " + "no freeze, no git divergence check.", + ) +``` + +- [ ] **Step 5: Run tests, format, type-check** + +Run: `venv/bin/black percona_obs/ && venv/bin/pyright && venv/bin/pytest -q` +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add percona_obs/cmd_sync.py percona_obs/cli.py tests/test_sync_release.py +git commit -s -m "sync release --from: hop 2, pre-release to the release project" +``` + +--- + +### Task 7: `project release` carries rendered `qa:` blocks into the mirrors + +**Goal:** Every mirror `project.yaml` (top and subprojects) gets the staging project's `qa:` block with `%!{…}` macros rendered and `:staging:` references rewritten to the release project, leaving `${…}` env tokens intact. + +**Files:** +- Modify: `percona_obs/cmd_project.py` (`_write_release_tree`, new `_rewrite_qa_project_refs`, `cmd_project_release` top-level `project_data`) +- Test: `tests/test_project_release.py` + +**Acceptance Criteria:** +- [ ] A staging subproject with `qa: [{pipeline: p, parameters: {OBS_PROJECT: "${OBS_ROOTPRJ}:ppg:staging:17:containers", VERSION: "ppg-%!{PG_VERSION}"}}]` and a `macros.yaml` defining `PG_VERSION: 17.11` yields a mirror `qa:` with `OBS_PROJECT: "${OBS_ROOTPRJ}:ppg:releases:17:containers"` and `VERSION: ppg-17.11` +- [ ] A staging project without `qa:` yields a mirror without `qa:` +- [ ] The top-level mirror gets the top-level staging `qa:` the same way + +**Verify:** `venv/bin/pytest -q tests/test_project_release.py -k qa` → PASS + +**Steps:** + +- [ ] **Step 1: Write the failing tests** (append to `tests/test_project_release.py`) + +```python +def test_rewrite_qa_project_refs_recurses_and_keeps_env_tokens(): + from percona_obs.cmd_project import _rewrite_qa_project_refs + + qa = [ + { + "pipeline": "p", + "parameters": { + "OBS_PROJECT": "${OBS_ROOTPRJ}:ppg:staging:17:containers", + "PLATFORMS": ["ppg:staging:17", "other"], + }, + } + ] + out = _rewrite_qa_project_refs(qa, "ppg:staging:17", "ppg:releases:17") + assert out[0]["parameters"]["OBS_PROJECT"] == "${OBS_ROOTPRJ}:ppg:releases:17:containers" + assert out[0]["parameters"]["PLATFORMS"] == ["ppg:releases:17", "other"] + assert qa[0]["parameters"]["PLATFORMS"][0] == "ppg:staging:17" # input untouched + + +def test_write_release_tree_carries_rendered_qa_blocks(tmp_path, monkeypatch): + import percona_obs.common as common + + src = _staging_tree(tmp_path) + monkeypatch.setattr(cmd_project, "_REPO_DIR", tmp_path) + monkeypatch.setattr(common, "REPO_ROOT", tmp_path / "root") + (tmp_path / "root/macros.yaml").write_text("- PG_VERSION: 17.11\n") + (src / "containers" / "project.yaml").write_text( + (src / "containers" / "project.yaml").read_text() + + "qa:\n - pipeline: p\n parameters:\n" + " OBS_PROJECT: ${OBS_ROOTPRJ}:ppg:staging:17:containers\n" + " VERSION: ppg-%!{PG_VERSION}\n" + ) + rel = tmp_path / "root/ppg/releases/17" + _write_release_tree( + rel, {"build": False, "repositories": []}, src, "ppg:staging:17", "ppg:releases:17", "ppg", "17" + ) + containers = yaml.safe_load((rel / "containers" / "project.yaml").read_text()) + assert containers["qa"][0]["parameters"] == { + "OBS_PROJECT": "${OBS_ROOTPRJ}:ppg:releases:17:containers", + "VERSION": "ppg-17.11", + } + tarballs = yaml.safe_load((rel / "tarballs" / "project.yaml").read_text()) + assert "qa" not in tarballs +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `venv/bin/pytest -q tests/test_project_release.py -k qa` +Expected: FAIL (`ImportError: _rewrite_qa_project_refs`). + +- [ ] **Step 3: Implement** + +Add above `_write_release_tree`: + +```python +def _rewrite_qa_project_refs(value, source_project_id: str, release_project_id: str): + """Deep-copy a loaded ``qa:`` value replacing ````/``:…`` + project references inside strings with the release project. Macros are + already rendered by the loader; ``${VAR}`` tokens are left as they are.""" + if isinstance(value, str): + if value == source_project_id: + return release_project_id + return re.sub( + rf"(?}` as the only `18.6-1` entry +- [ ] With the local tag present, a new id is derived and appended as a mapping entry; older entries keep their original form + +**Verify:** `venv/bin/pytest -q tests/test_project_release.py -k "refresh or source_commit or changelog_section"` → PASS + +**Steps:** + +- [ ] **Step 1: Write the failing tests** (append to `tests/test_project_release.py`) + +```python +def test_replace_changelog_section(): + from percona_obs.cmd_project import _replace_changelog_section + + text = "# Changelog\n\n## [18.6-1] - 2026-01-01\n\n### Added\n- a\n\n## [18.4-1] - 2025-12-01\n\n### Added\n- b\n" + out = _replace_changelog_section(text, "18.6-1", "## [18.6-1] - 2026-02-02\n\n### Added\n- c\n") + assert "- a" not in out and "- c" in out and "- b" in out + assert out.index("[18.6-1]") < out.index("[18.4-1]") + # absent → prepended above the first section + out2 = _replace_changelog_section(text, "18.7-1", "## [18.7-1] - x\n") + assert out2.index("[18.7-1]") < out2.index("[18.6-1]") + + +def test_resolve_source_commit_override_and_ancestor(monkeypatch): + from percona_obs import cmd_project as cp + + monkeypatch.setattr(cp, "_is_ancestor_of_head", lambda sha: sha == "good") + assert cp._resolve_source_commit("good", "https://github.com/o/r.git") == "good" + with pytest.raises(SystemExit, match="not an ancestor of HEAD"): + cp._resolve_source_commit("bad", "https://github.com/o/r.git") + monkeypatch.setattr(cp, "last_synced_head_sha", lambda slug: "good") + assert cp._resolve_source_commit(None, "https://github.com/o/r.git") == "good" + with pytest.raises(SystemExit, match="PERCONA_OBS_PACKAGING_REPO"): + cp._resolve_source_commit(None, "") + + +def test_open_release_is_refreshed_in_place(monkeypatch): + from percona_obs import cmd_project as cp + from percona_obs.release_record import ReleaseEntry + + monkeypatch.setattr(cp, "_tag_exists_locally", lambda tag: False) + entries = [ReleaseEntry("ppg/18.4-1"), ReleaseEntry("ppg/18.6-1", "old")] + rid, new_entries, prev = cp._plan_release_entries(entries, None, lambda: "18.6-2", "ppg", "newsha") + assert rid == "18.6-1" + assert new_entries == [ReleaseEntry("ppg/18.4-1"), ReleaseEntry("ppg/18.6-1", "newsha")] + assert prev == "18.4-1" + + +def test_shipped_release_appends_next_id(monkeypatch): + from percona_obs import cmd_project as cp + from percona_obs.release_record import ReleaseEntry + + monkeypatch.setattr(cp, "_tag_exists_locally", lambda tag: True) + entries = [ReleaseEntry("ppg/18.6-1", "old")] + rid, new_entries, prev = cp._plan_release_entries(entries, None, lambda: "18.6-2", "ppg", "newsha") + assert rid == "18.6-2" + assert new_entries == [ReleaseEntry("ppg/18.6-1", "old"), ReleaseEntry("ppg/18.6-2", "newsha")] + assert prev == "18.6-1" + + +def test_explicit_release_id_wins_over_refresh(monkeypatch): + from percona_obs import cmd_project as cp + from percona_obs.release_record import ReleaseEntry + + monkeypatch.setattr(cp, "_tag_exists_locally", lambda tag: False) + entries = [ReleaseEntry("ppg/18.6-1")] + with pytest.raises(SystemExit, match="already present"): + cp._plan_release_entries(entries, "18.6-1", lambda: "x", "ppg", "s") + rid, new_entries, _ = cp._plan_release_entries(entries, "18.6-9", lambda: "x", "ppg", "s") + assert rid == "18.6-9" and new_entries[-1] == ReleaseEntry("ppg/18.6-9", "s") +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `venv/bin/pytest -q tests/test_project_release.py -k "refresh or source_commit or changelog_section or appends_next or explicit_release_id"` +Expected: FAIL (`AttributeError`). + +- [ ] **Step 3: Implement the helpers** (place above `cmd_project_release`) + +```python +def _tag_exists_locally(tag: str) -> bool: + r = subprocess.run( + ["git", "rev-parse", "--verify", "--quiet", f"refs/tags/{tag}"], + capture_output=True, + cwd=_REPO_DIR, + ) + return r.returncode == 0 + + +def _is_ancestor_of_head(sha: str) -> bool: + r = subprocess.run( + ["git", "merge-base", "--is-ancestor", sha, "HEAD"], + capture_output=True, + cwd=_REPO_DIR, + ) + return r.returncode == 0 + + +def _resolve_source_commit(override: "str | None", repo_url: str) -> str: + """The commit of main whose staging tree the release is cut from. + + Default: head SHA of the last sync-main run with successful sync jobs + (``gh api``), so the record names what OBS actually holds. Must be an + ancestor of HEAD so the release branch contains the tree it claims. + """ + if override: + sha = override + else: + if not repo_url: + raise SystemExit( + "error: PERCONA_OBS_PACKAGING_REPO is not set in the profile; " + "set it or pass --source-commit" + ) + sha = last_synced_head_sha(repo_slug_from_url(repo_url)) + if not _is_ancestor_of_head(sha): + raise SystemExit( + f"error: source commit {sha} is not an ancestor of HEAD; " + "rebase the release branch on main (and fetch) first" + ) + return sha + + +def _plan_release_entries( + existing: "list[ReleaseEntry]", + explicit_id: "str | None", + derive_id: "Callable[[], str]", + product: str, + source_commit: str, +) -> "tuple[str, list[ReleaseEntry], str | None]": + """Decide the release id and the new ``releases:`` list. + + Returns (release_id, entries, prev_release_id). The last entry whose + tag is not yet a git tag is *open*: without ``--release-id`` it is + refreshed in place (same id, new source_commit) and prev_release_id is + the entry before it. Otherwise the next id is derived and appended. + """ + open_entry = existing[-1] if existing and not _tag_exists_locally(existing[-1].tag) else None + if explicit_id: + tag = f"{product}/{explicit_id}" + if any(e.tag == tag for e in existing): + raise SystemExit(f"error: release tag {tag} is already present in release.yaml") + prev = existing[-1].tag.split("/")[-1] if existing else None + return explicit_id, [*existing, ReleaseEntry(tag, source_commit)], prev + if open_entry is not None: + rid = open_entry.tag.split("/")[-1] + print( + f"note: {open_entry.tag} has no git tag locally — treating the release as " + "still open and refreshing it in place (fetch tags if it already shipped)", + flush=True, + ) + prev = existing[-2].tag.split("/")[-1] if len(existing) > 1 else None + return rid, [*existing[:-1], ReleaseEntry(open_entry.tag, source_commit)], prev + rid = derive_id() + tag = f"{product}/{rid}" + if any(e.tag == tag for e in existing): + raise SystemExit(f"error: release tag {tag} is already present in release.yaml") + prev = existing[-1].tag.split("/")[-1] if existing else None + return rid, [*existing, ReleaseEntry(tag, source_commit)], prev + + +def _replace_changelog_section(text: str, release_id: str, section: str) -> str: + """Replace the ``## [release_id]`` section (up to the next ``## [``) or, + when absent, prepend *section* above the first section.""" + pattern = re.compile( + r"## \[" + re.escape(release_id) + r"\].*?(?=\n## \[|\Z)", re.DOTALL + ) + if pattern.search(text): + return pattern.sub(lambda _m: section.rstrip("\n") + "\n", text, count=1) + m = re.search(r"\n(## \[)", text) + if m: + pos = m.start() + 1 + return text[:pos] + section.rstrip("\n") + "\n\n" + text[pos:] + return text.rstrip() + "\n\n" + section.rstrip("\n") + "\n" +``` + +Add imports: `from .github_api import last_synced_head_sha, repo_slug_from_url` and `from .release_record import ReleaseEntry, dump_release_entry, parse_release_entries` (merge with Task 1's line). + +- [ ] **Step 4: Rewrite the decision block in `cmd_project_release`** + +Replace from `# Auto-derive release-id from OBS if not provided.` down to and including `if tag in existing_releases: raise SystemExit(...)` with: + +```python + release_env: dict[str, str] = { + **(parse_env_overrides(args.env_overrides) if args.env_overrides else {}), + **auto_rootprj_env(args.rootprj), + } + source_commit = _resolve_source_commit( + getattr(args, "source_commit", None), + release_env.get("PERCONA_OBS_PACKAGING_REPO", ""), + ) + + def _derive() -> str: + pkg_archs = _fetch_all_pkg_archs(apiurl, source_obs_project) + return _derive_release_id( + pkg_archs, + existing_releases, + release_name, + lambda repo, arch, pkg: _fetch_pkg_versrel( + apiurl, source_obs_project, repo, arch, pkg + ), + counter_mode=not tree_has_server_package(source_path), + major=major, + ) + + release_id, new_entries, prev_release_id = _plan_release_entries( + existing_entries, args.release_id or None, _derive, product, source_commit + ) + tag = f"{product}/{release_id}" + is_refresh = bool(existing_entries) and existing_entries[-1].tag == tag +``` + +Then: +- delete the later duplicate `release_env` definition (it now lives above); +- replace `prev_release_id: str | None = None` and the `if not is_first_release and existing_releases: prev_release_id = existing_releases[-1].split("/")[-1]` assignment with just the `for rel_sub in ...` loop guarded by `if not is_first_release and prev_release_id:`; +- in the CVE scan, `prev_tag_for_scan = f"{product}/{prev_release_id}" if prev_release_id else None`; +- `release_data["releases"] = [dump_release_entry(e) for e in new_entries]`; +- in the update path, replace the manual "prepend changelog section" block (`m = re.search(r"\n(## \[)", ...)` … `new_content = ...`) with `new_content = _replace_changelog_section(existing_changelog, release_id, changelog_section)`; +- commit message: `f"Refresh release {release_project} ({release_id})"` when `is_refresh`, else as today; print `Refresh release:` in the preview when `is_refresh`. + +Note `existing_releases` (tags list) is still what `_derive_release_id` counts; on a refresh the open tag is counted too but `_derive` is never called, so the count is irrelevant. + +- [ ] **Step 5: CLI flag** + +```python + project_release_parser.add_argument( + "--source-commit", + metavar="SHA", + default=None, + dest="source_commit", + help="Commit of main whose staging tree is released (default: head of the " + "last sync-main run with successful sync jobs, via gh). Recorded in release.yaml.", + ) +``` + +- [ ] **Step 6: Run tests, format, type-check** + +Run: `venv/bin/black percona_obs/ && venv/bin/pyright && venv/bin/pytest -q` +Expected: PASS. + +- [ ] **Step 7: Commit** + +```bash +git add percona_obs/cmd_project.py percona_obs/cli.py tests/test_project_release.py +git commit -s -m "project release: record source_commit, refresh an open release in place" +``` + +--- + +### Task 9: Documentation and final verification + +**Goal:** The tool reference, the architecture reference and the key-files tables describe the two commands, the `release.yaml` mapping form and the refresh rule; the whole suite and type-check pass on the final tree. + +**Files:** +- Modify: `docs/PERCONA_OBS_TOOL.md` ("Releasing packages" chapter: Step 1, `release.yaml` format, "Running `sync release` manually", new "`sync prerelease`" section) +- Modify: `.github/copilot-instructions.md` (key files table; a short "Release hops" note next to the `sync release-pr` section; the obs-pr-cleanup one-liner mention) +- Modify: `CLAUDE.md` (key files table: `release_copy.py`, `release_record.py`, `github_api.py`) + +**Acceptance Criteria:** +- [ ] `docs/PERCONA_OBS_TOOL.md` documents: `--source-commit`, the refresh rule ("last entry without a git tag is open"), the mapping entry form, `sync prerelease` flags (`--branch-from`, `--synced-commit`, `--source-commit`, `--force`, `--no-freeze`, `--freeze-timeout`, `--verify-timeout`, `--dry-run`), `sync release --from`, and that the workflow integration follows in rollout step 2 +- [ ] Key-file tables list the three new modules +- [ ] `venv/bin/pytest -q`, `venv/bin/black --check percona_obs/`, `venv/bin/pyright` all pass + +**Verify:** `venv/bin/pytest -q && venv/bin/black --check percona_obs/ && venv/bin/pyright && grep -c "sync prerelease" docs/PERCONA_OBS_TOOL.md` → green, count ≥ 3 + +**Steps:** + +- [ ] **Step 1: `docs/PERCONA_OBS_TOOL.md`** + +In "Step 1 — Create or update a release": add after item 1 (auto-derive) a new item: + +```markdown +2. **Resolves the source commit** — the head SHA of the last `sync-main.yml` run on + `main` whose sync jobs succeeded (`gh api`), or `--source-commit `. It must be + an ancestor of HEAD. It is recorded in `release.yaml` so the release names the + staging tree its binaries were built from. +3. **Open-release refresh** — if the last `releases:` entry has no git tag yet, the + release is still in flight: the same id is kept, the mirror tree is regenerated, the + `CHANGELOG.md` section for that id is replaced and `source_commit` is updated. Only + when the tag exists is the next id derived and appended (`--release-id` always wins). +``` + +Renumber the following items. Replace the "`release.yaml` format" block with: + +```yaml +repository: ${PERCONA_OBS_PACKAGING_REPO} +project: ppg:staging:17 +releases: + - ppg/17.9-1 # historical entries stay plain tags + - tag: ppg/17.10-1 # written by project release + source_commit: 9e9d9738c0ffee… # full SHA of main the binaries were built from +``` + +and note that the last entry is the current release in either form. + +Add a new section before "Running `sync release` manually": + +```markdown +### `sync prerelease` — hop 1 (staging → PR pre-release) + +Run with the PR profile; `--branch-from` names the production profile holding staging: + + ./percona-obs -P pr-12 sync prerelease ppg:releases:18 --branch-from main + +Steps: (1) require a `source_commit` on the last `release.yaml` entry; (2) **record +check** — the freeze-scope paths (source tree, its `macros.yaml` chain, every +whole-frozen project, every aggregate-frozen package) must be identical between +`source_commit` and the last synced `main` (`--synced-commit`, else `gh api`), or it +fails with "staging changed since the cut"; (3) pre-release meta from the mirrors, +**publish forced on**; (4) drain, assert green, freeze, `osc release`, verify, +restore; (5) `source_commit: ` appended to each pre-release project description. +`--dry-run` runs (1), (2), the mirror/changelog/tag checks and the green check only. +`--force` skips (2). The workflow that calls this on release PRs lands in rollout step 2. + +### `sync release --from` — hop 2 (pre-release → release project) + + ./percona-obs -P main sync release ppg:releases:18 --from isv:percona:PR:pr-12:ppg:releases:18 + +No freeze, no git divergence check. The pre-release description's `source_commit` +must equal the tagged `release.yaml` entry's (`--force` skips); then the release +project meta is applied from the mirrors and binaries are copied with `osc release`. +``` + +In the flags table of "Running `sync release` manually" add the `--from` row. Add a one-line note at the top of the chapter: "Until rollout step 2 lands, CI still runs the single-hop `sync release` on merge; the two hops are available for manual and rehearsal use." + +- [ ] **Step 2: `.github/copilot-instructions.md` and `CLAUDE.md`** + +Add rows to both key-file tables: + +```markdown +| `percona_obs/release_copy.py` | Shared release hop mechanics (mirror meta apply, `osc release` copy, verify, orphans) | +| `percona_obs/release_record.py` | `release.yaml` `releases:` entries (tag strings and `{tag, source_commit}`) | +| `percona_obs/github_api.py` | `gh api` lookups (last synced `main` commit) | +``` + +In copilot-instructions, after the `sync release-pr` section, add a 6-line "Release hops" paragraph pointing at the spec and summarising `sync prerelease` / `sync release --from`. + +- [ ] **Step 3: Final verification** + +Run: `venv/bin/pytest -q && venv/bin/black --check percona_obs/ && venv/bin/pyright` +Expected: all pass, `0 errors`. Then `git status` must be clean apart from `venv/`. + +- [ ] **Step 4: Commit** + +```bash +git add docs/PERCONA_OBS_TOOL.md .github/copilot-instructions.md CLAUDE.md +git commit -s -m "docs: pre-release flow tool commands and release.yaml entry form" +``` + +--- + +## Self-review notes + +- Spec coverage (tool PR scope): §2 `release.yaml` format → Task 1; `project release` three changes → Tasks 7, 8; `sync prerelease` → Task 5; `sync release --from` → Task 6; `release_copy.py` → Task 3; marker/publish primitives → Task 4; §6 unit tests → each task; §7 docs (tool reference) → Task 9. Workflow changes (§3), the one-in-flight guard and the rehearsal are the step-2 plan. +- Deviation flagged in Global Constraints: temporary `releasetarget` entries are kept alongside `--target-project`. +- Type consistency: `collect_release_subprojects`, `filter_release_repo_names`, `apply_mirror_meta`, `copy_project_binaries`, `release_tree`, `warn_orphans` (release_copy); `set_project_description_marker`, `read_project_description_marker`, `publish_override` (obs_api); `parse_release_entries`, `release_tags`, `current_release`, `dump_release_entry`, `ReleaseEntry` (release_record); `repo_slug_from_url`, `last_synced_head_sha`, `_gh_api_json` (github_api); `_scope_local_paths`, `_changed_files_between`, `_run_frozen`, `_check_source_unchanged_since_tag`, `cmd_sync_prerelease`, `args.from_project`, `args.synced_commit`, `args.source_commit` (cmd_sync); `_rewrite_qa_project_refs`, `_tag_exists_locally`, `_is_ancestor_of_head`, `_resolve_source_commit`, `_plan_release_entries`, `_replace_changelog_section` (cmd_project). diff --git a/docs/superpowers/plans/2026-10-13-prerelease-flow-tool.md.tasks.json b/docs/superpowers/plans/2026-10-13-prerelease-flow-tool.md.tasks.json new file mode 100644 index 000000000..429d1859a --- /dev/null +++ b/docs/superpowers/plans/2026-10-13-prerelease-flow-tool.md.tasks.json @@ -0,0 +1,96 @@ +{ + "planPath": "docs/superpowers/plans/2026-10-13-prerelease-flow-tool.md", + "tasks": [ + { + "id": 0, + "subject": "Task 0: Worktree environment and baseline", + "status": "pending", + "description": "**Goal:** A working `venv/` in the worktree and a green baseline so later failures are attributable.\n\n**Files:**\n- Create: `venv/` (git-ignored) in `.claude/worktrees/prerelease-flow`\n\n**Acceptance Criteria:**\n- [ ] `venv/bin/pytest -q` passes on the untouched branch\n- [ ] `venv/bin/black --check percona_obs/` and `venv/bin/pyright` pass\n\n**Verify:** `venv/bin/pytest -q && venv/bin/black --check percona_obs/ && venv/bin/pyright` \u2192 all green, pyright prints `0 errors`\n\n```json:metadata\n{\"files\": [], \"verifyCommand\": \"venv/bin/pytest -q && venv/bin/black --check percona_obs/ && venv/bin/pyright\", \"acceptanceCriteria\": [\"`venv/bin/pytest -q` passes on the untouched branch\", \"`venv/bin/black --check percona_obs/` and `venv/bin/pyright` pass\"], \"modelTier\": \"mechanical\"}\n```" + }, + { + "id": 1, + "subject": "Task 1: `release_record.py` and mapping-form `release.yaml` entries", + "status": "pending", + "description": "**Goal:** One parser for `releases:` entries used by every reader (`compute_ppg_release`, `project release`, `sync release`, the cleanup workflow one-liner), accepting strings, `{tag, source_commit}` mappings and legacy `revision:`.\n\n**Files:**\n- Create: `percona_obs/release_record.py`\n- Modify: `percona_obs/common.py:319-346` (`compute_ppg_release`)\n- Modify: `percona_obs/cmd_project.py:2293-2300` (existing release list)\n- Modify: `percona_obs/cmd_sync.py:2628-2635` (release list in `cmd_sync_release`)\n- Modify: `.github/workflows/obs-pr-cleanup.yml:138`\n- Test: `tests/test_release_record.py`, `tests/test_computed_macros.py`\n\n**Acceptance Criteria:**\n- [ ] `parse_release_entries` returns `ReleaseEntry(tag, source_commit)` for both forms and raises `SystemExit` naming an unrecognised entry\n- [ ] `compute_ppg_release` counts a mapping entry's tag (`{tag: ppg/18.4-2}` counts for 18.4)\n- [ ] `project release` and `sync release` still work with string-only files (existing tests pass)\n- [ ] The cleanup one-liner prints the tag for a mapping last entry\n\n**Verify:** `venv/bin/pytest -q tests/test_release_record.py tests/test_computed_macros.py tests/test_project_release.py tests/test_sync_release.py` \u2192 PASS\n\n```json:metadata\n{\"files\": [\"percona_obs/release_record.py\", \"tests/test_computed_macros.py\", \"tests/test_release_record.py\"], \"verifyCommand\": \"venv/bin/pytest -q tests/test_release_record.py tests/test_computed_macros.py tests/test_project_release.py tests/test_sync_release.py\", \"acceptanceCriteria\": [\"`parse_release_entries` returns `ReleaseEntry(tag, source_commit)` for both forms and raises `SystemExit` naming an unrecognised entry\", \"`compute_ppg_release` counts a mapping entry's tag (`{tag: ppg/18.4-2}` counts for 18.4)\", \"`project release` and `sync release` still work with string-only files (existing tests pass)\", \"The cleanup one-liner prints the tag for a mapping last entry\"], \"modelTier\": \"standard\"}\n```", + "blockedBy": [ + 0 + ] + }, + { + "id": 2, + "subject": "Task 2: `github_api.py` \u2014 last synced head SHA", + "status": "pending", + "description": "**Goal:** Resolve \"the commit of `main` whose staging tree OBS currently holds\" as the head SHA of the newest completed `sync-main.yml` run on `main` whose sync jobs all concluded `success`, through `gh api`.\n\n**Files:**\n- Create: `percona_obs/github_api.py`\n- Test: `tests/test_github_api.py`\n\n**Acceptance Criteria:**\n- [ ] `repo_slug_from_url` handles `https://github.com/percona/obs-packaging.git`, `https://github.com/percona/obs-packaging`, `git@github.com:percona/obs-packaging.git`\n- [ ] `last_synced_head_sha` skips a run whose `percona-obs sync push (...)` job failed even if the run's other jobs passed, and returns the next older qualifying run's `head_sha`\n- [ ] A `gh` failure raises `SystemExit` with the stderr text; no qualifying run raises `SystemExit`\n\n**Verify:** `venv/bin/pytest -q tests/test_github_api.py` \u2192 PASS\n\n```json:metadata\n{\"files\": [\"percona_obs/github_api.py\", \"tests/test_github_api.py\"], \"verifyCommand\": \"venv/bin/pytest -q tests/test_github_api.py\", \"acceptanceCriteria\": [\"`repo_slug_from_url` handles `https://github.com/percona/obs-packaging.git`, `https://github.com/percona/obs-packaging`, `git@github.com:percona/obs-packaging.git`\", \"`last_synced_head_sha` skips a run whose `percona-obs sync push (...)` job failed even if the run's other jobs passed, and returns the next older qualifying run's `head_sha`\", \"A `gh` failure raises `SystemExit` with the stderr text; no qualifying run raises `SystemExit`\"], \"modelTier\": \"standard\"}\n```", + "blockedBy": [ + 0 + ] + }, + { + "id": 3, + "subject": "Task 3: `release_copy.py` and `cmd_sync_release` refactor (behaviour unchanged)", + "status": "pending", + "description": "**Goal:** Move the copy mechanics out of `cmd_sync.py` into `percona_obs/release_copy.py` and make `cmd_sync_release` a thin orchestration with a single code path for first and update releases. Behaviour must be identical; the existing tests prove it after their monkeypatch targets move.\n\n**Files:**\n- Create: `percona_obs/release_copy.py`\n- Modify: `percona_obs/cmd_sync.py` (`_collect_release_subprojects`, `_filter_release_repo_names`, `_sync_release_subprojects` removed; `cmd_sync_release` body)\n- Test: `tests/test_sync_release.py` (rewired), `tests/test_release_copy.py` (new)\n\n**Acceptance Criteria:**\n- [ ] `release_copy.copy_project_binaries` adds release targets, runs `osc release --no-delay --target-project `, removes targets in `finally` even when osc fails\n- [ ] `release_copy.release_tree` copies top then each pair, applies each pair's mirror meta before its copy, verifies when `verify_timeout > 0`, warns about orphans\n- [ ] `cmd_sync_release` first-release and update paths both run the freeze sequence and the same `release_tree`; all pre-existing `test_sync_release.py` assertions hold\n\n**Verify:** `venv/bin/pytest -q tests/test_sync_release.py tests/test_release_copy.py` \u2192 PASS\n\n```json:metadata\n{\"files\": [\"percona_obs/cmd_sync.py\", \"percona_obs/release_copy.py\", \"tests/test_release_copy.py\", \"tests/test_sync_release.py\"], \"verifyCommand\": \"venv/bin/pytest -q tests/test_sync_release.py tests/test_release_copy.py\", \"acceptanceCriteria\": [\"`release_copy.copy_project_binaries` adds release targets, runs `osc release --no-delay --target-project `, removes targets in `finally` even when osc fails\", \"`release_copy.release_tree` copies top then each pair, applies each pair's mirror meta before its copy, verifies when `verify_timeout > 0`, warns about orphans\", \"`cmd_sync_release` first-release and update paths both run the freeze sequence and the same `release_tree`; all pre-existing `test_sync_release.py` assertions hold\"], \"modelTier\": \"standard\"}\n```", + "blockedBy": [ + 1 + ] + }, + { + "id": 4, + "subject": "Task 4: `obs_api` \u2014 publish override and description marker", + "status": "pending", + "description": "**Goal:** Two small OBS-side primitives the new hops need: force-publish on a project built from a mirror, and a `source_commit:` marker in a project description.\n\n**Files:**\n- Modify: `percona_obs/obs_api.py:1215-1280` (`_apply_project_config`), new functions after `_disable_project_builds`\n- Test: `tests/test_obs_api_markers.py`\n\n**Acceptance Criteria:**\n- [ ] `_apply_project_config(..., publish_override=True)` emits `` regardless of the mirror's `publish:`; `None` keeps today's behaviour\n- [ ] `set_project_description_marker` appends `source_commit: ` once (replacing an existing marker line) and pushes the meta\n- [ ] `read_project_description_marker` returns the value or `None`\n\n**Verify:** `venv/bin/pytest -q tests/test_obs_api_markers.py` \u2192 PASS\n\n```json:metadata\n{\"files\": [\"tests/test_obs_api_markers.py\"], \"verifyCommand\": \"venv/bin/pytest -q tests/test_obs_api_markers.py\", \"acceptanceCriteria\": [\"`_apply_project_config(..., publish_override=True)` emits `` regardless of the mirror's `publish:`; `None` keeps today's behaviour\", \"`set_project_description_marker` appends `source_commit: ` once (replacing an existing marker line) and pushes the meta\", \"`read_project_description_marker` returns the value or `None`\"], \"modelTier\": \"standard\"}\n```", + "blockedBy": [ + 3 + ] + }, + { + "id": 5, + "subject": "Task 5: `sync prerelease` (hop 1)", + "status": "pending", + "description": "**Goal:** New command that, under the PR profile, checks the `source_commit` record against the last synced `main`, creates/re-applies the pre-release tree from the mirrors (publish forced on), freezes staging, copies, verifies, restores, and pins `source_commit` in the pre-release description.\n\n**Files:**\n- Modify: `percona_obs/cmd_sync.py` (new `cmd_sync_prerelease`, helpers `_scope_local_paths`, `_changed_files_between`)\n- Modify: `percona_obs/cli.py` (new `sync prerelease` parser after `sync release`)\n- Test: `tests/test_sync_prerelease.py`\n\n**Acceptance Criteria:**\n- [ ] Without a mapping entry carrying `source_commit`, the command fails with \"has no source_commit\"\n- [ ] A non-empty diff of scope paths between `source_commit` and the synced head aborts before `wait_for_quiesce` is called, listing the files\n- [ ] Order of operations: record check \u2192 top meta (publish True) \u2192 drain \u2192 green \u2192 freeze \u2192 copy tree \u2192 restore \u2192 marker; marker is not written when the copy raised\n- [ ] `--dry-run` runs the record check and the green check, writes nothing\n- [ ] `--synced-commit` bypasses `gh`; `--force` bypasses the record check only\n\n**Verify:** `venv/bin/pytest -q tests/test_sync_prerelease.py` \u2192 PASS\n\n```json:metadata\n{\"files\": [\"percona_obs/cli.py\", \"percona_obs/cmd_sync.py\", \"tests/test_sync_prerelease.py\"], \"verifyCommand\": \"venv/bin/pytest -q tests/test_sync_prerelease.py\", \"acceptanceCriteria\": [\"Without a mapping entry carrying `source_commit`, the command fails with \\\"has no source_commit\\\"\", \"A non-empty diff of scope paths between `source_commit` and the synced head aborts before `wait_for_quiesce` is called, listing the files\", \"Order of operations: record check \\u2192 top meta (publish True) \\u2192 drain \\u2192 green \\u2192 freeze \\u2192 copy tree \\u2192 restore \\u2192 marker; marker is not written when the copy raised\", \"`--dry-run` runs the record check and the green check, writes nothing\", \"`--synced-commit` bypasses `gh`; `--force` bypasses the record check only\"], \"modelTier\": \"standard\"}\n```", + "blockedBy": [ + 2, + 4 + ] + }, + { + "id": 6, + "subject": "Task 6: `sync release --from ` (hop 2)", + "status": "pending", + "description": "**Goal:** With `--from`, `sync release` skips the freeze and the git divergence check, verifies the pre-release's `source_commit` marker against the tagged `release.yaml`, applies the mirror meta to the release tree and copies from the pre-release.\n\n**Files:**\n- Modify: `percona_obs/cmd_sync.py` (`cmd_sync_release`)\n- Modify: `percona_obs/cli.py` (`--from` on `sync release`)\n- Test: `tests/test_sync_release.py`\n\n**Acceptance Criteria:**\n- [ ] `--from X`: no `wait_for_quiesce`/`freeze_builds` call, no git diff, `release_tree(apiurl, \"X\", , ...)` called once, meta applied from mirrors for a new release project too (not `_create_release_project`)\n- [ ] Marker mismatch \u2192 `SystemExit(\"source_commit mismatch\")` before any meta write; `--force` skips the check\n- [ ] Without `--from` behaviour is Task 3's\n\n**Verify:** `venv/bin/pytest -q tests/test_sync_release.py -k from_prerelease` \u2192 PASS\n\n```json:metadata\n{\"files\": [\"percona_obs/cli.py\", \"percona_obs/cmd_sync.py\", \"tests/test_sync_release.py\"], \"verifyCommand\": \"venv/bin/pytest -q tests/test_sync_release.py -k from_prerelease\", \"acceptanceCriteria\": [\"`--from X`: no `wait_for_quiesce`/`freeze_builds` call, no git diff, `release_tree(apiurl, \\\"X\\\", , ...)` called once, meta applied from mirrors for a new release project too (not `_create_release_project`)\", \"Marker mismatch \\u2192 `SystemExit(\\\"source_commit mismatch\\\")` before any meta write; `--force` skips the check\", \"Without `--from` behaviour is Task 3's\"], \"modelTier\": \"standard\"}\n```", + "blockedBy": [ + 5 + ] + }, + { + "id": 7, + "subject": "Task 7: `project release` carries rendered `qa:` blocks into the mirrors", + "status": "pending", + "description": "**Goal:** Every mirror `project.yaml` (top and subprojects) gets the staging project's `qa:` block with `%!{\u2026}` macros rendered and `:staging:` references rewritten to the release project, leaving `${\u2026}` env tokens intact.\n\n**Files:**\n- Modify: `percona_obs/cmd_project.py` (`_write_release_tree`, new `_rewrite_qa_project_refs`, `cmd_project_release` top-level `project_data`)\n- Test: `tests/test_project_release.py`\n\n**Acceptance Criteria:**\n- [ ] A staging subproject with `qa: [{pipeline: p, parameters: {OBS_PROJECT: \"${OBS_ROOTPRJ}:ppg:staging:17:containers\", VERSION: \"ppg-%!{PG_VERSION}\"}}]` and a `macros.yaml` defining `PG_VERSION: 17.11` yields a mirror `qa:` with `OBS_PROJECT: \"${OBS_ROOTPRJ}:ppg:releases:17:containers\"` and `VERSION: ppg-17.11`\n- [ ] A staging project without `qa:` yields a mirror without `qa:`\n- [ ] The top-level mirror gets the top-level staging `qa:` the same way\n\n**Verify:** `venv/bin/pytest -q tests/test_project_release.py -k qa` \u2192 PASS\n\n```json:metadata\n{\"files\": [\"percona_obs/cmd_project.py\", \"tests/test_project_release.py\"], \"verifyCommand\": \"venv/bin/pytest -q tests/test_project_release.py -k qa\", \"acceptanceCriteria\": [\"A staging subproject with `qa: [{pipeline: p, parameters: {OBS_PROJECT: \\\"${OBS_ROOTPRJ}:ppg:staging:17:containers\\\", VERSION: \\\"ppg-%!{PG_VERSION}\\\"}}]` and a `macros.yaml` defining `PG_VERSION: 17.11` yields a mirror `qa:` with `OBS_PROJECT: \\\"${OBS_ROOTPRJ}:ppg:releases:17:containers\\\"` and `VERSION: ppg-17.11`\", \"A staging project without `qa:` yields a mirror without `qa:`\", \"The top-level mirror gets the top-level staging `qa:` the same way\"], \"modelTier\": \"standard\"}\n```", + "blockedBy": [ + 1 + ] + }, + { + "id": 8, + "subject": "Task 8: `project release` \u2014 `source_commit` and open-release refresh", + "status": "pending", + "description": "**Goal:** `project release` records `source_commit` (last synced `main` via `gh`, or `--source-commit`) in a mapping entry, refuses a SHA that is not an ancestor of HEAD, and when the last entry's tag has no git tag yet it refreshes that release in place (same id, CHANGELOG section replaced, mirrors regenerated, `source_commit` updated).\n\n**Files:**\n- Modify: `percona_obs/cmd_project.py` (`cmd_project_release`, new helpers `_resolve_source_commit`, `_tag_exists_locally`, `_replace_changelog_section`)\n- Modify: `percona_obs/cli.py` (`--source-commit` on `project release`)\n- Test: `tests/test_project_release.py`\n\n**Acceptance Criteria:**\n- [ ] `_replace_changelog_section(text, \"18.6-1\", new)` replaces exactly the `## [18.6-1]` section and keeps the others; when absent it prepends like the update path does today\n- [ ] `_resolve_source_commit` returns the override when given, else the `gh` lookup; raises when the SHA is not an ancestor of HEAD\n- [ ] With last entry `ppg/18.6-1` and no local tag, the derived id is `18.6-1` (refresh) and `release.yaml` ends with `{tag: ppg/18.6-1, source_commit: }` as the only `18.6-1` entry\n- [ ] With the local tag present, a new id is derived and appended as a mapping entry; older entries keep their original form\n\n**Verify:** `venv/bin/pytest -q tests/test_project_release.py -k \"refresh or source_commit or changelog_section\"` \u2192 PASS\n\n```json:metadata\n{\"files\": [\"percona_obs/cli.py\", \"percona_obs/cmd_project.py\", \"tests/test_project_release.py\"], \"verifyCommand\": \"venv/bin/pytest -q tests/test_project_release.py -k \\\"refresh or source_commit or changelog_section\\\"\", \"acceptanceCriteria\": [\"`_replace_changelog_section(text, \\\"18.6-1\\\", new)` replaces exactly the `## [18.6-1]` section and keeps the others; when absent it prepends like the update path does today\", \"`_resolve_source_commit` returns the override when given, else the `gh` lookup; raises when the SHA is not an ancestor of HEAD\", \"With last entry `ppg/18.6-1` and no local tag, the derived id is `18.6-1` (refresh) and `release.yaml` ends with `{tag: ppg/18.6-1, source_commit: }` as the only `18.6-1` entry\", \"With the local tag present, a new id is derived and appended as a mapping entry; older entries keep their original form\"], \"modelTier\": \"standard\"}\n```", + "blockedBy": [ + 7, + 2 + ] + }, + { + "id": 9, + "subject": "Task 9: Documentation and final verification", + "status": "pending", + "description": "**Goal:** The tool reference, the architecture reference and the key-files tables describe the two commands, the `release.yaml` mapping form and the refresh rule; the whole suite and type-check pass on the final tree.\n\n**Files:**\n- Modify: `docs/PERCONA_OBS_TOOL.md` (\"Releasing packages\" chapter: Step 1, `release.yaml` format, \"Running `sync release` manually\", new \"`sync prerelease`\" section)\n- Modify: `.github/copilot-instructions.md` (key files table; a short \"Release hops\" note next to the `sync release-pr` section; the obs-pr-cleanup one-liner mention)\n- Modify: `CLAUDE.md` (key files table: `release_copy.py`, `release_record.py`, `github_api.py`)\n\n**Acceptance Criteria:**\n- [ ] `docs/PERCONA_OBS_TOOL.md` documents: `--source-commit`, the refresh rule (\"last entry without a git tag is open\"), the mapping entry form, `sync prerelease` flags (`--branch-from`, `--synced-commit`, `--source-commit`, `--force`, `--no-freeze`, `--freeze-timeout`, `--verify-timeout`, `--dry-run`), `sync release --from`, and that the workflow integration follows in rollout step 2\n- [ ] Key-file tables list the three new modules\n- [ ] `venv/bin/pytest -q`, `venv/bin/black --check percona_obs/`, `venv/bin/pyright` all pass\n\n**Verify:** `venv/bin/pytest -q && venv/bin/black --check percona_obs/ && venv/bin/pyright && grep -c \"sync prerelease\" docs/PERCONA_OBS_TOOL.md` \u2192 green, count \u2265 3\n\n```json:metadata\n{\"files\": [\".github/copilot-instructions.md\", \"CLAUDE.md\", \"docs/PERCONA_OBS_TOOL.md\", \"github_api.py\", \"release_copy.py\", \"release_record.py\"], \"verifyCommand\": \"venv/bin/pytest -q && venv/bin/black --check percona_obs/ && venv/bin/pyright && grep -c \\\"sync prerelease\\\" docs/PERCONA_OBS_TOOL.md\", \"acceptanceCriteria\": [\"`docs/PERCONA_OBS_TOOL.md` documents: `--source-commit`, the refresh rule (\\\"last entry without a git tag is open\\\"), the mapping entry form, `sync prerelease` flags (`--branch-from`, `--synced-commit`, `--source-commit`, `--force`, `--no-freeze`, `--freeze-timeout`, `--verify-timeout`, `--dry-run`), `sync release --from`, and that the workflow integration follows in rollout step 2\", \"Key-file tables list the three new modules\", \"`venv/bin/pytest -q`, `venv/bin/black --check percona_obs/`, `venv/bin/pyright` all pass\"], \"modelTier\": \"mechanical\"}\n```", + "blockedBy": [ + 6, + 8 + ] + } + ], + "lastUpdated": "2026-10-13T14:27:40.532018+00:00" +} From aeafb7126ae955a67f0f4c9370f4b0bff0aa07ce Mon Sep 17 00:00:00 2001 From: Ricardo Dias Date: Tue, 13 Oct 2026 15:43:54 +0100 Subject: [PATCH 03/16] release.yaml: accept {tag, source_commit} entries via release_record Signed-off-by: Ricardo Dias --- .github/workflows/obs-pr-cleanup.yml | 2 +- percona_obs/cmd_project.py | 13 +++--- percona_obs/cmd_sync.py | 6 +-- percona_obs/common.py | 10 +++-- percona_obs/release_record.py | 63 ++++++++++++++++++++++++++++ tests/test_computed_macros.py | 18 ++++++++ tests/test_release_record.py | 50 ++++++++++++++++++++++ 7 files changed, 146 insertions(+), 16 deletions(-) create mode 100644 percona_obs/release_record.py create mode 100644 tests/test_release_record.py diff --git a/.github/workflows/obs-pr-cleanup.yml b/.github/workflows/obs-pr-cleanup.yml index 7de53209f..7477efac8 100644 --- a/.github/workflows/obs-pr-cleanup.yml +++ b/.github/workflows/obs-pr-cleanup.yml @@ -135,7 +135,7 @@ jobs: while IFS= read -r release_file; do TAG=$(git show "${MERGE_SHA}:${release_file}" | venv/bin/python3 -c \ - "import sys,yaml; d=yaml.safe_load(sys.stdin); r=d.get('releases') or []; print(r[-1] if r else d.get('revision') or '')") + "import sys,yaml; d=yaml.safe_load(sys.stdin); r=d.get('releases') or []; last=r[-1] if r else d.get('revision') or ''; print(last.get('tag','') if isinstance(last, dict) else last)") [ -z "$TAG" ] && continue echo "Creating tag ${TAG} at ${MERGE_SHA}" if ! OUT=$(gh api "repos/${GITHUB_REPOSITORY}/git/refs" -X POST \ diff --git a/percona_obs/cmd_project.py b/percona_obs/cmd_project.py index fc19b7d9c..4307a1e71 100644 --- a/percona_obs/cmd_project.py +++ b/percona_obs/cmd_project.py @@ -79,6 +79,7 @@ ) from .cve_scan import ChangedPackage, scan_release_cves from .release_scope import has_container_images, tree_has_server_package +from .release_record import ReleaseEntry, parse_release_entries from .services import _git_head_sha, _git_tag_for_sha _YAML_FILENAMES = {"project.yaml", "package.yaml"} @@ -2290,14 +2291,12 @@ def cmd_project_release(args: argparse.Namespace) -> None: is_first_release = not release_dir.is_dir() # Load existing release list if updating. - existing_releases: list[str] = [] + existing_entries: list[ReleaseEntry] = [] if not is_first_release: - existing_data = load_yaml(release_dir / "release.yaml") - raw = existing_data.get("releases") - if isinstance(raw, list): - existing_releases = [str(r) for r in raw] - elif existing_data.get("revision"): - existing_releases = [str(existing_data["revision"])] + existing_entries = parse_release_entries( + load_yaml(release_dir / "release.yaml") + ) + existing_releases: list[str] = [e.tag for e in existing_entries] # Auto-derive release-id from OBS if not provided. release_id: str = args.release_id or "" diff --git a/percona_obs/cmd_sync.py b/percona_obs/cmd_sync.py index 6771d8227..42c761ab6 100644 --- a/percona_obs/cmd_sync.py +++ b/percona_obs/cmd_sync.py @@ -54,6 +54,7 @@ parse_env_overrides, resolve_project_path, ) +from .release_record import release_tags from .project_config import ( RepositoryFilter, package_in_slice, @@ -2636,10 +2637,7 @@ def cmd_sync_release(args) -> None: if not source_project_id: raise SystemExit(f"error: release.yaml missing 'project' field: {release_file}") - # Support both new plain-string list format and old revision field. - releases_list: list[str] = release_data.get("releases") or [] - if not releases_list and release_data.get("revision"): - releases_list = [str(release_data["revision"])] + releases_list = release_tags(release_data) tag = releases_list[-1] if releases_list else "" apiurl: str = osc.conf.config["apiurl"] diff --git a/percona_obs/common.py b/percona_obs/common.py index 55e69aa2b..d34cfb786 100644 --- a/percona_obs/common.py +++ b/percona_obs/common.py @@ -337,10 +337,12 @@ def compute_ppg_release( return "1" if not isinstance(data, dict): return "1" - raw = data.get("releases") - tags = [str(t) for t in raw] if isinstance(raw, list) else [] - if not tags and data.get("revision"): - tags = [str(data["revision"])] + from .release_record import release_tags # local: release_record is leaf + + try: + tags = release_tags(data) + except SystemExit: + return "1" prefix = f"{product}/{pg_version}-" return str(sum(1 for t in tags if t.startswith(prefix)) + 1) diff --git a/percona_obs/release_record.py b/percona_obs/release_record.py new file mode 100644 index 000000000..ea4ebe011 --- /dev/null +++ b/percona_obs/release_record.py @@ -0,0 +1,63 @@ +"""``release.yaml`` ``releases:`` entries. + +Two entry forms coexist: the plain tag string (``- ppg/18.4-1``, all +historical entries) and the mapping form written by ``project release`` +since the pre-release flow (``- {tag: ppg/18.7-1, source_commit: }``). +``source_commit`` is the full SHA of ``main`` whose staging tree the +release's binaries were built from. The legacy single ``revision:`` field +is still read as a one-element list. +""" + +from dataclasses import dataclass +from typing import Any + + +@dataclass(frozen=True) +class ReleaseEntry: + tag: str + source_commit: "str | None" = None + + +def parse_release_entries(data: Any) -> "list[ReleaseEntry]": + """Return the ``releases:`` list of a loaded release.yaml as entries. + + Raises SystemExit naming the offending item for anything that is neither + a string nor a mapping with a string ``tag``. + """ + if not isinstance(data, dict): + return [] + raw = data.get("releases") + if not raw and data.get("revision"): + return [ReleaseEntry(str(data["revision"]))] + if not isinstance(raw, list): + return [] + entries: list[ReleaseEntry] = [] + for item in raw: + if isinstance(item, str): + entries.append(ReleaseEntry(item)) + elif isinstance(item, dict) and isinstance(item.get("tag"), str): + sc = item.get("source_commit") + entries.append(ReleaseEntry(item["tag"], str(sc) if sc else None)) + else: + raise SystemExit( + f"error: release.yaml: unrecognised releases entry {item!r} " + "(expected a tag string or {tag: ..., source_commit: ...})" + ) + return entries + + +def release_tags(data: Any) -> "list[str]": + return [e.tag for e in parse_release_entries(data)] + + +def current_release(data: Any) -> "ReleaseEntry | None": + """The last entry is the release in flight (or just shipped).""" + entries = parse_release_entries(data) + return entries[-1] if entries else None + + +def dump_release_entry(entry: ReleaseEntry) -> "str | dict[str, str]": + """YAML-ready value: a bare tag when there is nothing else to record.""" + if entry.source_commit is None: + return entry.tag + return {"tag": entry.tag, "source_commit": entry.source_commit} diff --git a/tests/test_computed_macros.py b/tests/test_computed_macros.py index 4f2ca8873..65cdc1266 100644 --- a/tests/test_computed_macros.py +++ b/tests/test_computed_macros.py @@ -170,3 +170,21 @@ def test_load_macros_injects_prev_major(tmp_path, monkeypatch): root = _tree(tmp_path) monkeypatch.setattr(common, "REPO_ROOT", root) assert common.load_macros(root / "ppg/staging/18")["PG_PREV_MAJOR_VERSION"] == "17" + + +MAPPING_RELEASE_YAML = """\ +project: ppg:staging:18 +releases: +- ppg/18.3-1 +- tag: ppg/18.4-1 + source_commit: 9e9d9738c0ffee +""" + + +def test_compute_counts_mapping_entries(): + assert compute_ppg_release("ppg", "18.4", MAPPING_RELEASE_YAML) == "2" + assert compute_ppg_release("ppg", "18.3", MAPPING_RELEASE_YAML) == "2" + + +def test_compute_tolerates_garbage_entries(): + assert compute_ppg_release("ppg", "18.4", "releases: [{x: 1}]\n") == "1" diff --git a/tests/test_release_record.py b/tests/test_release_record.py new file mode 100644 index 000000000..64a4f9cf5 --- /dev/null +++ b/tests/test_release_record.py @@ -0,0 +1,50 @@ +"""Unit tests for percona_obs.release_record.""" + +import pytest + +from percona_obs.release_record import ( + ReleaseEntry, + current_release, + dump_release_entry, + parse_release_entries, + release_tags, +) + + +def test_parse_string_and_mapping_entries(): + data = { + "releases": [ + "ppg/18.4-1", + {"tag": "ppg/18.6-1"}, + {"tag": "ppg/18.7-1", "source_commit": "abc123"}, + ] + } + assert parse_release_entries(data) == [ + ReleaseEntry("ppg/18.4-1"), + ReleaseEntry("ppg/18.6-1"), + ReleaseEntry("ppg/18.7-1", "abc123"), + ] + assert release_tags(data) == ["ppg/18.4-1", "ppg/18.6-1", "ppg/18.7-1"] + assert current_release(data) == ReleaseEntry("ppg/18.7-1", "abc123") + + +def test_parse_legacy_revision_and_empty(): + assert parse_release_entries({"revision": "ppg/17.9-1"}) == [ + ReleaseEntry("ppg/17.9-1") + ] + assert parse_release_entries({}) == [] + assert parse_release_entries("not a dict") == [] + assert current_release({}) is None + + +def test_parse_rejects_garbage_entry(): + with pytest.raises(SystemExit, match="unrecognised releases entry"): + parse_release_entries({"releases": [{"source_commit": "x"}]}) + + +def test_dump_round_trip(): + assert dump_release_entry(ReleaseEntry("ppg/18.4-1")) == "ppg/18.4-1" + assert dump_release_entry(ReleaseEntry("ppg/18.7-1", "abc")) == { + "tag": "ppg/18.7-1", + "source_commit": "abc", + } From 69f19a1a318317dc1c46261b98364f27158944cf Mon Sep 17 00:00:00 2001 From: Ricardo Dias Date: Tue, 13 Oct 2026 15:48:30 +0100 Subject: [PATCH 04/16] github_api: look up the last synced main commit via gh Signed-off-by: Ricardo Dias --- percona_obs/github_api.py | 67 +++++++++++++++++++++++++++++++++++ tests/test_github_api.py | 73 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 140 insertions(+) create mode 100644 percona_obs/github_api.py create mode 100644 tests/test_github_api.py diff --git a/percona_obs/github_api.py b/percona_obs/github_api.py new file mode 100644 index 000000000..a00dd36a3 --- /dev/null +++ b/percona_obs/github_api.py @@ -0,0 +1,67 @@ +"""GitHub REST lookups through the ``gh`` CLI. + +Only ``gh`` is used (no token handling in the tool): locally the operator is +logged in, in CI ``GH_TOKEN`` is set. Every call is a subprocess so tests +monkeypatch ``_gh_api_json``. +""" + +import json +import re +import subprocess +from typing import Any + +_SLUG_RE = re.compile(r"github\.com[/:]([^/\s]+/[^/\s]+?)(?:\.git)?/?$") + +# Job name prefix of sync-main.yml's upload legs ("percona-obs sync push ()"). +SYNC_JOB_PREFIX = "percona-obs sync push" + + +def repo_slug_from_url(url: str) -> str: + """``https://github.com/o/r.git`` / ``git@github.com:o/r.git`` → ``o/r``.""" + m = _SLUG_RE.search(url.strip()) + if not m: + raise SystemExit(f"error: cannot derive owner/repo from {url!r}") + return m.group(1) + + +def _gh_api_json(path: str) -> Any: + r = subprocess.run(["gh", "api", path], capture_output=True, text=True) + if r.returncode != 0: + raise SystemExit(f"error: gh api {path} failed: {r.stderr.strip()}") + return json.loads(r.stdout or "null") + + +def last_synced_head_sha( + repo_slug: str, + *, + workflow_file: str = "sync-main.yml", + branch: str = "main", + job_prefix: str = SYNC_JOB_PREFIX, + max_runs: int = 20, +) -> str: + """Head SHA of the newest completed *workflow_file* run on *branch* whose + sync jobs (name starting with *job_prefix*) all concluded ``success``. + + The run's own conclusion is ignored on purpose: the build poll job fails + the run after the upload already happened, and that upload is what OBS + holds. + """ + data = _gh_api_json( + f"repos/{repo_slug}/actions/workflows/{workflow_file}/runs" + f"?branch={branch}&status=completed&per_page={max_runs}" + ) + for run in data.get("workflow_runs") or []: + jobs = ( + _gh_api_json( + f"repos/{repo_slug}/actions/runs/{run['id']}/jobs?per_page=100" + ).get("jobs") + or [] + ) + sync_jobs = [j for j in jobs if str(j.get("name", "")).startswith(job_prefix)] + if sync_jobs and all(j.get("conclusion") == "success" for j in sync_jobs): + return str(run["head_sha"]) + raise SystemExit( + f"error: no completed {workflow_file} run on {branch} of {repo_slug} " + f"with successful sync jobs in the last {max_runs} runs; " + "pass --source-commit explicitly" + ) diff --git a/tests/test_github_api.py b/tests/test_github_api.py new file mode 100644 index 000000000..d15968f7f --- /dev/null +++ b/tests/test_github_api.py @@ -0,0 +1,73 @@ +"""Unit tests for percona_obs.github_api (gh is monkeypatched).""" + +import pytest + +import percona_obs.github_api as gha + + +@pytest.mark.parametrize( + "url", + [ + "https://github.com/percona/obs-packaging.git", + "https://github.com/percona/obs-packaging", + "git@github.com:percona/obs-packaging.git", + ], +) +def test_repo_slug_from_url(url): + assert gha.repo_slug_from_url(url) == "percona/obs-packaging" + + +def test_repo_slug_rejects_unknown(): + with pytest.raises(SystemExit, match="cannot derive owner/repo"): + gha.repo_slug_from_url("file:///tmp/x") + + +def _fake_api(monkeypatch, runs, jobs_by_run): + def fake(path): + if "/workflows/sync-main.yml/runs" in path: + return {"workflow_runs": runs} + for run_id, jobs in jobs_by_run.items(): + if f"/runs/{run_id}/jobs" in path: + return {"jobs": jobs} + raise AssertionError(path) + + monkeypatch.setattr(gha, "_gh_api_json", fake) + + +def test_last_synced_skips_runs_with_failed_sync_job(monkeypatch): + runs = [ + {"id": 3, "head_sha": "c" * 40}, + {"id": 2, "head_sha": "b" * 40}, + {"id": 1, "head_sha": "a" * 40}, + ] + jobs = { + 3: [ + {"name": "percona-obs sync push (boo)", "conclusion": "success"}, + {"name": "percona-obs sync push (labs)", "conclusion": "failure"}, + {"name": "Wait for OBS builds (boo)", "conclusion": "success"}, + ], + 2: [ + {"name": "percona-obs sync push (boo)", "conclusion": "success"}, + {"name": "percona-obs sync push (labs)", "conclusion": "success"}, + {"name": "Wait for OBS builds (boo)", "conclusion": "failure"}, + ], + } + _fake_api(monkeypatch, runs, jobs) + assert gha.last_synced_head_sha("percona/obs-packaging") == "b" * 40 + + +def test_last_synced_no_qualifying_run(monkeypatch): + _fake_api(monkeypatch, [{"id": 1, "head_sha": "a" * 40}], {1: []}) + with pytest.raises(SystemExit, match="no completed sync-main.yml run"): + gha.last_synced_head_sha("percona/obs-packaging") + + +def test_gh_failure_is_reported(monkeypatch): + class R: + returncode = 1 + stdout = "" + stderr = "gh: not logged in" + + monkeypatch.setattr(gha.subprocess, "run", lambda *a, **k: R()) + with pytest.raises(SystemExit, match="not logged in"): + gha._gh_api_json("repos/x/y") From c8c7b38486300d00304dff887d8b919d21a65d0a Mon Sep 17 00:00:00 2001 From: Ricardo Dias Date: Tue, 13 Oct 2026 15:52:16 +0100 Subject: [PATCH 05/16] sync release: move copy mechanics into release_copy, one code path Signed-off-by: Ricardo Dias --- percona_obs/cmd_sync.py | 425 +++++------------------------------- percona_obs/obs_api.py | 7 +- percona_obs/release_copy.py | 225 +++++++++++++++++++ tests/test_release_copy.py | 113 ++++++++++ tests/test_sync_release.py | 161 ++++---------- 5 files changed, 451 insertions(+), 480 deletions(-) create mode 100644 percona_obs/release_copy.py create mode 100644 tests/test_release_copy.py diff --git a/percona_obs/cmd_sync.py b/percona_obs/cmd_sync.py index 42c761ab6..07e925ea7 100644 --- a/percona_obs/cmd_sync.py +++ b/percona_obs/cmd_sync.py @@ -54,6 +54,11 @@ parse_env_overrides, resolve_project_path, ) +from .release_copy import ( + apply_mirror_meta, + collect_release_subprojects, + release_tree, +) from .release_record import release_tags from .project_config import ( RepositoryFilter, @@ -79,12 +84,10 @@ save_manifest, ) from .obs_api import ( - _add_release_targets, _apply_package_config, _flag_entries, _apply_project_config, _disable_package_builds, - _disable_project_builds, _create_project_skeleton, _copy_project_conf, _create_release_project, @@ -101,8 +104,6 @@ _fetch_obs_project_repository_names, _fetch_obs_subproject_names, _obs_project_exists, - _read_project_release_source, - _remove_release_targets, _upload_obs_files, ) from .release_scope import collect_release_scope @@ -112,7 +113,6 @@ freeze_packages, restore_builds, restore_packages, - verify_release_landed, wait_for_quiesce, ) from .services import ( @@ -139,10 +139,6 @@ _OBS_SUBSTITUTABLE = {"_service", "_aggregate", "_link"} -# Resolve osc relative to the running Python so it works inside a venv even -# when the venv's bin/ directory is not on PATH (e.g. CI runners). -_OSC_BIN = os.path.join(os.path.dirname(sys.executable), "osc") - # Stem of the OBS sub-project that holds per-PR build environments. # This sub-project is managed exclusively by CI workflows and must never be # deleted by orphan cleanup or created by sync push. @@ -2438,156 +2434,6 @@ def cmd_sync_promote(args) -> None: _print_ok(f"promote successful{suffix} ({promoted} promoted, {skipped} skipped)") -def _collect_release_subprojects( - source_project_id: str, - release_path: Path, -) -> "tuple[list[tuple[str, Path]], list[str]]": - """Map source subprojects to their release-mirror directories. - - Returns (pairs, missing): - pairs — [(subproject_name, release_sub_path)] for every source - subproject whose mirror project.yaml exists. - missing — subproject names with no mirror project.yaml. Full-snapshot - releases require a mirror for every subproject; callers must - treat a non-empty list as a hard error. - - Subprojects out of the active slice are skipped on both sides; `missing` - lists only in-slice source subprojects without a mirror. - """ - source_path = resolve_project_path(source_project_id) - pairs: list[tuple[str, Path]] = [] - missing: list[str] = [] - for sub_obs_id, sub_path in find_projects(source_path, source_project_id): - if sub_obs_id == source_project_id: - continue - if not (sub_path / "project.yaml").is_file(): - continue # intermediate directory, not a real OBS project - subproject_name = sub_obs_id[len(source_project_id) + 1 :] - release_sub_path = release_path / Path(*subproject_name.split(":")) - if (release_sub_path / "project.yaml").is_file(): - if project_in_slice(release_sub_path): - pairs.append((subproject_name, release_sub_path)) - # else: this instance does not hold the subproject — nothing to release - elif project_in_slice(sub_path): - missing.append(subproject_name) - # else: source out of slice too; a missing mirror is irrelevant here - return pairs, missing - - -def _filter_release_repo_names( - apiurl: str, - source_repo_names: "list[str]", - release_obs_project: str, -) -> "list[str]": - """Keep only source repos that exist in the release project. - - Adding a for a repo the target lacks makes osc release - fail or drop binaries; a mismatch here means the mirror regeneration and - the live release project disagree, which is worth a loud warning but not - an abort (the mirror was just applied — a leftover mismatch is repo - removal pending on OBS's side). - """ - target = _fetch_obs_project_repository_names(apiurl, release_obs_project) - kept = [r for r in source_repo_names if r in target] - for r in source_repo_names: - if r not in target: - print( - f"warning: source repo {r} has no counterpart in " - f"{release_obs_project}; skipped", - flush=True, - ) - return kept - - -def _sync_release_subprojects( - apiurl: str, - args, - source_project_id: str, - release_obs_project: str, - release_path: Path, - env_vars: dict[str, str], - pairs: "list[tuple[str, Path]] | None" = None, -) -> None: - """Create and populate OBS subprojects for a release (e.g. containers). - - Every source subproject MUST have a mirror project.yaml under the - release tree — a missing mirror aborts the release (a release is a full - snapshot; nothing is skipped silently). Release-tier OBS projects with - no local mirror are reported as orphans (manual `sync delete`). - - ``pairs`` may be passed pre-collected (e.g. by ``cmd_sync_release``, - which needs the list before this function runs in order to compute the - build-freeze scope) to avoid walking the tree twice; when omitted this - still performs its own collection and hard-error check for direct - callers. - """ - if pairs is None: - pairs, missing = _collect_release_subprojects(source_project_id, release_path) - if missing: - listing = "\n".join( - f" {source_project_id}:{name}" for name in sorted(missing) - ) - raise SystemExit( - "error: staging subprojects have no release mirror " - f"(missing project.yaml under {release_path.relative_to(_REPO_DIR)}):\n" - f"{listing}\n" - "Re-run 'percona-obs project release' to regenerate the release tree." - ) - - for subproject_name, release_sub_path in pairs: - release_sub_obs = f"{release_obs_project}:{subproject_name}" - sub_obs_id = f"{source_project_id}:{subproject_name}" - - paths_stripped, _ = _apply_project_config( - apiurl, - release_sub_obs, - release_sub_path, - args.rootprj, - env_vars=env_vars, - ) - if paths_stripped: - _apply_project_config( - apiurl, - release_sub_obs, - release_sub_path, - args.rootprj, - env_vars=env_vars, - ) - _disable_project_builds(apiurl, release_sub_obs) - - source_sub_obs = f"{args.rootprj}:{sub_obs_id}" - source_sub_repo_elems, _ = _read_project_release_source(apiurl, source_sub_obs) - source_sub_repo_names = _filter_release_repo_names( - apiurl, - [r.get("name", "") for r in source_sub_repo_elems if r.get("name")], - release_sub_obs, - ) - _add_release_targets( - apiurl, source_sub_obs, release_sub_obs, source_sub_repo_names - ) - try: - _print_pending(f"releasing {source_sub_obs} → {release_sub_obs}") - subprocess.run( - [_OSC_BIN, "-A", apiurl, "release", source_sub_obs, "--no-delay"], - check=True, - ) - finally: - _remove_release_targets(apiurl, source_sub_obs, release_sub_obs) - _print_ok(f"released {source_sub_obs} → {release_sub_obs}") - - # Orphan detection: release-tier OBS projects with no local mirror. - local_names = {name for name, _ in pairs} - for obs_sub in sorted(_fetch_obs_subproject_names(apiurl, release_obs_project)): - sub_name = obs_sub[len(release_obs_project) + 1 :] - if sub_name not in local_names: - print( - f"warning: orphaned release subproject on OBS: {obs_sub} " - "(no local mirror) — remove manually with " - f"'percona-obs sync delete {obs_sub[len(args.rootprj) + 1 :]}'", - flush=True, - ) - - def _build_freeze_scope( source_obs_project: str, source_sub_obs_projects: "list[str]", @@ -2611,6 +2457,41 @@ def _build_freeze_scope( return whole, packages +def _check_source_unchanged_since_tag(source_project_id: str, tag: str, args) -> None: + """Abort when the source project's tree changed since *tag* (legacy + single-hop release only; ``--skip-tag-check`` skips, no tag = nothing).""" + if not tag: + return + if getattr(args, "skip_tag_check", False): + _print_same(f"skipping tag divergence check for {tag} (--skip-tag-check)") + return + source_path = resolve_project_path(source_project_id) + _print_pending(f"checking for changes in {source_project_id} since tag {tag}") + result = subprocess.run( + ["git", "diff", "--name-only", f"{tag}..HEAD", "--", str(source_path)], + capture_output=True, + text=True, + cwd=_REPO_DIR, + ) + if result.returncode != 0: + print(result.stderr, end="", file=sys.stderr) + raise SystemExit( + f"error: git diff failed — is the tag '{tag}' present locally?" + ) + changed = [l.strip() for l in result.stdout.splitlines() if l.strip()] + if changed: + print( + f"error: source project {source_project_id} has changed since tag {tag}:", + file=sys.stderr, + ) + for f in changed: + print(f" {f}", file=sys.stderr) + raise SystemExit( + "The source project has diverged from the release tag. " + "Use --force to skip this check." + ) + + def cmd_sync_release(args) -> None: """Release packages from an OBS source project to a release target. @@ -2649,7 +2530,7 @@ def cmd_sync_release(args) -> None: **auto_rootprj_env(args.rootprj), } - subproject_pairs, missing_mirrors = _collect_release_subprojects( + subproject_pairs, missing_mirrors = collect_release_subprojects( source_project_id, release_path ) source_sub_obs_projects = [ @@ -2755,219 +2636,31 @@ def _freeze_and_run(release_all) -> None: _print_pending(f"checking release project {release_obs_project}") project_exists = _obs_project_exists(apiurl, release_obs_project) - if project_exists: - # Update-in-place: re-run osc release to copy updated binaries. - # Also covers idempotent re-runs of a first-release that partially succeeded. - if not args.force: - source_path = resolve_project_path(source_project_id) - if tag and getattr(args, "skip_tag_check", False): - _print_same( - f"skipping tag divergence check for {tag} (--skip-tag-check)" - ) - elif tag: - _print_pending( - f"checking for changes in {source_project_id} since tag {tag}" - ) - result = subprocess.run( - [ - "git", - "diff", - "--name-only", - f"{tag}..HEAD", - "--", - str(source_path), - ], - capture_output=True, - text=True, - cwd=_REPO_DIR, - ) - if result.returncode != 0: - print(result.stderr, end="", file=sys.stderr) - raise SystemExit( - f"error: git diff failed — is the tag '{tag}' present locally?" - ) - changed_files = [ - line.strip() for line in result.stdout.splitlines() if line.strip() - ] - if changed_files: - print( - f"error: source project {source_project_id} has changed since " - f"tag {tag}:", - file=sys.stderr, - ) - for f in changed_files: - print(f" {f}", file=sys.stderr) - raise SystemExit( - "The source project has diverged from the release tag. " - "Use --force to skip this check." - ) + if not args.force: + _check_source_unchanged_since_tag(source_project_id, tag, args) + if project_exists: # Re-apply the local release project config so repo additions, # publish flags and prjconf changes reach OBS on update releases too. - paths_stripped, _ = _apply_project_config( - apiurl, - release_obs_project, - release_path, - args.rootprj, - env_vars=shared_env_vars, + apply_mirror_meta( + apiurl, release_obs_project, release_path, args.rootprj, shared_env_vars ) - if paths_stripped: - _apply_project_config( - apiurl, - release_obs_project, - release_path, - args.rootprj, - env_vars=shared_env_vars, - ) - _disable_project_builds(apiurl, release_obs_project) + else: + # First release: snapshot the source topology, builds disabled. + _create_release_project(apiurl, source_obs_project, release_obs_project) + _copy_project_conf(apiurl, source_obs_project, release_obs_project) - _remove_release_targets(apiurl, source_obs_project, release_obs_project) - source_repo_elems, _ = _read_project_release_source(apiurl, source_obs_project) - repo_names = _filter_release_repo_names( + _freeze_and_run( + lambda: release_tree( apiurl, - [r.get("name", "") for r in source_repo_elems if r.get("name")], + source_obs_project, release_obs_project, + subproject_pairs, + args.rootprj, + shared_env_vars, + verify_timeout=args.verify_timeout, ) - - def _release_all() -> None: - _add_release_targets( - apiurl, source_obs_project, release_obs_project, repo_names - ) - try: - _print_pending( - f"releasing {source_obs_project} → {release_obs_project}" - ) - subprocess.run( - [ - _OSC_BIN, - "-A", - apiurl, - "release", - source_obs_project, - "--no-delay", - ], - check=True, - ) - finally: - _remove_release_targets(apiurl, source_obs_project, release_obs_project) - - _sync_release_subprojects( - apiurl, - args, - source_project_id, - release_obs_project, - release_path, - env_vars=shared_env_vars, - pairs=subproject_pairs, - ) - if args.verify_timeout > 0: - verify_release_landed( - apiurl, - source_obs_project, - release_obs_project, - timeout_s=args.verify_timeout, - ) - for name, _ in subproject_pairs: - verify_release_landed( - apiurl, - f"{source_obs_project}:{name}", - f"{release_obs_project}:{name}", - timeout_s=args.verify_timeout, - ) - - _freeze_and_run(_release_all) - - update_suffix = f" (update: {tag})" if tag else "" - _print_ok( - f"~ release {source_obs_project} → {release_obs_project}{update_suffix}" - ) - return - - # First release: check that staging hasn't changed since tag, then create. - if not args.force: - source_path = resolve_project_path(source_project_id) - if tag and getattr(args, "skip_tag_check", False): - _print_same(f"skipping tag divergence check for {tag} (--skip-tag-check)") - elif tag: - _print_pending( - f"checking for changes in {source_project_id} since tag {tag}" - ) - result = subprocess.run( - ["git", "diff", "--name-only", f"{tag}..HEAD", "--", str(source_path)], - capture_output=True, - text=True, - cwd=_REPO_DIR, - ) - if result.returncode != 0: - print(result.stderr, end="", file=sys.stderr) - raise SystemExit( - f"error: git diff failed — is the tag '{tag}' present locally?" - ) - changed_files = [ - line.strip() for line in result.stdout.splitlines() if line.strip() - ] - if changed_files: - print( - f"error: source project {source_project_id} has changed since " - f"tag {tag}:", - file=sys.stderr, - ) - for f in changed_files: - print(f" {f}", file=sys.stderr) - raise SystemExit( - "The source project has diverged from the release tag. " - "Use --force to skip this check." - ) - - # Create the release target project on OBS (builds disabled, same repos as source). - source_repo_elems = _create_release_project( - apiurl, source_obs_project, release_obs_project ) - repo_names = _filter_release_repo_names( - apiurl, - [r.get("name", "") for r in source_repo_elems if r.get("name")], - release_obs_project, - ) - - _copy_project_conf(apiurl, source_obs_project, release_obs_project) - - def _release_all() -> None: - _add_release_targets( - apiurl, source_obs_project, release_obs_project, repo_names - ) - try: - _print_pending(f"releasing {source_obs_project} → {release_obs_project}") - subprocess.run( - [_OSC_BIN, "-A", apiurl, "release", source_obs_project, "--no-delay"], - check=True, - ) - finally: - _remove_release_targets(apiurl, source_obs_project, release_obs_project) - - _sync_release_subprojects( - apiurl, - args, - source_project_id, - release_obs_project, - release_path, - env_vars=shared_env_vars, - pairs=subproject_pairs, - ) - if args.verify_timeout > 0: - verify_release_landed( - apiurl, - source_obs_project, - release_obs_project, - timeout_s=args.verify_timeout, - ) - for name, _ in subproject_pairs: - verify_release_landed( - apiurl, - f"{source_obs_project}:{name}", - f"{release_obs_project}:{name}", - timeout_s=args.verify_timeout, - ) - - _freeze_and_run(_release_all) - _print_ok(f"release {source_obs_project} → {release_obs_project}") + suffix = f" (update: {tag})" if project_exists and tag else "" + _print_ok(f"release {source_obs_project} → {release_obs_project}{suffix}") diff --git a/percona_obs/obs_api.py b/percona_obs/obs_api.py index 69927983e..44848b8aa 100644 --- a/percona_obs/obs_api.py +++ b/percona_obs/obs_api.py @@ -1223,6 +1223,7 @@ def _apply_project_config( active_projects: "set[str] | None" = None, branch_rootprj: str | None = None, existing_branch_projects: "dict[str, set[str]] | None" = None, + publish_override: "dict[str, bool] | bool | None" = None, ) -> "tuple[bool, bool]": """Create or update OBS project metadata and build config from project.yaml. @@ -1271,7 +1272,11 @@ def _apply_project_config( project_config.get("description", ""), repos, rootprj, - publish=project_config.get("publish"), + publish=( + publish_override + if publish_override is not None + else project_config.get("publish") + ), build=build, debuginfo=project_config.get("debuginfo"), active_projects=active_projects, diff --git a/percona_obs/release_copy.py b/percona_obs/release_copy.py new file mode 100644 index 000000000..425762dc9 --- /dev/null +++ b/percona_obs/release_copy.py @@ -0,0 +1,225 @@ +"""Shared mechanics of the two release hops. + +Hop 1 (``sync prerelease``): staging → per-PR pre-release project. +Hop 2 (``sync release --from``): pre-release → release project. +Both apply mirror project.yaml meta to the target tree, copy binaries with +``osc release`` and verify the copy landed. The legacy single-hop +``sync release`` (no ``--from``) uses the same primitives. + +``osc release`` requires ```` entries on the +source, so a copy adds them, runs osc with ``--target-project`` and removes +them again in a ``finally`` — the source meta is unchanged at rest. +""" + +import os +import subprocess +import sys +from pathlib import Path + +from .common import ( + _REPO_DIR, + _print_ok, + _print_pending, + find_projects, + resolve_project_path, +) +from .obs_api import ( + _add_release_targets, + _apply_project_config, + _disable_project_builds, + _fetch_obs_project_repository_names, + _fetch_obs_subproject_names, + _read_project_release_source, + _remove_release_targets, +) +from .project_config import project_in_slice +from .release_freeze import verify_release_landed + +_OSC_BIN = os.path.join(os.path.dirname(sys.executable), "osc") + + +def collect_release_subprojects( + source_project_id: str, + release_path: Path, +) -> "tuple[list[tuple[str, Path]], list[str]]": + """Map source subprojects to their release-mirror directories. + + Returns (pairs, missing): + pairs — [(subproject_name, release_sub_path)] for every source + subproject whose mirror project.yaml exists. + missing — subproject names with no mirror project.yaml. Full-snapshot + releases require a mirror for every subproject; callers must + treat a non-empty list as a hard error. + + Subprojects out of the active slice are skipped on both sides; `missing` + lists only in-slice source subprojects without a mirror. + """ + source_path = resolve_project_path(source_project_id) + pairs: list[tuple[str, Path]] = [] + missing: list[str] = [] + for sub_obs_id, sub_path in find_projects(source_path, source_project_id): + if sub_obs_id == source_project_id: + continue + if not (sub_path / "project.yaml").is_file(): + continue # intermediate directory, not a real OBS project + subproject_name = sub_obs_id[len(source_project_id) + 1 :] + release_sub_path = release_path / Path(*subproject_name.split(":")) + if (release_sub_path / "project.yaml").is_file(): + if project_in_slice(release_sub_path): + pairs.append((subproject_name, release_sub_path)) + # else: this instance does not hold the subproject — nothing to release + elif project_in_slice(sub_path): + missing.append(subproject_name) + # else: source out of slice too; a missing mirror is irrelevant here + return pairs, missing + + +def filter_release_repo_names( + apiurl: str, + source_repo_names: "list[str]", + release_obs_project: str, +) -> "list[str]": + """Keep only source repos that exist in the release project. + + Adding a for a repo the target lacks makes osc release + fail or drop binaries; a mismatch here means the mirror regeneration and + the live release project disagree, which is worth a loud warning but not + an abort (the mirror was just applied — a leftover mismatch is repo + removal pending on OBS's side). + """ + target = _fetch_obs_project_repository_names(apiurl, release_obs_project) + kept = [r for r in source_repo_names if r in target] + for r in source_repo_names: + if r not in target: + print( + f"warning: source repo {r} has no counterpart in " + f"{release_obs_project}; skipped", + flush=True, + ) + return kept + + +def apply_mirror_meta( + apiurl: str, + target_obs_project: str, + mirror_path: Path, + rootprj: str, + env_vars: "dict[str, str]", + *, + publish_override: "dict[str, bool] | bool | None" = None, +) -> None: + """Create or update *target_obs_project* from the mirror project.yaml and + make sure builds are disabled. Two passes when OBS stripped paths.""" + paths_stripped, _ = _apply_project_config( + apiurl, + target_obs_project, + mirror_path, + rootprj, + env_vars=env_vars, + publish_override=publish_override, + ) + if paths_stripped: + _apply_project_config( + apiurl, + target_obs_project, + mirror_path, + rootprj, + env_vars=env_vars, + publish_override=publish_override, + ) + _disable_project_builds(apiurl, target_obs_project) + + +def copy_project_binaries( + apiurl: str, source_obs_project: str, target_obs_project: str +) -> None: + """``osc release`` every repo of *source* that exists in *target*.""" + repo_elems, _ = _read_project_release_source(apiurl, source_obs_project) + repo_names = filter_release_repo_names( + apiurl, + [r.get("name", "") for r in repo_elems if r.get("name")], + target_obs_project, + ) + _add_release_targets(apiurl, source_obs_project, target_obs_project, repo_names) + try: + _print_pending(f"releasing {source_obs_project} → {target_obs_project}") + subprocess.run( + [ + _OSC_BIN, + "-A", + apiurl, + "release", + source_obs_project, + "--no-delay", + "--target-project", + target_obs_project, + ], + check=True, + ) + finally: + _remove_release_targets(apiurl, source_obs_project, target_obs_project) + _print_ok(f"released {source_obs_project} → {target_obs_project}") + + +def warn_orphans( + apiurl: str, + target_obs_project: str, + pairs: "list[tuple[str, Path]]", + rootprj: str, +) -> None: + """Report target-tier OBS subprojects with no local mirror (never deleted).""" + local_names = {name for name, _ in pairs} + for obs_sub in sorted(_fetch_obs_subproject_names(apiurl, target_obs_project)): + sub_name = obs_sub[len(target_obs_project) + 1 :] + if sub_name not in local_names: + print( + f"warning: orphaned release subproject on OBS: {obs_sub} " + "(no local mirror) — remove manually with " + f"'percona-obs sync delete {obs_sub[len(rootprj) + 1 :]}'", + flush=True, + ) + + +def release_tree( + apiurl: str, + source_obs_project: str, + target_obs_project: str, + pairs: "list[tuple[str, Path]]", + rootprj: str, + env_vars: "dict[str, str]", + *, + publish_override: "dict[str, bool] | bool | None" = None, + verify_timeout: int = 600, +) -> None: + """Copy the top project and every (subproject, mirror) pair, then verify. + + The caller has already applied the top project's meta (that happens + before any freeze); subproject meta is applied here, right before each + subproject's copy, exactly as the single-hop release always did. + """ + copy_project_binaries(apiurl, source_obs_project, target_obs_project) + for subproject_name, mirror_path in pairs: + target_sub = f"{target_obs_project}:{subproject_name}" + apply_mirror_meta( + apiurl, + target_sub, + mirror_path, + rootprj, + env_vars, + publish_override=publish_override, + ) + copy_project_binaries( + apiurl, f"{source_obs_project}:{subproject_name}", target_sub + ) + if verify_timeout > 0: + verify_release_landed( + apiurl, source_obs_project, target_obs_project, timeout_s=verify_timeout + ) + for subproject_name, _ in pairs: + verify_release_landed( + apiurl, + f"{source_obs_project}:{subproject_name}", + f"{target_obs_project}:{subproject_name}", + timeout_s=verify_timeout, + ) + warn_orphans(apiurl, target_obs_project, pairs, rootprj) diff --git a/tests/test_release_copy.py b/tests/test_release_copy.py new file mode 100644 index 000000000..f1dcde331 --- /dev/null +++ b/tests/test_release_copy.py @@ -0,0 +1,113 @@ +"""Unit tests for percona_obs.release_copy (OBS and osc monkeypatched).""" + +from pathlib import Path + +import pytest + +import percona_obs.release_copy as rc + + +def _stub_obs(monkeypatch, calls): + monkeypatch.setattr( + rc, "_read_project_release_source", lambda apiurl, prj: ([], None) + ) + monkeypatch.setattr( + rc, "filter_release_repo_names", lambda apiurl, names, target: names + ) + monkeypatch.setattr( + rc, "_add_release_targets", lambda *a, **k: calls.append(("add", a[1], a[2])) + ) + monkeypatch.setattr( + rc, + "_remove_release_targets", + lambda *a, **k: calls.append(("remove", a[1], a[2])), + ) + monkeypatch.setattr( + rc, + "_apply_project_config", + lambda *a, **k: calls.append(("meta", a[1])) or (False, True), + ) + monkeypatch.setattr(rc, "_disable_project_builds", lambda *a, **k: None) + monkeypatch.setattr(rc, "_fetch_obs_subproject_names", lambda apiurl, prefix: set()) + + +def test_copy_project_binaries_uses_target_project_and_restores(monkeypatch): + calls = [] + _stub_obs(monkeypatch, calls) + seen = {} + monkeypatch.setattr( + rc.subprocess, "run", lambda cmd, **k: seen.setdefault("cmd", cmd) + ) + rc.copy_project_binaries("http://obs", "S", "T") + assert "--target-project" in seen["cmd"] and "T" in seen["cmd"] + assert seen["cmd"][seen["cmd"].index("release") + 1] == "S" + assert calls == [("add", "S", "T"), ("remove", "S", "T")] + + +def test_copy_project_binaries_restores_on_failure(monkeypatch): + calls = [] + _stub_obs(monkeypatch, calls) + + def boom(*a, **k): + raise RuntimeError("osc failed") + + monkeypatch.setattr(rc.subprocess, "run", boom) + with pytest.raises(RuntimeError): + rc.copy_project_binaries("http://obs", "S", "T") + assert calls[-1] == ("remove", "S", "T") + + +def test_release_tree_order_and_verify(monkeypatch, tmp_path): + calls = [] + _stub_obs(monkeypatch, calls) + monkeypatch.setattr( + rc.subprocess, "run", lambda cmd, **k: calls.append(("osc", cmd[4])) + ) + monkeypatch.setattr( + rc, + "verify_release_landed", + lambda apiurl, s, t, timeout_s=600: calls.append(("verify", s, t)), + ) + pairs = [("containers", tmp_path / "containers")] + rc.release_tree("http://obs", "S", "T", pairs, "home:Admin", {}, verify_timeout=5) + assert [c for c in calls if c[0] in ("osc", "meta")] == [ + ("osc", "S"), + ("meta", "T:containers"), + ("osc", "S:containers"), + ] + assert ("verify", "S", "T") in calls + assert ("verify", "S:containers", "T:containers") in calls + + +def test_release_tree_publish_override_reaches_meta(monkeypatch, tmp_path): + seen = {} + calls = [] + _stub_obs(monkeypatch, calls) + monkeypatch.setattr( + rc, + "_apply_project_config", + lambda *a, **k: (seen.setdefault("publish", k.get("publish_override")), True), + ) + monkeypatch.setattr(rc.subprocess, "run", lambda *a, **k: None) + rc.release_tree( + "http://obs", + "S", + "T", + [("x", tmp_path)], + "home:Admin", + {}, + publish_override=True, + verify_timeout=0, + ) + assert seen["publish"] is True + + +def test_warn_orphans(monkeypatch, capsys): + monkeypatch.setattr( + rc, + "_fetch_obs_subproject_names", + lambda apiurl, prefix: {"T:containers", "T:containers:ubi9"}, + ) + rc.warn_orphans("http://obs", "T", [("containers", Path("/x"))], "home:Admin") + out = capsys.readouterr().out + assert "T:containers:ubi9" in out and "sync delete" in out diff --git a/tests/test_sync_release.py b/tests/test_sync_release.py index 2b3136a7a..f4d222402 100644 --- a/tests/test_sync_release.py +++ b/tests/test_sync_release.py @@ -1,8 +1,8 @@ """Unit tests for the sync release subproject handling (percona_obs.cmd_sync). -_collect_release_subprojects maps source subprojects to release mirrors; -_sync_release_subprojects must hard-error on missing mirrors and report -OBS-side orphans. OBS interaction is monkeypatched out. +collect_release_subprojects maps source subprojects to release mirrors; +cmd_sync_release must hard-error on missing mirrors. OBS interaction is +monkeypatched out. """ from pathlib import Path @@ -11,10 +11,8 @@ import pytest import percona_obs.cmd_sync as cmd_sync -from percona_obs.cmd_sync import ( - _collect_release_subprojects, - _sync_release_subprojects, -) +import percona_obs.release_copy as rc +from percona_obs.release_copy import collect_release_subprojects _REPO_YAML = "repositories:\n - name: {name}\n archs: [x86_64]\n paths: []\n" @@ -53,77 +51,12 @@ def _mk_tree(tmp_path: Path, monkeypatch=None): def test_collect_pairs_and_missing(tmp_path, monkeypatch): src, rel = _mk_tree(tmp_path, monkeypatch) - monkeypatch.setattr(cmd_sync, "resolve_project_path", lambda pid: src) - pairs, missing = _collect_release_subprojects("ppg:staging:17", rel) + monkeypatch.setattr(rc, "resolve_project_path", lambda pid: src) + pairs, missing = collect_release_subprojects("ppg:staging:17", rel) assert [name for name, _ in pairs] == ["containers"] assert missing == ["tarballs"] -def test_missing_mirror_is_hard_error(tmp_path, monkeypatch): - src, rel = _mk_tree(tmp_path, monkeypatch) - monkeypatch.setattr(cmd_sync, "resolve_project_path", lambda pid: src) - monkeypatch.setattr(cmd_sync, "_REPO_DIR", tmp_path) - - class Args: - rootprj = "home:Admin" - - with pytest.raises(SystemExit, match="tarballs"): - _sync_release_subprojects( - "http://obs", - Args(), - "ppg:staging:17", - "home:Admin:ppg:releases:17", - rel, - env_vars={}, - ) - - -def test_orphan_reporting(tmp_path, monkeypatch, capsys): - src, rel = _mk_tree(tmp_path, monkeypatch) - # complete the mirror so no hard error fires - (rel / "tarballs").mkdir() - (rel / "tarballs" / "project.yaml").write_text( - "build: false\n" + _REPO_YAML.format(name="ssl3") - ) - monkeypatch.setattr(cmd_sync, "resolve_project_path", lambda pid: src) - monkeypatch.setattr(cmd_sync, "_REPO_DIR", tmp_path) - # neutralize the OBS side of the per-subproject body - monkeypatch.setattr( - cmd_sync, "_apply_project_config", lambda *a, **k: (False, None) - ) - monkeypatch.setattr(cmd_sync, "_disable_project_builds", lambda *a, **k: None) - monkeypatch.setattr( - cmd_sync, "_read_project_release_source", lambda *a, **k: ([], None) - ) - monkeypatch.setattr(cmd_sync, "_add_release_targets", lambda *a, **k: None) - monkeypatch.setattr(cmd_sync, "_remove_release_targets", lambda *a, **k: None) - monkeypatch.setattr(cmd_sync.subprocess, "run", lambda *a, **k: None) - monkeypatch.setattr( - cmd_sync, - "_fetch_obs_subproject_names", - lambda apiurl, prefix: { - "home:Admin:ppg:releases:17:containers", - "home:Admin:ppg:releases:17:containers:ubi9", # old layout = orphan - }, - ) - - class Args: - rootprj = "home:Admin" - - _sync_release_subprojects( - "http://obs", - Args(), - "ppg:staging:17", - "home:Admin:ppg:releases:17", - rel, - env_vars={}, - ) - out = capsys.readouterr().out - assert "orphaned release subproject" in out - assert "containers:ubi9" in out - assert "sync delete" in out - - class _Args: """Minimal args namespace matching what cmd_sync_release touches.""" @@ -148,24 +81,29 @@ def _wire_release_update_path(monkeypatch, src, rel): "resolve_project_path", lambda pid: src if pid == "ppg:staging:17" else rel, ) + monkeypatch.setattr( + rc, "resolve_project_path", lambda pid: src if pid == "ppg:staging:17" else rel + ) monkeypatch.setattr(cmd_sync, "_REPO_DIR", rel.parents[2]) monkeypatch.setattr(cmd_sync.osc.conf, "config", {"apiurl": "http://obs"}) monkeypatch.setattr(cmd_sync, "_obs_project_exists", lambda *a, **k: True) + monkeypatch.setattr(rc, "_apply_project_config", lambda *a, **k: (False, None)) + monkeypatch.setattr(rc, "_disable_project_builds", lambda *a, **k: None) + monkeypatch.setattr(rc, "_read_project_release_source", lambda *a, **k: ([], None)) monkeypatch.setattr( - cmd_sync, "_apply_project_config", lambda *a, **k: (False, None) - ) - monkeypatch.setattr(cmd_sync, "_disable_project_builds", lambda *a, **k: None) - monkeypatch.setattr( - cmd_sync, "_read_project_release_source", lambda *a, **k: ([], None) - ) - monkeypatch.setattr( - cmd_sync, "_filter_release_repo_names", lambda apiurl, names, target: names - ) - monkeypatch.setattr(cmd_sync, "_add_release_targets", lambda *a, **k: None) - monkeypatch.setattr(cmd_sync, "_remove_release_targets", lambda *a, **k: None) - monkeypatch.setattr( - cmd_sync, "_fetch_obs_subproject_names", lambda apiurl, prefix: set() + rc, "filter_release_repo_names", lambda apiurl, names, target: names ) + monkeypatch.setattr(rc, "_add_release_targets", lambda *a, **k: None) + monkeypatch.setattr(rc, "_remove_release_targets", lambda *a, **k: None) + monkeypatch.setattr(rc, "_fetch_obs_subproject_names", lambda apiurl, prefix: set()) + + +def test_missing_mirror_is_hard_error(tmp_path, monkeypatch): + src, rel = _mk_tree(tmp_path, monkeypatch) + _wire_release_update_path(monkeypatch, src, rel) + args = _Args(project="ppg:releases:17", rootprj="home:Admin") + with pytest.raises(SystemExit, match="tarballs"): + cmd_sync.cmd_sync_release(args) def test_freeze_order_and_restore_on_failure(tmp_path, monkeypatch): @@ -191,13 +129,13 @@ def test_freeze_order_and_restore_on_failure(tmp_path, monkeypatch): monkeypatch.setattr( cmd_sync, "restore_builds", lambda *a, **k: calls.append("restore") ) - monkeypatch.setattr(cmd_sync, "verify_release_landed", lambda *a, **k: None) + monkeypatch.setattr(rc, "verify_release_landed", lambda *a, **k: None) def boom(*a, **k): calls.append("release") raise RuntimeError("osc failed") - monkeypatch.setattr(cmd_sync.subprocess, "run", boom) + monkeypatch.setattr(rc.subprocess, "run", boom) args = _Args(project="ppg:releases:17", rootprj="home:Admin") @@ -225,9 +163,9 @@ def test_red_staging_aborts_before_freeze(tmp_path, monkeypatch): cmd_sync, "freeze_builds", lambda *a, **k: freeze_called.append(1) or {} ) monkeypatch.setattr(cmd_sync, "restore_builds", lambda *a, **k: None) - monkeypatch.setattr(cmd_sync, "verify_release_landed", lambda *a, **k: None) + monkeypatch.setattr(rc, "verify_release_landed", lambda *a, **k: None) monkeypatch.setattr( - cmd_sync.subprocess, + rc.subprocess, "run", lambda *a, **k: pytest.fail("osc release must not run"), ) @@ -255,11 +193,11 @@ def fail_if_called(*a, **k): monkeypatch.setattr(cmd_sync, "assert_all_green", fail_if_called) monkeypatch.setattr(cmd_sync, "freeze_builds", fail_if_called) monkeypatch.setattr(cmd_sync, "restore_builds", lambda *a, **k: None) - monkeypatch.setattr(cmd_sync.subprocess, "run", lambda *a, **k: None) + monkeypatch.setattr(rc.subprocess, "run", lambda *a, **k: None) verified = [] monkeypatch.setattr( - cmd_sync, + rc, "verify_release_landed", lambda apiurl, src_prj, rel_prj, timeout_s=600: verified.append( (src_prj, rel_prj) @@ -294,26 +232,23 @@ def _wire_release_first_path(monkeypatch, src, rel): "resolve_project_path", lambda pid: src if pid == "ppg:staging:17" else rel, ) + monkeypatch.setattr( + rc, "resolve_project_path", lambda pid: src if pid == "ppg:staging:17" else rel + ) monkeypatch.setattr(cmd_sync, "_REPO_DIR", rel.parents[2]) monkeypatch.setattr(cmd_sync.osc.conf, "config", {"apiurl": "http://obs"}) monkeypatch.setattr(cmd_sync, "_obs_project_exists", lambda *a, **k: False) monkeypatch.setattr(cmd_sync, "_create_release_project", lambda *a, **k: []) monkeypatch.setattr(cmd_sync, "_copy_project_conf", lambda *a, **k: None) + monkeypatch.setattr(rc, "_apply_project_config", lambda *a, **k: (False, None)) + monkeypatch.setattr(rc, "_disable_project_builds", lambda *a, **k: None) + monkeypatch.setattr(rc, "_read_project_release_source", lambda *a, **k: ([], None)) monkeypatch.setattr( - cmd_sync, "_apply_project_config", lambda *a, **k: (False, None) - ) - monkeypatch.setattr(cmd_sync, "_disable_project_builds", lambda *a, **k: None) - monkeypatch.setattr( - cmd_sync, "_read_project_release_source", lambda *a, **k: ([], None) - ) - monkeypatch.setattr( - cmd_sync, "_filter_release_repo_names", lambda apiurl, names, target: names - ) - monkeypatch.setattr(cmd_sync, "_add_release_targets", lambda *a, **k: None) - monkeypatch.setattr(cmd_sync, "_remove_release_targets", lambda *a, **k: None) - monkeypatch.setattr( - cmd_sync, "_fetch_obs_subproject_names", lambda apiurl, prefix: set() + rc, "filter_release_repo_names", lambda apiurl, names, target: names ) + monkeypatch.setattr(rc, "_add_release_targets", lambda *a, **k: None) + monkeypatch.setattr(rc, "_remove_release_targets", lambda *a, **k: None) + monkeypatch.setattr(rc, "_fetch_obs_subproject_names", lambda apiurl, prefix: set()) def test_first_release_does_not_shell_out_to_dry_run_sync(tmp_path, monkeypatch): @@ -330,7 +265,7 @@ def test_first_release_does_not_shell_out_to_dry_run_sync(tmp_path, monkeypatch) monkeypatch.setattr(cmd_sync, "assert_all_green", lambda *a, **k: []) monkeypatch.setattr(cmd_sync, "freeze_builds", lambda *a, **k: {}) monkeypatch.setattr(cmd_sync, "restore_builds", lambda *a, **k: None) - monkeypatch.setattr(cmd_sync, "verify_release_landed", lambda *a, **k: None) + monkeypatch.setattr(rc, "verify_release_landed", lambda *a, **k: None) run_calls = [] @@ -338,7 +273,7 @@ def fake_run(cmd, *a, **k): run_calls.append(cmd) return SimpleNamespace(returncode=0, stdout="", stderr="") - monkeypatch.setattr(cmd_sync.subprocess, "run", fake_run) + monkeypatch.setattr(rc.subprocess, "run", fake_run) args = _Args(project="ppg:releases:17", rootprj="home:Admin") @@ -416,11 +351,11 @@ def test_dry_run_passes_when_clean(tmp_path, monkeypatch, capsys): def test_filter_release_repo_names_warns_and_drops(monkeypatch, capsys): monkeypatch.setattr( - cmd_sync, + rc, "_fetch_obs_project_repository_names", lambda apiurl, prj: {"RockyLinux_9", "Debian_13"}, ) - kept = cmd_sync._filter_release_repo_names( + kept = rc.filter_release_repo_names( "http://obs", ["RockyLinux_9", "UBI_9", "Debian_13"], "x:rel" ) assert kept == ["RockyLinux_9", "Debian_13"] @@ -432,16 +367,16 @@ def test_collect_release_subprojects_respects_slice(tmp_path, monkeypatch): from percona_obs.project_config import RepositoryFilter src, rel = _mk_tree(tmp_path, monkeypatch) - monkeypatch.setattr(cmd_sync, "resolve_project_path", lambda pid: src) + monkeypatch.setattr(rc, "resolve_project_path", lambda pid: src) try: # boo: containers is out of slice (mirror and source) → not a pair, not missing; # tarballs is in slice on boo and has no mirror → missing common.set_default_repository_filter(RepositoryFilter(exclude_repos=("ubi*",))) - pairs, missing = _collect_release_subprojects("ppg:staging:17", rel) + pairs, missing = collect_release_subprojects("ppg:staging:17", rel) assert pairs == [] and missing == ["tarballs"] # labs: containers released, tarballs is out of slice → not missing common.set_default_repository_filter(RepositoryFilter(include_repos=("ubi*",))) - pairs, missing = _collect_release_subprojects("ppg:staging:17", rel) + pairs, missing = collect_release_subprojects("ppg:staging:17", rel) assert [n for n, _ in pairs] == ["containers"] and missing == [] finally: common.set_default_repository_filter(None) From 6cc277ad517c614fb529df2be2a2da0104d3dbe8 Mon Sep 17 00:00:00 2001 From: Ricardo Dias Date: Tue, 13 Oct 2026 15:56:04 +0100 Subject: [PATCH 06/16] obs_api: publish override and project description markers Signed-off-by: Ricardo Dias --- percona_obs/obs_api.py | 41 ++++++++++++++++++++++++++++ tests/test_obs_api_markers.py | 51 +++++++++++++++++++++++++++++++++++ 2 files changed, 92 insertions(+) create mode 100644 tests/test_obs_api_markers.py diff --git a/percona_obs/obs_api.py b/percona_obs/obs_api.py index 44848b8aa..f721fe90e 100644 --- a/percona_obs/obs_api.py +++ b/percona_obs/obs_api.py @@ -1212,6 +1212,44 @@ def _disable_project_builds(apiurl: str, obs_project_name: str) -> None: _print_update(f"project meta {obs_project_name} (builds disabled)") +def set_project_description_marker( + apiurl: str, obs_project_name: str, key: str, value: str +) -> None: + """Append (or replace) a ``: `` line in the project description. + + Used to pin ``source_commit`` on a pre-release project so the final hop + can check it without git. ``_apply_project_config`` rewrites the + description from the mirror, so callers set the marker last. + """ + raw = _decode_obs_response(osc.core.show_project_meta(apiurl, obs_project_name)) + root = ET.fromstring(raw) + desc = root.find("description") + if desc is None: + desc = ET.SubElement(root, "description") + kept = [ + line + for line in (desc.text or "").splitlines() + if not line.startswith(f"{key}: ") + ] + desc.text = "\n".join([*kept, f"{key}: {value}"]).strip() + ET.indent(root, space=" ") + _edit_project_meta( + apiurl, obs_project_name, ET.tostring(root, encoding="unicode"), force=False + ) + _print_update(f"project meta {obs_project_name} ({key}: {value})") + + +def read_project_description_marker( + apiurl: str, obs_project_name: str, key: str +) -> "str | None": + raw = _decode_obs_response(osc.core.show_project_meta(apiurl, obs_project_name)) + desc = ET.fromstring(raw).findtext("description") or "" + for line in desc.splitlines(): + if line.startswith(f"{key}: "): + return line[len(key) + 2 :].strip() or None + return None + + def _apply_project_config( apiurl: str, obs_project_name: str, @@ -1256,6 +1294,9 @@ def _apply_project_config( branch-source projects that don't exist on OBS are filtered out before upload (avoids ``repository_access_failure`` when a PR adds a brand-new subproject that has no production counterpart yet). + + ``publish_override`` replaces the mirror's ``publish:`` flags (the pre-release + forces publishing on so QA always has repositories). """ project_config = _load_project_config_with_inheritance(project_path, env_vars) # In --branch-from (PR) context, always enable builds regardless of the diff --git a/tests/test_obs_api_markers.py b/tests/test_obs_api_markers.py new file mode 100644 index 000000000..4c2eb0cc9 --- /dev/null +++ b/tests/test_obs_api_markers.py @@ -0,0 +1,51 @@ +"""Unit tests for the project description marker and publish override (percona_obs.obs_api).""" + +from pathlib import Path + +import percona_obs.obs_api as obs_api + +_META = ( + 'tMirror of X\n' + "source_commit: old" +) + + +def test_marker_round_trip(monkeypatch): + pushed = {} + monkeypatch.setattr(obs_api.osc.core, "show_project_meta", lambda a, p: _META) + monkeypatch.setattr( + obs_api, "_edit_project_meta", lambda a, p, meta, force: pushed.setdefault("meta", meta) or False + ) + obs_api.set_project_description_marker("http://obs", "P", "source_commit", "new") + assert pushed["meta"].count("source_commit:") == 1 + assert "source_commit: new" in pushed["meta"] + assert "Mirror of X" in pushed["meta"] + + monkeypatch.setattr(obs_api.osc.core, "show_project_meta", lambda a, p: pushed["meta"]) + assert obs_api.read_project_description_marker("http://obs", "P", "source_commit") == "new" + assert obs_api.read_project_description_marker("http://obs", "P", "other") is None + + +def test_publish_override_forces_enable(monkeypatch, tmp_path): + (tmp_path / "project.yaml").write_text( + "publish: {r: false}\nrepositories:\n - name: r\n archs: [x86_64]\n paths: []\n" + ) + seen = {} + monkeypatch.setattr( + obs_api, + "build_project_meta", + lambda *a, **k: seen.setdefault("publish", k.get("publish")) or "", + ) + import urllib.error + + def not_found(*a, **k): + raise urllib.error.HTTPError("u", 404, "nf", {}, None) + + monkeypatch.setattr(obs_api.osc.core, "show_project_meta", not_found) + monkeypatch.setattr(obs_api.osc.core, "show_project_conf", not_found) + monkeypatch.setattr(obs_api, "_fetch_root_project_managed_elements", lambda a, r: []) + monkeypatch.setattr(obs_api, "_inject_obs_managed_elements", lambda m, e: m) + obs_api._apply_project_config( + "http://obs", "root:P", tmp_path, "root", dry_run=True, publish_override=True + ) + assert seen["publish"] is True From 7806ab7fba844057bd9dcc5f614c508108a6abf2 Mon Sep 17 00:00:00 2001 From: Ricardo Dias Date: Tue, 13 Oct 2026 15:58:16 +0100 Subject: [PATCH 07/16] obs_api: refuse marker writes that strip repo paths; keep description order Signed-off-by: Ricardo Dias --- percona_obs/obs_api.py | 12 +++++++-- tests/test_obs_api_markers.py | 48 +++++++++++++++++++++++++++++++---- 2 files changed, 53 insertions(+), 7 deletions(-) diff --git a/percona_obs/obs_api.py b/percona_obs/obs_api.py index f721fe90e..4b15cd978 100644 --- a/percona_obs/obs_api.py +++ b/percona_obs/obs_api.py @@ -1225,7 +1225,10 @@ def set_project_description_marker( root = ET.fromstring(raw) desc = root.find("description") if desc is None: - desc = ET.SubElement(root, "description") + desc = ET.Element("description") + title = root.find("title") + insert_at = 0 if title is None else list(root).index(title) + 1 + root.insert(insert_at, desc) kept = [ line for line in (desc.text or "").splitlines() @@ -1233,9 +1236,14 @@ def set_project_description_marker( ] desc.text = "\n".join([*kept, f"{key}: {value}"]).strip() ET.indent(root, space=" ") - _edit_project_meta( + stripped = _edit_project_meta( apiurl, obs_project_name, ET.tostring(root, encoding="unicode"), force=False ) + if stripped: + raise SystemExit( + f"error: writing {key} to {obs_project_name} stripped its repository paths " + "(repository_access_failure); the project meta must be re-applied before retrying" + ) _print_update(f"project meta {obs_project_name} ({key}: {value})") diff --git a/tests/test_obs_api_markers.py b/tests/test_obs_api_markers.py index 4c2eb0cc9..76cd7d1c6 100644 --- a/tests/test_obs_api_markers.py +++ b/tests/test_obs_api_markers.py @@ -6,7 +6,7 @@ _META = ( 'tMirror of X\n' - "source_commit: old" + 'source_commit: old' ) @@ -14,15 +14,22 @@ def test_marker_round_trip(monkeypatch): pushed = {} monkeypatch.setattr(obs_api.osc.core, "show_project_meta", lambda a, p: _META) monkeypatch.setattr( - obs_api, "_edit_project_meta", lambda a, p, meta, force: pushed.setdefault("meta", meta) or False + obs_api, + "_edit_project_meta", + lambda a, p, meta, force: pushed.setdefault("meta", meta) and False, ) obs_api.set_project_description_marker("http://obs", "P", "source_commit", "new") assert pushed["meta"].count("source_commit:") == 1 assert "source_commit: new" in pushed["meta"] assert "Mirror of X" in pushed["meta"] - monkeypatch.setattr(obs_api.osc.core, "show_project_meta", lambda a, p: pushed["meta"]) - assert obs_api.read_project_description_marker("http://obs", "P", "source_commit") == "new" + monkeypatch.setattr( + obs_api.osc.core, "show_project_meta", lambda a, p: pushed["meta"] + ) + assert ( + obs_api.read_project_description_marker("http://obs", "P", "source_commit") + == "new" + ) assert obs_api.read_project_description_marker("http://obs", "P", "other") is None @@ -43,9 +50,40 @@ def not_found(*a, **k): monkeypatch.setattr(obs_api.osc.core, "show_project_meta", not_found) monkeypatch.setattr(obs_api.osc.core, "show_project_conf", not_found) - monkeypatch.setattr(obs_api, "_fetch_root_project_managed_elements", lambda a, r: []) + monkeypatch.setattr( + obs_api, "_fetch_root_project_managed_elements", lambda a, r: [] + ) monkeypatch.setattr(obs_api, "_inject_obs_managed_elements", lambda m, e: m) obs_api._apply_project_config( "http://obs", "root:P", tmp_path, "root", dry_run=True, publish_override=True ) assert seen["publish"] is True + + +def test_marker_refuses_when_repository_paths_stripped(monkeypatch): + monkeypatch.setattr(obs_api.osc.core, "show_project_meta", lambda a, p: _META) + monkeypatch.setattr(obs_api, "_edit_project_meta", lambda a, p, meta, force: True) + import pytest + + with pytest.raises(SystemExit, match="stripped its repository paths"): + obs_api.set_project_description_marker( + "http://obs", "P", "source_commit", "new" + ) + + +def test_marker_inserts_missing_description_after_title(monkeypatch): + no_desc = 't' + pushed = {} + monkeypatch.setattr(obs_api.osc.core, "show_project_meta", lambda a, p: no_desc) + monkeypatch.setattr( + obs_api, + "_edit_project_meta", + lambda a, p, meta, force: pushed.setdefault("meta", meta) and False, + ) + obs_api.set_project_description_marker("http://obs", "P", "source_commit", "abc") + import xml.etree.ElementTree as ET + + root = ET.fromstring(pushed["meta"]) + tags = [child.tag for child in root] + assert tags == ["title", "description", "repository"] + assert root.findtext("description") == "source_commit: abc" From d87e11468d94ce6abb6044cb8daa4e855929cc5b Mon Sep 17 00:00:00 2001 From: Ricardo Dias Date: Tue, 13 Oct 2026 16:00:15 +0100 Subject: [PATCH 08/16] sync prerelease: hop 1, staging to the PR pre-release project Signed-off-by: Ricardo Dias --- percona_obs/cli.py | 64 ++++++++ percona_obs/cmd_sync.py | 296 +++++++++++++++++++++++++++++----- tests/test_sync_prerelease.py | 184 +++++++++++++++++++++ 3 files changed, 507 insertions(+), 37 deletions(-) create mode 100644 tests/test_sync_prerelease.py diff --git a/percona_obs/cli.py b/percona_obs/cli.py index 8a876daf1..15915aa3b 100644 --- a/percona_obs/cli.py +++ b/percona_obs/cli.py @@ -33,6 +33,7 @@ cmd_sync_delete, cmd_sync_promote, cmd_sync_release, + cmd_sync_prerelease, ) from .common import ( _DIM, @@ -354,6 +355,69 @@ def build_parser() -> argparse.ArgumentParser: ) sync_release_parser.set_defaults(func=cmd_sync_release) + sync_prerelease_parser = sync_subparsers.add_parser( + "prerelease", + help="Hop 1 of a release: copy a staging tree into this PR's pre-release project.", + ) + sync_prerelease_parser.add_argument( + "project", help="Release project identifier (e.g. ppg:releases:18)." + ) + sync_prerelease_parser.add_argument( + "--branch-from", + required=True, + metavar="PROFILE", + dest="branch_from", + help="Profile of the production instance holding the staging source.", + ) + sync_prerelease_parser.add_argument( + "--synced-commit", + metavar="SHA", + default=None, + dest="synced_commit", + help="Head SHA of the last successful sync-main run (default: looked up with gh).", + ) + sync_prerelease_parser.add_argument( + "--source-commit", + metavar="SHA", + default=None, + dest="source_commit", + help="Override the source_commit recorded in release.yaml.", + ) + sync_prerelease_parser.add_argument( + "--force", + action="store_true", + default=False, + help="Skip the source_commit record check.", + ) + sync_prerelease_parser.add_argument( + "--no-freeze", + action="store_true", + default=False, + help="Skip the drain/green-check/build-freeze sequence around osc release.", + ) + sync_prerelease_parser.add_argument( + "--freeze-timeout", + type=int, + default=3600, + dest="freeze_timeout", + help="Seconds to wait for the source scheduler to drain (default: 3600).", + ) + sync_prerelease_parser.add_argument( + "--verify-timeout", + type=int, + default=600, + dest="verify_timeout", + help="Seconds to wait for copied packages to land (default: 600; 0 disables).", + ) + sync_prerelease_parser.add_argument( + "--dry-run", + action="store_true", + default=False, + dest="dry_run", + help="Run the record check, mirror/changelog/tag checks and the green check only.", + ) + sync_prerelease_parser.set_defaults(func=cmd_sync_prerelease) + build_parser_ = subparsers.add_parser( "build", help="Trigger OBS builds or check their status.", diff --git a/percona_obs/cmd_sync.py b/percona_obs/cmd_sync.py index 07e925ea7..11ed6f069 100644 --- a/percona_obs/cmd_sync.py +++ b/percona_obs/cmd_sync.py @@ -30,6 +30,7 @@ REPO_ROOT, _REPO_DIR, _load_project_config_with_inheritance, + _macros_chain_files, _print_action, _print_aggregate, _print_ok, @@ -59,7 +60,8 @@ collect_release_subprojects, release_tree, ) -from .release_record import release_tags +from .github_api import last_synced_head_sha, repo_slug_from_url +from .release_record import current_release, release_tags from .project_config import ( RepositoryFilter, package_in_slice, @@ -105,6 +107,7 @@ _fetch_obs_subproject_names, _obs_project_exists, _upload_obs_files, + set_project_description_marker, ) from .release_scope import collect_release_scope from .release_freeze import ( @@ -2492,6 +2495,42 @@ def _check_source_unchanged_since_tag(source_project_id: str, tag: str, args) -> ) +def _run_frozen( + apiurl, quiesce_scope, freeze_scope, freeze_packages_map, args, action +) -> None: + """Drain, green-check and freeze the source scope, run ``action``, restore.""" + snapshots: dict[str, str] = {} + pkg_snapshots: dict[tuple[str, str], str] = {} + if not args.no_freeze: + _print_pending(f"draining scheduler for {len(quiesce_scope)} project(s)") + wait_for_quiesce( + apiurl, + quiesce_scope, + timeout_s=args.freeze_timeout, + packages=freeze_packages_map, + ) + problems = assert_all_green(apiurl, quiesce_scope, packages=freeze_packages_map) + if problems: + listing = "\n".join(f" {p}" for p in problems[:50]) + raise SystemExit( + f"error: source projects are not fully green:\n{listing}\n" + "Fix the failures (or use --no-freeze to bypass at your own risk)." + ) + snapshots = freeze_builds(apiurl, freeze_scope) + try: + pkg_snapshots = freeze_packages(apiurl, freeze_packages_map) + except Exception: + restore_builds(apiurl, snapshots) + raise + try: + action() + finally: + if pkg_snapshots: + restore_packages(apiurl, pkg_snapshots) + if snapshots: + restore_builds(apiurl, snapshots) + + def cmd_sync_release(args) -> None: """Release packages from an OBS source project to a release target. @@ -2599,40 +2638,6 @@ def cmd_sync_release(args) -> None: "Re-run 'percona-obs project release' to regenerate the release tree." ) - def _freeze_and_run(release_all) -> None: - snapshots: dict[str, str] = {} - pkg_snapshots: dict[tuple[str, str], str] = {} - if not args.no_freeze: - _print_pending(f"draining scheduler for {len(quiesce_scope)} project(s)") - wait_for_quiesce( - apiurl, - quiesce_scope, - timeout_s=args.freeze_timeout, - packages=freeze_packages_map, - ) - problems = assert_all_green( - apiurl, quiesce_scope, packages=freeze_packages_map - ) - if problems: - listing = "\n".join(f" {p}" for p in problems[:50]) - raise SystemExit( - f"error: source projects are not fully green:\n{listing}\n" - "Fix the failures (or use --no-freeze to bypass at your own risk)." - ) - snapshots = freeze_builds(apiurl, freeze_scope) - try: - pkg_snapshots = freeze_packages(apiurl, freeze_packages_map) - except Exception: - restore_builds(apiurl, snapshots) - raise - try: - release_all() - finally: - if pkg_snapshots: - restore_packages(apiurl, pkg_snapshots) - if snapshots: - restore_builds(apiurl, snapshots) - _print_pending(f"checking release project {release_obs_project}") project_exists = _obs_project_exists(apiurl, release_obs_project) @@ -2650,7 +2655,12 @@ def _freeze_and_run(release_all) -> None: _create_release_project(apiurl, source_obs_project, release_obs_project) _copy_project_conf(apiurl, source_obs_project, release_obs_project) - _freeze_and_run( + _run_frozen( + apiurl, + quiesce_scope, + freeze_scope, + freeze_packages_map, + args, lambda: release_tree( apiurl, source_obs_project, @@ -2659,8 +2669,220 @@ def _freeze_and_run(release_all) -> None: args.rootprj, shared_env_vars, verify_timeout=args.verify_timeout, - ) + ), ) suffix = f" (update: {tag})" if project_exists and tag else "" _print_ok(f"release {source_obs_project} → {release_obs_project}{suffix}") + + +def _scope_local_paths( + source_path: Path, + freeze_scope: "list[str]", + freeze_packages_map: "dict[str, set[str]]", + rootprj: str, +) -> "list[Path]": + """Local git paths whose content decides the released binaries: the + source tree, its macros chain, every whole-frozen project and every + aggregate-frozen package.""" + paths: list[Path] = [source_path, *_macros_chain_files(source_path)] + for prj in freeze_scope: + p = resolve_project_path(prj[len(rootprj) + 1 :]) + if p.is_dir() and p not in paths: + paths.append(p) + for prj, pkgs in freeze_packages_map.items(): + base = resolve_project_path(prj[len(rootprj) + 1 :]) + paths.extend(base / pkg for pkg in sorted(pkgs)) + return paths + + +def _changed_files_between(a: str, b: str, paths: "list[Path]") -> "list[str]": + """``git diff --name-only a b -- paths`` (both commits must exist locally).""" + for sha in (a, b): + r = subprocess.run( + ["git", "cat-file", "-e", f"{sha}^{{commit}}"], + cwd=_REPO_DIR, + capture_output=True, + ) + if r.returncode != 0: + raise SystemExit( + f"error: commit {sha} is not available locally; " + "fetch the repository (full history) and retry" + ) + rel = [str(p.relative_to(_REPO_DIR)) if p.is_absolute() else str(p) for p in paths] + r = subprocess.run( + ["git", "diff", "--name-only", a, b, "--", *rel], + cwd=_REPO_DIR, + capture_output=True, + text=True, + ) + if r.returncode != 0: + raise SystemExit(f"error: git diff {a} {b} failed: {r.stderr.strip()}") + return [l.strip() for l in r.stdout.splitlines() if l.strip()] + + +def cmd_sync_prerelease(args) -> None: + """Hop 1: copy a staging project tree into this PR's pre-release project. + + Runs with the PR profile (``-P pr-N``); ``--branch-from`` names the + profile of the production instance that holds staging. Steps (spec §2): + record check, pre-release meta from the mirrors with publish forced on, + drain/green/freeze staging, osc release, verify, restore, pin + ``source_commit`` in the pre-release description. + """ + release_path = resolve_project_path(args.project) + release_file = release_path / "release.yaml" + if not release_file.is_file(): + raise SystemExit( + f"error: release.yaml not found at {release_path.relative_to(_REPO_DIR)}" + ) + env_vars = parse_env_overrides(args.env_overrides) if args.env_overrides else {} + release_data = load_yaml_with_env(release_file, env_vars) + source_project_id = release_data.get("project") + if not source_project_id: + raise SystemExit(f"error: release.yaml missing 'project' field: {release_file}") + entry = current_release(release_data) + if entry is None: + raise SystemExit("error: release.yaml has no releases: entries") + source_commit = args.source_commit or entry.source_commit + if not source_commit: + raise SystemExit( + f"error: release {entry.tag} has no source_commit in release.yaml; " + "re-run 'percona-obs project release' (or pass --source-commit)" + ) + + branch_profile = _load_profile(args.branch_from) + source_rootprj = branch_profile.get("rootprj", "") + if not source_rootprj: + raise SystemExit(f"error: profile {args.branch_from!r} has no rootprj") + apiurl: str = osc.conf.config["apiurl"] + source_obs_project = f"{source_rootprj}:{source_project_id}" + prerelease_obs_project = f"{args.rootprj}:{args.project}" + source_path = resolve_project_path(source_project_id) + # Mirror paths are rewritten to the PR root through ${OBS_ROOTPRJ}. + shared_env_vars = {**env_vars, **auto_rootprj_env(args.rootprj)} + # The freeze scope lives on the production root. + source_env_vars = {**env_vars, **auto_rootprj_env(source_rootprj)} + + subproject_pairs, missing_mirrors = collect_release_subprojects( + source_project_id, release_path + ) + source_sub_obs_projects = [ + f"{source_obs_project}:{name}" for name, _ in subproject_pairs + ] + freeze_scope, freeze_packages_map = _build_freeze_scope( + source_obs_project, + source_sub_obs_projects, + source_path, + source_project_id, + source_rootprj, + source_env_vars, + ) + quiesce_scope = freeze_scope + sorted(freeze_packages_map) + + # --- record check (spec §2, step 2) --- + if not args.force: + synced = args.synced_commit or last_synced_head_sha( + repo_slug_from_url( + shared_env_vars.get("PERCONA_OBS_PACKAGING_REPO") + or _load_profile_env(args.branch_from).get( + "PERCONA_OBS_PACKAGING_REPO", "" + ) + ) + ) + changed = _changed_files_between( + source_commit, + synced, + _scope_local_paths( + source_path, freeze_scope, freeze_packages_map, source_rootprj + ), + ) + if changed: + listing = "\n".join(f" {f}" for f in changed[:50]) + raise SystemExit( + f"error: staging changed since the cut ({source_commit[:12]} → " + f"{synced[:12]}):\n{listing}\n" + "Re-run 'percona-obs project release' on a rebased branch." + ) + _print_same( + f"source record {source_commit[:12]} matches synced main {synced[:12]}" + ) + + if missing_mirrors: + listing = "\n".join( + f" {source_project_id}:{n}" for n in sorted(missing_mirrors) + ) + raise SystemExit( + f"error: staging subprojects have no release mirror:\n{listing}\n" + "Re-run 'percona-obs project release' to regenerate the release tree." + ) + + if args.dry_run: + failures: list[str] = [] + release_id = entry.tag.split("/")[-1] + changelog_file = release_path / "CHANGELOG.md" + if not changelog_file.is_file(): + failures.append("CHANGELOG.md is missing") + elif f"## [{release_id}]" not in changelog_file.read_text("utf-8"): + failures.append(f"CHANGELOG.md has no '## [{release_id}]' section") + r = subprocess.run( + ["git", "rev-parse", "--verify", "--quiet", f"refs/tags/{entry.tag}"], + capture_output=True, + cwd=_REPO_DIR, + ) + if r.returncode == 0: + failures.append(f"tag {entry.tag} already exists") + if not _obs_project_exists(apiurl, source_obs_project): + failures.append(f"source project {source_obs_project} not found on OBS") + else: + failures.extend( + f"not green: {p}" + for p in assert_all_green( + apiurl, quiesce_scope, packages=freeze_packages_map + ) + ) + if failures: + for f in failures: + print(f" ✗ {f}", file=sys.stderr) + raise SystemExit("error: prerelease dry-run failed") + _print_ok(f"prerelease dry-run passed for {entry.tag} (dry run)") + return + + # Top-level pre-release meta from the mirror, publish forced on (spec §1). + apply_mirror_meta( + apiurl, + prerelease_obs_project, + release_path, + args.rootprj, + shared_env_vars, + publish_override=True, + ) + + _run_frozen( + apiurl, + quiesce_scope, + freeze_scope, + freeze_packages_map, + args, + lambda: release_tree( + apiurl, + source_obs_project, + prerelease_obs_project, + subproject_pairs, + args.rootprj, + shared_env_vars, + publish_override=True, + verify_timeout=args.verify_timeout, + ), + ) + + set_project_description_marker( + apiurl, prerelease_obs_project, "source_commit", source_commit + ) + for name, _ in subproject_pairs: + set_project_description_marker( + apiurl, f"{prerelease_obs_project}:{name}", "source_commit", source_commit + ) + _print_ok( + f"prerelease {source_obs_project} → {prerelease_obs_project} ({entry.tag})" + ) diff --git a/tests/test_sync_prerelease.py b/tests/test_sync_prerelease.py new file mode 100644 index 000000000..2ccdd98e2 --- /dev/null +++ b/tests/test_sync_prerelease.py @@ -0,0 +1,184 @@ +"""Unit tests for `sync prerelease` (percona_obs.cmd_sync.cmd_sync_prerelease). + +OBS, osc, git and gh are all monkeypatched; the staging/release trees come +from tests.test_sync_release._mk_tree. +""" + +import pytest + +import percona_obs.cmd_sync as cmd_sync +import percona_obs.release_copy as rc +from tests.test_sync_release import _REPO_YAML, _mk_tree + +SHA = "a" * 40 + + +class _Args: + def __init__(self, **kw): + self.project = "ppg:releases:17" + self.rootprj = "home:Admin:PR:pr-5" + self.branch_from = "main" + self.synced_commit = SHA + self.source_commit = None + self.force = False + self.dry_run = False + self.no_freeze = False + self.freeze_timeout = 1 + self.verify_timeout = 0 + self.env_overrides = None + self.profile = None + self.__dict__.update(kw) + + +def _tree_with_record(tmp_path, monkeypatch, source_commit=SHA): + src, rel = _mk_tree(tmp_path, monkeypatch) + (rel / "tarballs").mkdir() + (rel / "tarballs" / "project.yaml").write_text( + "build: false\n" + _REPO_YAML.format(name="ssl3") + ) + entry = ( + f" - tag: ppg/17.11-1\n source_commit: {source_commit}\n" + if source_commit + else " - ppg/17.11-1\n" + ) + (rel / "release.yaml").write_text(f"project: ppg:staging:17\nreleases:\n{entry}") + return src, rel + + +def _wire(monkeypatch, src, rel, calls): + monkeypatch.setattr( + cmd_sync, + "resolve_project_path", + lambda pid: src if pid == "ppg:staging:17" else rel, + ) + monkeypatch.setattr( + rc, "resolve_project_path", lambda pid: src if pid == "ppg:staging:17" else rel + ) + monkeypatch.setattr(cmd_sync, "_REPO_DIR", rel.parents[2]) + monkeypatch.setattr(cmd_sync.osc.conf, "config", {"apiurl": "http://obs"}) + monkeypatch.setattr( + cmd_sync, + "_load_profile", + lambda name: {"rootprj": "home:Admin", "apiurl": "http://obs"}, + ) + monkeypatch.setattr(cmd_sync, "_obs_project_exists", lambda *a, **k: True) + monkeypatch.setattr(cmd_sync, "_changed_files_between", lambda *a, **k: []) + monkeypatch.setattr( + cmd_sync, + "apply_mirror_meta", + lambda apiurl, prj, path, rootprj, env, publish_override=None: calls.append( + ("meta", prj, publish_override) + ), + ) + monkeypatch.setattr( + cmd_sync, "wait_for_quiesce", lambda *a, **k: calls.append("drain") + ) + monkeypatch.setattr( + cmd_sync, "assert_all_green", lambda *a, **k: calls.append("green") or [] + ) + monkeypatch.setattr( + cmd_sync, + "freeze_builds", + lambda *a, **k: calls.append("freeze") or {"p": ""}, + ) + monkeypatch.setattr(cmd_sync, "freeze_packages", lambda *a, **k: {}) + monkeypatch.setattr(cmd_sync, "restore_packages", lambda *a, **k: None) + monkeypatch.setattr( + cmd_sync, "restore_builds", lambda *a, **k: calls.append("restore") + ) + monkeypatch.setattr( + cmd_sync, + "release_tree", + lambda *a, **k: calls.append(("copy", a[1], a[2], k.get("publish_override"))), + ) + monkeypatch.setattr( + cmd_sync, + "set_project_description_marker", + lambda apiurl, prj, key, value: calls.append(("marker", prj, value)), + ) + + +def test_requires_source_commit(tmp_path, monkeypatch): + src, rel = _tree_with_record(tmp_path, monkeypatch, source_commit=None) + _wire(monkeypatch, src, rel, []) + with pytest.raises(SystemExit, match="has no source_commit"): + cmd_sync.cmd_sync_prerelease(_Args()) + + +def test_record_mismatch_aborts_before_drain(tmp_path, monkeypatch): + src, rel = _tree_with_record(tmp_path, monkeypatch) + calls = [] + _wire(monkeypatch, src, rel, calls) + monkeypatch.setattr( + cmd_sync, + "_changed_files_between", + lambda *a, **k: ["root/ppg/staging/17/x/rpm/x.spec"], + ) + with pytest.raises(SystemExit, match="staging changed since the cut"): + cmd_sync.cmd_sync_prerelease(_Args()) + assert "drain" not in calls + + +def test_happy_path_order(tmp_path, monkeypatch): + src, rel = _tree_with_record(tmp_path, monkeypatch) + calls = [] + _wire(monkeypatch, src, rel, calls) + cmd_sync.cmd_sync_prerelease(_Args()) + assert calls[0] == ("meta", "home:Admin:PR:pr-5:ppg:releases:17", True) + assert calls[1:4] == ["drain", "green", "freeze"] + assert calls[4] == ( + "copy", + "home:Admin:ppg:staging:17", + "home:Admin:PR:pr-5:ppg:releases:17", + True, + ) + assert calls[5] == "restore" + assert calls[6] == ("marker", "home:Admin:PR:pr-5:ppg:releases:17", SHA) + + +def test_marker_not_written_when_copy_fails(tmp_path, monkeypatch): + src, rel = _tree_with_record(tmp_path, monkeypatch) + calls = [] + _wire(monkeypatch, src, rel, calls) + + def boom(*a, **k): + raise RuntimeError("osc") + + monkeypatch.setattr(cmd_sync, "release_tree", boom) + with pytest.raises(RuntimeError): + cmd_sync.cmd_sync_prerelease(_Args()) + assert "restore" in calls and not any( + c[0] == "marker" for c in calls if isinstance(c, tuple) + ) + + +def test_dry_run_checks_only(tmp_path, monkeypatch): + src, rel = _tree_with_record(tmp_path, monkeypatch) + (rel / "CHANGELOG.md").write_text("# Changelog\n\n## [17.11-1] - 2026-01-01\n") + calls = [] + _wire(monkeypatch, src, rel, calls) + monkeypatch.setattr( + cmd_sync.subprocess, + "run", + lambda *a, **k: type("R", (), {"returncode": 1, "stdout": "", "stderr": ""})(), + ) + cmd_sync.cmd_sync_prerelease(_Args(dry_run=True)) + assert calls == ["green"] + + +def test_scope_local_paths_maps_obs_names_back(tmp_path, monkeypatch): + src, rel = _tree_with_record(tmp_path, monkeypatch) + monkeypatch.setattr( + cmd_sync, + "resolve_project_path", + lambda pid: src.parents[2] / pid.replace(":", "/"), + ) + paths = cmd_sync._scope_local_paths( + src, + ["home:Admin:ppg:staging:17", "home:Admin:ppg:staging:17:containers"], + {"home:Admin:ppg:common:deps": {"geos"}}, + "home:Admin", + ) + assert src in paths + assert src.parents[2] / "ppg/common/deps/geos" in paths + assert any(p.name == "macros.yaml" for p in paths) From a44bbd750806988ac69aa51c13fb632c837f7f9e Mon Sep 17 00:00:00 2001 From: Ricardo Dias Date: Tue, 13 Oct 2026 16:03:16 +0100 Subject: [PATCH 09/16] sync prerelease: record check follows _shared symlink targets Signed-off-by: Ricardo Dias --- percona_obs/cmd_sync.py | 23 +++++++++++++++--- tests/test_sync_prerelease.py | 46 +++++++++++++++++++++++++++++++++++ 2 files changed, 66 insertions(+), 3 deletions(-) diff --git a/percona_obs/cmd_sync.py b/percona_obs/cmd_sync.py index 11ed6f069..249cbcd79 100644 --- a/percona_obs/cmd_sync.py +++ b/percona_obs/cmd_sync.py @@ -69,6 +69,7 @@ prune_flag_map, ) from .git_utils import ( + _package_pathspecs, _generate_sync_message, _has_package_changes_since, _head_full_sha, @@ -2686,13 +2687,29 @@ def _scope_local_paths( source tree, its macros chain, every whole-frozen project and every aggregate-frozen package.""" paths: list[Path] = [source_path, *_macros_chain_files(source_path)] + + def _add(path: Path) -> None: + if path not in paths: + paths.append(path) + + project_dirs = [source_path] for prj in freeze_scope: p = resolve_project_path(prj[len(rootprj) + 1 :]) - if p.is_dir() and p not in paths: - paths.append(p) + if p.is_dir(): + _add(p) + project_dirs.append(p) + # Package dirs that are symlinks into _shared/: git sees only the link + # blob, so the link target must be a pathspec of its own. + for d in project_dirs: + for child in sorted(d.iterdir()): + if child.is_symlink(): + for target in _package_pathspecs(child)[1:]: + _add(target) for prj, pkgs in freeze_packages_map.items(): base = resolve_project_path(prj[len(rootprj) + 1 :]) - paths.extend(base / pkg for pkg in sorted(pkgs)) + for pkg in sorted(pkgs): + for path in _package_pathspecs(base / pkg): + _add(path) return paths diff --git a/tests/test_sync_prerelease.py b/tests/test_sync_prerelease.py index 2ccdd98e2..3fe3a6143 100644 --- a/tests/test_sync_prerelease.py +++ b/tests/test_sync_prerelease.py @@ -182,3 +182,49 @@ def test_scope_local_paths_maps_obs_names_back(tmp_path, monkeypatch): assert src in paths assert src.parents[2] / "ppg/common/deps/geos" in paths assert any(p.name == "macros.yaml" for p in paths) + + +def test_shared_symlink_target_edit_is_detected(tmp_path, monkeypatch): + import subprocess + + import percona_obs.common as common + + def git(*a): + return subprocess.run( + ["git", "-c", "user.name=t", "-c", "user.email=t@x", *a], + cwd=tmp_path, + check=True, + capture_output=True, + text=True, + ).stdout.strip() + + git("init", "-q") + shared = tmp_path / "root/ppg/staging/_shared/pkgx/rpm" + shared.mkdir(parents=True) + spec = shared / "x.spec" + spec.write_text("Version: 1\n") + src = tmp_path / "root/ppg/staging/17" + src.mkdir(parents=True) + (src / "project.yaml").write_text("title: S\n") + (src / "pkgx").symlink_to("../_shared/pkgx") + (tmp_path / "root/macros.yaml").write_text("- M: 1\n") + git("add", "-A") + git("commit", "-q", "-m", "a") + a = git("rev-parse", "HEAD") + spec.write_text("Version: 2\n") + git("commit", "-q", "-am", "b") + b = git("rev-parse", "HEAD") + + monkeypatch.setattr(cmd_sync, "_REPO_DIR", tmp_path) + monkeypatch.setattr(common, "REPO_ROOT", tmp_path / "root") + monkeypatch.setattr( + cmd_sync, + "resolve_project_path", + lambda pid: tmp_path / "root" / pid.replace(":", "/"), + ) + paths = cmd_sync._scope_local_paths( + src, ["home:Admin:ppg:staging:17"], {}, "home:Admin" + ) + assert cmd_sync._changed_files_between(a, b, paths) == [ + "root/ppg/staging/_shared/pkgx/rpm/x.spec" + ] From b2843443e0d8499c08399af18170b52b4ca70b92 Mon Sep 17 00:00:00 2001 From: Ricardo Dias Date: Tue, 13 Oct 2026 16:05:08 +0100 Subject: [PATCH 10/16] sync release --from: hop 2, pre-release to the release project Signed-off-by: Ricardo Dias --- percona_obs/cli.py | 9 ++++++ percona_obs/cmd_sync.py | 33 ++++++++++++++++++++ tests/test_sync_release.py | 64 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 106 insertions(+) diff --git a/percona_obs/cli.py b/percona_obs/cli.py index 15915aa3b..23d874396 100644 --- a/percona_obs/cli.py +++ b/percona_obs/cli.py @@ -353,6 +353,15 @@ def build_parser() -> argparse.ArgumentParser: dest="dry_run", help="Validate the release (mirrors, changelog, tag, green staging) without writing to OBS.", ) + sync_release_parser.add_argument( + "--from", + metavar="OBS_PROJECT", + default=None, + dest="from_project", + help="Hop 2: copy from this pre-release OBS project (full name, e.g. " + "isv:percona:PR:pr-12:ppg:releases:18) instead of from staging; " + "no freeze, no git divergence check.", + ) sync_release_parser.set_defaults(func=cmd_sync_release) sync_prerelease_parser = sync_subparsers.add_parser( diff --git a/percona_obs/cmd_sync.py b/percona_obs/cmd_sync.py index 249cbcd79..ff5349104 100644 --- a/percona_obs/cmd_sync.py +++ b/percona_obs/cmd_sync.py @@ -108,6 +108,7 @@ _fetch_obs_subproject_names, _obs_project_exists, _upload_obs_files, + read_project_description_marker, set_project_description_marker, ) from .release_scope import collect_release_scope @@ -2642,6 +2643,38 @@ def cmd_sync_release(args) -> None: _print_pending(f"checking release project {release_obs_project}") project_exists = _obs_project_exists(apiurl, release_obs_project) + from_project: "str | None" = getattr(args, "from_project", None) + if from_project: + # Hop 2 (spec §2): no staging involvement, no freeze, no git check. + entry = current_release(release_data) + if entry is None: + raise SystemExit("error: release.yaml has no releases: entries") + if not args.force: + marker = read_project_description_marker( + apiurl, from_project, "source_commit" + ) + if marker != entry.source_commit: + raise SystemExit( + f"error: source_commit mismatch: {from_project} was populated from " + f"{marker or ''} but release.yaml {entry.tag} records " + f"{entry.source_commit or ''}; repopulate the pre-release " + "or cut a new release" + ) + apply_mirror_meta( + apiurl, release_obs_project, release_path, args.rootprj, shared_env_vars + ) + release_tree( + apiurl, + from_project, + release_obs_project, + subproject_pairs, + args.rootprj, + shared_env_vars, + verify_timeout=args.verify_timeout, + ) + _print_ok(f"release {from_project} → {release_obs_project} ({entry.tag})") + return + if not args.force: _check_source_unchanged_since_tag(source_project_id, tag, args) diff --git a/tests/test_sync_release.py b/tests/test_sync_release.py index f4d222402..d4320c301 100644 --- a/tests/test_sync_release.py +++ b/tests/test_sync_release.py @@ -71,6 +71,7 @@ def __init__(self, project, rootprj, no_freeze=False, verify_timeout=600): self.rootprj = rootprj self.env_overrides = None self.message = None + self.from_project = None def _wire_release_update_path(monkeypatch, src, rel): @@ -411,3 +412,66 @@ def test_build_freeze_scope_merges_extra_sources(tmp_path, monkeypatch): "home:Admin:ppg:staging:18", ] assert packages == {"home:Admin:ppg:staging:common:tools": {"percona-pgbouncer"}} + + +def _args_from(rel, **kw): + a = _Args(project="ppg:releases:17", rootprj="home:Admin") + a.from_project = "home:Admin:PR:pr-5:ppg:releases:17" + a.__dict__.update(kw) + return a + + +def _mapping_release_yaml(rel, sha="b" * 40): + (rel / "release.yaml").write_text( + f"project: ppg:staging:17\nreleases:\n - tag: ppg/17.11-1\n source_commit: {sha}\n" + ) + + +def test_from_prerelease_skips_freeze_and_copies_from_it(tmp_path, monkeypatch): + src, rel = _mk_tree(tmp_path, monkeypatch) + (rel / "tarballs").mkdir() + (rel / "tarballs" / "project.yaml").write_text( + "build: false\n" + _REPO_YAML.format(name="ssl3") + ) + _mapping_release_yaml(rel) + _wire_release_update_path(monkeypatch, src, rel) + monkeypatch.setattr(cmd_sync, "_obs_project_exists", lambda *a, **k: False) + monkeypatch.setattr( + cmd_sync, "read_project_description_marker", lambda a, p, k: "b" * 40 + ) + fail = lambda *a, **k: pytest.fail("freeze must be skipped with --from") + monkeypatch.setattr(cmd_sync, "wait_for_quiesce", fail) + monkeypatch.setattr(cmd_sync, "freeze_builds", fail) + monkeypatch.setattr(cmd_sync, "_create_release_project", fail) + metas, copies = [], [] + monkeypatch.setattr( + cmd_sync, "apply_mirror_meta", lambda a, prj, *r, **k: metas.append(prj) + ) + monkeypatch.setattr( + cmd_sync, "release_tree", lambda a, s, t, *r, **k: copies.append((s, t)) + ) + cmd_sync.cmd_sync_release(_args_from(rel)) + assert metas == ["home:Admin:ppg:releases:17"] + assert copies == [ + ("home:Admin:PR:pr-5:ppg:releases:17", "home:Admin:ppg:releases:17") + ] + + +def test_from_prerelease_marker_mismatch(tmp_path, monkeypatch): + src, rel = _mk_tree(tmp_path, monkeypatch) + (rel / "tarballs").mkdir() + (rel / "tarballs" / "project.yaml").write_text( + "build: false\n" + _REPO_YAML.format(name="ssl3") + ) + _mapping_release_yaml(rel) + _wire_release_update_path(monkeypatch, src, rel) + monkeypatch.setattr( + cmd_sync, "read_project_description_marker", lambda a, p, k: "c" * 40 + ) + monkeypatch.setattr( + cmd_sync, + "apply_mirror_meta", + lambda *a, **k: pytest.fail("no meta on mismatch"), + ) + with pytest.raises(SystemExit, match="source_commit mismatch"): + cmd_sync.cmd_sync_release(_args_from(rel)) From c4ef20bc3e6b45b126467e2b3bc71109d6fdcd09 Mon Sep 17 00:00:00 2001 From: Ricardo Dias Date: Tue, 13 Oct 2026 16:06:46 +0100 Subject: [PATCH 11/16] sync release --from: refuse entries without source_commit Signed-off-by: Ricardo Dias --- percona_obs/cmd_sync.py | 5 +++++ tests/test_sync_release.py | 45 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 50 insertions(+) diff --git a/percona_obs/cmd_sync.py b/percona_obs/cmd_sync.py index ff5349104..438ad59ca 100644 --- a/percona_obs/cmd_sync.py +++ b/percona_obs/cmd_sync.py @@ -2650,6 +2650,11 @@ def cmd_sync_release(args) -> None: if entry is None: raise SystemExit("error: release.yaml has no releases: entries") if not args.force: + if not entry.source_commit: + raise SystemExit( + f"error: release.yaml entry {entry.tag} has no source_commit; " + "re-run 'percona-obs project release' (or pass --force)" + ) marker = read_project_description_marker( apiurl, from_project, "source_commit" ) diff --git a/tests/test_sync_release.py b/tests/test_sync_release.py index d4320c301..11ba99324 100644 --- a/tests/test_sync_release.py +++ b/tests/test_sync_release.py @@ -473,5 +473,50 @@ def test_from_prerelease_marker_mismatch(tmp_path, monkeypatch): "apply_mirror_meta", lambda *a, **k: pytest.fail("no meta on mismatch"), ) + monkeypatch.setattr( + cmd_sync, "release_tree", lambda *a, **k: pytest.fail("no copy on mismatch") + ) with pytest.raises(SystemExit, match="source_commit mismatch"): cmd_sync.cmd_sync_release(_args_from(rel)) + + +def _bare_tag_setup(tmp_path, monkeypatch): + src, rel = _mk_tree(tmp_path, monkeypatch) + (rel / "tarballs").mkdir() + (rel / "tarballs" / "project.yaml").write_text( + "build: false\n" + _REPO_YAML.format(name="ssl3") + ) + (rel / "release.yaml").write_text( + "project: ppg:staging:17\nreleases:\n - ppg/17.11-1\n" + ) + _wire_release_update_path(monkeypatch, src, rel) + monkeypatch.setattr(cmd_sync, "_obs_project_exists", lambda *a, **k: False) + monkeypatch.setattr( + cmd_sync, "read_project_description_marker", lambda a, p, k: None + ) + return rel + + +def test_from_prerelease_bare_tag_entry_refused(tmp_path, monkeypatch): + rel = _bare_tag_setup(tmp_path, monkeypatch) + monkeypatch.setattr( + cmd_sync, "apply_mirror_meta", lambda *a, **k: pytest.fail("no meta") + ) + monkeypatch.setattr( + cmd_sync, "release_tree", lambda *a, **k: pytest.fail("no copy") + ) + with pytest.raises(SystemExit, match="has no source_commit"): + cmd_sync.cmd_sync_release(_args_from(rel)) + + +def test_from_prerelease_force_skips_provenance_checks(tmp_path, monkeypatch): + rel = _bare_tag_setup(tmp_path, monkeypatch) + metas, copies = [], [] + monkeypatch.setattr( + cmd_sync, "apply_mirror_meta", lambda a, prj, *r, **k: metas.append(prj) + ) + monkeypatch.setattr( + cmd_sync, "release_tree", lambda a, s, t, *r, **k: copies.append((s, t)) + ) + cmd_sync.cmd_sync_release(_args_from(rel, force=True)) + assert len(metas) == 1 and len(copies) == 1 From 88f713ee1d2b0b0176b9521a3fe93bd590aaffc3 Mon Sep 17 00:00:00 2001 From: Ricardo Dias Date: Tue, 13 Oct 2026 16:08:27 +0100 Subject: [PATCH 12/16] project release: carry rendered qa: blocks into the release mirrors Signed-off-by: Ricardo Dias --- percona_obs/cmd_project.py | 36 ++++++++++++++++++++++++ tests/test_project_release.py | 53 +++++++++++++++++++++++++++++++++++ 2 files changed, 89 insertions(+) diff --git a/percona_obs/cmd_project.py b/percona_obs/cmd_project.py index 4307a1e71..faa9d41f0 100644 --- a/percona_obs/cmd_project.py +++ b/percona_obs/cmd_project.py @@ -1707,6 +1707,31 @@ def _is_relative_to(child: Path, parent: Path) -> bool: return False +def _rewrite_qa_project_refs(value, source_project_id: str, release_project_id: str): + """Deep-copy a loaded ``qa:`` value replacing ````/``:...`` + project references inside strings with the release project. Macros are + already rendered by the loader; ``${VAR}`` tokens are left as they are.""" + if isinstance(value, str): + if value == source_project_id: + return release_project_id + return re.sub( + rf"(? None: project_data["debuginfo"] = source_debuginfo if source_prjconf: project_data["project-config"] = source_prjconf + top_cfg_no_env = _load_project_config_with_inheritance( + source_path, repo_filter=RepositoryFilter.EMPTY + ) + if top_cfg_no_env.get("qa") is not None: + project_data["qa"] = _rewrite_qa_project_refs( + top_cfg_no_env["qa"], args.project, release_project + ) if is_first_release: commit_msg = f"Release {release_project} ({release_id})" diff --git a/tests/test_project_release.py b/tests/test_project_release.py index fe7ab13fd..6ef226b2a 100644 --- a/tests/test_project_release.py +++ b/tests/test_project_release.py @@ -708,3 +708,56 @@ def test_derive_release_id_without_server_package_errors_unless_counter_mode(): release_name="17", versrel_lookup=lambda repo, arch, pkg: None, ) + + +def test_rewrite_qa_project_refs_recurses_and_keeps_env_tokens(): + from percona_obs.cmd_project import _rewrite_qa_project_refs + + qa = [ + { + "pipeline": "p", + "parameters": { + "OBS_PROJECT": "${OBS_ROOTPRJ}:ppg:staging:17:containers", + "PLATFORMS": ["ppg:staging:17", "other"], + }, + } + ] + out = _rewrite_qa_project_refs(qa, "ppg:staging:17", "ppg:releases:17") + assert ( + out[0]["parameters"]["OBS_PROJECT"] + == "${OBS_ROOTPRJ}:ppg:releases:17:containers" + ) + assert out[0]["parameters"]["PLATFORMS"] == ["ppg:releases:17", "other"] + assert qa[0]["parameters"]["PLATFORMS"][0] == "ppg:staging:17" # input untouched + + +def test_write_release_tree_carries_rendered_qa_blocks(tmp_path, monkeypatch): + import percona_obs.common as common + + src = _staging_tree(tmp_path) + monkeypatch.setattr(cmd_project, "_REPO_DIR", tmp_path) + monkeypatch.setattr(common, "REPO_ROOT", tmp_path / "root") + (tmp_path / "root/macros.yaml").write_text("- PG_VERSION: 17.11\n") + (src / "containers" / "project.yaml").write_text( + (src / "containers" / "project.yaml").read_text() + + "qa:\n - pipeline: p\n parameters:\n" + " OBS_PROJECT: ${OBS_ROOTPRJ}:ppg:staging:17:containers\n" + " VERSION: ppg-%!{PG_VERSION}\n" + ) + rel = tmp_path / "root/ppg/releases/17" + _write_release_tree( + rel, + {"build": False, "repositories": []}, + src, + "ppg:staging:17", + "ppg:releases:17", + "ppg", + "17", + ) + containers = yaml.safe_load((rel / "containers" / "project.yaml").read_text()) + assert containers["qa"][0]["parameters"] == { + "OBS_PROJECT": "${OBS_ROOTPRJ}:ppg:releases:17:containers", + "VERSION": "ppg-17.11", + } + tarballs = yaml.safe_load((rel / "tarballs" / "project.yaml").read_text()) + assert "qa" not in tarballs From fd7e97b9a285781160b85797d70c37e8be3287c6 Mon Sep 17 00:00:00 2001 From: Ricardo Dias Date: Tue, 13 Oct 2026 16:11:08 +0100 Subject: [PATCH 13/16] project release: record source_commit, refresh an open release in place Signed-off-by: Ricardo Dias --- .github/scripts/update_version_lists.py | 5 +- percona_obs/cli.py | 8 ++ percona_obs/cmd_project.py | 158 +++++++++++++++++++----- tests/test_project_release.py | 75 +++++++++++ tests/test_update_version_lists.py | 38 ++++++ 5 files changed, 250 insertions(+), 34 deletions(-) create mode 100644 tests/test_update_version_lists.py diff --git a/.github/scripts/update_version_lists.py b/.github/scripts/update_version_lists.py index 6c888f142..64b24e61d 100644 --- a/.github/scripts/update_version_lists.py +++ b/.github/scripts/update_version_lists.py @@ -156,8 +156,9 @@ def get_latest_release_id(project_id: str) -> str: releases = data.get("releases") or [] if not releases: return "—" - last = releases[-1] # e.g. "ppg/17.9-1" - return last.split("/", 1)[1] if "/" in last else last + last = releases[-1] # "ppg/17.9-1" or {tag: ppg/17.9-1, source_commit: ...} + tag = last.get("tag", "") if isinstance(last, dict) else str(last) + return tag.split("/", 1)[1] if "/" in tag else tag # --------------------------------------------------------------------------- diff --git a/percona_obs/cli.py b/percona_obs/cli.py index 23d874396..b8ec7cb41 100644 --- a/percona_obs/cli.py +++ b/percona_obs/cli.py @@ -896,6 +896,14 @@ def build_parser() -> argparse.ArgumentParser: dest="no_cve_scan", help="Skip the best-effort CVE scan when generating the CHANGELOG Security section.", ) + project_release_parser.add_argument( + "--source-commit", + metavar="SHA", + default=None, + dest="source_commit", + help="Commit of main whose staging tree is released (default: head of the " + "last sync-main run with successful sync jobs, via gh). Recorded in release.yaml.", + ) project_release_parser.set_defaults(func=cmd_project_release) return parser diff --git a/percona_obs/cmd_project.py b/percona_obs/cmd_project.py index faa9d41f0..88e5bb00b 100644 --- a/percona_obs/cmd_project.py +++ b/percona_obs/cmd_project.py @@ -79,7 +79,8 @@ ) from .cve_scan import ChangedPackage, scan_release_cves from .release_scope import has_container_images, tree_has_server_package -from .release_record import ReleaseEntry, parse_release_entries +from .release_record import ReleaseEntry, dump_release_entry, parse_release_entries +from .github_api import last_synced_head_sha, repo_slug_from_url from .services import _git_head_sha, _git_tag_for_sha _YAML_FILENAMES = {"project.yaml", "package.yaml"} @@ -2285,6 +2286,105 @@ def _derive_release_id( return f"{major_minor}-{matching_count + 1}" +def _tag_exists_locally(tag: str) -> bool: + r = subprocess.run( + ["git", "rev-parse", "--verify", "--quiet", f"refs/tags/{tag}"], + capture_output=True, + cwd=_REPO_DIR, + ) + return r.returncode == 0 + + +def _is_ancestor_of_head(sha: str) -> bool: + r = subprocess.run( + ["git", "merge-base", "--is-ancestor", sha, "HEAD"], + capture_output=True, + cwd=_REPO_DIR, + ) + return r.returncode == 0 + + +def _resolve_source_commit(override: "str | None", repo_url: str) -> str: + """The commit of main whose staging tree the release is cut from. + + Default: head SHA of the last sync-main run with successful sync jobs + (``gh api``), so the record names what OBS actually holds. Must be an + ancestor of HEAD so the release branch contains the tree it claims. + """ + if override: + sha = override + else: + if not repo_url: + raise SystemExit( + "error: PERCONA_OBS_PACKAGING_REPO is not set in the profile; " + "set it or pass --source-commit" + ) + sha = last_synced_head_sha(repo_slug_from_url(repo_url)) + if not _is_ancestor_of_head(sha): + raise SystemExit( + f"error: source commit {sha} is not an ancestor of HEAD; " + "rebase the release branch on main (and fetch) first" + ) + return sha + + +def _plan_release_entries( + existing: "list[ReleaseEntry]", + explicit_id: "str | None", + derive_id: "Callable[[], str]", + product: str, + source_commit: str, +) -> "tuple[str, list[ReleaseEntry], str | None]": + """Decide the release id and the new ``releases:`` list. + + Returns (release_id, entries, prev_release_id). The last entry whose + tag is not yet a git tag is *open*: without ``--release-id`` it is + refreshed in place (same id, new source_commit) and prev_release_id is + the entry before it. Otherwise the next id is derived and appended. + """ + open_entry = ( + existing[-1] if existing and not _tag_exists_locally(existing[-1].tag) else None + ) + if explicit_id: + tag = f"{product}/{explicit_id}" + if any(e.tag == tag for e in existing): + raise SystemExit( + f"error: release tag {tag} is already present in release.yaml" + ) + prev = existing[-1].tag.split("/")[-1] if existing else None + return explicit_id, [*existing, ReleaseEntry(tag, source_commit)], prev + if open_entry is not None: + rid = open_entry.tag.split("/")[-1] + print( + f"note: {open_entry.tag} has no git tag locally — treating the release as " + "still open and refreshing it in place (fetch tags if it already shipped)", + flush=True, + ) + prev = existing[-2].tag.split("/")[-1] if len(existing) > 1 else None + return rid, [*existing[:-1], ReleaseEntry(open_entry.tag, source_commit)], prev + rid = derive_id() + tag = f"{product}/{rid}" + if any(e.tag == tag for e in existing): + raise SystemExit(f"error: release tag {tag} is already present in release.yaml") + prev = existing[-1].tag.split("/")[-1] if existing else None + return rid, [*existing, ReleaseEntry(tag, source_commit)], prev + + +def _replace_changelog_section(text: str, release_id: str, section: str) -> str: + """Replace the ``## [release_id]`` section (up to the next ``## [``) or, + when absent, prepend *section* above the first section.""" + pattern = re.compile( + r"## \[" + re.escape(release_id) + r"\].*?(?=\n## \[|\Z)", re.DOTALL + ) + if pattern.search(text): + return pattern.sub(lambda _m: section.rstrip("\n") + "\n", text, count=1) + m = re.search(r"\n(## \[)", text) + if m: + pos = m.start() + 1 + return text[:pos] + section.rstrip("\n") + "\n\n" + text[pos:] + return text.rstrip() + "\n\n" + section.rstrip("\n") + "\n" + + def cmd_project_release(args: argparse.Namespace) -> None: """Create or update a release: release.yaml, CHANGELOG.md, project.yaml, commit, and open a PR.""" product = args.project.split(":")[0] @@ -2327,11 +2427,18 @@ def cmd_project_release(args: argparse.Namespace) -> None: ) existing_releases: list[str] = [e.tag for e in existing_entries] - # Auto-derive release-id from OBS if not provided. - release_id: str = args.release_id or "" - if not release_id: + release_env: dict[str, str] = { + **(parse_env_overrides(args.env_overrides) if args.env_overrides else {}), + **auto_rootprj_env(args.rootprj), + } + source_commit = _resolve_source_commit( + getattr(args, "source_commit", None), + release_env.get("PERCONA_OBS_PACKAGING_REPO", ""), + ) + + def _derive() -> str: pkg_archs = _fetch_all_pkg_archs(apiurl, source_obs_project) - release_id = _derive_release_id( + return _derive_release_id( pkg_archs, existing_releases, release_name, @@ -2342,18 +2449,15 @@ def cmd_project_release(args: argparse.Namespace) -> None: major=major, ) + release_id, new_entries, prev_release_id = _plan_release_entries( + existing_entries, args.release_id or None, _derive, product, source_commit + ) tag = f"{product}/{release_id}" - - if tag in existing_releases: - raise SystemExit(f"error: release tag {tag} is already present in release.yaml") + is_refresh = bool(existing_entries) and existing_entries[-1].tag == tag # Source project topology comes from the tree, unfiltered: the release # snapshot is instance-agnostic and each instance's `sync release` slices # it (spec Section 4). Env vars follow the same precedence as sync. - release_env: dict[str, str] = { - **(parse_env_overrides(args.env_overrides) if args.env_overrides else {}), - **auto_rootprj_env(args.rootprj), - } source_cfg = resolve_project_config( source_path, release_env, repo_filter=RepositoryFilter.EMPTY ) @@ -2403,7 +2507,6 @@ def cmd_project_release(args: argparse.Namespace) -> None: _print_pending("fetching container image package lists for CHANGELOG") source_container_pkgs: dict[str, dict[str, str]] = {} release_container_pkgs: dict[str, dict[str, str]] = {} - prev_release_id: str | None = None for subproject_name in container_subs: sub_full = f"{args.rootprj}:{args.project}:{subproject_name}" _merge_container_pkgs( @@ -2413,8 +2516,7 @@ def cmd_project_release(args: argparse.Namespace) -> None: ), sub_full, ) - if not is_first_release and existing_releases: - prev_release_id = existing_releases[-1].split("/")[-1] + if not is_first_release and prev_release_id: # Diff against whatever release subprojects actually exist on OBS — # this is what makes the ubi9 old-layout → new-layout migration diff # correctly instead of dumping every image as "add". @@ -2430,7 +2532,7 @@ def cmd_project_release(args: argparse.Namespace) -> None: security_lines: list[str] | None = None if not getattr(args, "no_cve_scan", False): _print_pending("scanning upstream release notes for CVE fixes") - prev_tag_for_scan = existing_releases[-1] if existing_releases else None + prev_tag_for_scan = f"{product}/{prev_release_id}" if prev_release_id else None try: changed_pkgs = _build_changed_packages( source_versions, release_versions, source_path @@ -2457,11 +2559,10 @@ def cmd_project_release(args: argparse.Namespace) -> None: ) # Build release.yaml content. - new_releases_list = existing_releases + [tag] release_data: dict = { "repository": "${PERCONA_OBS_PACKAGING_REPO}", "project": args.project, - "releases": new_releases_list, + "releases": [dump_release_entry(e) for e in new_entries], } # Build project.yaml for the release directory. @@ -2489,11 +2590,14 @@ def cmd_project_release(args: argparse.Namespace) -> None: if is_first_release: commit_msg = f"Release {release_project} ({release_id})" + elif is_refresh: + commit_msg = f"Refresh release {release_project} ({release_id})" else: commit_msg = f"Update release {release_project} ({release_id})" # Preview and confirm. - print(f"{'First' if is_first_release else 'Update'} release: {release_project}") + kind = "First" if is_first_release else "Refresh" if is_refresh else "Update" + print(f"{kind} release: {release_project}") print(f" Tag: {tag}") print(f" Directory: {release_dir.relative_to(_REPO_DIR)}/") if not is_first_release: @@ -2564,19 +2668,9 @@ def cmd_project_release(args: argparse.Namespace) -> None: changelog_file = release_dir / "CHANGELOG.md" if changelog_file.is_file(): existing_changelog = changelog_file.read_text("utf-8") - m = re.search(r"\n(## \[)", existing_changelog) - if m: - insert_pos = m.start() + 1 - new_content = ( - existing_changelog[:insert_pos] - + changelog_section - + "\n" - + existing_changelog[insert_pos:] - ) - else: - new_content = ( - existing_changelog.rstrip() + "\n\n" + changelog_section + "\n" - ) + new_content = _replace_changelog_section( + existing_changelog, release_id, changelog_section + ) else: new_content = _CHANGELOG_HEADER + changelog_section + "\n" changelog_file.write_text(new_content, encoding="utf-8") diff --git a/tests/test_project_release.py b/tests/test_project_release.py index 6ef226b2a..9eb8be207 100644 --- a/tests/test_project_release.py +++ b/tests/test_project_release.py @@ -761,3 +761,78 @@ def test_write_release_tree_carries_rendered_qa_blocks(tmp_path, monkeypatch): } tarballs = yaml.safe_load((rel / "tarballs" / "project.yaml").read_text()) assert "qa" not in tarballs + + +def test_replace_changelog_section(): + from percona_obs.cmd_project import _replace_changelog_section + + text = "# Changelog\n\n## [18.6-1] - 2026-01-01\n\n### Added\n- a\n\n## [18.4-1] - 2025-12-01\n\n### Added\n- b\n" + out = _replace_changelog_section( + text, "18.6-1", "## [18.6-1] - 2026-02-02\n\n### Added\n- c\n" + ) + assert "- a" not in out and "- c" in out and "- b" in out + assert out.index("[18.6-1]") < out.index("[18.4-1]") + # absent -> prepended above the first section + out2 = _replace_changelog_section(text, "18.7-1", "## [18.7-1] - x\n") + assert out2.index("[18.7-1]") < out2.index("[18.6-1]") + + +def test_resolve_source_commit_override_and_ancestor(monkeypatch): + from percona_obs import cmd_project as cp + + monkeypatch.setattr(cp, "_is_ancestor_of_head", lambda sha: sha == "good") + assert cp._resolve_source_commit("good", "https://github.com/o/r.git") == "good" + with pytest.raises(SystemExit, match="not an ancestor of HEAD"): + cp._resolve_source_commit("bad", "https://github.com/o/r.git") + monkeypatch.setattr(cp, "last_synced_head_sha", lambda slug: "good") + assert cp._resolve_source_commit(None, "https://github.com/o/r.git") == "good" + with pytest.raises(SystemExit, match="PERCONA_OBS_PACKAGING_REPO"): + cp._resolve_source_commit(None, "") + + +def test_open_release_is_refreshed_in_place(monkeypatch): + from percona_obs import cmd_project as cp + from percona_obs.release_record import ReleaseEntry + + monkeypatch.setattr(cp, "_tag_exists_locally", lambda tag: False) + entries = [ReleaseEntry("ppg/18.4-1"), ReleaseEntry("ppg/18.6-1", "old")] + rid, new_entries, prev = cp._plan_release_entries( + entries, None, lambda: "18.6-2", "ppg", "newsha" + ) + assert rid == "18.6-1" + assert new_entries == [ + ReleaseEntry("ppg/18.4-1"), + ReleaseEntry("ppg/18.6-1", "newsha"), + ] + assert prev == "18.4-1" + + +def test_shipped_release_appends_next_id(monkeypatch): + from percona_obs import cmd_project as cp + from percona_obs.release_record import ReleaseEntry + + monkeypatch.setattr(cp, "_tag_exists_locally", lambda tag: True) + entries = [ReleaseEntry("ppg/18.6-1", "old")] + rid, new_entries, prev = cp._plan_release_entries( + entries, None, lambda: "18.6-2", "ppg", "newsha" + ) + assert rid == "18.6-2" + assert new_entries == [ + ReleaseEntry("ppg/18.6-1", "old"), + ReleaseEntry("ppg/18.6-2", "newsha"), + ] + assert prev == "18.6-1" + + +def test_explicit_release_id_wins_over_refresh(monkeypatch): + from percona_obs import cmd_project as cp + from percona_obs.release_record import ReleaseEntry + + monkeypatch.setattr(cp, "_tag_exists_locally", lambda tag: False) + entries = [ReleaseEntry("ppg/18.6-1")] + with pytest.raises(SystemExit, match="already present"): + cp._plan_release_entries(entries, "18.6-1", lambda: "x", "ppg", "s") + rid, new_entries, _ = cp._plan_release_entries( + entries, "18.6-9", lambda: "x", "ppg", "s" + ) + assert rid == "18.6-9" and new_entries[-1] == ReleaseEntry("ppg/18.6-9", "s") diff --git a/tests/test_update_version_lists.py b/tests/test_update_version_lists.py new file mode 100644 index 000000000..274260936 --- /dev/null +++ b/tests/test_update_version_lists.py @@ -0,0 +1,38 @@ +"""Latest release-id lookup in .github/scripts/update_version_lists.py.""" + +import importlib.util +from pathlib import Path + +_SPEC = importlib.util.spec_from_file_location( + "update_version_lists", + Path(__file__).parent.parent / ".github/scripts/update_version_lists.py", +) +assert _SPEC is not None + + +def _load(): + assert _SPEC is not None and _SPEC.loader is not None + mod = importlib.util.module_from_spec(_SPEC) + _SPEC.loader.exec_module(mod) + return mod + + +def _latest(tmp_path, monkeypatch, releases_yaml: str) -> str: + m = _load() + d = tmp_path / "ppg" / "releases" / "18" + d.mkdir(parents=True) + (d / "release.yaml").write_text(releases_yaml) + monkeypatch.setattr(m, "ROOT_DIR", tmp_path) + return m.get_latest_release_id("ppg:releases:18") + + +def test_latest_release_id_string_entries(tmp_path, monkeypatch): + assert ( + _latest(tmp_path, monkeypatch, "releases:\n- ppg/18.4-1\n- ppg/18.6-1\n") + == "18.6-1" + ) + + +def test_latest_release_id_mapping_entry(tmp_path, monkeypatch): + y = "releases:\n- ppg/18.4-1\n- tag: ppg/18.6-1\n source_commit: abc\n" + assert _latest(tmp_path, monkeypatch, y) == "18.6-1" From d138ede458affa6af92e1b727e6dd06ce0424974 Mon Sep 17 00:00:00 2001 From: Ricardo Dias Date: Tue, 13 Oct 2026 16:15:01 +0100 Subject: [PATCH 14/16] docs: pre-release flow tool commands and release.yaml entry form Signed-off-by: Ricardo Dias --- .github/copilot-instructions.md | 16 +++++- CLAUDE.md | 3 ++ docs/PERCONA_OBS_TOOL.md | 91 +++++++++++++++++++++++++++++---- 3 files changed, 98 insertions(+), 12 deletions(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index d97df0760..7245c22e1 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -542,6 +542,17 @@ merge a non-release PR (one that does not create a version tag) once its OBS bui Options: - `--yes` / `-y` — skip the confirmation prompt. +### Release hops (`sync prerelease`, `sync release --from`) + +A release ships in two hops (spec: `docs/superpowers/specs/2026-10-13-prerelease-project-release-flow-design.md`). +`sync prerelease --branch-from ` copies staging into a PR pre-release +project after checking the `source_commit` recorded in `release.yaml` against the last synced `main` +and stamps that SHA into the pre-release project descriptions. `sync release --from ` +then copies the pre-release into the release project (no freeze, no git divergence check) after +verifying the marker equals the tagged entry's `source_commit`. `release.yaml` entries are plain tag +strings (historical) or `{tag, source_commit}` mappings; `obs-pr-cleanup.yml` and +`.github/scripts/update_version_lists.py` read both. CI integration follows in rollout step 2. + ### `sync promote [--dry-run] [--no-services] [--no-cache] [-m MSG] [project] [package]` Promotes branch packages (created by a prior `--branch-from` sync) back to full source syncs. For each targeted package whose latest OBS revision comment matches the `branch:` pattern, the `_aggregate` is replaced with the local `obs/` source files (running any `mode="manual"` services as needed). Packages that already hold real sources are skipped (`=` output). @@ -1036,6 +1047,9 @@ When branching is involved, always confirm which OBS instance is being queried: | Root project config | `root/project.yaml` | | Repo-owned OBS service (npm lockfile generation) | `tools/obs-services/npm_lockfile` | | Version-independent tool with its own dependency project (pgAdmin) | `root/ppg/staging/common/tools/percona-pgadmin4/` + `root/ppg/common/deps/pgadmin/` | +| Shared release hop mechanics (mirror meta apply, `osc release` copy, verify, orphans) | `percona_obs/release_copy.py` | +| `release.yaml` `releases:` entries (tag strings and `{tag, source_commit}`) | `percona_obs/release_record.py` | +| `gh api` lookups (last synced `main` commit) | `percona_obs/github_api.py` | | Management script | `percona-obs` (commands: `sync push`, `sync delete`, `sync promote`, `build trigger`, `build status`, `build dependency`, `profile create`, `profile list`, `project verify`) | --- @@ -1099,7 +1113,7 @@ and dispatches the release before the shared cleanup step runs. **Release PR path** (merged, and every changed `root/**` file is under `root/*/releases/`): 1. For each `release.yaml` changed by the PR, reads the last entry of its `releases:` - list from the **merge commit** blob (not the PR head) and creates that git tag + list (a plain tag string or a `{tag, source_commit}` mapping — both are understood) from the **merge commit** blob (not the PR head) and creates that git tag (e.g. `ppg/17.9-1`) pointing at `merge_commit_sha`, using `GITHUB_TOKEN` (`contents: write`). A tag-creation error other than "already exists" fails the job. 2. Dispatches `obs-release.yml` for that tag via diff --git a/CLAUDE.md b/CLAUDE.md index 8d8a0f7d0..09616a9ab 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -47,6 +47,9 @@ Always use `git commit -s` (signed-off-by). Do not add `Co-Authored-By: Claude` | `percona_obs/cmd_build.py` | `build trigger` / `build status` implementation | | `percona_obs/cmd_profile.py` | `profile` command implementation | | `percona_obs/obs_api.py` | OBS API wrappers | +| `percona_obs/release_copy.py` | Shared release hop mechanics (mirror meta apply, `osc release` copy, verify, orphans) | +| `percona_obs/release_record.py` | `release.yaml` `releases:` entries (tag strings and `{tag, source_commit}`) | +| `percona_obs/github_api.py` | `gh api` lookups (last synced `main` commit) | | `percona_obs/common.py` | Shared constants, colour helpers, data utilities | | `percona_obs/targets.py` | Target resolution helpers | | `percona_obs/services.py` | Local OBS service execution + caching | diff --git a/docs/PERCONA_OBS_TOOL.md b/docs/PERCONA_OBS_TOOL.md index 30894be49..9de33802e 100644 --- a/docs/PERCONA_OBS_TOOL.md +++ b/docs/PERCONA_OBS_TOOL.md @@ -987,6 +987,10 @@ devel/staging/releases tier layout). Release IDs include the PG minor version an counter (e.g. `17.9-1`, `17.9-2`, `17.10-1`). A single OBS release project (`ppg:releases:17`) covers all minor versions for a PG major. +Until rollout step 2 (the workflows PR) lands, CI still runs the single-hop `sync release` +on merge; the two hops (`sync prerelease`, `sync release --from`) are available for +manual and rehearsal use. + The process is PR-based: 1. **`project release`** — auto-derives the release ID from OBS, regenerates the full @@ -1045,8 +1049,17 @@ to override the arithmetic. # Override both ./percona-obs -P local project release ppg:staging:17 --release-name 17 --release-id 17.9-1 + +# Name the source commit explicitly (skips the gh lookup) +./percona-obs -P local project release ppg:staging:17 --source-commit ``` +Before cutting, run `git fetch --tags` (or `git fetch percona --tags`): the open-release +detection below reads **local** git tags, so without a fresh fetch a shipped release could +be mistaken for an open one (the command prints a note when it refreshes). The command +also needs `gh` (logged in, with network access) on every run unless `--source-commit` is +passed, because it resolves the last synced `main` commit through the GitHub API. + `source-project` must be a `staging`-tier project — any other tier is rejected with an error. `project release ` does the following: @@ -1058,23 +1071,32 @@ error. `project release ` does the following: - If no `percona-postgresql` / `percona-postgresqlNN` package directory exists anywhere under the source tree (read from git; e.g. `ppg/staging/common`), a plain counter `-N` is used instead (tag `ppg/common-`). A PG major whose OBS listing is transiently empty still fails with the old error (use `--release-id` to override), so an OBS blip can never produce a counter tag for a major. -2. **Fetches** the source project's repository topology from OBS. -3. **Regenerates the full mirror tree**, on every release, not just the first: the +2. **Resolves the source commit** — the head SHA of the last `sync-main.yml` run on + `main` whose sync jobs succeeded (`gh api`), or `--source-commit `. It must be + an ancestor of HEAD. It is recorded in `release.yaml` so the release names the + staging tree its binaries were built from. +3. **Open-release refresh** — if the last `releases:` entry has no git tag yet, the + release is still in flight: the same id is kept, the mirror tree is regenerated, the + `CHANGELOG.md` section for that id is replaced and `source_commit` is updated. Only + when the tag exists is the next id derived and appended (`--release-id` always wins). +4. **Fetches** the source project's repository topology from OBS. +5. **Regenerates the full mirror tree**, on every release, not just the first: the top-level `project.yaml` plus one nested mirror directory per staging subproject (`containers/`, `extras/`, `extras/containers/`, `tarballs/`). All mirrors get `build: false`, `publish:` flags are carried over, and subproject paths — including intra-project sibling paths — are rewritten to the release namespace. A mirror directory whose staging source no longer exists is **deleted**. -4. **Builds `CHANGELOG.md`** — compares built package versions from the top-level staging +6. **Builds `CHANGELOG.md`** — compares built package versions from the top-level staging project, `extras`, and `tarballs` against the release OBS project (for updates) and generates entries with upstream URLs. Container images are listed per flavor (`ubi8` and `ubi9` as distinct entries); the previous-release image diff is queried from whichever project actually holds the previous release's images, producing real `prev → new` diffs instead of full package dumps (a full dump remains only for a true first release). -5. **Appends** the new tag to `release.yaml`'s `releases:` list and prepends the new - section to `CHANGELOG.md`. -6. **Commits** the generated files locally with `git commit -s`. Pushing the branch and +7. **Appends** a `{tag, source_commit}` entry to `release.yaml`'s `releases:` list (or + updates the open one) and prepends the new section to `CHANGELOG.md` (or replaces the + open one's). +8. **Commits** the generated files locally with `git commit -s`. Pushing the branch and opening the PR remain manual steps — `project release` does not do either. The git tag (e.g. `ppg/17.9-1`) is created automatically by `obs-pr-cleanup.yml`, on the @@ -1088,13 +1110,15 @@ anything by itself. repository: ${PERCONA_OBS_PACKAGING_REPO} project: ppg:staging:17 releases: - - ppg/17.9-1 # first release - - ppg/17.9-2 # update release - - ppg/17.10-1 # minor version bump + - ppg/17.9-1 # historical entries stay plain tags + - tag: ppg/17.10-1 # written by project release + source_commit: 9e9d9738c0ffee… # full SHA of main the binaries were built from ``` -`sync release` always uses the **last** entry as the current tag. Old entries are -kept for history. The backward-compat `revision:` field (plain string) is still +The last entry is the current release, in either form (plain tag string or +`{tag, source_commit}` mapping). `sync release` always uses it as the current tag. Old +entries are kept for history. `sync release --from` and `sync prerelease` require the +last entry to carry a `source_commit`. The backward-compat `revision:` field (plain string) is still accepted and treated as a single-element list. ### CHANGELOG.md format @@ -1177,6 +1201,50 @@ the pending `obs-release.yml` run is **cancelled** rather than requeued — the a cancelled "OBS Release" run with no release having happened. Recovery is the same manual command above, re-run once the concurrency group is free. +### `sync prerelease` — hop 1 (staging → PR pre-release) + +Run with the PR profile; `--branch-from` names the production profile holding staging: + +```sh +./percona-obs -P pr-12 sync prerelease ppg:releases:18 --branch-from main +``` + +Steps: (1) require a `source_commit` on the last `release.yaml` entry; (2) **record +check** — the freeze-scope paths (source tree, its `macros.yaml` chain, every +whole-frozen project, every aggregate-frozen package) must be identical between +`source_commit` and the last synced `main` (`--synced-commit`, else `gh api`), or it +fails with "staging changed since the cut"; edits under `root//staging/_shared/...` +count as staging changes because the check follows the `_shared/` symlink targets of +packages; (3) pre-release meta from the mirrors, **publish forced on**; (4) drain, assert +green, freeze, `osc release`, verify, restore; (5) `source_commit: ` appended to each +pre-release project description. + +| Flag | Default | Effect | +|---|---|---| +| `--branch-from PROFILE` | required | Profile of the production instance holding the staging source. | +| `--synced-commit SHA` | `gh` lookup | Head SHA of the last successful sync-main run. | +| `--source-commit SHA` | last entry's | Override the `source_commit` from `release.yaml`. | +| `--force` | off | Skips the record check (2). | +| `--no-freeze` | off | Skips the drain/green-check/freeze sequence. | +| `--freeze-timeout` | `3600` | Bound on the drain step, seconds. | +| `--verify-timeout` | `600` | Bound on the post-release verify poll (`0` disables). | +| `--dry-run` | off | Runs (1), (2), the mirror/changelog/tag checks and the green check only. | + +The workflow that calls this on release PRs lands in rollout step 2. + +### `sync release --from` — hop 2 (pre-release → release project) + +```sh +./percona-obs -P main sync release ppg:releases:18 --from isv:percona:PR:pr-12:ppg:releases:18 +``` + +No freeze, no git divergence check. The last `release.yaml` entry must have a +`source_commit` and the pre-release description's marker must equal it; a missing or +different marker, or an entry without `source_commit`, is refused (`--force` skips both +checks). Then the release project meta is applied from the mirrors and binaries are copied +with `osc release`. Known limitation: `--dry-run` together with `--from` still only runs +the staging-side dry-run checks. + ### Running `sync release` manually For local testing or recovery from a failed CI run: @@ -1195,6 +1263,7 @@ Flags: | `--no-freeze` | off | Skips the drain/freeze/restore sequence (§ below) — for recovery scenarios where staging is already known idle. | | `--freeze-timeout` | `3600` (seconds) | Bound on the drain step; the release fails rather than waiting forever for a stuck package. | | `--verify-timeout` | `600` (seconds, `0` disables) | Bound on the post-release poll that confirms each released repo actually holds binaries. | +| `--from OBS_PROJECT` | off | Hop 2: copy from this pre-release OBS project (full name) instead of staging; no freeze, no git divergence check. See above. | `sync release ` reads `release.yaml` and performs: From 1f96ca94f055ca01355e82f139078135466efa7f Mon Sep 17 00:00:00 2001 From: Ricardo Dias Date: Tue, 13 Oct 2026 16:27:12 +0100 Subject: [PATCH 15/16] Final review fixes: hop 2 mirror-derived subprojects, record check config layers, osc release without --target-project, remote tag check, nightly QA skips release tier Signed-off-by: Ricardo Dias --- .github/scripts/list_qa_matrix.py | 10 ++ docs/PERCONA_OBS_TOOL.md | 6 +- ...-prerelease-project-release-flow-design.md | 14 ++- percona_obs/cmd_project.py | 40 ++++++- percona_obs/cmd_sync.py | 106 ++++++++++++------ percona_obs/release_copy.py | 6 +- tests/test_list_qa_matrix.py | 8 ++ tests/test_project_release.py | 38 +++++-- tests/test_release_copy.py | 6 +- tests/test_sync_prerelease.py | 2 + tests/test_sync_release.py | 49 ++++++++ 11 files changed, 222 insertions(+), 63 deletions(-) diff --git a/.github/scripts/list_qa_matrix.py b/.github/scripts/list_qa_matrix.py index 0961efe20..e3f526227 100644 --- a/.github/scripts/list_qa_matrix.py +++ b/.github/scripts/list_qa_matrix.py @@ -41,6 +41,11 @@ def qa_project_type(full_name: str) -> str: return "packages" +def is_release_tier(full_name: str) -> bool: + """True for production release projects (``...:releases:[:sub]``).""" + return ":releases:" in full_name + + def normalize_entry(entry: dict) -> dict: """Ensure a combo carries ready-to-use ``name_filter``/``package_filter`` strings. @@ -113,6 +118,11 @@ def main() -> None: matrix: list = [] for full_name in subs: project = full_name[len(prefix) :] + # Release mirrors now carry qa: blocks meant for the per-PR + # pre-release; the frozen production snapshots must not be re-tested + # by the nightly (production-root enumeration, i.e. not PR mode). + if not present_only and is_release_tier(full_name): + continue proj_type = qa_project_type(full_name) if qa_types and proj_type not in qa_types: continue diff --git a/docs/PERCONA_OBS_TOOL.md b/docs/PERCONA_OBS_TOOL.md index 9de33802e..6e7a87483 100644 --- a/docs/PERCONA_OBS_TOOL.md +++ b/docs/PERCONA_OBS_TOOL.md @@ -1055,8 +1055,10 @@ to override the arithmetic. ``` Before cutting, run `git fetch --tags` (or `git fetch percona --tags`): the open-release -detection below reads **local** git tags, so without a fresh fetch a shipped release could -be mistaken for an open one (the command prints a note when it refreshes). The command +detection below reads local git tags and also consults the remote +(`PERCONA_OBS_PACKAGING_REPO`, via `git ls-remote --tags`), so a stale local clone only +matters when the remote is unreachable (the command then warns, and prints a note when it +refreshes an open release). The command also needs `gh` (logged in, with network access) on every run unless `--source-commit` is passed, because it resolves the last synced `main` commit through the GitHub API. diff --git a/docs/superpowers/specs/2026-10-13-prerelease-project-release-flow-design.md b/docs/superpowers/specs/2026-10-13-prerelease-project-release-flow-design.md index 315c6c3f5..ca4dded3c 100644 --- a/docs/superpowers/specs/2026-10-13-prerelease-project-release-flow-design.md +++ b/docs/superpowers/specs/2026-10-13-prerelease-project-release-flow-design.md @@ -192,9 +192,11 @@ With `--from` the command loses the freeze and the git divergence check: tagged `release.yaml` entry's. 2. Apply the release project meta from the mirror (repo additions, publish flags, prjconf), builds disabled. -3. `osc release --target-project --no-delay` from the - pre-release for the top project and every in-slice subproject. No - temporary `releasetarget` entries are needed on the pre-release. +3. `osc release --no-delay` from the pre-release for the top project and every + in-slice subproject, with the same temporary `` entries + (added before, removed in a `finally`) that hop 1 and the legacy path use; + OBS requires release targets for a project-level release, so the + pre-release meta carries them only during the copy. 4. Verify binaries landed (bounded by `--verify-timeout`). Without `--from` the command keeps its current staging-sourced behaviour as a @@ -348,5 +350,7 @@ re-dispatch of hop 2. - `source_commit` is recorded by `project release` on the developer's machine, not committed back by CI: a workflow-token push would become an untested PR head and block required checks. -- `osc release --target-project` is used for hop 2 so the pre-release meta - stays a pure mirror with no `releasetarget` entries. +- Hop 2 uses `osc release --no-delay` with temporary `` + entries (removed in a `finally`), as hop 1 does; OBS requires release + targets for a project-level release, so the pre-release meta carries them + only during the copy. diff --git a/percona_obs/cmd_project.py b/percona_obs/cmd_project.py index 88e5bb00b..2d42d0939 100644 --- a/percona_obs/cmd_project.py +++ b/percona_obs/cmd_project.py @@ -2295,6 +2295,28 @@ def _tag_exists_locally(tag: str) -> bool: return r.returncode == 0 +def _tag_exists_on_remote(repo_url: str, tag: str) -> bool: + """True when *repo_url* has refs/tags/; a failed lookup warns, returns False.""" + try: + r = subprocess.run( + ["git", "ls-remote", "--tags", repo_url, f"refs/tags/{tag}"], + capture_output=True, + text=True, + cwd=_REPO_DIR, + timeout=60, + ) + except (subprocess.TimeoutExpired, OSError) as e: + print(f"warning: git ls-remote {repo_url} failed: {e}", flush=True) + return False + if r.returncode != 0: + print( + f"warning: git ls-remote {repo_url} failed: {r.stderr.strip()}", + flush=True, + ) + return False + return bool(r.stdout.strip()) + + def _is_ancestor_of_head(sha: str) -> bool: r = subprocess.run( ["git", "merge-base", "--is-ancestor", sha, "HEAD"], @@ -2334,6 +2356,7 @@ def _plan_release_entries( derive_id: "Callable[[], str]", product: str, source_commit: str, + tag_exists: "Callable[[str], bool]", ) -> "tuple[str, list[ReleaseEntry], str | None]": """Decide the release id and the new ``releases:`` list. @@ -2342,9 +2365,7 @@ def _plan_release_entries( refreshed in place (same id, new source_commit) and prev_release_id is the entry before it. Otherwise the next id is derived and appended. """ - open_entry = ( - existing[-1] if existing and not _tag_exists_locally(existing[-1].tag) else None - ) + open_entry = existing[-1] if existing and not tag_exists(existing[-1].tag) else None if explicit_id: tag = f"{product}/{explicit_id}" if any(e.tag == tag for e in existing): @@ -2356,8 +2377,8 @@ def _plan_release_entries( if open_entry is not None: rid = open_entry.tag.split("/")[-1] print( - f"note: {open_entry.tag} has no git tag locally — treating the release as " - "still open and refreshing it in place (fetch tags if it already shipped)", + f"note: {open_entry.tag} has no git tag (local or remote) — treating the release " + "as still open and refreshing it in place", flush=True, ) prev = existing[-2].tag.split("/")[-1] if len(existing) > 1 else None @@ -2449,8 +2470,15 @@ def _derive() -> str: major=major, ) + repo_url = release_env.get("PERCONA_OBS_PACKAGING_REPO", "") release_id, new_entries, prev_release_id = _plan_release_entries( - existing_entries, args.release_id or None, _derive, product, source_commit + existing_entries, + args.release_id or None, + _derive, + product, + source_commit, + lambda t: _tag_exists_locally(t) + or (bool(repo_url) and _tag_exists_on_remote(repo_url, t)), ) tag = f"{product}/{release_id}" is_refresh = bool(existing_entries) and existing_entries[-1].tag == tag diff --git a/percona_obs/cmd_sync.py b/percona_obs/cmd_sync.py index 438ad59ca..e62b34166 100644 --- a/percona_obs/cmd_sync.py +++ b/percona_obs/cmd_sync.py @@ -66,6 +66,7 @@ RepositoryFilter, package_in_slice, project_in_slice, + _chain, prune_flag_map, ) from .git_utils import ( @@ -2533,6 +2534,22 @@ def _run_frozen( restore_builds(apiurl, snapshots) +def _mirror_subproject_pairs( + release_path: Path, release_project_id: str +) -> "list[tuple[str, Path]]": + """(subproject_name, mirror_path) for every in-slice subproject of the + release mirror tree (hop 2 copies what the mirror declares, not staging).""" + pairs: list[tuple[str, Path]] = [] + for sub_id, sub_path in find_projects(release_path, release_project_id): + if sub_id == release_project_id: + continue + if not (sub_path / "project.yaml").is_file(): + continue + if project_in_slice(sub_path): + pairs.append((sub_id[len(release_project_id) + 1 :], sub_path)) + return pairs + + def cmd_sync_release(args) -> None: """Release packages from an OBS source project to a release target. @@ -2571,6 +2588,53 @@ def cmd_sync_release(args) -> None: **auto_rootprj_env(args.rootprj), } + from_project: "str | None" = getattr(args, "from_project", None) + if from_project: + # Hop 2 (spec §2): no staging involvement, no freeze, no git check. + # The subproject set comes from the release mirror tree itself. + entry = current_release(release_data) + if entry is None: + raise SystemExit("error: release.yaml has no releases: entries") + if not args.force: + if not entry.source_commit: + raise SystemExit( + f"error: release.yaml entry {entry.tag} has no source_commit; " + "re-run 'percona-obs project release' (or pass --force)" + ) + marker = read_project_description_marker( + apiurl, from_project, "source_commit" + ) + if marker != entry.source_commit: + raise SystemExit( + f"error: source_commit mismatch: {from_project} was populated from " + f"{marker or ''} but release.yaml {entry.tag} records " + f"{entry.source_commit or ''}; repopulate the pre-release " + "or cut a new release" + ) + mirror_pairs = _mirror_subproject_pairs(release_path, args.project) + if getattr(args, "dry_run", False): + for name, _ in mirror_pairs: + _print_same(f"+ {release_obs_project}:{name}") + _print_ok( + f"release dry-run passed for {entry.tag} " + f"({from_project} → {release_obs_project}) (dry run)" + ) + return + apply_mirror_meta( + apiurl, release_obs_project, release_path, args.rootprj, shared_env_vars + ) + release_tree( + apiurl, + from_project, + release_obs_project, + mirror_pairs, + args.rootprj, + shared_env_vars, + verify_timeout=args.verify_timeout, + ) + _print_ok(f"release {from_project} → {release_obs_project} ({entry.tag})") + return + subproject_pairs, missing_mirrors = collect_release_subprojects( source_project_id, release_path ) @@ -2643,43 +2707,6 @@ def cmd_sync_release(args) -> None: _print_pending(f"checking release project {release_obs_project}") project_exists = _obs_project_exists(apiurl, release_obs_project) - from_project: "str | None" = getattr(args, "from_project", None) - if from_project: - # Hop 2 (spec §2): no staging involvement, no freeze, no git check. - entry = current_release(release_data) - if entry is None: - raise SystemExit("error: release.yaml has no releases: entries") - if not args.force: - if not entry.source_commit: - raise SystemExit( - f"error: release.yaml entry {entry.tag} has no source_commit; " - "re-run 'percona-obs project release' (or pass --force)" - ) - marker = read_project_description_marker( - apiurl, from_project, "source_commit" - ) - if marker != entry.source_commit: - raise SystemExit( - f"error: source_commit mismatch: {from_project} was populated from " - f"{marker or ''} but release.yaml {entry.tag} records " - f"{entry.source_commit or ''}; repopulate the pre-release " - "or cut a new release" - ) - apply_mirror_meta( - apiurl, release_obs_project, release_path, args.rootprj, shared_env_vars - ) - release_tree( - apiurl, - from_project, - release_obs_project, - subproject_pairs, - args.rootprj, - shared_env_vars, - verify_timeout=args.verify_timeout, - ) - _print_ok(f"release {from_project} → {release_obs_project} ({entry.tag})") - return - if not args.force: _check_source_unchanged_since_tag(source_project_id, tag, args) @@ -2736,6 +2763,11 @@ def _add(path: Path) -> None: if p.is_dir(): _add(p) project_dirs.append(p) + # Inherited config layers (project.yaml / subprojects.yaml up the chain). + for pd in list(project_dirs): + for d in _chain(pd): + _add(d / "project.yaml") + _add(d / "subprojects.yaml") # Package dirs that are symlinks into _shared/: git sees only the link # blob, so the link target must be a pathspec of its own. for d in project_dirs: diff --git a/percona_obs/release_copy.py b/percona_obs/release_copy.py index 425762dc9..98bf675fc 100644 --- a/percona_obs/release_copy.py +++ b/percona_obs/release_copy.py @@ -7,7 +7,7 @@ ``sync release`` (no ``--from``) uses the same primitives. ``osc release`` requires ```` entries on the -source, so a copy adds them, runs osc with ``--target-project`` and removes +source, so a copy adds them, runs ``osc release`` and removes them again in a ``finally`` — the source meta is unchanged at rest. """ @@ -134,6 +134,8 @@ def copy_project_binaries( apiurl: str, source_obs_project: str, target_obs_project: str ) -> None: """``osc release`` every repo of *source* that exists in *target*.""" + # Self-heal leftovers of a killed earlier run before reading the meta. + _remove_release_targets(apiurl, source_obs_project, target_obs_project) repo_elems, _ = _read_project_release_source(apiurl, source_obs_project) repo_names = filter_release_repo_names( apiurl, @@ -151,8 +153,6 @@ def copy_project_binaries( "release", source_obs_project, "--no-delay", - "--target-project", - target_obs_project, ], check=True, ) diff --git a/tests/test_list_qa_matrix.py b/tests/test_list_qa_matrix.py index 42faa3aff..5744cddd0 100644 --- a/tests/test_list_qa_matrix.py +++ b/tests/test_list_qa_matrix.py @@ -251,3 +251,11 @@ def test_qa_show_failure_is_skipped_in_nightly_mode(monkeypatch, capsys): captured = capsys.readouterr() assert captured.out.strip() == "[]" assert "warning: qa show" in captured.err + + +def test_is_release_tier(): + m = _load() + assert m.is_release_tier("a:ppg:releases:18") + assert m.is_release_tier("a:ppg:releases:18:containers") + assert not m.is_release_tier("a:ppg:staging:18") + assert not m.is_release_tier("a:ppg:devel:18:containers") diff --git a/tests/test_project_release.py b/tests/test_project_release.py index 9eb8be207..632bc65c4 100644 --- a/tests/test_project_release.py +++ b/tests/test_project_release.py @@ -794,10 +794,9 @@ def test_open_release_is_refreshed_in_place(monkeypatch): from percona_obs import cmd_project as cp from percona_obs.release_record import ReleaseEntry - monkeypatch.setattr(cp, "_tag_exists_locally", lambda tag: False) entries = [ReleaseEntry("ppg/18.4-1"), ReleaseEntry("ppg/18.6-1", "old")] rid, new_entries, prev = cp._plan_release_entries( - entries, None, lambda: "18.6-2", "ppg", "newsha" + entries, None, lambda: "18.6-2", "ppg", "newsha", lambda t: False ) assert rid == "18.6-1" assert new_entries == [ @@ -811,10 +810,9 @@ def test_shipped_release_appends_next_id(monkeypatch): from percona_obs import cmd_project as cp from percona_obs.release_record import ReleaseEntry - monkeypatch.setattr(cp, "_tag_exists_locally", lambda tag: True) entries = [ReleaseEntry("ppg/18.6-1", "old")] rid, new_entries, prev = cp._plan_release_entries( - entries, None, lambda: "18.6-2", "ppg", "newsha" + entries, None, lambda: "18.6-2", "ppg", "newsha", lambda t: True ) assert rid == "18.6-2" assert new_entries == [ @@ -828,11 +826,37 @@ def test_explicit_release_id_wins_over_refresh(monkeypatch): from percona_obs import cmd_project as cp from percona_obs.release_record import ReleaseEntry - monkeypatch.setattr(cp, "_tag_exists_locally", lambda tag: False) entries = [ReleaseEntry("ppg/18.6-1")] with pytest.raises(SystemExit, match="already present"): - cp._plan_release_entries(entries, "18.6-1", lambda: "x", "ppg", "s") + cp._plan_release_entries( + entries, "18.6-1", lambda: "x", "ppg", "s", lambda t: False + ) rid, new_entries, _ = cp._plan_release_entries( - entries, "18.6-9", lambda: "x", "ppg", "s" + entries, "18.6-9", lambda: "x", "ppg", "s", lambda t: False ) assert rid == "18.6-9" and new_entries[-1] == ReleaseEntry("ppg/18.6-9", "s") + + +def test_tag_exists_on_remote(monkeypatch): + from types import SimpleNamespace + + from percona_obs import cmd_project as cp + + monkeypatch.setattr( + cp.subprocess, + "run", + lambda *a, **k: SimpleNamespace(returncode=0, stdout="abc\trefs/tags/t\n"), + ) + assert cp._tag_exists_on_remote("u", "t") is True + monkeypatch.setattr( + cp.subprocess, + "run", + lambda *a, **k: SimpleNamespace(returncode=0, stdout=""), + ) + assert cp._tag_exists_on_remote("u", "t") is False + monkeypatch.setattr( + cp.subprocess, + "run", + lambda *a, **k: SimpleNamespace(returncode=1, stdout="", stderr="boom"), + ) + assert cp._tag_exists_on_remote("u", "t") is False diff --git a/tests/test_release_copy.py b/tests/test_release_copy.py index f1dcde331..9aab9369c 100644 --- a/tests/test_release_copy.py +++ b/tests/test_release_copy.py @@ -31,7 +31,7 @@ def _stub_obs(monkeypatch, calls): monkeypatch.setattr(rc, "_fetch_obs_subproject_names", lambda apiurl, prefix: set()) -def test_copy_project_binaries_uses_target_project_and_restores(monkeypatch): +def test_copy_project_binaries_restores_targets(monkeypatch): calls = [] _stub_obs(monkeypatch, calls) seen = {} @@ -39,9 +39,9 @@ def test_copy_project_binaries_uses_target_project_and_restores(monkeypatch): rc.subprocess, "run", lambda cmd, **k: seen.setdefault("cmd", cmd) ) rc.copy_project_binaries("http://obs", "S", "T") - assert "--target-project" in seen["cmd"] and "T" in seen["cmd"] + assert "--target-project" not in seen["cmd"] assert seen["cmd"][seen["cmd"].index("release") + 1] == "S" - assert calls == [("add", "S", "T"), ("remove", "S", "T")] + assert calls == [("remove", "S", "T"), ("add", "S", "T"), ("remove", "S", "T")] def test_copy_project_binaries_restores_on_failure(monkeypatch): diff --git a/tests/test_sync_prerelease.py b/tests/test_sync_prerelease.py index 3fe3a6143..ee02fefe6 100644 --- a/tests/test_sync_prerelease.py +++ b/tests/test_sync_prerelease.py @@ -182,6 +182,8 @@ def test_scope_local_paths_maps_obs_names_back(tmp_path, monkeypatch): assert src in paths assert src.parents[2] / "ppg/common/deps/geos" in paths assert any(p.name == "macros.yaml" for p in paths) + assert src.parent / "project.yaml" in paths + assert src.parents[1] / "project.yaml" in paths def test_shared_symlink_target_edit_is_detected(tmp_path, monkeypatch): diff --git a/tests/test_sync_release.py b/tests/test_sync_release.py index 11ba99324..6f8426498 100644 --- a/tests/test_sync_release.py +++ b/tests/test_sync_release.py @@ -520,3 +520,52 @@ def test_from_prerelease_force_skips_provenance_checks(tmp_path, monkeypatch): ) cmd_sync.cmd_sync_release(_args_from(rel, force=True)) assert len(metas) == 1 and len(copies) == 1 + + +def test_mirror_subproject_pairs(tmp_path, monkeypatch): + src, rel = _mk_tree(tmp_path, monkeypatch) + (rel / "tarballs").mkdir() + (rel / "tarballs" / "project.yaml").write_text("build: false\n") + (rel / "empty").mkdir() # no project.yaml: ignored + monkeypatch.setattr( + cmd_sync, "project_in_slice", lambda p, *a, **k: p.name != "tarballs" + ) + pairs = cmd_sync._mirror_subproject_pairs(rel, "ppg:releases:17") + assert [(n, p.name) for n, p in pairs] == [("containers", "containers")] + + +def test_from_prerelease_proceeds_without_staging_mirror(tmp_path, monkeypatch): + # staging has a tarballs subproject, the release mirror does not + src, rel = _mk_tree(tmp_path, monkeypatch) + _mapping_release_yaml(rel) + _wire_release_update_path(monkeypatch, src, rel) + monkeypatch.setattr( + cmd_sync, "read_project_description_marker", lambda a, p, k: "b" * 40 + ) + monkeypatch.setattr(cmd_sync, "apply_mirror_meta", lambda *a, **k: None) + got = [] + monkeypatch.setattr( + cmd_sync, "release_tree", lambda a, s, t, pairs, *r, **k: got.append(pairs) + ) + cmd_sync.cmd_sync_release(_args_from(rel)) + assert [n for n, _ in got[0]] == ["containers"] + + +def test_from_prerelease_dry_run_writes_nothing(tmp_path, monkeypatch, capsys): + src, rel = _mk_tree(tmp_path, monkeypatch) + _mapping_release_yaml(rel) + _wire_release_update_path(monkeypatch, src, rel) + monkeypatch.setattr( + cmd_sync, "read_project_description_marker", lambda a, p, k: "b" * 40 + ) + monkeypatch.setattr( + cmd_sync, "apply_mirror_meta", lambda *a, **k: pytest.fail("no write") + ) + monkeypatch.setattr( + cmd_sync, "release_tree", lambda *a, **k: pytest.fail("no write") + ) + monkeypatch.setattr( + cmd_sync, "assert_all_green", lambda *a, **k: pytest.fail("no green check") + ) + cmd_sync.cmd_sync_release(_args_from(rel, dry_run=True)) + assert "containers" in capsys.readouterr().out From cf4d023cb4b17570419fde8f29810d76a2a8b280 Mon Sep 17 00:00:00 2001 From: Ricardo Dias Date: Tue, 13 Oct 2026 16:29:15 +0100 Subject: [PATCH 16/16] docs: mark the pre-release flow tool plan tasks complete Signed-off-by: Ricardo Dias --- ...6-10-13-prerelease-flow-tool.md.tasks.json | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/superpowers/plans/2026-10-13-prerelease-flow-tool.md.tasks.json b/docs/superpowers/plans/2026-10-13-prerelease-flow-tool.md.tasks.json index 429d1859a..525e0ff95 100644 --- a/docs/superpowers/plans/2026-10-13-prerelease-flow-tool.md.tasks.json +++ b/docs/superpowers/plans/2026-10-13-prerelease-flow-tool.md.tasks.json @@ -4,13 +4,13 @@ { "id": 0, "subject": "Task 0: Worktree environment and baseline", - "status": "pending", + "status": "completed", "description": "**Goal:** A working `venv/` in the worktree and a green baseline so later failures are attributable.\n\n**Files:**\n- Create: `venv/` (git-ignored) in `.claude/worktrees/prerelease-flow`\n\n**Acceptance Criteria:**\n- [ ] `venv/bin/pytest -q` passes on the untouched branch\n- [ ] `venv/bin/black --check percona_obs/` and `venv/bin/pyright` pass\n\n**Verify:** `venv/bin/pytest -q && venv/bin/black --check percona_obs/ && venv/bin/pyright` \u2192 all green, pyright prints `0 errors`\n\n```json:metadata\n{\"files\": [], \"verifyCommand\": \"venv/bin/pytest -q && venv/bin/black --check percona_obs/ && venv/bin/pyright\", \"acceptanceCriteria\": [\"`venv/bin/pytest -q` passes on the untouched branch\", \"`venv/bin/black --check percona_obs/` and `venv/bin/pyright` pass\"], \"modelTier\": \"mechanical\"}\n```" }, { "id": 1, "subject": "Task 1: `release_record.py` and mapping-form `release.yaml` entries", - "status": "pending", + "status": "completed", "description": "**Goal:** One parser for `releases:` entries used by every reader (`compute_ppg_release`, `project release`, `sync release`, the cleanup workflow one-liner), accepting strings, `{tag, source_commit}` mappings and legacy `revision:`.\n\n**Files:**\n- Create: `percona_obs/release_record.py`\n- Modify: `percona_obs/common.py:319-346` (`compute_ppg_release`)\n- Modify: `percona_obs/cmd_project.py:2293-2300` (existing release list)\n- Modify: `percona_obs/cmd_sync.py:2628-2635` (release list in `cmd_sync_release`)\n- Modify: `.github/workflows/obs-pr-cleanup.yml:138`\n- Test: `tests/test_release_record.py`, `tests/test_computed_macros.py`\n\n**Acceptance Criteria:**\n- [ ] `parse_release_entries` returns `ReleaseEntry(tag, source_commit)` for both forms and raises `SystemExit` naming an unrecognised entry\n- [ ] `compute_ppg_release` counts a mapping entry's tag (`{tag: ppg/18.4-2}` counts for 18.4)\n- [ ] `project release` and `sync release` still work with string-only files (existing tests pass)\n- [ ] The cleanup one-liner prints the tag for a mapping last entry\n\n**Verify:** `venv/bin/pytest -q tests/test_release_record.py tests/test_computed_macros.py tests/test_project_release.py tests/test_sync_release.py` \u2192 PASS\n\n```json:metadata\n{\"files\": [\"percona_obs/release_record.py\", \"tests/test_computed_macros.py\", \"tests/test_release_record.py\"], \"verifyCommand\": \"venv/bin/pytest -q tests/test_release_record.py tests/test_computed_macros.py tests/test_project_release.py tests/test_sync_release.py\", \"acceptanceCriteria\": [\"`parse_release_entries` returns `ReleaseEntry(tag, source_commit)` for both forms and raises `SystemExit` naming an unrecognised entry\", \"`compute_ppg_release` counts a mapping entry's tag (`{tag: ppg/18.4-2}` counts for 18.4)\", \"`project release` and `sync release` still work with string-only files (existing tests pass)\", \"The cleanup one-liner prints the tag for a mapping last entry\"], \"modelTier\": \"standard\"}\n```", "blockedBy": [ 0 @@ -19,7 +19,7 @@ { "id": 2, "subject": "Task 2: `github_api.py` \u2014 last synced head SHA", - "status": "pending", + "status": "completed", "description": "**Goal:** Resolve \"the commit of `main` whose staging tree OBS currently holds\" as the head SHA of the newest completed `sync-main.yml` run on `main` whose sync jobs all concluded `success`, through `gh api`.\n\n**Files:**\n- Create: `percona_obs/github_api.py`\n- Test: `tests/test_github_api.py`\n\n**Acceptance Criteria:**\n- [ ] `repo_slug_from_url` handles `https://github.com/percona/obs-packaging.git`, `https://github.com/percona/obs-packaging`, `git@github.com:percona/obs-packaging.git`\n- [ ] `last_synced_head_sha` skips a run whose `percona-obs sync push (...)` job failed even if the run's other jobs passed, and returns the next older qualifying run's `head_sha`\n- [ ] A `gh` failure raises `SystemExit` with the stderr text; no qualifying run raises `SystemExit`\n\n**Verify:** `venv/bin/pytest -q tests/test_github_api.py` \u2192 PASS\n\n```json:metadata\n{\"files\": [\"percona_obs/github_api.py\", \"tests/test_github_api.py\"], \"verifyCommand\": \"venv/bin/pytest -q tests/test_github_api.py\", \"acceptanceCriteria\": [\"`repo_slug_from_url` handles `https://github.com/percona/obs-packaging.git`, `https://github.com/percona/obs-packaging`, `git@github.com:percona/obs-packaging.git`\", \"`last_synced_head_sha` skips a run whose `percona-obs sync push (...)` job failed even if the run's other jobs passed, and returns the next older qualifying run's `head_sha`\", \"A `gh` failure raises `SystemExit` with the stderr text; no qualifying run raises `SystemExit`\"], \"modelTier\": \"standard\"}\n```", "blockedBy": [ 0 @@ -28,7 +28,7 @@ { "id": 3, "subject": "Task 3: `release_copy.py` and `cmd_sync_release` refactor (behaviour unchanged)", - "status": "pending", + "status": "completed", "description": "**Goal:** Move the copy mechanics out of `cmd_sync.py` into `percona_obs/release_copy.py` and make `cmd_sync_release` a thin orchestration with a single code path for first and update releases. Behaviour must be identical; the existing tests prove it after their monkeypatch targets move.\n\n**Files:**\n- Create: `percona_obs/release_copy.py`\n- Modify: `percona_obs/cmd_sync.py` (`_collect_release_subprojects`, `_filter_release_repo_names`, `_sync_release_subprojects` removed; `cmd_sync_release` body)\n- Test: `tests/test_sync_release.py` (rewired), `tests/test_release_copy.py` (new)\n\n**Acceptance Criteria:**\n- [ ] `release_copy.copy_project_binaries` adds release targets, runs `osc release --no-delay --target-project `, removes targets in `finally` even when osc fails\n- [ ] `release_copy.release_tree` copies top then each pair, applies each pair's mirror meta before its copy, verifies when `verify_timeout > 0`, warns about orphans\n- [ ] `cmd_sync_release` first-release and update paths both run the freeze sequence and the same `release_tree`; all pre-existing `test_sync_release.py` assertions hold\n\n**Verify:** `venv/bin/pytest -q tests/test_sync_release.py tests/test_release_copy.py` \u2192 PASS\n\n```json:metadata\n{\"files\": [\"percona_obs/cmd_sync.py\", \"percona_obs/release_copy.py\", \"tests/test_release_copy.py\", \"tests/test_sync_release.py\"], \"verifyCommand\": \"venv/bin/pytest -q tests/test_sync_release.py tests/test_release_copy.py\", \"acceptanceCriteria\": [\"`release_copy.copy_project_binaries` adds release targets, runs `osc release --no-delay --target-project `, removes targets in `finally` even when osc fails\", \"`release_copy.release_tree` copies top then each pair, applies each pair's mirror meta before its copy, verifies when `verify_timeout > 0`, warns about orphans\", \"`cmd_sync_release` first-release and update paths both run the freeze sequence and the same `release_tree`; all pre-existing `test_sync_release.py` assertions hold\"], \"modelTier\": \"standard\"}\n```", "blockedBy": [ 1 @@ -37,7 +37,7 @@ { "id": 4, "subject": "Task 4: `obs_api` \u2014 publish override and description marker", - "status": "pending", + "status": "completed", "description": "**Goal:** Two small OBS-side primitives the new hops need: force-publish on a project built from a mirror, and a `source_commit:` marker in a project description.\n\n**Files:**\n- Modify: `percona_obs/obs_api.py:1215-1280` (`_apply_project_config`), new functions after `_disable_project_builds`\n- Test: `tests/test_obs_api_markers.py`\n\n**Acceptance Criteria:**\n- [ ] `_apply_project_config(..., publish_override=True)` emits `` regardless of the mirror's `publish:`; `None` keeps today's behaviour\n- [ ] `set_project_description_marker` appends `source_commit: ` once (replacing an existing marker line) and pushes the meta\n- [ ] `read_project_description_marker` returns the value or `None`\n\n**Verify:** `venv/bin/pytest -q tests/test_obs_api_markers.py` \u2192 PASS\n\n```json:metadata\n{\"files\": [\"tests/test_obs_api_markers.py\"], \"verifyCommand\": \"venv/bin/pytest -q tests/test_obs_api_markers.py\", \"acceptanceCriteria\": [\"`_apply_project_config(..., publish_override=True)` emits `` regardless of the mirror's `publish:`; `None` keeps today's behaviour\", \"`set_project_description_marker` appends `source_commit: ` once (replacing an existing marker line) and pushes the meta\", \"`read_project_description_marker` returns the value or `None`\"], \"modelTier\": \"standard\"}\n```", "blockedBy": [ 3 @@ -46,7 +46,7 @@ { "id": 5, "subject": "Task 5: `sync prerelease` (hop 1)", - "status": "pending", + "status": "completed", "description": "**Goal:** New command that, under the PR profile, checks the `source_commit` record against the last synced `main`, creates/re-applies the pre-release tree from the mirrors (publish forced on), freezes staging, copies, verifies, restores, and pins `source_commit` in the pre-release description.\n\n**Files:**\n- Modify: `percona_obs/cmd_sync.py` (new `cmd_sync_prerelease`, helpers `_scope_local_paths`, `_changed_files_between`)\n- Modify: `percona_obs/cli.py` (new `sync prerelease` parser after `sync release`)\n- Test: `tests/test_sync_prerelease.py`\n\n**Acceptance Criteria:**\n- [ ] Without a mapping entry carrying `source_commit`, the command fails with \"has no source_commit\"\n- [ ] A non-empty diff of scope paths between `source_commit` and the synced head aborts before `wait_for_quiesce` is called, listing the files\n- [ ] Order of operations: record check \u2192 top meta (publish True) \u2192 drain \u2192 green \u2192 freeze \u2192 copy tree \u2192 restore \u2192 marker; marker is not written when the copy raised\n- [ ] `--dry-run` runs the record check and the green check, writes nothing\n- [ ] `--synced-commit` bypasses `gh`; `--force` bypasses the record check only\n\n**Verify:** `venv/bin/pytest -q tests/test_sync_prerelease.py` \u2192 PASS\n\n```json:metadata\n{\"files\": [\"percona_obs/cli.py\", \"percona_obs/cmd_sync.py\", \"tests/test_sync_prerelease.py\"], \"verifyCommand\": \"venv/bin/pytest -q tests/test_sync_prerelease.py\", \"acceptanceCriteria\": [\"Without a mapping entry carrying `source_commit`, the command fails with \\\"has no source_commit\\\"\", \"A non-empty diff of scope paths between `source_commit` and the synced head aborts before `wait_for_quiesce` is called, listing the files\", \"Order of operations: record check \\u2192 top meta (publish True) \\u2192 drain \\u2192 green \\u2192 freeze \\u2192 copy tree \\u2192 restore \\u2192 marker; marker is not written when the copy raised\", \"`--dry-run` runs the record check and the green check, writes nothing\", \"`--synced-commit` bypasses `gh`; `--force` bypasses the record check only\"], \"modelTier\": \"standard\"}\n```", "blockedBy": [ 2, @@ -56,7 +56,7 @@ { "id": 6, "subject": "Task 6: `sync release --from ` (hop 2)", - "status": "pending", + "status": "completed", "description": "**Goal:** With `--from`, `sync release` skips the freeze and the git divergence check, verifies the pre-release's `source_commit` marker against the tagged `release.yaml`, applies the mirror meta to the release tree and copies from the pre-release.\n\n**Files:**\n- Modify: `percona_obs/cmd_sync.py` (`cmd_sync_release`)\n- Modify: `percona_obs/cli.py` (`--from` on `sync release`)\n- Test: `tests/test_sync_release.py`\n\n**Acceptance Criteria:**\n- [ ] `--from X`: no `wait_for_quiesce`/`freeze_builds` call, no git diff, `release_tree(apiurl, \"X\", , ...)` called once, meta applied from mirrors for a new release project too (not `_create_release_project`)\n- [ ] Marker mismatch \u2192 `SystemExit(\"source_commit mismatch\")` before any meta write; `--force` skips the check\n- [ ] Without `--from` behaviour is Task 3's\n\n**Verify:** `venv/bin/pytest -q tests/test_sync_release.py -k from_prerelease` \u2192 PASS\n\n```json:metadata\n{\"files\": [\"percona_obs/cli.py\", \"percona_obs/cmd_sync.py\", \"tests/test_sync_release.py\"], \"verifyCommand\": \"venv/bin/pytest -q tests/test_sync_release.py -k from_prerelease\", \"acceptanceCriteria\": [\"`--from X`: no `wait_for_quiesce`/`freeze_builds` call, no git diff, `release_tree(apiurl, \\\"X\\\", , ...)` called once, meta applied from mirrors for a new release project too (not `_create_release_project`)\", \"Marker mismatch \\u2192 `SystemExit(\\\"source_commit mismatch\\\")` before any meta write; `--force` skips the check\", \"Without `--from` behaviour is Task 3's\"], \"modelTier\": \"standard\"}\n```", "blockedBy": [ 5 @@ -65,7 +65,7 @@ { "id": 7, "subject": "Task 7: `project release` carries rendered `qa:` blocks into the mirrors", - "status": "pending", + "status": "completed", "description": "**Goal:** Every mirror `project.yaml` (top and subprojects) gets the staging project's `qa:` block with `%!{\u2026}` macros rendered and `:staging:` references rewritten to the release project, leaving `${\u2026}` env tokens intact.\n\n**Files:**\n- Modify: `percona_obs/cmd_project.py` (`_write_release_tree`, new `_rewrite_qa_project_refs`, `cmd_project_release` top-level `project_data`)\n- Test: `tests/test_project_release.py`\n\n**Acceptance Criteria:**\n- [ ] A staging subproject with `qa: [{pipeline: p, parameters: {OBS_PROJECT: \"${OBS_ROOTPRJ}:ppg:staging:17:containers\", VERSION: \"ppg-%!{PG_VERSION}\"}}]` and a `macros.yaml` defining `PG_VERSION: 17.11` yields a mirror `qa:` with `OBS_PROJECT: \"${OBS_ROOTPRJ}:ppg:releases:17:containers\"` and `VERSION: ppg-17.11`\n- [ ] A staging project without `qa:` yields a mirror without `qa:`\n- [ ] The top-level mirror gets the top-level staging `qa:` the same way\n\n**Verify:** `venv/bin/pytest -q tests/test_project_release.py -k qa` \u2192 PASS\n\n```json:metadata\n{\"files\": [\"percona_obs/cmd_project.py\", \"tests/test_project_release.py\"], \"verifyCommand\": \"venv/bin/pytest -q tests/test_project_release.py -k qa\", \"acceptanceCriteria\": [\"A staging subproject with `qa: [{pipeline: p, parameters: {OBS_PROJECT: \\\"${OBS_ROOTPRJ}:ppg:staging:17:containers\\\", VERSION: \\\"ppg-%!{PG_VERSION}\\\"}}]` and a `macros.yaml` defining `PG_VERSION: 17.11` yields a mirror `qa:` with `OBS_PROJECT: \\\"${OBS_ROOTPRJ}:ppg:releases:17:containers\\\"` and `VERSION: ppg-17.11`\", \"A staging project without `qa:` yields a mirror without `qa:`\", \"The top-level mirror gets the top-level staging `qa:` the same way\"], \"modelTier\": \"standard\"}\n```", "blockedBy": [ 1 @@ -74,7 +74,7 @@ { "id": 8, "subject": "Task 8: `project release` \u2014 `source_commit` and open-release refresh", - "status": "pending", + "status": "completed", "description": "**Goal:** `project release` records `source_commit` (last synced `main` via `gh`, or `--source-commit`) in a mapping entry, refuses a SHA that is not an ancestor of HEAD, and when the last entry's tag has no git tag yet it refreshes that release in place (same id, CHANGELOG section replaced, mirrors regenerated, `source_commit` updated).\n\n**Files:**\n- Modify: `percona_obs/cmd_project.py` (`cmd_project_release`, new helpers `_resolve_source_commit`, `_tag_exists_locally`, `_replace_changelog_section`)\n- Modify: `percona_obs/cli.py` (`--source-commit` on `project release`)\n- Test: `tests/test_project_release.py`\n\n**Acceptance Criteria:**\n- [ ] `_replace_changelog_section(text, \"18.6-1\", new)` replaces exactly the `## [18.6-1]` section and keeps the others; when absent it prepends like the update path does today\n- [ ] `_resolve_source_commit` returns the override when given, else the `gh` lookup; raises when the SHA is not an ancestor of HEAD\n- [ ] With last entry `ppg/18.6-1` and no local tag, the derived id is `18.6-1` (refresh) and `release.yaml` ends with `{tag: ppg/18.6-1, source_commit: }` as the only `18.6-1` entry\n- [ ] With the local tag present, a new id is derived and appended as a mapping entry; older entries keep their original form\n\n**Verify:** `venv/bin/pytest -q tests/test_project_release.py -k \"refresh or source_commit or changelog_section\"` \u2192 PASS\n\n```json:metadata\n{\"files\": [\"percona_obs/cli.py\", \"percona_obs/cmd_project.py\", \"tests/test_project_release.py\"], \"verifyCommand\": \"venv/bin/pytest -q tests/test_project_release.py -k \\\"refresh or source_commit or changelog_section\\\"\", \"acceptanceCriteria\": [\"`_replace_changelog_section(text, \\\"18.6-1\\\", new)` replaces exactly the `## [18.6-1]` section and keeps the others; when absent it prepends like the update path does today\", \"`_resolve_source_commit` returns the override when given, else the `gh` lookup; raises when the SHA is not an ancestor of HEAD\", \"With last entry `ppg/18.6-1` and no local tag, the derived id is `18.6-1` (refresh) and `release.yaml` ends with `{tag: ppg/18.6-1, source_commit: }` as the only `18.6-1` entry\", \"With the local tag present, a new id is derived and appended as a mapping entry; older entries keep their original form\"], \"modelTier\": \"standard\"}\n```", "blockedBy": [ 7, @@ -84,7 +84,7 @@ { "id": 9, "subject": "Task 9: Documentation and final verification", - "status": "pending", + "status": "completed", "description": "**Goal:** The tool reference, the architecture reference and the key-files tables describe the two commands, the `release.yaml` mapping form and the refresh rule; the whole suite and type-check pass on the final tree.\n\n**Files:**\n- Modify: `docs/PERCONA_OBS_TOOL.md` (\"Releasing packages\" chapter: Step 1, `release.yaml` format, \"Running `sync release` manually\", new \"`sync prerelease`\" section)\n- Modify: `.github/copilot-instructions.md` (key files table; a short \"Release hops\" note next to the `sync release-pr` section; the obs-pr-cleanup one-liner mention)\n- Modify: `CLAUDE.md` (key files table: `release_copy.py`, `release_record.py`, `github_api.py`)\n\n**Acceptance Criteria:**\n- [ ] `docs/PERCONA_OBS_TOOL.md` documents: `--source-commit`, the refresh rule (\"last entry without a git tag is open\"), the mapping entry form, `sync prerelease` flags (`--branch-from`, `--synced-commit`, `--source-commit`, `--force`, `--no-freeze`, `--freeze-timeout`, `--verify-timeout`, `--dry-run`), `sync release --from`, and that the workflow integration follows in rollout step 2\n- [ ] Key-file tables list the three new modules\n- [ ] `venv/bin/pytest -q`, `venv/bin/black --check percona_obs/`, `venv/bin/pyright` all pass\n\n**Verify:** `venv/bin/pytest -q && venv/bin/black --check percona_obs/ && venv/bin/pyright && grep -c \"sync prerelease\" docs/PERCONA_OBS_TOOL.md` \u2192 green, count \u2265 3\n\n```json:metadata\n{\"files\": [\".github/copilot-instructions.md\", \"CLAUDE.md\", \"docs/PERCONA_OBS_TOOL.md\", \"github_api.py\", \"release_copy.py\", \"release_record.py\"], \"verifyCommand\": \"venv/bin/pytest -q && venv/bin/black --check percona_obs/ && venv/bin/pyright && grep -c \\\"sync prerelease\\\" docs/PERCONA_OBS_TOOL.md\", \"acceptanceCriteria\": [\"`docs/PERCONA_OBS_TOOL.md` documents: `--source-commit`, the refresh rule (\\\"last entry without a git tag is open\\\"), the mapping entry form, `sync prerelease` flags (`--branch-from`, `--synced-commit`, `--source-commit`, `--force`, `--no-freeze`, `--freeze-timeout`, `--verify-timeout`, `--dry-run`), `sync release --from`, and that the workflow integration follows in rollout step 2\", \"Key-file tables list the three new modules\", \"`venv/bin/pytest -q`, `venv/bin/black --check percona_obs/`, `venv/bin/pyright` all pass\"], \"modelTier\": \"mechanical\"}\n```", "blockedBy": [ 6, @@ -92,5 +92,5 @@ ] } ], - "lastUpdated": "2026-10-13T14:27:40.532018+00:00" + "lastUpdated": "2026-10-13T15:16:04.121336+00:00" }