Research which PMCs are currently enabled to release via CI and what types of artifacts

[dave2wave]

Please research which PMCs are currently enabled for CI and list for each their latest releases by project and their artifacts - how many source and how many binary.

[potiuk]

can we add magpie PMC to that?

I would love to use @ppkarwasz action to release Magpie and facilitate reproducibility checks for our source artifact.

[dave2wave]

We follow security's procedures which is (1) security accepts your claim that you create reproducible binaries, (2) Infra issues you an OpenPGP key pair and places the private key in GH, and (3) the project places the public key in their KEYS file (or enable ATR to manage your KEYS file and have an admin add the key.) If you do all three you will be enabled.

However @ppkarwasz action may not be approved as is since building a release artifact is out of ATR's scope.

[dave2wave]

It's up to the Security and Infra to change the bit-for-bit requirement expressed here: https://infra.apache.org/release-signing.html#automated-release-signing

[potiuk]

We follow security's procedures which is (1) security accepts your claim that you create reproducible binaries, (2) Infra issues you an OpenPGP key pair and places the private key in GH, and (3) the project places the public key in their KEYS file (or enable ATR to manage your KEYS file and have an admin add the key.) If you do all three you will be enabled.

Oh absolutely. I am happy to follow the procedure. I can even build my own action to do it - it's not as super necessary to get a common action, but it would be nice.

[potiuk]

You might be right - iit might not have to be good for central "tooling" action @dave2wave. But what I will do instead - I am encoding all that in Magpie's release family skills - so that anyone using might either add it for manual reproducible source package preparation and adding "ASF specific" skill where people will be able to apply for reproducibility using an action that Magpie will add them - this way we will leave it in the hands of the PMCs who might want to use Magpie to drive their releases.

[dave2wave]

It may be that it is easy to accommodate certain build types, but every project differs and some have very elaborate builds. I know of at lease one case where attempts to change the build from ant to mvn was a one or two PMC members against the whole PMC, the results were truly awful.

Another part of the hard problem of builds is determining and capturing all of the binary conveniences and other distribution platforms. The maven team has volunteered an ATR plugin

I do see value in providing a way in tooling to sign and checksum every artifact in a stash before doing all of the constituent parts of composing an atr release.

[potiuk]

It may be that it is easy to accommodate certain build types, but every project differs and some have very elaborate builds. I know of at lease one case where attempts to change the build from ant to mvn was a one or two PMC members against the whole PMC, the results were truly awful.

Another part of the hard problem of builds is determining and capturing all of the binary conveniences and other distribution platforms. The maven team has volunteered an ATR plugin

I do see value in providing a way in tooling to sign and checksum every artifact in a stash before doing all of the constituent parts of composing an atr release.

Yeah. That's why Magpie is probably way better for it. Magpie - by definition - treats all SKILLs as "base" and each project will define (and Magpie will even automatically propose it) the overrides that will be specific for the project. For example when project already has release process, Magpie adoption will find out by reading it, and will propose an override - especially that our release process as of now is very explicit about having optional binary artifacts and the need to decide about it during adoption.

Similarly as we have binary reproduction as an optional step in the release process. This is where agentic execution of the process via SKILL and built-in flexibility pays off.

It's the first time in history of developer tools that I developed, where we can both - provide a tool, but not bind the users with it, allowing them not only to diverge, but also our own tool detecting when divergence is needed and proposing it to the user - where user automatically becomes a co-creator of the tool they get - without impacting other users of the tool - and even gets the possibility of upstreaming the changes if they are generic ones - straight from the tool.

If there is a definition of "have cake and eat it too" - that would be it.

See apache/magpie#1296

[sbp]

The twelve that ATR currently knows about are: Polaris, Cordova, Pekko, Arrow, Grails, Daffodil, Directory, Baremaps (incubating), KIE (incubating), OpenDAL, Beam, and Logging Services. This is the list from the new reproducible builds tab in the admin catalogue dashboard. You must be an admin to see this tab.

There may be others whose keys haven't been allocated yet or aren't in ATR.

[potiuk]

Yeah. I will do Magpie "regular" reproducible release first - and then for the next one I will start switching over to CI releases - both for Magpie and Airflow and apply for reproducible CI releases there. We are fully reproducible in Airflow for years now.

[potiuk]

It's a bit risky to start with Airflow given our complexity - so doing it for Magpie first seems like a good idea.

[dave2wave]

The twelve that ATR currently knows about are: Polaris, Cordova, Pekko, Arrow, Grails, Daffodil, Directory, Baremaps (incubating), KIE (incubating), OpenDAL, Beam, and Logging Services. This is the list from the new reproducible builds tab in the admin catalogue dashboard. You must be an admin to see this tab.

We will have further details about artifacts signed by such keys tomorrow. With that information we can understand if adding a source release build action could be in scope for ATR.

[sbp]

Current releases by automated release committees on downloads.apache.org, as of today, 22nd September 2026:

Committee	Releases	Binary	Source only
Arrow	12	0	12
Baremaps	0	0	0
Beam	17	2	15
Cordova	53	53	0
Daffodil	3	2	1
Directory	11	9	2
Grails	8	4	4
KIE	3	3	0
Logging Services	16	14	2
OpenDAL	11	0	11
Pekko	20	0	20
Polaris	3	3	0
Total	157	90	67

Source only (3): Arrow, OpenDAL, Pekko.

Binary only (3): Cordova, KIE, Polaris.

Others (6): Baremaps (no releases), Beam (mostly source), Daffodil (mix), Directory (mostly binary), Grails (mix), Logging Services (mostly binary).

These are rough counts, using ATR's filename heuristics and a few fixups, just to inform this conversation as quickly as possible. More details coming. I will also implement the full artifacts-by-key catalogue in ATR.

[sbp]

And here's the subset of packages that were only signed by the automated release keys:

Committee	Releases	Binary	Source only
Arrow	1	0	1
Baremaps	0	0	0
Beam	2	2	0
Cordova	6	6	0
Daffodil	3	2	1
Directory	0	0	0
Grails	8	4	4
KIE	3	3	0
Logging Services	10	8	2
OpenDAL	2	0	2
Pekko	15	0	15
Polaris	2	2	0
Total	52	27	25

Source only (3): Arrow, OpenDAL, Pekko.

Binary only (4): Beam, Cordova, KIE, Polaris.

Others (5): Baremaps (no releases), Directory (no releases signed by its automated key), Daffodil (mix), Grails (mix), Logging Services (mostly binary).