DOC/REL: update docs on how release tags are signed

[rgommers]

We've used PGP signing for releases to date, with the exact keys published in the docs. Unfortunately that sets an expectation that takes work and doc updates to keep - for very little extra real-world protection. So drop that going forward, and instead explain how to check signing.

See https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification#ssh-commit-signature-verification for relevant docs.

Also amend the 0.21.0 Changelog entry for this change, and fix the release date in the Changelog while we're at it.

We've used PGP signing for releases to date. Unfortunately that has poor usability, and it doesn't add much real-world protection especially if keys aren't exercised regularly, don't have a broad web of trust, and cannot be revoked.

The poor usability now bit us concretely - I don't have access to the machine I normally do releases from, so the key in the docs isn't available. This isn't fixable on a reasonable time-scale. So, do the obvious thing (short of stopping signing completely) and switch to SSH-signed tags. Those are much easier to use, and in case of new machines, lost keys, etc. a maintainer can easily create a new key and use the features GitHub offers to make it a signing key. The .github/allowed_signers file should then be updated for it.

@dnicolodi I realized this too late, sorry about that. We should merge this before pushing the release tag. If you want to add your own signing key as well, feel free to push to this PR (or add it later).

[eli-schwartz]

The poor usability now bit us concretely - I don't have access to the machine I normally do releases from, so the key in the docs isn't available. This isn't fixable on a reasonable time-scale. So, do the obvious thing (short of stopping signing completely) and switch to SSH-signed tags. Those are much easier to use, and in case of new machines, lost keys, etc. a maintainer can easily create a new key and use the features GitHub offers to make it a signing key. The .github/allowed_signers file should then be updated for it.

This argument doesn't really make sense. Both a PGP key and an SSH key are "cryptographic private key material shoved into a text file", both come with trivially usable subcommands to easily generate a new key, both can be trivially uploaded to GitHub and redownloaded as trusted account-linked key material.

Ssh signing has bad usability and equally bad tooling among people who verify signatures. And delegating trust to a file checked into git is inherently illogical -- any attack that could forge a commit and switch to new ssh signatures would also update the signing file.

"Don't trust an unidentified file with a security signature to also contain a field / unpacked file telling you what public key to trust" is only not the number one rule of cryptographic security because people assumed it went without saying, like "1+1 == 2" and "water is wet".

[eli-schwartz]

We've used PGP signing for releases to date. Unfortunately that has poor usability, and it doesn't add much real-world protection especially if keys aren't exercised regularly, don't have a broad web of trust, and cannot be revoked.

• web of trust is a cute theory but doesn't accurately model how people actually use it, which is actually to rely on the builtin "TOFU" model provided by PGP. Anyways, software signing has always relied quite heavily on TOFU by way of embedding a list of valid signing keys anywhere you check it. That trust list is generally scraped from "other sources the legitimate maintainer controls", and e.g. GitHub's associated PGP keys for a user account can be trusted to be reliable even if a PAT or a CI workflow compromise allowed an attacker to commit to a repository and create malicious tags, because adding ssh / pgp keys in the settings page is a much more trusted action than pushing.
• PGP keys can absolutely be revoked... unlike ssh keys, which can only be deleted / delisted leaving no real clue as to why and people tracking that ssh key in their own automation will usually not even know it was deleted (without significantly more effortful work). And you cannot allow an ssh key to be permitted to sign old releases and not new releases.

[rgommers]

PGP keys can absolutely be revoked

Yes I know - but you need access to the original key, so if that's on a temporarily shelved machine that I don't have access to, it cannot be done in practice.

I'm not interested in a PGP vs. SSH debate. I'm using PGP twice a year, for meson-python releases, and it's a pain, with little upside. SSH keys on the other hand I'm using constantly. It's just much nicer. And in case something went wrong with it, I'd actually notice, unlike PGP. So, switching to that.

[dnicolodi]

I am with @eli-schwartz on this. I don't think that switching to SSH keys for signing has any practical advantage offers much weaker protection that what is potentially allowed with a PGP key.

Those are much easier to use, and in case of new machines, lost keys, etc. a maintainer can easily create a new key and use the features GitHub offers to make it a signing key. The .github/allowed_signers file should then be updated for it.

I don't see how this is any different from generating a new PGP key and adding it to the documentation next to the old one with a note saying that the old one has been used till the last version and the new one will be used from the next version onward. AFAIK, doing this would have the exact same guarantees that adding the hash of an SSH key to a text file in the repository.

If the issue is that a new PGP key would not be signed by anyone, I am happy to sign your new PGP key if we find a minute any day to get into a video chat. I am sure @eli-schwartz could do the same.

[rgommers]

If the issue is that a new PGP key would not be signed by anyone

No, that's not the main issue. The main issue is usability: I do not have any PGP program installed at all, and there is no great way to do that on macOS. SSH you get from Apple, so nothing to trust and always installed for me. GnuPG or similar, not so much. The main recommendation seems to be Homebrew, which I deliberately did not install on this machine (reasons are both "don't want to trust, going for minimal trust here" and "don't like globally-installed tools"). The second recommendation is a non-free GUI program (https://gpgtools.org/), which currently has a banner at the top that only a beta version of that GUI program supports macOS 26 (out for quite a while now).

I don't think that switching to SSH keys for signing has any practical advantage offers much weaker protection that what is potentially allowed with a PGP key.

It has a major advantage, since it's actually installed and has up-to-date keys. Use of PGP and the way the docs are written has now delayed the release for 3 days, so it's self-evident that it does matter. I don't want a PGP key and the programs needed to manage it on my machine, it's extra friction and costing time I do not want to spend.

And much weaker protection isn't really true (given Eli said above it's trust-on-first-use anyway) nor really relevant - almost no popular Python projects do this, and I've never gotten a request for this - it just doesn't matter all that much.

What seems to be the essence of what we need is this:

1. Ensure tags are signed than unsigned
2. Using a key that shows the tag as "verified" by GitHub, and the key retrievable from a maintainer's signing keys
3. Adding a note in the docs that we do (1) and (2)

Anything else isn't really relevant.

If I were setting this up from scratch, I wouldn't add keys to about.rst nor add allowed_signers, and keep it minimal. I only extended it because it was set up this way originally; happy to trim this PR down.

[dnicolodi]

What seems to be the essence of what we need is this:

1. Ensure tags are signed than unsigned

EPARSE

2. Using a key that shows the tag as "verified" by GitHub, and the key retrievable from a maintainer's signing keys

Does this mean that the ultimate goal is just to have the green checkmark next to the tag in the GitHub interface?

3. Adding a note in the docs that we do (1) and (2)

Well, this is more a consequence of the decision of doing anything, more than a goal 🙂

We should agree on what we want to achieve indeed. However, I don't understand what you mean with the points above. I just found this talk https://www.youtube.com/watch?v=w9SN_15gUhc which seems very relevant, but I haven't had time to listen to it yet.

If I were setting this up from scratch, I wouldn't add keys to about.rst nor add allowed_signers, and keep it minimal. I only extended it because it was set up this way originally; happy to trim this PR down.

I would keep this as lean as possible too. I don't know why things were set up the way they are originally. I don't understand the role of the allowed_signers file. As Eli says, verifying the content of the repo against a file contained in the repo does not seem particularly useful.

The main recommendation seems to be Homebrew, which I deliberately did not install on this machine (reasons are both "don't want to trust, going for minimal trust here" and "don't like globally-installed tools")

It is very likely that you exceptions for this rule, unless you use only the software that comes with the macOS installation. For example, I believe you may have installed pixi, as a global tool. I don't see how that is different from installing Sequoia PGP.

[rgommers]

I don't know why things were set up the way they are originally. I

I don't either. That was Filipe, I just followed the existing pattern, which was always a bit of an annoyance, it's just worse now that I switched machines and am traveling.

Does this mean that the ultimate goal is just to have the green checkmark next to the tag in the GitHub interface?

Pretty much, yes. Nothing more is needed - the rest is just security theater. If we do this, we're already doing much better than most other projects (including the likes of our trusted dependency meson, which doesn't use signed tags and yolo's the release process to the extent that a none-any wheel has repeatedly gone missing).

Well, this is more a consequence of the decision of doing anything, more than a goal 🙂

Yeah the only reason to list it is because we already have docs that talk about this, so in case anyone happens to rely on the existing keys listed in the docs, they can see that we changed it on purpose. I wouldn't touch the docs if we didn't already have content in about.rst.

For example, I believe you may have installed pixi, as a global tool.

Actually I use pixi and uv in aarch64 containers for daily development, exactly because I wanted to stop trusting too many packages from PyPI and conda-forge. So even they aren't global.

It is very likely that you exceptions for this rule,

Yes, there are a few exceptions - as few as possible.

Take a step back please. You're arguing with me for no real reason - there's no need to be "right" here: PGP and SSH both serve the job equally well, so why insist I have to do extra work and install a tool I do not have installed nor want to?

We should agree on what we want to achieve indeed.

The key goal is this: releases are done with signed tags and trusted publishing from those signed tags, with both you and me able to make releases, efficiently. Not listing exact keys in the docs would be a practical conclusion that follows (but just linking signing keys of you and me, as retrievable from github), as well as allowing signing with both SSH and PGP.

Right now we're just in a poor situation: neither of us can do a release without breaking too-detailed expectations set in about.rst.

[dnicolodi]

Take a step back please. You're arguing with me for no real reason - there's no need to be "right" here: PGP and SSH both serve the job equally well, so why insist I have to do extra work and install a tool I do not have installed nor want to?

Sorry, I didn't want to argue or use a confrontational tone. However, I had the impression that we had a shift in goals regarding the signing of release tags driven "it needs to work with the software already installed on Ralph' machine". I was pushing back on this as an a-priori worthwhile goal to pursue. However, of course, a simpler solution is nicer to have.

[rgommers]

Thanks Daniele.

However, of course, a simpler solution is nicer to have.

Okay, let me strip this PR down to the absolute minimum, with the goals articulated above - and update the PR description as well.

[dnicolodi]

I have been thinking about this a tiny bit more.

Does this mean that the ultimate goal is just to have the green checkmark next to the tag in the GitHub interface?

Pretty much, yes. Nothing more is needed - the rest is just security theater. If we do this, we're already doing much better than most other projects

AFAIU, the green check mark comes from GitHub verifying that the tag or commit is signed with the private side of a public key stored in the user account. For PGP and SSH keys, the key just needs to be there, with no further restriction possible. Therefore, a compromised GitHub account would result in someone in being able to push commits to the repository and adding a key to the user account. My conclusion is that the green check mark is not much of a protection against compromised accounts. Why do we want to have the green check marks for the release tag then? Does anyone care about them? I honestly do not have a feeling for that.

If we switch to SSH key tag signing in the form described in this PR, it is may opinion that we can very well do not sign the tags and have the pretty much the same protection against compromised GitHub accounts. The only extra protection comes from listing the keys used for signing somewhere else. However, for the current project infrastructure, there is a space where to publish them that is not linked to the project git repository and thus the GitHub accounts. And listing the keys somewhere breaks in the same way as the current situation if a key becomes unavailable (with some added risk, because while PGP keys normally shared across systems, SSH keys are usually generated per system, thus moving from one device to another to tag a release would normally mean using a different SSH key).

Conclusion: the green check marks do not add protection against compromised GitHub accounts. If for some reason we want the green check marks anyway, I am fine using SSH keys. If we use SSH keys, I would not mention them in the documentation. I would go as far as to remove any mention of PGP or SSH signing from the documentation.

[rgommers]

Conclusion: the green check marks do not add protection against compromised GitHub accounts.

Agreed, there is very little that protects against that 100%, because a compromised account gives you admin access. But it's more likely that for example a GITHUB_TOKEN with write permissions leaks, which allows pushing a commit or tag. That's also what Eli said above:

That trust list is generally scraped from "other sources the legitimate maintainer controls", and e.g. GitHub's associated PGP keys for a user account can be trusted to be reliable even if a PAT or a CI workflow compromise allowed an attacker to commit to a repository and create malicious tags, because adding ssh / pgp keys in the settings page is a much more trusted action than pushing.

If we use SSH keys, I would not mention them in the documentation. I would go as far as to remove any mention of PGP or SSH signing from the documentation.

I'm fine with that. I don't see the harm in keeping the historical ones, but I don't really care too much either way, I just want a normal smooth process.

[dnicolodi]

If we use SSH keys, I would not mention them in the documentation. I would go as far as to remove any mention of PGP or SSH signing from the documentation.

I'm fine with that. I don't see the harm in keeping the historical ones, but I don't really care too much either way, I just want a normal smooth process.

Let's keep the list of PGP keys. This should make this PR even smaller.

[dnicolodi]

Thanks @rgommers. Just a few minor tweaks (that simplify this even more).

[rgommers]

Thanks, accepted all suggestions - we can squash-merge when green I think.

[dnicolodi]

Merged. Thanks @rgommers

[rgommers]

https://pypi.org/project/meson-python/0.21.0/ is up

[eli-schwartz]

Sorry for the late reply.

PGP keys can absolutely be revoked

Yes I know - but you need access to the original key, so if that's on a temporarily shelved machine that I don't have access to, it cannot be done in practice.

Then I don't understand your goal because ssh doesn't allow you to do it even if you do have the key. But also revocation is something you do to mark a key as compromised. If you revoke a key every time you do releases from a different machine, then you break reproducibility of old releases! Please don't break reproducibility of old releases!

I'm not interested in a PGP vs. SSH debate. I'm using PGP twice a year, for meson-python releases, and it's a pain, with little upside. SSH keys on the other hand I'm using constantly. It's just much nicer. And in case something went wrong with it, I'd actually notice, unlike PGP. So, switching to that.

I'm not trying to tell you to use PGP, and I'm not trying to convince you that PGP is good. I'm pointing out that from my perspective as a downstream, whatever PGP is, ssh signing is "even worse". But the arguments you're making for not using it are actually the arguments most people use for not providing code signing at all. Providing code signing is usually a very specific choice to provide certain guarantees to downstream consumers. It's not about having a signing stamp -- it's about providing people with the tools to assure themselves that the same person guarantees this release, who also guaranteed the last release. Key rotations are always a hindrance here, and generally imply signing a transition document with the old key (this functionality is built into GPG). An example of doing this is at mesonbuild/meson@ac3ee31 (I did this, just a couple months ago. gpg --list-sigs "EF51 D308 B1CE 2308 88BE 0D84 EA42 3E93 A163 43F1" -- signed by 0x84818A6819AF4A9B 2024-06-11)

And if downstream doesn't verify it anyway because it's a pain and the tooling doesn't exist, then... who are you helping, exactly?

I should also note, if you occasionally use a PGP key to sign releases it isn't hard to also use it to sign everything. I have my git config --global commit.gpgsign=true, so every commit I create and push to github across any repo is authenticated as coming from me (and can be verified at HEAD instead of only at release time). I would notice if something went wrong.

...

ssh signing is not championed by people who consume it, it is championed by people who want to produce it. It was introduced as a "because we can" option by and for people who want to be able to say "we implement cryptography, therefore we have Checked The Box™. It is raw cryptography, in the raw, no usability warrantied. Unfortunately, the genuinely hard part of cryptography isn't using an ed25519 library to convert a message blob, the hard part is providing a UI that makes people want to use it. SSH doesn't ship with one. GnuPG does, but many people are very opinionated about it. Git did not make any effort to provide a UI for either one, but for PGP, you can rely on the one that ships with GnuPG.

If you're not going to use PGP because you've had a change of heart and now think that key management is tricky and "doesn't add much real-world protection" then just do not sign things, and tell people to rely on access management by popular SaaS providers working out okay. Saying that it doesn't add much real-world protection "therefore we will switch to raw ssh" feels like making a point without a goal.

Switching without a certificate handover and then committing in advance to the benefit of:

a maintainer can easily create a new key and use the features GitHub offers to make it a signing key. The .github/allowed_signers file should then be updated for it.

just tells me that I cannot assume any future updates to the signing key will be signed by transition documents either, so the only thing I can possibly know is that "someone with commit access to the repo has added a new verification key", but if "commit access to the repo" is the only bar to pass, then great! I already trust the code without a signature!

[eli-schwartz]

The main issue is usability: I do not have any PGP program installed at all, and there is no great way to do that on macOS. SSH you get from Apple, so nothing to trust and always installed for me. GnuPG or similar, not so much. The main recommendation seems to be Homebrew, which I deliberately did not install on this machine (reasons are both "don't want to trust, going for minimal trust here" and "don't like globally-installed tools"). The second recommendation is a non-free GUI program (https://gpgtools.org/), which currently has a banner at the top that only a beta version of that GUI program supports macOS 26 (out for quite a while now).

AFAICT you can get it from conda-forge too, which I assume is not globally installed. You can also minimally install https://crates.io/crates/sequoia-chameleon-gnupg anywhere that rust is installed, which should provide a drop-in replacement (as a single file, no need for 16 different programs and libraries that all depend on each other and half of which launch themselves as system daemons). For Thunderbird email, PGP support is built in via librnp and cannot actually be switched off, you don't need to install some proprietary GUI even if you never installed a single tool intentionally. Any of these solutions sounds like far more of a "recommendation" in my mind than "nonfree GUI", so maybe I'm weird but I don't know why gnupg bothers to list gpgtools at all.

Not trying to say you have to use PGP, just saying if I wanted to the first place I'd look would not have been gpgtools.

Does this mean that the ultimate goal is just to have the green checkmark next to the tag in the GitHub interface?

Pretty much, yes. Nothing more is needed - the rest is just security theater. If we do this, we're already doing much better than most other projects (including the likes of our trusted dependency meson, which doesn't use signed tags and yolo's the release process to the extent that a none-any wheel has repeatedly gone missing).

I would never even look at the green checkmark, I would want to validate it myself because the green checkmark doesn't really help my download workflows know that what I downloaded is the same thing that has a green checkmark.

Meson certainly does PGP sign the tag:

$ git --no-pager show 1.12.0
tag 1.12.0
Tagger:     Jussi Pakkanen <jussi.pakkanen@mailbox.org>
TaggerDate: Mon Aug 10 18:08:36 2026

1.12.0
-----BEGIN PGP SIGNATURE-----

iQJPBAABCgA5FiEEGeLW2bRtjapiiPh3wk5jG6ux/nAFAmp6S+gbHGp1c3NpLnBh
a2thbmVuQG1haWxib3gub3JnAAoJEMJOYxursf5w0rIP/RuHRpc/q6XMbvZRIrIz
v1FQF6qDjySj/m8cMjOwENNqUtdGFCZoqz8Z7p2gWu29KxA73Pg+El/aFZJzMrOM
5rynri8QCtv/jgCSiNrSpV23lyWHORIQQpbRnIrGlSRSEht09TOBnI2GhdOrDyDu
cqwiLFuYPqSONMAfoim+sBDplLOQ0Y86kBbZwS0ltoa2AC/d3/m8VouKLR7cZRl1
s5/y5kzeuV5lVRuspfmmguM1/HFBJltF5SiYWF2gICO0aYSpOh6N0//hW3h8b6jl
sP2F51vE01qodJbhOOxpYwuvfVqs/EoH+tSBdjMt4+oGgK8r7KPQDJn9xBMoMl43
uu1l/t2zqx2BIIOqeA+wSIL7paTKFjVb33f/0w8RU3nATubKZfQjKye12p0g/+FM
doNjQ+ozN0ZN+AEEgSZB2Yf0i1mqjevmWRcRXCVMMIol0OdytQ5IMZuD0UoZKBnG
18u0ImJJZd5SdsVY/Xp4Cd906XfW4R/71eju85sTAoz2zrxIT32+0N9wQL7trYD3
YS1SWSVGSGvG0BzPUmUf9JuVg+aw53d4rQx03+XhRk0UdjZSLWDURh4dOf4ylMVi
bl8DSAboVIZwxnJgGF3za67CpQiRVaJ+lhsJS4b1hiZo3sPrP+lzuZlEWWRPN4PR
/fiDj3hPrwuJWXyaYBP8w66g
=h6Hi
-----END PGP SIGNATURE-----

commit ab4508e6abe338a3bd2e6eb5da1d18b8cf55dc93 (tag: 1.12.0, origin/1.12)
Author:     Jussi Pakkanen <jussi.pakkanen@mailbox.org>
AuthorDate: Mon Aug 10 16:56:48 2026
Commit:     Jussi Pakkanen <jussi.pakkanen@mailbox.org>
CommitDate: Mon Aug 10 16:56:48 2026

    Bump version number for release.

https://github.com/mesonbuild/meson/releases/download/1.12.0/meson-1.12.0.tar.gz
https://github.com/mesonbuild/meson/releases/download/1.12.0/meson-1.12.0.tar.gz.asc

https://github.com/mesonbuild/meson/releases/tag/1.11.1 -- green checkmark for me (and PGP-signed wheels).

https://github.com/mesonbuild/meson/releases/tag/1.12.0 -- no green checkmark and I do not know why, because as I pointed out, it is tagged and the primary subkey is the same one that was added to Gentoo in sec-keys/openpgp-keys-jpakkane: new package, add 20231105...

We walk the talk that I talk, useful or not.

[eli-schwartz]

I do appreciate the clarity in the version of the PR that was merged (re "the goal is for github to show it as verified").

[rgommers]

just tells me that I cannot assume any future updates to the signing key will be signed by transition documents either,

Indeed, you can't. The usability is just too poor for this, as this issue and the delayed release has shown. Hence, we do not guarantee this.

And if downstream doesn't verify it anyway because it's a pain and the tooling doesn't exist, then... who are you helping, exactly?

Ourselves, for one. And anyone who is looking into a possible issue. If someone who is not me or Daniele moves a tag, that will be clear, because they cannot sign it with our keys. There's still a large difference between signed and unsigned here.

There's also trusted publishing; that's a good tool - but it currently has a small gap for full traceability. I'll open a separate issue for that in a bit.

https://github.com/mesonbuild/meson/releases/tag/1.11.1 -- green checkmark for me (and PGP-signed wheels).

https://github.com/mesonbuild/meson/releases/tag/1.12.0 -- no green checkmark and I do not know why, because as I pointed out, it is tagged and the primary subkey is the same one that was added to Gentoo

All of the tags signed by you are listed as verified, and none of the recent ones signed by Jussi are. They were though until meson 1.8.0 (see https://github.com/mesonbuild/meson/tags?after=1.8.4). That should mean that Jussi hasn't updated the signing keys at https://api.github.com/users/jpakkane/gpg_keys after he started using a new key.

[eli-schwartz]

It is the same key he used before...

... ah, maybe it is because he recently added a new email address and started committing via that email address in git config user.email. @jpakkane maybe you need to re-upload the existing key to make github register the new uid?

(Does github really display nothing in this case? For commits, it definitely shows a "signed but cannot confirm by who" bubble if you use a PGP key that isn't associated with your account, etc. Also a "half verified" if the author / committer differ because the author cannot sign, which, "lol".)