ci: retry the toolchain PPA instead of failing the job on a Launchpad 500

[Yaraslaut]

What is actually wrong with CI

Nothing is failing right now — 1 failure in the last 40 runs, and that one
was a stale pre-rebase run I have since fixed. What went wrong today was not a
defect in the pipeline's logic but its dependence on a third-party service with
no retry.

Six install steps — five in ci.yml, one in docs.yml — need GCC 15 from
ppa:ubuntu-toolchain-r/test, and each called add-apt-repository exactly
once. Launchpad serves that call from a service that returns HTTP 500 often
enough to matter. On 2026-08-25 it took out roughly a dozen jobs across six pull
requests in one afternoon:

add-apt-repository -y ppa:ubuntu-toolchain-r/test
lazr.restfulclient.errors.ServerError: HTTP Error 500: Internal Server Error
  b'GPGKeyTemporarilyNotFoundError'

Every one of them died in the dependency-install step before a single file was
compiled. The affected legs were exactly those that pull from that PPA —
Application ladder, Valgrind memcheck, Linux / gcc-release, Linux / Qt6 WebSockets, Linux / all optional features, Build documentation. gcc-debug
survived only because its apt cache hit and it never called Launchpad.

The cost is not the red tick. It is that a Launchpad 500 and a real regression
look identical in the checks list, so each one costs a reviewer the time to open
the log and rule it out — and costs whoever is watching a round of reruns. I
spent a fair part of today doing exactly that.

The fix

All six sites now call scripts/add_apt_repository_retrying.sh: five attempts,
linear backoff, each retry announced so a slow green is distinguishable from a
healthy one.

It still fails after the last attempt, deliberately. Continuing without the
PPA would let the following apt-get install pull the distro's older GCC, and
the job would build and pass while testing a compiler the project does not
target — a green tick for the wrong thing, which is worse than a red one. The
final error names Launchpad and links the status page, so the next person does
not start by suspecting their own change.

I verified actions/checkout precedes the install step in every one of the six
jobs, since the script does not exist before it:

workflow	job	checkout	script use
ci.yml	linux-compilers	124	138
ci.yml	linux-qt	570	583
ci.yml	ladder-tests	651	697
ci.yml	linux-all-features	1023	1036
ci.yml	valgrind	1193	1206
docs.yml	build-docs	32	38

The helper is tested, and the test can fail

A retry wrapper nobody tests is worth less than no wrapper: it looks like
resilience while possibly retrying nothing, and the only way to find out is
during the outage it exists to survive. So it gets a self-test, run in CI beside
the repo's existing checker self-tests, following the scripts/test_check_*.sh
convention already in the tree.

The test drives the script against a stub whose failure pattern it controls and
asserts the call count, so "it retried" is distinguishable from "it happened
to succeed":

ok    succeeds first time: exit status
ok    succeeds first time: one call only
ok    recovers after two failures: exit status
ok    recovers after two failures: called three times
ok    persistent outage: exit status
ok    persistent outage: exhausted every attempt
ok    persistent outage: emits an ::error:: naming Launchpad
ok    announces each retry

Mutation-tested in both directions:

mutation	checks failed
ATTEMPTS cut to 1 (no retry at all)	4
persistent outage returns 0 instead of 1	1
unmodified	0

Not done

• Removing the PPA dependency. Ubuntu 24.04 does not ship GCC 15, so the PPA
  is how the project gets the compiler it targets. Vendoring or a container
  image would remove the dependency entirely, but that is a much larger change
  and this one is worth having regardless.
• Retrying apt.llvm.org. The clang legs fetch from a different host, which
  has not been flaky in the runs I looked at. Adding retries there speculatively
  would be guessing; the same helper will fit if it ever is.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!