fix(postgres): make pgbackrest backups survive transient GCS 429s - #11
Draft
thenav56 wants to merge 1 commit into
Draft
fix(postgres): make pgbackrest backups survive transient GCS 429s#11thenav56 wants to merge 1 commit into
thenav56 wants to merge 1 commit into
Conversation
Scheduled and manual backups were failing outright with: ERROR: [039]: HTTP request failed with 429 (Too Many Requests) The object .../backup.manifest.copy exceeded the rate limit for object mutation operations GCS allows ~1 mutation/sec per object, and backup.manifest.copy is written repeatedly at the end of a backup, so it occasionally trips the limit. Two independent problems, hence two changes: * pgbackrest 2.55.1 only retried 5xx responses. 408/429 were added to the retry class in 2.57.0 (pgbackrest/pgbackrest#2661), so a single 429 aborted the whole backup. Bumped to 2.59.0. The download URL had to change too: since 2.59.0 the git archive export-ignores src/build, so the auto-generated GitHub tarball no longer builds ("File build/common/regExp.c does not exist"). Use the official release asset instead, which ships the generated sources. * Ofelia has no retry or on-error support, so a failed scheduled job is simply skipped until the next window -- ie. a missed monthly full backup. All three jobs now go through pgbackrest-with-retry (3 attempts, 300s apart), which still exits non-zero on final failure so the Slack alert is preserved. Compatibility checked against the existing prod repository: * PG support in 2.59.0 is 9.6-19 and the image is PG 17. Only 9.5 was dropped (2.58.0). * REPOSITORY_FORMAT is 5 in both versions, so the existing repo and the current full backup stay valid. No stanza-upgrade needed. * Every option in pgbackrest.conf and every PGBACKREST_REPO1_* env var is still accepted by 2.59.0. One behavior change to be aware of: from 2.59.0 only `restore` may run as root. The ofelia jobs and archive_command already run as postgres, but ad-hoc `docker compose exec postgres pgbackrest ...` now needs `-u postgres`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scheduled and manual backups were failing outright with:
GCS allows ~1 mutation/sec per object, and backup.manifest.copy is written repeatedly at the end of a backup, so it occasionally trips the limit. Two independent problems, hence two changes:
Currently testing this using staging.