StorageBackupReconciler: recover from duplicate snapshot/backup creation on retry#226
Merged
Conversation
noctarius
marked this pull request as draft
July 13, 2026 10:12
Make snapshot and backup creation idempotent across reconcile retries. Previously, if the operator restarted (or a request otherwise failed) between the backend create call succeeding and the status write that records the resulting ID, the CR's status field stayed empty and the next reconcile issued a second, unconditional create call -- producing an orphaned duplicate snapshot or backup on the backend. Creation is now attempted directly (no pre-creation lookup, since that would cost an extra network call on every normal reconcile for no benefit in the common case). If the backend rejects the retried create as a duplicate, the reconciler recovers the existing resource's ID with a single lookup instead of failing: - Snapshots: the backend's v2 API returns 409 Conflict with an "already exists" body on a duplicate name. isDuplicateNameError matches on the 409 status. - Backups: the backend has no name uniqueness constraint for backups; it instead returns 400 with "already has a backup" when the target snapshot is already backed up. isDuplicateBackupError matches on that specific 400 condition. storagebackupsync_controller.go also retries the status patch for a StorageBackup CR that was previously imported (labeled) but whose status write failed, instead of attempting to recreate it.
boddumanohar
marked this pull request as ready for review
July 24, 2026 09:39
boddumanohar
changed the title
geoffrey1330
approved these changes
Jul 24, 2026
boddumanohar
changed the title
noctarius
approved these changes
Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fixes: #144
Problem
When the operator creates a backup, it does two things in sequence:
But If the operator crashed or restarted between those two steps, it creates the same backup again and we get 409 conflict. which is not handled on the operator side.
Fix
as a fix we handle 409 conflict and get the ID using
findSnapshotByName. Since the backend doesn't have support get by Name, we list snapshots and get the name.