[omicron-nexus] add an error injection test for deleting local storage#9997
Merged
sunshowers merged 2 commits intomainfrom Mar 10, 2026
Conversation
Created using spr 1.3.6-beta.1
This was referenced Mar 7, 2026
jmpesp
approved these changes
Mar 9, 2026
sunshowers
added a commit
to oxidecomputer/progenitor-extras
that referenced
this pull request
Mar 9, 2026
…ly (#3) See #2. I've tested this with the new test added in oxidecomputer/omicron#9997. I've also tested a local change which enables the clippy::disallowed_methods lint in omicron for `retry_operation` and `retry_operation_while`, suggesting use of `_indefinitely` instead. I believe that should be a complete structural fix for #2.
sunshowers
added a commit
that referenced
this pull request
Mar 9, 2026
…9998) While working on #9997 I saw this error in the logs: ``` saga ACTION error at node "delete_local_storage": deserialize failed: unknown variant failed to delete local storage: failed at attempt 4: retries exhausted: Error Response: status: 503 Service Unavailable; headers: {\"content-type\": \"application/json\", \"x-request-id\": \"4a21fdec-b7b2-4f37-a99d-c218efa1c701\", \"content-length\": \"94\", \"date\": \"Sat, 07 Mar 2026 05:34:47 GMT\"}; value: Error { error_code: None, message: \"Service Unavailable\", request_id: \"4a21fdec-b7b2-4f37-a99d-c218efa1c701\" }, expected one of ObjectNotFound, ObjectAlreadyExists, InvalidRequest, Unauthenticated, InvalidValue, Forbidden, InternalError, ServiceUnavailable, InsufficientCapacity, TypeVersionMismatch, Conflict, NotFound, Gone ``` The root cause for that was that we were passing in a string as an error rather than a structured omicron-common error. That's because steno's `ActionError::action_failed` accepts [anything](https://docs.rs/steno/latest/steno/trait.ActionData.html) that implements `Debug + DeserializeOwned`. Fix this by: * adding `nexus_types::saga::saga_action_failed` * banning most uses of `ActionError::action_failed` through clippy disallowed-methods * updating all the call sites There are two instances of `ActionError::action_failed` that remain, both of which are marked with `expect(clippy::disallowed_methods)`: 1. The invocation inside `saga_action_failed`. 2. In `nexus/src/app/sagas/instance_update/mod.rs`, the instance updater lock error which is handled specially.
Created using spr 1.3.6-beta.1
sunshowers
deleted the
sunshowers/spr/omicron-nexus-add-an-error-injection-test-for-deleting-local-storage
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In #9993 we found that we'd accidentally dropped the indefinite retry logic. Add a test that fails if we retry fewer than 8 times. I've confirmed that the test fails before #9993.
I tried bumping the number of tries up to 32, but the test took far too long even with Tokio's
time::pausesupport. The reason for that is that all the background tasks get kicked off many times. 8 seems like a decent number of retries to check against.