Feature - Declare the S3RDMA ioctx and add backend integration seams - #181
Open
ran-yuan-rui wants to merge 6 commits into
Open
Feature - Declare the S3RDMA ioctx and add backend integration seams#181ran-yuan-rui wants to merge 6 commits into
ran-yuan-rui wants to merge 6 commits into
Conversation
ran-yuan-rui
changed the title
ran-yuan-rui
force-pushed
the
feature-s3rdma-s1-seams
branch
2 times, most recently
from
August 6, 2026 14:40
4098802 to
7bc2084
Compare
A new backend identifier for an S3-over-RDMA ioctx. The registry is map-driven (no switch over the enum), so the value is inert until a backend registers under it.
Listing is prefix resolution on the control plane, independent of how a backend's data plane reads the resolved keys. A glob / LIST layer can now depend on this interface instead of a concrete ioctx type. rest_ioctx implements it in the same change (its existing list_objects_paged / list_max_matches become overrides — no behavior change), so the interface lands with a production implementation. The ListObjectsV2 page/entry types stay in rest/s3: they are S3-protocol shapes, not REST-transport shapes; the include is documented as such.
A protected virtual with an empty default body, called from the exception handlers of both device-plane dispatch bodies before the failure is returned through an errored future. A backend whose contract makes a poisoned CUDA context fatal — because registered GPU memory must not be reused before RDMA writes and CUDA work are quiescent — invokes its fatal policy in the override; every existing backend inherits the no-op and is behavior-identical.
Hands one type's registration to a new backend type in a single stroke — unlike an unregister+register pair there is no observable no-claimant gap and no call-it-early-enough discipline left to convention. Throws invalid_argument on a missing old type, an already-registered new type, or null checker/factory, with the strong guarantee (the emplace precedes an erase by key — an iterator would not survive a rehash). The registry latches its first lookup_path; replace_ioctx afterwards throws logic_error (bootstrap is single-threaded by contract).
A non-constructible declaration for the planned S3-over-RDMA backend: an ioctx subclass with a deleted constructor and every override declared but not defined, claiming no paths, plus a placeholder translation unit so CI compiles the header. The backend is developed in Sirius (design: experimental/s3-rdma-transport-design.md there); its implementation replaces this declaration when the backend is contributed upstream.
…ject_store_listing Registry: replace success re-routes s3:// and proves the old claimant is gone (a counting factory registered over it stays at zero calls); invalid_argument on missing old / registered new / null checker / null factory, each leaving routing unchanged; logic_error after the first lookup_path; register_ioctx stays legal after lookup. Hook: a stub reactor drives both device-plane dispatch bodies through prep- and enqueue-stage throws (hook fires exactly once per failed dispatch), empty reactor selection and success paths (no call), and the non-overriding default (error future preserved). Listing: rest_ioctx exercised purely through the object_store_listing interface against a test-local scripted LIST server (multi-page with continuation tokens, query recording; the accept thread owns a fixed fd, shut down before join): pagination, early-stop sink, on-the-wire max-keys clamp, the max_scanned cap, and the configured match cap.
ran-yuan-rui
force-pushed
the
feature-s3rdma-s1-seams
branch
from
August 8, 2026 12:40
7bc2084 to
dedb58c
Compare
ran-yuan-rui
changed the title
ran-yuan-rui
marked this pull request as ready for review
August 8, 2026 12:47
ran-yuan-rui
changed the title
Contributor
Author
|
@aminaramoon @felipeblazing Please help to review and start to test |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR declares the cuCascade interfaces needed by the S3RDMA backend. It does not implement the S3RDMA transport.
The transport implementation remains in Sirius. The non-constructible
s3rdma_ioctxheader links to the detailed design document.Changes
io_context_type::s3rdma.s3rdma_ioctxdeclaration.object_store_listingand implement it with the existing REST listing methods.io_context_registry::replace_ioctx().templated_ioctx::on_device_dispatch_failure().replace_ioctx()cuCascade initially assigns
s3://paths to the REST backend. When S3RDMA is selected, Sirius uses this API before the first path lookup to replace the REST registration with the S3RDMA checker and factory.The replacement runs under the registry lock and leaves one explicit claimant for
s3://. It changes registration metadata only; it does not create, stop, or migrate an I/O context.on_device_dispatch_failure()templated_ioctxnormally converts synchronous device-dispatch exceptions into errored futures. S3RDMA needs to check whether such a failure came from a sticky CUDA context error before returning it as a normal request failure.The new protected hook provides that check point. Its default implementation is a no-op, so existing backends keep their current behavior.
Testing
cucascade_io_tests113 → 133 cases).