diff --git a/pkg/raftstore/README.md b/pkg/raftstore/README.md index 40dcd4d93..1330de77b 100644 --- a/pkg/raftstore/README.md +++ b/pkg/raftstore/README.md @@ -43,6 +43,91 @@ Most of the concepts defined below come from the Raft consensus algorithm itself - **Snapshot:** A serialized copy of the application state at a given point in time. - **WAL (write-ahead log):** The on-disk files a node uses to persist its Raft log entries before they're considered durable. +## Architecture + +

+ raftstore architecture: repo, Store, registry, and consensus layers +
+ Overall architecture and request path through the Store and Consensus layers. +

+ +### The repo layer + +Each service (`rid`, `scd`, `aux_`) defines its own `repos.Repository` interface (e.g. +`pkg/rid/repos`) describing the basic operations needed from storage (`GetSubscription`, +`InsertSubscription`, `DeleteSubscription`...). Every storage backend (`sqlstore`, `raftstore`, `memstore`) implements it. + +The `memstore` holds the data and `raftstore` is the replication layer that wraps it: + +- The `memstore` `repos.Repository` ([pkg/memstore/store.go](../memstore/store.go)) is the most basic implementation. It holds data in-memory and every interface method call directly reads from or mutates it. +- The `raftstore` `repos.Repository` does not hold application data itself. It embeds a +`memstore.Store[R]` instance, which is where the data lives: + ```go + type repo struct { + consensus *consensus.Consensus + memStore *memstore.Store[R] + } + ``` + Since individual `repos.Repository` method calls still need strong consistency guarantees, the `raftstore` `repos.Repository` implementations issue a request to be replicated by Raft. + Once the proposal for that call has been committed, `raftstore`'s `Apply` + (`raftstore.RaftRepo[R].Apply`) calls `r.memStore` for the data reads and writes. + +### The Store layer: `Interact` and `Transact` + +Above the repo layer, every backend implements the [`store.Store[R]`](../store/store.go) interface: + +```go +type Store[R any] interface { + io.Closer + Interact(context.Context) (R, error) + Transact(ctx context.Context, request OperationRequest) (any, error) +} +``` + +In the `raftstore`, both `Interact` and `Transact` end up proposing to Raft. +The difference is what gets replicated as an atomically-applied unit: + +- `Interact` returns a repo (`R`) on which its methods are called one at a time. Each such method + proposes to consensus and applies its own change atomically. + + Because each call is its own independent proposal, there is no atomicity *across* multiple + `Interact` calls. + +- `Transact` takes a single `OperationRequest`; a whole operation consisting of a set of `repos.Repository` + calls. This `OperationRequest` is replicated as one `Proposal` and is then applied atomically. + + `Transact` requests for each service are defined and looked up in a `registry map[string]store.OperationHandler[R]`. An `OperationHandler[R]` bundles everything needed for each operation: + + ```go + type OperationHandler[R any] struct { + Encode func(req OperationRequest) ([]byte, error) + Decode func(buf []byte) (OperationRequest, error) + Execute func(ctx context.Context, repo R, request OperationRequest) (any, error) + IsReadOnly bool + } + ``` + + On every node, once the Raft entry is committed, `Apply` looks the handler up using the proposal's + `RequestType` (the `OperationID`), and uses `Execute` + to run the operation's business logic which is a set of calls against the `memstore` repo. + +### The consensus layer + +The link between the consensus layer and the storage layer for both `Transact` and `repo` methods is `Consensus.HandleClientRequest` +([consensus/consensus.go](consensus/consensus.go)). From there, a proposal is generated: + +1. The proposal is serialized, tracked in an in-memory `proposalsTracker`, and passed down to the + embedded etcd Raft node via `node.Propose(ctx, buf)`. +2. Once a quorum has durably appended the entry, it surfaces in the `startRaftUpdatesConsumer` goroutine as a committed entry. +3. It is then passed as an `EntryCommit{Prop, Done}` + into a channel read by the store. +4. `Store.processCommits` ([store.go](store.go)) calls, on each entry, `raftRepo.Apply(ctx, + proposal)` which maps the `Proposal` to the appropriate type and business logic and mutates the `memstore`. This runs identically on every node. +5. `Apply`'s result is sent back on the `EntryCommit`'s `Done` channel. Only the proposing node is + actually waiting on it: it gets untracked from the `proposalsTracker`, which unblocks the waiting + `select` in `HandleClientRequest` and returns the result. On every other node, the result is + computed and then discarded to have consistent states across nodes. + ## Future Work - **Reads go through Raft:** There's no `ReadIndex`-based read path yet. Read-only requests are still proposed and committed like writes. See the `TODO` in [consensus/proposal.go](consensus/proposal.go). diff --git a/pkg/raftstore/docs/architecture.png b/pkg/raftstore/docs/architecture.png new file mode 100644 index 000000000..3338cf23d Binary files /dev/null and b/pkg/raftstore/docs/architecture.png differ diff --git a/pkg/raftstore/docs/architecture.xml b/pkg/raftstore/docs/architecture.xml new file mode 100644 index 000000000..8c47e98de --- /dev/null +++ b/pkg/raftstore/docs/architecture.xml @@ -0,0 +1,2 @@ + +7V3Zkpu4Gn6Wc+Gq7gu72JfL3jJJVeZMVyez5GpKBtlmGhADuNvO048Eko0ENouR7ZM5TlVsJCHE//27lp7oD9HmpxQkq5+RD8OJpvibif440TRVUR38RUq2ZYmlaGXBMg182mhf8CX4DtmdtHQd+DDjGuYIhXmQ8IUeimPo5VwZSFP0zjdboJB/agKWsFbwxQNhvfT3wM9XZaljKvvyjzBYrtiTVYXWRIA1pgXZCvjovVKkP030hxShvPwVbR5gSIjH6FLe9+FA7W5gKYzzLjckv8Z//uU+3i3fXhcw+9P4LXr9Nm3ohRZl+ZbRIIcbXHe/yqMQF6j45wLFOcVKNdg1vYHUgzBYxvi3h3uFKW3wgEKUFj3qlufA+QKXl896A+GaPmuiWSF5WJaAmBuF9feaUOreK3u5w5Xpcn6D3/kB/2Rft0WfrC3+tSTfX1MQZ8DLAxSz/jGdykeULeg4YIrftUIGSsmfIIpgnm5xk1UFbJ0i+75nDM2hZRvWhF5T9tdceg0oXy53Xe+hwz8oej2Q1C6DpKPNdcsahiQeaj7tgqcyBPRPZMAXAd1wzgW61QX0FK1jH/oU0PdVkMMvCfBI7TtW2gI/BGFYgdcH0Fl4BMU8Ra+wlwj7wRuj9JccpXBi3r9MzMcZk0bWbp7uAdnhVLk5hcsgI9Qy739JMKQE0U+PuKdB+Fl1/FRLEFqLx083XUn4qYp0AE3o+EYTgIclNwWLPCsQw+IGE5TNXsj/AS7aNoB7g8V4ncZ4gJoyJ+RmkndzW9AZkxn3w/q8bQZZFpamIWBpyZJFkwIH/ZpHUQcXrVMPHmMMrcYFpF+mmVGar9ASxSB82pfe7/lEwVf7Np8RSih3/AXzfEv1PVjniOcduAnyPyq/v5GuZia9etzQnouLLbuIMfH+qF5U7iKX+9uKK3Yf9eVAuoT5MaoqzWyQwhArgjee0uNjqnPeJZgzMBrk9ihTCDh+BnPsJ3O0rxlaIgsBdkTvaEUU+H4JM8yC73QohJIJCuK8eHPznmjFBol+ThEWYNhZyBqpS2+ZKjPFPkxv2tkzGVSlCVosMgy0CMjumSfo0E6ezyk6dLGAltdoBH3bnStKE8U/ghij9RAGeEwv8O81zBrs3c08RN5rhu9cx3lA3jRflUo3W4d5wXcRJNVz4L3ejqUideO4uTM1WSrSGVVFGpdUkXu1+K1S06Yi91rxG6cUm1UkjP07EsLiyxjFsCz5EBCa91GhzkVVKIPpunzUioszkt8hCJUhCJUrLQZQ7Wv0ISMYHXYhRyK5K5BcNzmSq7opjeZOF5pnK5CQn942DDDxU32A5VloByyPNbdMQvl5CeznOeuiBkUQTzEaBeHx6EA+jOftdl/bMHmeV01FFgDMRHWyJE1SoAhq/YiKrhEUsSj0RL0quEqMimLEIprj0jLSu6oJv3pHM8ao+0h2plQ+wuBKU1LreSxPTdNGBm2ALa6BKQdDIe1nqM7MMgbCqLS4Z7JRs6SJmqPpfNCoqLsWB30ifPUM0wC/FgmTTga1EtUoitMZ6MNxTQ1AU0gHGkMFGnfUJL17vrigbFuUdIMjZcZmF46Ub2iw9nWLTXeRXn4G2xABn+StQETMdTzPkj4Zq2NshyHTdB7HaW9tI9wgL9TWXFkKXBsUTWkOpz8KGbZGUSAkPiAe6+wuScLtTVLkT0B4K8daYLnWbXf3sW2OIRwTV1sDjYftztzqh+u5FpRIVhK6Lot/bMWoMhDmA/c4E+GLZiZoDadZvFXhll9iFufgL9zlCvlEFQJCEQn8YmruzLQPgaqqJo+5M4x1yFMsIaLaRQHnYhhDFsPU9E2fAOAFAp+k5cj7asrvKQ7pMjlI60JEZahDfQexp1psJhtK6WmKIWnaBxRnMM7WWWdr3paU0PmYWAyJVWnzT3qnnMS5CZxhtspfsDX9NfEBFhNC8HWEdW59HnGJ8ODygOhSmTOE4my9o/EQ2dIydWYf96kt/a27NWwvlv5WjtvaA6q1LdTzQbbacWpH42xaJ+rh05ZjUM4ZPl3YDOq5gyAPRVGQzx5R4dfsAp9CKKe45IXOT40QAGGfHfsrY8TdoxkqV7YeHTJhUaygmeE4xIMZVqIEn+yCOlQRMr/6uXSo0Sfx26pDnQGuY1XvtayNGKLBGPvVxPGhURSfiMos+UGKB+qqArLGQAdUNdyWniQ7oEafyef/6eTFXj2PH5AIuUzXcQcntk0h7cGHsY44eSebPfqkudsUi2G28NEQ14uxjlplm/aFVkNmSWqKa3RGUhXVmjm2sQOfzzRY4+QwXIvPifMPsZ2jD5HNcPaJzqJxHRnz0jrlxfLPgvMa3JKArAAnUSD+WpeB4IkOSptPaWhWZ5aU71MyfSAxNjfJv0ltfX75afQ1i0+j/SiB8sJ1ljdG6jcJhGkmeyWv4GOKnogl+g/jxel9so5tpoCZsWuI0+UsU+vo2JYycKnQ3DRP1LaMKX6slbyqZvGTTd0NuXytaTZgc/mMJpkanP0X+eMtHdQERde24GM8oZC+f2gIgZ/pZGv2NQXeKzFAA+jctE2rJWkh0aDYPQzKD20nZMyd88LiDMxPiB3VuEFyOGCeuoCGMdmFDVQhtP8K82SNkVD6N8qsmESqiVqPDNJx4Ze96E07UWatZqV/bpkdeT5nuNCyJXBCRlGiDA9bkVTZmeOFIMsCj0eLl/GGaYDWtapVX4b5wY10rzg4x7Y7HBAgcXWCPlASVaetJ9mieGp8xzjh0tm0FcAvQF43gllGDicZSSoVW+cQ0q7JknaacqWHV7QEIA3ydd5zKBSSgZsuQBSE27LpRxi+QcIhlfqssO6kVlWTTbWiHB+piVEagbBS9wbSAOBvD5DTbxqbvFO9QSoNGl8pIfYnYDrFFPCCeFm/E6UJZjvapVaWEWpPKauT4h23szqypSmmT1LYKxQ1OTngYYH7Z08q9i4SXUWgK4axe0MKYKnIitN6KmN4R6nPj3r3IPyi89cAP4s8sAwup1QVcu3IztlloY5FhDWSEiK4VX/cVl7Dhx6i+3yE9/CDLAkBRTeIw2K+XflPECXYnQNxTvEIEciFW8VDSvosNx/vDBOzIQlh8sGxLgTHhrR1a3aDhr6S9RaDiDtgUaAmLfNg9wxJznTAiKwdoS10rm2lG4/Oumw67yZ4Dk/kHJr6qdH/aYNVOQ45CSU8DMDZVr8a0jaE2p22mV/JjtxHkINiYNi+9PDw2rbjthFfjI7HO9eBOpOjTJixUV7/0Tc9/PDuIZltiZGU6s4cff8x+R4lx1XOqSfgMN64fFpyfwTgOBGVpggbtdgZfFcRUznd3KoViubrrF0NFprzU5wFPmwLtLi1AN2OeBv31urJf/qHEVcpOHWlKx79JyxyVBV5SrfPpFKr0m1eS3TNs089l8GV79dxd9+lVik4p04COdcxCfTzmpzQGx7aFdnd32xRwYbLDljrv3n6DCr4Ko/n3J2hWR5Rls6eNtBbDzwVpikGEDIJNYdGkbY5gHV8kVNh5O/7tbAnaPP8rou9dE3Yk74c99BZErp1ybMk3ItPrOrc7raZolnHbdKB3eTjr3aw3ZnFIdVy6kefcwL4jsmRNYZxXty1E62fW2eK/0+nnnM61R3jgIcfUnKt+iGbnNwKoWNnNS50ewmpNU6V2uuYef0RpRZf7v/ARNl8/2c69Kd/AA== \ No newline at end of file