[raftstore/documentation] Document overall architecture and request path

pkg/raftstore/README.md

@@ -43,6 +43,91 @@ Most of the concepts defined below come from the Raft consensus algorithm itself
 - **Snapshot:** A serialized copy of the application state at a given point in time.
 - **WAL (write-ahead log):** The on-disk files a node uses to persist its Raft log entries before they're considered durable.
 
+## Architecture
+
+<p align="center">
+  <img src="docs/architecture.png" alt="raftstore architecture: repo, Store, registry, and consensus layers" />
+  <br>
+  <em>Overall architecture and request path through the Store and Consensus layers.</em>
+</p>
+
+### The repo layer
+
+Each service (`rid`, `scd`, `aux_`) defines its own `repos.Repository` interface (e.g.
+`pkg/rid/repos`) describing the basic operations needed from storage (`GetSubscription`,
+`InsertSubscription`, `DeleteSubscription`...). Every storage backend (`sqlstore`, `raftstore`, `memstore`) implements it.
+
+The `memstore` holds the data and `raftstore` is the replication layer that wraps it:
+
+- The `memstore` `repos.Repository` ([pkg/memstore/store.go](../memstore/store.go)) is the most basic implementation. It holds data in-memory and every interface method call directly reads from or mutates it.
+- The `raftstore` `repos.Repository` does not hold application data itself. It embeds a
+`memstore.Store[R]` instance, which is where the data lives:
+  ```go
+  type repo struct {
+      consensus *consensus.Consensus
+      memStore  *memstore.Store[R]
+  }
+  ```
+  Since individual `repos.Repository` method calls still need strong consistency guarantees, the `raftstore` `repos.Repository` implementations issue a request to be replicated by Raft.
+  Once the proposal for that call has been committed, `raftstore`'s `Apply`
+  (`raftstore.RaftRepo[R].Apply`) calls `r.memStore` for the data reads and writes.
+
+### The Store layer: `Interact` and `Transact`
+
+Above the repo layer, every backend implements the [`store.Store[R]`](../store/store.go) interface:
+
+```go
+type Store[R any] interface {
+    io.Closer
+    Interact(context.Context) (R, error)
+    Transact(ctx context.Context, request OperationRequest) (any, error)
+}
+```
+
+In the `raftstore`, both `Interact` and `Transact` end up proposing to Raft.
+The difference is what gets replicated as an atomically-applied unit:
+
+- `Interact` returns a repo (`R`) on which its methods are called one at a time. Each such method
+  proposes to consensus and applies its own change atomically.
+
+  Because each call is its own independent proposal, there is no atomicity *across* multiple
+  `Interact` calls.
+
+- `Transact` takes a single `OperationRequest`; a whole operation consisting of a set of `repos.Repository`
+  calls. This `OperationRequest` is replicated as one `Proposal` and is then applied atomically.
+
+  `Transact` requests for each service are defined and looked up in a `registry map[string]store.OperationHandler[R]`. An `OperationHandler[R]` bundles everything needed for each operation:
+
+  ```go
+  type OperationHandler[R any] struct {
+      Encode     func(req OperationRequest) ([]byte, error)
+      Decode     func(buf []byte) (OperationRequest, error)
+      Execute    func(ctx context.Context, repo R, request OperationRequest) (any, error)
+      IsReadOnly bool
+  }
+  ```
+
+  On every node, once the Raft entry is committed, `Apply` looks the handler up using the proposal's
+    `RequestType` (the `OperationID`), and uses `Execute`
+    to run the operation's business logic which is a set of calls against the `memstore` repo.
+
+### The consensus layer
+
+The link between the consensus layer and the storage layer for both `Transact` and `repo` methods is `Consensus.HandleClientRequest`
+([consensus/consensus.go](consensus/consensus.go)). From there, a proposal is generated:
+
+1. The proposal is serialized, tracked in an in-memory `proposalsTracker`, and passed down to the
+   embedded etcd Raft node via `node.Propose(ctx, buf)`.
+2. Once a quorum has durably appended the entry, it surfaces in the `startRaftUpdatesConsumer` goroutine as a committed entry.
+3. It is then passed as an `EntryCommit{Prop, Done}`
+   into a channel read by the store.
+4. `Store.processCommits` ([store.go](store.go)) calls, on each entry, `raftRepo.Apply(ctx,
+   proposal)` which maps the `Proposal` to the appropriate type and business logic and mutates the `memstore`. This runs identically on every node.
+5. `Apply`'s result is sent back on the `EntryCommit`'s `Done` channel. Only the proposing node is
+   actually waiting on it: it gets untracked from the `proposalsTracker`, which unblocks the waiting
+   `select` in `HandleClientRequest` and returns the result. On every other node, the result is
+   computed and then discarded to have consistent states across nodes.
+
 ## Future Work
 
 - **Reads go through Raft:** There's no `ReadIndex`-based read path yet. Read-only requests are still proposed and committed like writes. See the `TODO` in [consensus/proposal.go](consensus/proposal.go).

pkg/raftstore/docs/architecture.xml

@@ -0,0 +1,2 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<mxfile host="app.diagrams.net"><diagram name="Raftstore &amp; memstore" id="l1H3WYYalsFc6ekI36-O">7V3Zkpu4Gn6Wc+Gq7gu72JfL3jJJVeZMVyez5GpKBtlmGhADuNvO048Eko0ENouR7ZM5TlVsJCHE//27lp7oD9HmpxQkq5+RD8OJpvibif440TRVUR38RUq2ZYmlaGXBMg182mhf8CX4DtmdtHQd+DDjGuYIhXmQ8IUeimPo5VwZSFP0zjdboJB/agKWsFbwxQNhvfT3wM9XZaljKvvyjzBYrtiTVYXWRIA1pgXZCvjovVKkP030hxShvPwVbR5gSIjH6FLe9+FA7W5gKYzzLjckv8Z//uU+3i3fXhcw+9P4LXr9Nm3ohRZl+ZbRIIcbXHe/yqMQF6j45wLFOcVKNdg1vYHUgzBYxvi3h3uFKW3wgEKUFj3qlufA+QKXl896A+GaPmuiWSF5WJaAmBuF9feaUOreK3u5w5Xpcn6D3/kB/2Rft0WfrC3+tSTfX1MQZ8DLAxSz/jGdykeULeg4YIrftUIGSsmfIIpgnm5xk1UFbJ0i+75nDM2hZRvWhF5T9tdceg0oXy53Xe+hwz8oej2Q1C6DpKPNdcsahiQeaj7tgqcyBPRPZMAXAd1wzgW61QX0FK1jH/oU0PdVkMMvCfBI7TtW2gI/BGFYgdcH0Fl4BMU8Ra+wlwj7wRuj9JccpXBi3r9MzMcZk0bWbp7uAdnhVLk5hcsgI9Qy739JMKQE0U+PuKdB+Fl1/FRLEFqLx083XUn4qYp0AE3o+EYTgIclNwWLPCsQw+IGE5TNXsj/AS7aNoB7g8V4ncZ4gJoyJ+RmkndzW9AZkxn3w/q8bQZZFpamIWBpyZJFkwIH/ZpHUQcXrVMPHmMMrcYFpF+mmVGar9ASxSB82pfe7/lEwVf7Np8RSih3/AXzfEv1PVjniOcduAnyPyq/v5GuZia9etzQnouLLbuIMfH+qF5U7iKX+9uKK3Yf9eVAuoT5MaoqzWyQwhArgjee0uNjqnPeJZgzMBrk9ihTCDh+BnPsJ3O0rxlaIgsBdkTvaEUU+H4JM8yC73QohJIJCuK8eHPznmjFBol+ThEWYNhZyBqpS2+ZKjPFPkxv2tkzGVSlCVosMgy0CMjumSfo0E6ezyk6dLGAltdoBH3bnStKE8U/ghij9RAGeEwv8O81zBrs3c08RN5rhu9cx3lA3jRflUo3W4d5wXcRJNVz4L3ejqUideO4uTM1WSrSGVVFGpdUkXu1+K1S06Yi91rxG6cUm1UkjP07EsLiyxjFsCz5EBCa91GhzkVVKIPpunzUioszkt8hCJUhCJUrLQZQ7Wv0ISMYHXYhRyK5K5BcNzmSq7opjeZOF5pnK5CQn942DDDxU32A5VloByyPNbdMQvl5CeznOeuiBkUQTzEaBeHx6EA+jOftdl/bMHmeV01FFgDMRHWyJE1SoAhq/YiKrhEUsSj0RL0quEqMimLEIprj0jLSu6oJv3pHM8ao+0h2plQ+wuBKU1LreSxPTdNGBm2ALa6BKQdDIe1nqM7MMgbCqLS4Z7JRs6SJmqPpfNCoqLsWB30ifPUM0wC/FgmTTga1EtUoitMZ6MNxTQ1AU0gHGkMFGnfUJL17vrigbFuUdIMjZcZmF46Ub2iw9nWLTXeRXn4G2xABn+StQETMdTzPkj4Zq2NshyHTdB7HaW9tI9wgL9TWXFkKXBsUTWkOpz8KGbZGUSAkPiAe6+wuScLtTVLkT0B4K8daYLnWbXf3sW2OIRwTV1sDjYftztzqh+u5FpRIVhK6Lot/bMWoMhDmA/c4E+GLZiZoDadZvFXhll9iFufgL9zlCvlEFQJCEQn8YmruzLQPgaqqJo+5M4x1yFMsIaLaRQHnYhhDFsPU9E2fAOAFAp+k5cj7asrvKQ7pMjlI60JEZahDfQexp1psJhtK6WmKIWnaBxRnMM7WWWdr3paU0PmYWAyJVWnzT3qnnMS5CZxhtspfsDX9NfEBFhNC8HWEdW59HnGJ8ODygOhSmTOE4my9o/EQ2dIydWYf96kt/a27NWwvlv5WjtvaA6q1LdTzQbbacWpH42xaJ+rh05ZjUM4ZPl3YDOq5gyAPRVGQzx5R4dfsAp9CKKe45IXOT40QAGGfHfsrY8TdoxkqV7YeHTJhUaygmeE4xIMZVqIEn+yCOlQRMr/6uXSo0Sfx26pDnQGuY1XvtayNGKLBGPvVxPGhURSfiMos+UGKB+qqArLGQAdUNdyWniQ7oEafyef/6eTFXj2PH5AIuUzXcQcntk0h7cGHsY44eSebPfqkudsUi2G28NEQ14uxjlplm/aFVkNmSWqKa3RGUhXVmjm2sQOfzzRY4+QwXIvPifMPsZ2jD5HNcPaJzqJxHRnz0jrlxfLPgvMa3JKArAAnUSD+WpeB4IkOSptPaWhWZ5aU71MyfSAxNjfJv0ltfX75afQ1i0+j/SiB8sJ1ljdG6jcJhGkmeyWv4GOKnogl+g/jxel9so5tpoCZsWuI0+UsU+vo2JYycKnQ3DRP1LaMKX6slbyqZvGTTd0NuXytaTZgc/mMJpkanP0X+eMtHdQERde24GM8oZC+f2gIgZ/pZGv2NQXeKzFAA+jctE2rJWkh0aDYPQzKD20nZMyd88LiDMxPiB3VuEFyOGCeuoCGMdmFDVQhtP8K82SNkVD6N8qsmESqiVqPDNJx4Ze96E07UWatZqV/bpkdeT5nuNCyJXBCRlGiDA9bkVTZmeOFIMsCj0eLl/GGaYDWtapVX4b5wY10rzg4x7Y7HBAgcXWCPlASVaetJ9mieGp8xzjh0tm0FcAvQF43gllGDicZSSoVW+cQ0q7JknaacqWHV7QEIA3ydd5zKBSSgZsuQBSE27LpRxi+QcIhlfqssO6kVlWTTbWiHB+piVEagbBS9wbSAOBvD5DTbxqbvFO9QSoNGl8pIfYnYDrFFPCCeFm/E6UJZjvapVaWEWpPKauT4h23szqypSmmT1LYKxQ1OTngYYH7Z08q9i4SXUWgK4axe0MKYKnIitN6KmN4R6nPj3r3IPyi89cAP4s8sAwup1QVcu3IztlloY5FhDWSEiK4VX/cVl7Dhx6i+3yE9/CDLAkBRTeIw2K+XflPECXYnQNxTvEIEciFW8VDSvosNx/vDBOzIQlh8sGxLgTHhrR1a3aDhr6S9RaDiDtgUaAmLfNg9wxJznTAiKwdoS10rm2lG4/Oumw67yZ4Dk/kHJr6qdH/aYNVOQ45CSU8DMDZVr8a0jaE2p22mV/JjtxHkINiYNi+9PDw2rbjthFfjI7HO9eBOpOjTJixUV7/0Tc9/PDuIZltiZGU6s4cff8x+R4lx1XOqSfgMN64fFpyfwTgOBGVpggbtdgZfFcRUznd3KoViubrrF0NFprzU5wFPmwLtLi1AN2OeBv31urJf/qHEVcpOHWlKx79JyxyVBV5SrfPpFKr0m1eS3TNs089l8GV79dxd9+lVik4p04COdcxCfTzmpzQGx7aFdnd32xRwYbLDljrv3n6DCr4Ko/n3J2hWR5Rls6eNtBbDzwVpikGEDIJNYdGkbY5gHV8kVNh5O/7tbAnaPP8rou9dE3Yk74c99BZErp1ybMk3ItPrOrc7raZolnHbdKB3eTjr3aw3ZnFIdVy6kefcwL4jsmRNYZxXty1E62fW2eK/0+nnnM61R3jgIcfUnKt+iGbnNwKoWNnNS50ewmpNU6V2uuYef0RpRZf7v/ARNl8/2c69Kd/AA==</diagram></mxfile>