Remove ringbuffer usage for ledger mutations and range reads - #8405
Eddy Ashton (eddyashton) wants to merge 13 commits into
Conversation
Replace the ledger_init/append/truncate/commit/open and ledger_get_range/ entry_range/no_entry_range ringbuffer messages with typed ledger interfaces. A host-owned LedgerSubsystem runs mutations, range classification and reads of uncommitted state in FIFO order on one OrderedTasks lane, and reads that lie wholly within committed files as ordinary concurrent tasks. Entries are moved into owned storage before submission and results are delivered as owned values through typed callbacks. Ledger::init now lowers the committed-file classification boundary when it un-commits later files, so a read classified after init can never target a file about to be replayed into. Range results distinguish an absent range from an entry exceeding the read budget; recovery fails explicitly on the latter rather than treating it as end-of-ledger. The read budget remains derived from memory.max_msg_size so no configuration changes. host may now depend on tasks (a leaf component). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Ledger mutations now reach the host through the ledger OrderedTasks lane rather than the ringbuffer, but node-to-node messages still arrive on the ringbuffer. The two queues lost the emission order the enclave relies on: an AppendEntries message could be processed before the append it refers to had been applied, and the host dropped it. Restore a single order by submitting each outbound node message to the same lane as ledger mutations. The lane action reads any ledger entries and assembles the complete frame; the result is queued for the libuv thread, which owns the sockets and drains the queue on the existing 1ms cadence. Address and close updates take the same path so their order relative to sends is preserved. This is temporary until node-to-node transport leaves the ringbuffer. Also fix node_connections_test for the Ledger constructor change and the messaging.h include that ledger.h no longer provides. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Review findings on the typed ledger subsystem: - shutdown() ran after the enclave task workers had exited and only closed the gate, so an append or commit which had been accepted but not yet executed was never written. The ringbuffer design drained remaining messages before stopping the loop. Drain the ordered lane synchronously in shutdown() before closing the gate. - Reads of committed files bypassed the ledger state lock. A read already dispatched before init() could then observe a file being un-committed and rewritten. The libuv threadpool read previously took that lock; do so again, so committed reads are dispatched off the lane but their file access is serialised against mutations as before. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…edger-ringbuffer-removal-implementation
There was a problem hiding this comment.
🟡 Changes recommended
Shutdown can continue recovery callbacks after enclave threads stop, and new oversized-entry messages reference a nonexistent configuration setting.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Moves ledger mutations and range reads from ringbuffer messages to typed, task-backed interfaces, while preserving ledger ordering and node-to-node AppendEntries assembly.
Changes:
- Adds typed ledger reader/writer interfaces and FIFO host processing.
- Reworks node transport ordering and ledger shutdown handling.
- Updates ledger, consensus, historical-query, indexing, and host tests.
Custom instructions used
.github/copilot-instructions.md.github/instructions/reviewing.instructions.md
File summaries
| File | Description |
|---|---|
src/host/ledger_subsystem.h |
Implements typed ledger operations and task ordering. |
src/host/ledger.h |
Removes ringbuffer handlers and adds range status handling. |
src/host/node_connections.h |
Orders outbound node frames with ledger operations. |
src/host/run.cpp |
Constructs and shuts down the ledger subsystem. |
src/node/node_state.h |
Uses typed ledger recovery and mutation calls. |
src/node/historical_queries.h |
Uses typed historical range reads. |
src/consensus/ledger_enclave.h |
Sends owned typed ledger mutations. |
src/consensus/ledger_enclave_types.h |
Defines typed ledger interfaces and results. |
src/node/rpc/ledger_interface.h |
Defines the combined ledger subsystem interface. |
src/enclave/enclave.h |
Passes and shuts down the ledger subsystem. |
src/enclave/main.cpp |
Updates enclave creation interface. |
src/enclave/entry_points.h |
Updates enclave entry-point declaration. |
src/node/rpc/file_serving_handlers.h |
Uses the renamed ledger subsystem interface. |
src/consensus/test/ledger_stub.h |
Adds typed ledger test stubs. |
src/consensus/aft/test/enclave.cpp |
Tests typed mutation submission. |
src/node/test/historical_queries.cpp |
Migrates historical-query tests. |
src/indexing/test/indexing.cpp |
Migrates indexing tests. |
src/host/test/ledger.cpp |
Adds subsystem ordering and lifecycle tests. |
src/host/test/node_connections.cpp |
Exercises ledger-backed node connections. |
scripts/source-dependencies.json |
Records the host-to-tasks dependency. |
CMakeLists.txt |
Updates affected test link dependencies. |
Review details
- Files reviewed: 21/21 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
performance-enum-size on LedgerRangeStatus, a redundant access specifier in NodeConnectionsImpl, and std::move of a const shared_ptr in Enclave. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ages Review findings on #8405: - The shutdown drain executed queued range reads as well as mutations. A recovery read callback submits the next batch, so stopping mid-recovery could keep the host thread recovering the whole ledger after the enclave threads had joined. Set a draining flag before the drain: it rejects new submissions and makes queued reads (and committed-read tasks) skip their callback, while mutations still reach disk. - The oversized-entry log messages named ledger.max_read_size, which is not a setting. Describe the derived budget instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The test must SIGTERM the primary while it is reading the private ledger, which takes about 0.8s for the 2000 entries the test writes. It did not start polling until recover_with_shares() returned, which is 0.3-1.5s after the final share is accepted depending on client request latency, so on a slow client the read had finished before the first poll. Measured against main with identical read start (about 180ms after the final share) and replay rate (about 3.5k entries/s): the product behaviour is the same; only the client-side slack differs. Submit the shares from a thread and poll the primary immediately over a pre-established connection, so the first observation is bounded by the poll round trip rather than by the share-submission tail. Poll only the primary: followers begin reading later, once the primary broadcasts the ledger secrets, and need not be observed for this scenario. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
eb0f257 to
f3cf10e
Compare
Extract the body of the leader-only branch at the end of private ledger recovery into open_recovered_service(), which takes only the transaction, share manager and service key it uses. It is now testable without a NodeState harness, of which the repository has none. open_recovered_service_test covers a store in the recovering state: the service becomes OPEN, submitted shares are cleared, fresh shares are issued and the previous identity is endorsed; and that it throws for any status other than WAITING_FOR_RECOVERY_SHARES, which is the at-most-once guard that prevents a second node from opening a service another has already opened. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The test tried to SIGTERM the primary inside the window where it is reading the private ledger, by polling quickly enough. Nothing guaranteed that: the window is under a second and shrank relative to the client's turnaround between accepting the final share and the first poll, so the test failed deterministically in CI. Instead, submit the shares to the primary and SIGTERM it as soon as the final share is accepted, then assert the outcome: a new view is elected, every survivor reaches PartOfNetwork, the service is open, and no survivor died attempting a second opening. Whether the stop notice lands before or after the primary finishes reading is a race the test no longer needs to win: if the primary wrote the opening transaction and then stepped down before it replicated, the new leader rolls it back and opens the service itself, which both local runs exercised. A node which is not primary refusing to open, and opening being at-most-once, are covered by open_recovered_service_test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
f3cf10e to
c240d85
Compare
…edger-ringbuffer-removal-implementation
PreviousServiceIdentityEndorsement became a map keyed by IdentityType in #8401, so look up the CLASSICAL endorsement. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
DescriptionComparing 3 available runs from this branch (#8405) against the trend of the last 30 Each chart plots every benchmark as an axis, with values normalized so 100 is the EWMA baseline of recent Axis labels show the latest branch value and its difference from the main EWMA baseline, where 0% is on the baseline. They are coloured green where the latest run improves on the baseline, red where it regresses, and grey where the difference is within one std dev of the baseline (within noise). Higher is better for throughput and rate, lower for latency and memory. A benchmark which does not exist on Throughput (tx/s)---
config:
radar:
width: 620
height: 620
marginTop: 90
marginRight: 220
marginBottom: 60
marginLeft: 220
axisLabelFactor: 1.12
curveTension: 0.08
theme: base
themeCSS: |
.radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
.radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.40!important}
.radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.50!important}
.radarCurve-6{stroke-width:1.75px!important;stroke-opacity:1.00!important}
.radarAxisLabel:nth-of-type(1){fill:#808A94!important}
.radarAxisLabel:nth-of-type(2){fill:#808A94!important}
.radarAxisLabel:nth-of-type(3){fill:#808A94!important}
.radarAxisLabel:nth-of-type(4){fill:#808A94!important}
.radarAxisLabel:nth-of-type(5){fill:#808A94!important}
.radarAxisLabel:nth-of-type(6){fill:#808A94!important}
.radarAxisLabel:nth-of-type(7){fill:#808A94!important}
themeVariables:
cScale0: "#62B5E5"
cScale1: "#62B5E5"
cScale2: "#62B5E5"
cScale3: "#62B5E5"
cScale4: "#F97316"
cScale5: "#F97316"
cScale6: "#F97316"
radar:
axisColor: "#9CA3AF"
graticuleColor: "#E5E7EB"
graticuleOpacity: 0
axisStrokeWidth: 1
curveOpacity: 0
---
radar-beta
axis b0["Basic Blocking 100ms: 3,095 tx/s ▬ 0%"]
axis b1["Basic Blocking 20ms: 15,330 tx/s ▬ 0%"]
axis b2["Basic Blocking 2ms: 50,440 tx/s ▬ -1%"]
axis b3["Basic JS: 16,142 tx/s ▬ 0%"]
axis b4["Historical Queries: 1,002,241 tx/s ▬ 0%"]
axis b5["L…g Certificate Blocking: 29,553 tx/s ▬ 0%"]
axis b6["Logging JWT Blocking: 15,369 tx/s ▬ 0%"]
curve stddev2_high["main EWMA + 2 std dev"]{100.81, 100.87, 111.95, 117.14, 114.81, 101.60, 100.71}
curve stddev1_high["main EWMA + 1 std dev"]{100.41, 100.43, 105.98, 108.57, 107.41, 100.80, 100.35}
curve stddev1_low["main EWMA - 1 std dev"]{99.59, 99.57, 94.02, 91.43, 92.59, 99.20, 99.65}
curve stddev2_low["main EWMA - 2 std dev"]{99.19, 99.13, 88.05, 82.86, 85.19, 98.40, 99.29}
curve branch_0["#8405 (2 runs earlier)"]{99.87, 99.98, 98.74, 99.14, 100.00, 100.12, 100.07}
curve branch_1["#8405 (1 run earlier)"]{98.95, 99.28, 84.02, 77.18, 80.12, 97.95, 99.12}
curve branch_2["#8405"]{100.01, 99.95, 98.80, 100.43, 100.37, 100.28, 100.09}
graticule polygon
max 132
min 63
ticks 0
showLegend false
Latency (ms)---
config:
radar:
width: 620
height: 620
marginTop: 90
marginRight: 220
marginBottom: 60
marginLeft: 220
axisLabelFactor: 1.12
curveTension: 0.08
theme: base
themeCSS: |
.radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
.radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.40!important}
.radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.50!important}
.radarCurve-6{stroke-width:1.75px!important;stroke-opacity:1.00!important}
.radarAxisLabel:nth-of-type(1){fill:#808A94!important}
.radarAxisLabel:nth-of-type(2){fill:#808A94!important}
.radarAxisLabel:nth-of-type(3){fill:#E5484D!important}
.radarAxisLabel:nth-of-type(4){fill:#808A94!important}
.radarAxisLabel:nth-of-type(5){fill:#808A94!important}
.radarAxisLabel:nth-of-type(6){fill:#808A94!important}
.radarAxisLabel:nth-of-type(7){fill:#808A94!important}
themeVariables:
cScale0: "#62B5E5"
cScale1: "#62B5E5"
cScale2: "#62B5E5"
cScale3: "#62B5E5"
cScale4: "#F97316"
cScale5: "#F97316"
cScale6: "#F97316"
radar:
axisColor: "#9CA3AF"
graticuleColor: "#E5E7EB"
graticuleOpacity: 0
axisStrokeWidth: 1
curveOpacity: 0
---
radar-beta
axis b0["Basic Blocking 100ms: 99 ms ▬ 0%"]
axis b1["Basic Blocking 20ms: 19 ms ▬ 0%"]
axis b2["Basic Blocking 2ms: 6 ms ▲ 16%"]
axis b3["Basic JS: 19 ms ▬ +1%"]
axis b4["Historical Queries: 31 ms ▬ +3%"]
axis b5["Logging Certificate Blocking: 19 ms ▬ 0%"]
axis b6["Logging JWT Blocking: 19 ms ▬ 0%"]
curve stddev2_high["main EWMA + 2 std dev"]{100.86, 100.00, 125.14, 120.59, 119.18, 100.00, 103.57}
curve stddev1_high["main EWMA + 1 std dev"]{100.43, 100.00, 112.57, 110.30, 109.59, 100.00, 101.78}
curve stddev1_low["main EWMA - 1 std dev"]{99.57, 100.00, 87.43, 89.70, 90.41, 100.00, 98.22}
curve stddev2_low["main EWMA - 2 std dev"]{99.14, 100.00, 74.86, 79.41, 80.82, 100.00, 96.43}
curve branch_0["#8405 (2 runs earlier)"]{100.39, 100.00, 96.74, 100.60, 95.95, 100.00, 99.74}
curve branch_1["#8405 (1 run earlier)"]{100.39, 100.00, 135.44, 127.08, 129.03, 100.00, 104.99}
curve branch_2["#8405"]{100.39, 100.00, 116.09, 100.60, 102.57, 100.00, 99.74}
graticule polygon
max 157
min 53
ticks 0
showLegend false
Memory (bytes)---
config:
radar:
width: 620
height: 620
marginTop: 90
marginRight: 220
marginBottom: 60
marginLeft: 220
axisLabelFactor: 1.12
curveTension: 0.08
theme: base
themeCSS: |
.radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
.radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.40!important}
.radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.50!important}
.radarCurve-6{stroke-width:1.75px!important;stroke-opacity:1.00!important}
.radarAxisLabel:nth-of-type(1){fill:#808A94!important}
.radarAxisLabel:nth-of-type(2){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(3){fill:#808A94!important}
.radarAxisLabel:nth-of-type(4){fill:#808A94!important}
.radarAxisLabel:nth-of-type(5){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(6){fill:#808A94!important}
.radarAxisLabel:nth-of-type(7){fill:#808A94!important}
themeVariables:
cScale0: "#62B5E5"
cScale1: "#62B5E5"
cScale2: "#62B5E5"
cScale3: "#62B5E5"
cScale4: "#F97316"
cScale5: "#F97316"
cScale6: "#F97316"
radar:
axisColor: "#9CA3AF"
graticuleColor: "#E5E7EB"
graticuleOpacity: 0
axisStrokeWidth: 1
curveOpacity: 0
---
radar-beta
axis b0["Basic Blocking 100ms: 88 MiB ▬ 0%"]
axis b1["Basic Blocking 20ms: 87.7 MiB ▼ 2%"]
axis b2["Basic Blocking 2ms: 92.5 MiB ▬ +2%"]
axis b3["Basic JS: 97.5 MiB ▬ +1%"]
axis b4["Historical Queries: 143 MiB ▼ 4%"]
axis b5["Logging Certificate Blocking: 114 MiB ▬ -1%"]
axis b6["Logging JWT Blocking: 87.2 MiB ▬ 0%"]
curve stddev2_high["main EWMA + 2 std dev"]{102.90, 103.37, 103.08, 104.12, 101.14, 102.55, 103.68}
curve stddev1_high["main EWMA + 1 std dev"]{101.45, 101.68, 101.54, 102.06, 100.57, 101.28, 101.84}
curve stddev1_low["main EWMA - 1 std dev"]{98.55, 98.32, 98.46, 97.94, 99.43, 98.72, 98.16}
curve stddev2_low["main EWMA - 2 std dev"]{97.10, 96.63, 96.92, 95.88, 98.86, 97.45, 96.32}
curve branch_0["#8405 (2 runs earlier)"]{98.68, 98.89, 100.07, 100.60, 94.47, 97.09, 99.66}
curve branch_1["#8405 (1 run earlier)"]{97.01, 99.69, 97.79, 95.52, 93.53, 96.94, 98.06}
curve branch_2["#8405"]{100.01, 98.12, 101.51, 101.18, 96.00, 99.36, 99.79}
graticule polygon
max 109
min 89
ticks 0
showLegend false
Rate (ops/s)---
config:
radar:
width: 620
height: 620
marginTop: 90
marginRight: 220
marginBottom: 60
marginLeft: 220
axisLabelFactor: 1.12
curveTension: 0.08
theme: base
themeCSS: |
.radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
.radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
.radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.40!important}
.radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.50!important}
.radarCurve-6{stroke-width:1.75px!important;stroke-opacity:1.00!important}
.radarAxisLabel:nth-of-type(1){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(2){fill:#808A94!important}
.radarAxisLabel:nth-of-type(3){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(4){fill:#2DA44E!important}
.radarAxisLabel:nth-of-type(5){fill:#808A94!important}
.radarAxisLabel:nth-of-type(6){fill:#808A94!important}
.radarAxisLabel:nth-of-type(7){fill:#808A94!important}
.radarAxisLabel:nth-of-type(8){fill:#808A94!important}
.radarAxisLabel:nth-of-type(9){fill:#808A94!important}
themeVariables:
cScale0: "#62B5E5"
cScale1: "#62B5E5"
cScale2: "#62B5E5"
cScale3: "#62B5E5"
cScale4: "#F97316"
cScale5: "#F97316"
cScale6: "#F97316"
radar:
axisColor: "#9CA3AF"
graticuleColor: "#E5E7EB"
graticuleOpacity: 0
axisStrokeWidth: 1
curveOpacity: 0
---
radar-beta
axis b0["CCF c…n context lifecycle: 21,801 ops/s ▲ 2%"]
axis b1["CCF fresh JS invocation: 19,833 ops/s ▬ +1%"]
axis b2["CHAMP get: 66,179,797 ops/s ▲ 3%"]
axis b3["CHAMP put: 8,418,214 ops/s ▲ 4%"]
axis b4["KV deserialisation: 2,820,874 ops/s ▬ +3%"]
axis b5["KV serialisation: 2,459,420 ops/s ▬ +1%"]
axis b6["KV s…t deserialisation: 6,329 ops/s ▬ +2%"]
axis b7["KV snapshot serialisation: 4,892 ops/s ▬ +1%"]
axis b8["Q…S s…d c…t lifecycle: 26,552 ops/s ▬ +1%"]
curve stddev2_high["main EWMA + 2 std dev"]{103.30, 103.85, 104.67, 105.35, 112.93, 114.84, 104.02, 105.49, 103.46}
curve stddev1_high["main EWMA + 1 std dev"]{101.65, 101.92, 102.33, 102.67, 106.46, 107.42, 102.01, 102.74, 101.73}
curve stddev1_low["main EWMA - 1 std dev"]{98.35, 98.08, 97.67, 97.33, 93.54, 92.58, 97.99, 97.26, 98.27}
curve stddev2_low["main EWMA - 2 std dev"]{96.70, 96.15, 95.33, 94.65, 87.07, 85.16, 95.98, 94.51, 96.54}
curve branch_0["#8405 (2 runs earlier)"]{99.41, 100.71, 99.38, 101.26, 88.24, 84.77, 100.20, 98.07, 100.51}
curve branch_1["#8405 (1 run earlier)"]{99.82, 100.60, 99.22, 95.52, 100.99, 99.27, 98.14, 105.69, 99.91}
curve branch_2["#8405"]{101.76, 101.22, 103.04, 104.08, 102.73, 101.46, 101.80, 100.60, 101.34}
graticule polygon
max 126
min 74
ticks 0
showLegend false
|
…edger-ringbuffer-removal-implementation
…edger-ringbuffer-removal-implementation
Motivation
Next step of the ringbuffer-removal plan, after #8395 and #8394: take all ledger traffic off the host/enclave ringbuffer. This covers the five mutation messages (
ledger_init,ledger_append,ledger_truncate,ledger_commit,ledger_open) and the range-read request/response messages (ledger_get_range,ledger_entry_range,ledger_no_entry_range), replacing serialised messages with typed C++ interfaces and owned values.Node-to-node transport and
tickremain on the ringbuffer and are out of scope.Implementation summary
Typed interfaces.
consensus::AbstractLedgerWriter(init/append/truncate/commit/open) andconsensus::AbstractLedgerReader(get_rangewith a typed callback delivering an ownedLedgerRangeResult).LedgerEnclave,NodeStaterecovery, and the historicalStateCachetake these instead of a ringbuffer writer.Host
LedgerSubsystem(src/host/ledger_subsystem.h) wraps the existing synchronousLedger:ccf::tasks::OrderedTaskslane on the main job board. Each append owns its entry bytes before submission. This preserves the single-FIFO ordering the ringbuffer provided (append before commit/truncate, init before first read, open after recovery mutations).Ledgerstate lock, as the previous libuv-threadpool read did.WorkerShutdownGateand waits for in-flight callbacks. Draining is what the ringbuffer design achieved by reading remaining messages before stopping the loop: an accepted append/commit must reach disk.Range results distinguish
Found,NotFound, andTooLarge. Previously an entry exceeding the read budget producedno_entry_range, which recovery treated as end-of-ledger and silently completed early. Recovery now fails explicitly; historical queries log and drop the request. Only reachable with a ledger written beforeledger.max_transaction_sizeexisted, so no CHANGELOG entry. The read budget staysmemory.max_msg_size - ledger_range_response_metadata_size; no configuration changes. A ledger-named setting will replace it whenmemory.*is removed.Ledger::initfix.initun-commits later files so they can be replayed into, but did not lowerend_of_committed_files_idx, so those files were still classified as committed for readers. It now does. (Latent in the old design too.)Node-to-node ordering (second commit). Node messages still arrive on the ringbuffer, on a different queue from ledger mutations, which lost the emission order that AppendEntries attachment relied on: the host could read the ledger before the append had landed and dropped the AE (observed 5-8 times per e2e run at
[fail]). Each outbound node message is now submitted to the ledger lane, where any entries are read and the frame assembled as owned bytes, then queued as typed data for the libuv thread, which drains it on a 1ms timer (the same cadence the ringbuffer is drained at). Address and close updates take the same path so their order relative to sends is preserved.The ordering argument: on the enclave thread,
ledger->put_entry(N)pushes to the lane, then writesnode_outbound(AE ...N])to the ringbuffer (raft.hreplicate()); the host uv thread drains that ringbuffer message and pushes a lane action;SubTaskQueue::pushis mutex-ordered, so the lane seesappend(N)before the AE action; the lane is FIFO, soassemble_framereads after the append has landed. The host applies everything in enclave emission order, as the single ringbuffer did. Heartbeats queue behind ledger IO as before. This routing is temporary until node-to-node transport leaves the ringbuffer.Also:
hostmay now depend ontasks(a leaf component);ds/messaging.his included explicitly whereledger.hpreviously provided it transitively;node_connections_testgains a realLedgerSubsystemon a testJobBoard.Follow-up, separate PR: make the enclave dispatch loop ingress-only and require
worker_threads >= 1, so opaque blocking tasks can never run on the consensus ingress thread.Safety and compatibility
init: serialised on theLedgerstate lock as before; classification is now truthful afterinit.LedgerEnclavethrows if the subsystem rejects a mutation after shutdown has begun.Validation: unit suites
ledger_test,node_connections_test,raft_test,raft_enclave_test,historical_queries_test,indexing_test; e2erecovery_testandschema_test(file operations exercise the read budget) with zeroUnable to send AppendEntriesin node logs; format, include-policy, ASCII, copyright checks. Reviewed independently (two findings, both fixed in the third commit).