Fluentd exporter for observability of the raft tracing - #8386
Draft
cjen1-msft wants to merge 17 commits into
Draft
cjen1-msft wants to merge 17 commits into
cjen1-msft wants to merge 17 commits into
Conversation
Preserve the current source, configuration, and test work before narrowing the production change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove the one-off upstream-driver comparison CLI and unrelated session-limit argument changes. Keep the generic Fluentd sink, Raft encoders, transport regressions, and real-collector e2e coverage. Pass tracing mode explicitly from CMake so the default tracing-disabled scenario test does not require a collector. Clarify the configuration contract and shorten the changelog. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep Fluentd configuration and validation alongside upstream connection timeouts and host include changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved critical and moderate findings affect resource safety, producer behavior, configuration handling, reconnects, and trace tooling.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 5
Open (7)
What changed in this PR
This POC adds an optional Fluentd TCP exporter for Raft tracing using MessagePack and per-producer SPSC queues.
Changes:
- Adds trace serialization, queueing, transport lifecycle, reconnects, and drop reporting.
- Updates Raft tooling, TLA validation, visualization, configuration, and e2e workflows.
- Adds tests, benchmarks, CI integration, documentation, and release metadata.
| File | Reviewed change / final finding |
|---|---|
tla/tlc.py |
Enables traced scenario generation. Critical (1 vote): Collector mode is selected unconditionally although tracing is disabled by default, so the default build fails before trace generation. |
tla/consensus/Traceccfraft.tla |
Uses flattened trace fields. |
tests/trace_viz.py |
Displays flattened commit indices. |
tests/raft_trace.py |
Collects Fluentd MessagePack records. Critical (1 vote): msgpack is missing from the required test dependencies. |
tests/raft_trace_test.py |
Tests trace collection and replay. Moderate (3 votes): The module is not registered in CMake or invoked by CI. |
tests/raft_trace_e2e.py |
Tests exporter startup and outages. |
tests/raft_scenarios_runner.py |
Integrates traced drivers and comparisons. Critical (3 votes): Legacy baselines emit stdout JSON rather than TCP MessagePack. Moderate (1 vote): Raw MessagePack byte comparison makes map order significant. |
tests/infra/network.py |
Forwards observability configuration. |
tests/infra/e2e_args.py |
Adds observability CLI support. Moderate (1 vote): The mapped optional schema property lacks a default, causing KeyError with host-config defaults. |
tests/fluentd_benchmark.py |
Adds exporter benchmark collection. |
tests/fluentd_benchmark_test.py |
Tests benchmark collection. |
tests/config.jinja |
Renders observability settings. |
src/tracing/trace.h |
Defines trace event emission. Critical (1 vote): Uncaught MessagePack allocation failures can terminate the node from the producer path. |
src/tracing/test/fluentd.cpp |
Tests sink and queue behavior. |
src/tracing/test/events.h |
Declares test events. |
src/tracing/test/events.cpp |
Defines cross-translation-unit test events. |
src/tracing/spsc_queue.h |
Implements producer queues. Moderate (1 vote): Per-record vector allocation and copying can block the Raft producer path. |
src/tracing/fluentd_sink.h |
Implements Fluentd transport and lifecycle. Critical (1 vote): Independent queue limits permit excessive total allocation. Moderate (1 vote): Only the first resolved address is retained for reconnects. |
src/msgpack/test/msgpack_unit.cpp |
Tests MessagePack serialization. |
src/msgpack/serialization.h |
Adds structured serializers. |
src/host/run.cpp |
Configures and binds the exporter. |
src/consensus/aft/test/driver.h |
Adds dropped-message tracing. |
src/consensus/aft/test/driver.cpp |
Adds driver exporter setup. |
src/consensus/aft/raft.h |
Emits Raft trace events. |
src/consensus/aft/raft_trace_msgpack.h |
Defines Raft MessagePack formats. |
src/common/configuration.h |
Adds JSON configuration mappings. |
python/pyproject.toml |
Bumps the package version. |
include/ccf/node/startup_config.h |
Defines exporter configuration. |
doc/host_config_schema/host_config.json |
Documents exporter settings. Moderate (1 vote): The mapped property lacks a default. Moderate (2 votes): New nested objects allow unknown properties instead of rejecting typos. |
doc/architecture/raft_tla.rst |
Documents flattened trace fields. |
CMakeLists.txt |
Registers tracing tests and benchmarks. |
CHANGELOG.md |
Records the Fluentd exporter. |
.github/workflows/README.md |
Documents tracing benchmarks. |
.github/workflows/bencher.yml |
Builds and runs tracing benchmarks. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+41
to
+42
| producers == 0 || producers > MAX_PRODUCERS || size == 0 || | ||
| size > SPSCQueue::MAX_CAPACITY) |
Comment on lines
+53
to
+60
| const auto sequence = next_sequence(); | ||
| auto& buffer = event_buffer(); | ||
| buffer.clear(); | ||
| msgpack::write_array_header(buffer, 3); | ||
| msgpack::write_str(buffer, tag); | ||
| msgpack::write_fluentd_event_time( | ||
| buffer, | ||
| msgpack::FluentdEventTime::make(std::chrono::system_clock::now())); |
| if args.raft_tracing: | ||
| proc, records = run_driver(args.driver, scenario) | ||
| if args.compare_driver: | ||
| baseline, baseline_records = run_driver(args.compare_driver, scenario) |
| import subprocess | ||
| import threading | ||
|
|
||
| import msgpack |
| sys.executable, | ||
| args.scenarios_runner, | ||
| args.raft_driver, | ||
| "--raft-tracing", |
Comment on lines
+33
to
+37
| }, | ||
| "required": ["host", "port"] | ||
| } | ||
| } | ||
| }, |
| self.assertEqual(records, [record]) | ||
|
|
||
|
|
||
| if __name__ == "__main__": |
Preserve Fluentd tracing alongside upstream successor locking and ledger subsystem initialization. Declare build dependencies for the tracing e2e tests required by upstream CMake. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
cjen1-msft
changed the title
Keep ordinary tracing-disabled measurements intact, then run the existing paired export-disabled and export-enabled benchmark with a tracing-enabled build. Upload collector counts and node diagnostics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove the tracing compile guards while preserving ordinary log timestamps and JSON string messages. Register trace tests in default builds and verify unconfigured emission does not allocate. Run the existing A/B workloads without a collector and exclude the separate paired export benchmark so this commit isolates unconditional tracing overhead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
cjen1-msft
marked this pull request as draft
September 22, 2026 17:12
Reuse the collector for every existing Locust workload and signature interval, configuring all nodes and requiring trace reception from each. Preserve benchmark names and worker settings so the radar charts compare export-enabled results with main and earlier branch runs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a proof of concept for the fluentd observability sink.
I've implemented it using the raft tracing as a benchmark (as getting that data out is a major outcome of this work).
Architecture
sequenceDiagram participant H as Host thread participant E as Enclave thread (each) participant Q as Per-thread SPSC queue participant L as Logger consumer thread participant F as Fluentd H->>H: Configure exporter H->>Q: Preallocate one queue per producer H->>L: Start consumer activate L L->>F: Attempt TCP connection H->>H: Bind host's queue H->>E: Start enclave threads activate E E->>E: Bind assigned queue to thread-local pointer Note over E,Q: Queues are statically allocated by host loop Emit events E->>E: Serialize event to MessagePack E->>Q: Enqueue owned record activate Q alt Slot available Q-->>E: Accepted else Queue full Q-->>E: Rejected - drop end deactivate Q Note over E: Continue without waiting end loop Drain queues independently L->>Q: Read next record activate Q Q-->>L: Record bytes, retained until send finishes L->>F: Send bytes L->>Q: Destroy record and release slot deactivate Q L->>L: Report drop thresholds opt No work available L->>L: Sleep 1 ms end end H->>E: Stop and join producers deactivate E H->>L: Request shutdown L->>Q: Drain for up to 2 seconds L->>L: Count remaining records as dropped L->>F: Close TCP connection L-->>H: Consumer joined deactivate LTo head off some questions. The primary thing that impacts throughput is any delays on the raft side.
So the entire implementation needs to be non-blocking on the producer side, which unfortunately seems to include notifying waiting consumers. Hence the slightly odd consumer design.
Benchmarking
This is using the blocking basic benchmark, two nodes, 1k clients, 4k slots in the ringbuffer, and a RAM disk as this was running on an ACI machine.
The specific results are the median throughput run for each.
(with #8117 reverted locally)
Open questions