Expose recent committed ledger prefixes

[Amaury Chamayou (achamayou)]

Summary

• Add opt-in include_committed_prefix=true discovery for committed ledger entries not yet available in canonical .committed files.
• Serve immutable .committed_prefix resources with 307 Temporary Redirect, Cache-Control: no-store, range requests, ETags, and representation digests.
• Preserve physical ledger boundaries and keep prefix files excluded from committed-only discovery and recovery.
• Materialise prefixes without stalling the ledger writer: the chunk header and positions table are built under the ledger state lock, then the entry bytes are read with pread() on a duplicated descriptor with no ledger lock held. pread() never touches the writer's FILE* position, and the range is stable because Ledger::truncate never regresses past committed_idx. Committed bytes are always flushed (committable entries and completed files fflush), so a violation surfaces as a short read (HTTP 500), never as incorrect bytes.
• Ignore .committed_prefix files in ccf.ledger.Ledger directory discovery, matching the host; they remain readable individually with ccf.ledger.LedgerChunk.
• Add host and Python tests, endpoint e2e coverage, a 64 KiB-50 MiB benchmark matrix, OpenAPI updates, and operator documentation.

Design discussion: #8207

Testing

• ledger_test (including a case with non-committable entries inside the served range and an unflushed tail after it) and files_cleanup_test
• Python SDK test suite, including Ledger ignoring .committed_prefix files
• ledger_bench: pread() materialisation runs at 0.59-1.08x of a plain read of the same file across 64 KiB-50 MiB
• schema_test committed-prefix-download scenario, pinned with a high snapshot_tx_interval so a due snapshot cannot promote the source file to .committed mid-test
• Node OpenAPI validation
• Full logging sample build

[Copilot]

Pull request overview

Adds opt-in access to recent committed ledger prefixes while preserving canonical ledger and recovery behavior.

Changes:

• Adds prefix discovery, immutable downloads, ranges, digests, and metadata.
• Extends host and Python ledger parsing.
• Adds tests, benchmarks, OpenAPI definitions, and operator documentation.

Custom instructions used:

• .github/copilot-instructions.md
• .github/instructions/changelog.instructions.md
• .github/instructions/reviewing.instructions.md

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
CHANGELOG.md	Records the new capability.
include/ccf/http_consts.h	Adds the chunk-kind header.
src/host/ledger.h	Materializes committed prefixes.
src/host/ledger_filenames.h	Recognizes and excludes prefix files.
src/host/test/ledger.cpp	Tests prefix creation and recovery exclusion.
src/host/test/ledger_bench.cpp	Benchmarks prefix materialization.
src/host/test/files_cleanup_test.cpp	Tests strict prefix filenames.
src/node/rpc/file_serving_handlers.h	Implements discovery and download endpoints.
src/node/rpc/ledger_interface.h	Extends the ledger subsystem interface.
src/node/rpc/ledger_subsystem.h	Connects RPC and host ledger operations.
src/node/rpc/node_frontend.h	Bumps the API version.
python/src/ccf/ledger.py	Parses committed-prefix filenames.
python/tests/test_ledger.py	Tests Python filename handling.
tests/e2e_operations.py	Adds endpoint integration coverage.
tests/schema.py	Registers the new e2e scenario.
doc/operations/configuration.rst	Documents operator feature gating.
doc/operations/ledger_snapshot.rst	Documents prefix semantics and usage.
doc/schemas/node_openapi.json	Updates the generated API schema.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Andrea Piccione (andpiccione)]

What's the expected pattern that operators should follow to keep canonical backup progress separate from prefix progress?

With the current API, an external synchronization process advances since to the downloaded chunk’s end plus one. By passing this new prefix parameter, after downloading a prefix covering 101–140, it would ask for 141. If the physical chunk later becomes ledger_101-140.committed, for example through a forced ledger chunk before 141, that cursor cannot discover it anymore.

Since prefixes cannot replace canonical files for recovery, would operators need an independent canonical reconciliation pass that removes prefix files older than the latest committed file? An example covering this specific transition phase would be helpful for clarifying this part.

[Amaury Chamayou (achamayou)]

The committed prefix files deliberately have a distinct suffix to differentiate them, and allow the nodes to ignore them on DR if they somehow get written to a ledger directory.

It's not clear to me that the synchroniser is necessarily the job that should be downloading these files, but if it is, it must not update its persistency watermark when it does. If since=101 and the return value is a 101_138.committed_prefix, then the value of since must remain 101, and the next download attempt from since=101 may yield a final 101_140.committed.

The goal of this API is not to be used to allow the synchroniser to back up files more aggressively (the disk size benefit would be negligible), it is to allow exposing a complete ledger to auditors on a relatively slow-moving ledger, where the current chunk stays open for longer than it would take for the auditor to fetch the files. Incomplete chunks ideally get downloaded to a separate directory, and are treated as ephemeral.

[Andrea Piccione (andpiccione)]

Could we clarify the scope of “immutable once returned,” especially across disaster recovery?

For example, let's say we persist a prefix covering 101–140 while the latest available canonical backup ends at 100. If recovery resumes from that older backup, the saved prefix and indexed rows are no longer necessarily part of the recovered history.

What's a good way for consumers to detect that a prefix file is no longer part of the recovered history and how can they reconcile their saved prefixes?

[Amaury Chamayou (achamayou)]

Ideally, the operator makes sure that this does not happen. If it does though, receipts are still the way to assign blame: https://ccf.dev/main/audit/receipts.html#denounce-an-invalid-recovery.

If desired though, it's possible to scan through the chunk and find the view change and/or service identity update transaction.

[Andrea Piccione (andpiccione)]

Does this allocate and fill the whole response buffer with zeros while holding the ledger state lock? If ledger writes and commits use the same lock, large downloads could potentially delay writes, right?

I'm wondering if this could be a problem, especially for large chunks (we use 100 MB chunks). Could we move the large allocation outside the lock?

[Amaury Chamayou (achamayou)]

So that files_lock I think is no longer necessary, and the locks in this file don't use the clang scoping macros we have elsewhere and that are quite good for safety, so I will make a small detour and tidy this up before coming back to this. In short though, yes we could allocate outside I think, although the impact is probably not very high.

[Andrea Piccione (andpiccione)]

A request for just eight bytes, such as Range: bytes=0-7, still allocates and reads the entire prefix before returning those bytes, even when no full digest is requested. Invalid ranges also do this work before being rejected.

Could we validate the range first and avoid reading the whole prefix when possible? Otherwise, downloading a large prefix in small pieces repeatedly makes the node read the same data.