Skip to content

fix(mem_forest): make deserialize total on untrusted input - #160

Open
USCMig wants to merge 2 commits into
mit-dci:mainfrom
USCMig:fix-memforest-deserialize-untrusted
Open

USCMig wants to merge 2 commits into
mit-dci:mainfrom
USCMig:fix-memforest-deserialize-untrusted

Conversation

@USCMig

@USCMig USCMig commented Sep 14, 2026

Copy link
Copy Markdown

MemForest::deserialize is not total on untrusted input

deserialize returns io::Result, but two inputs make it abort or unwind
instead of returning Err. Both are reachable by anyone who can hand a node a
serialized forest — a snapshot file on disk, or a peer.

Found while fuzzing snapshot decoding in a downstream crate. Two of our seven
fuzz targets reach MemForest::deserialize and both died within seconds, so
they had to be excluded from the campaign entirely until this was fixed.

1. An out-of-range node type panics

The type tag is a u64 read straight from the input and matched against 0
(branch) and 1 (leaf). Everything else hit panic!("Invalid node type").
25 bytes are enough. Now an InvalidData error.

2. Unbounded recursion overflows the stack

_read_one recurses once per branch node with no bound, so the input chooses
the recursion depth
. A left spine of nested branches overflows the stack —
~1.2 MB of input in a debug build, ~4 MB in release.

This is the more serious of the two. A stack overflow aborts the process
rather than unwinding
, so a caller cannot contain it with catch_unwind the
way it can contain a panic. We had wrapped the call in catch_unwind
specifically to contain issue 1, and discovered while fixing it that the
containment had never covered issue 2 at all.

The bound is MAX_FOREST_ROWS, and it is not arbitrary: a root is a perfect
tree of at most that many rows, so its leaves sit at most that many levels below
it, and anything deeper is not a forest this crate could have serialized. A
spine exactly MAX_FOREST_ROWS deep still round-trips; 64 is rejected.

3. serialize unwraps a writer error (separate, second commit)

serialize returns io::Result<()> and ?s its two length prefixes, then
unwrap()ed the result of writing each root. A writer that fails partway — full
disk, closed pipe, socket gone — panicked instead of returning the Err the
signature promises.

This one is about a failing writer, not untrusted input. It is a separate
commit and can be dropped if you would rather take only the deserialize fixes.

Testing

Four regression tests, all in src/mem_forest/mod.rs:

test covers
test_deserialize_rejects_invalid_node_type the exact fuzzer seed
test_deserialize_rejects_every_out_of_range_node_type 2, 3, 255, 256, u32::MAX, u64::MAX
test_deserialize_rejects_unbounded_nesting accepts depth MAX_FOREST_ROWS, rejects 64 / 100 / 10,000
test_serialize_reports_writer_errors a writer that runs out of room mid-root

Each fix was reverted individually and confirmed to fail exactly its own
test and nothing else
— re-verified against this branch's merge base, not just
when it was written:

reverted node-type guard   -> test_deserialize_rejects_invalid_node_type,
                              test_deserialize_rejects_every_out_of_range_node_type
reverted depth bound       -> test_deserialize_rejects_unbounded_nesting
reverted serialize `?`     -> test_serialize_reports_writer_errors

The five crash artifacts our fuzzer produced all replay clean against this
branch. With it applied, the two previously-excluded targets ran 10M and 300k
executions with no crashes, taking one corpus from 8 inputs to 514.

cargo test --all green (93 tests), cargo clippy --all-targets clean,
cargo +nightly fmt --check clean. No public API change; no behaviour change
for any input the crate itself can produce.

Note on the base

This branch is only these two commits, on top of main. It does not include
the calculate_hashes fix from #152, which we also carry downstream — that is
yours to land separately and I did not want to entangle the two.

`MemForest::deserialize` returns `io::Result`, but two inputs made it
abort or unwind instead of returning `Err`. Both are reachable by anyone
who can hand a node a serialized forest -- a snapshot file, a peer.

1. A node's type tag is a `u64` read straight from the input and matched
   against 0 (branch) and 1 (leaf); everything else hit `panic!("Invalid
   node type")`. 25 bytes are enough. Now an `InvalidData` error.

2. `_read_one` recurses once per branch node with no bound, so the input
   chose the recursion depth. A left spine of nested branches overflows
   the stack: ~1.2 MB in a debug build, ~4 MB in release. That one is
   the more serious of the two, because a stack overflow aborts the
   process rather than unwinding, so a caller cannot contain it with
   `catch_unwind` the way it can contain a panic.

   A root is a perfect tree of at most `MAX_FOREST_ROWS` rows, so its
   leaves sit at most that many levels below it and anything deeper
   cannot be a forest this crate serialized. Bounding the recursion
   there rejects the crafted input and accepts every real one: the
   existing suite, including the JSON forest cases, is unchanged, and a
   spine exactly `MAX_FOREST_ROWS` deep still round-trips.

Found while fuzzing snapshot decoding in a downstream crate. Each fix
has a regression test, and each test was confirmed to fail with only its
own fix reverted.
…king

`serialize` returns `io::Result<()>` and uses `?` for the two length
prefixes, but then `unwrap()`ed the result of writing each root. A
writer that fails partway -- a full disk, a closed pipe, a socket that
went away -- panics rather than returning the `Err` the signature
promises.

Separate from the deserialize fixes in the previous commit and safe to
take on its own: this one is about a failing writer, not untrusted
input.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant