bsv: validate replay checkpoint sizes

[acts-1631]

Replay checkpoints serialize independent state, encoded-state, and
compressed-state sizes. For raw and uncompressed checkpoints, these
values must agree before the data is read into the state buffer.

A crafted replay could otherwise cause RetroArch to write the declared
compressed size into an allocation sized for the smaller state, resulting
in a heap buffer overflow before core deserialization.

Validate those relationships, handle allocation failures, and require
zlib and zstd decompression to produce the declared encoded size.

[hizzlekizzle]

@JoeOsborn thoughts?

[JoeOsborn]

I don’t hate the idea here. The main thing is to ensure compressed encoded size is no bigger than encoded size in uncompressed checkpoints, and encoded size is no bigger than size in raw checkpoints (I prefer “no bigger than” to “exactly equal to” since sometimes cores’ savestate sizes change during play, and we just want to avoid an overflow). Something like the checks in the first part of the PR are fine. The later checks around zlib or zstd decompression size seem like they are too late to prevent any bad behavior since the read and decompression write have already happened. Maybe a warning if they are different, but it’s not wrong to have some padding or extra space since it’s hard to know in advance what e.g. the compressed size would have been.

The matter of checking that allocations don’t fail is also worth discussing but not related to this issue; if they fail we likely have bigger problems than just a failed checkpoint. I also would rather allocations be reused across calls rather than made fresh every time but that work hasn’t been done yet.

It would be nice to have an example of an evil replay as a test case that shows the overflow under a sanitizer.

[acts-1631]

Thanks for the review. I pushed an update that uses the requested
upper-bound checks:

• compressed encoded size must not exceed encoded size when compression
  is none;
• encoded size must not exceed state size for raw checkpoints.

I removed the allocation handling and exact zlib/zstd output-size checks.

I also added an ASan regression fixture for a raw, uncompressed
checkpoint with state and encoded sizes of 1 and a compressed encoded
size of 4096. It verifies that the size prelude is rejected before the
copy into the state allocation.

Please let me know if you would prefer the test to exercise the full
replay loader rather than the checkpoint load path directly.