Skip to content

Various fixes to handling of chunk buffers#6184

Open
fortnern wants to merge 14 commits intoHDFGroup:developfrom
fortnern:chunk_buffer_fixes
Open

Various fixes to handling of chunk buffers#6184
fortnern wants to merge 14 commits intoHDFGroup:developfrom
fortnern:chunk_buffer_fixes

Conversation

@fortnern
Copy link
Copy Markdown
Member

@fortnern fortnern commented Jan 30, 2026

Verify uncompressed chunks are the right size after being uncompressed (reverse filtered)
Verify that the buffer returned from the filter callback is large enough to hold the returned data size
Fix bug in deflate filter that caused it to report the wrong buffer size
Fix bug in chunk copy code that could cause a background buffer overflow
Other general cleanup

The verification of the size of chunks after decompression causes a failure in NetCDF. I suspect this is an issue in that NetCDF test.


Important

Fixes and enhancements to chunk buffer handling, including size verification and buffer overflow prevention in H5Dchunk.c, H5Z.c, and H5Zdeflate.c.

  • Buffer Size Verification:
    • In H5Dchunk.c, verify uncompressed chunk size in H5D__chunk_lock() and H5D__chunk_copy_cb().
    • In H5Z.c, ensure buffer size is sufficient after filter callback in H5Z_pipeline().
  • Bug Fixes:
    • Fix buffer overflow in H5D__chunk_copy_cb() by ensuring background buffer size is adequate.
    • Correct buffer size reporting in H5Z__filter_deflate() in H5Zdeflate.c.
  • Miscellaneous:
    • Add bkg_size and src_chunk_size to H5D_chunk_it_ud3_t in H5Dchunk.c.
    • Minor cleanup and assertions added in H5Dchunk.c and H5Z.c.

This description was created by Ellipsis for 7788329. You can customize this summary. It will automatically update as commits are pushed.

@hyoklee hyoklee added the Component - C Library Core C library issues (usually in the src directory) label Jan 30, 2026
@hyoklee hyoklee moved this from To be triaged to In progress in HDF5 - TRIAGE & TRACK Jan 30, 2026
@fortnern fortnern marked this pull request as draft January 30, 2026 19:49
@fortnern
Copy link
Copy Markdown
Member Author

Not scheduled for 2.1.0. Needs a test and doc updates to clarify the expected behavior of filters

@vchoi-hdfgroup vchoi-hdfgroup added this to the HDF5 2.2.0 milestone Feb 16, 2026
@sonarqubecloud
Copy link
Copy Markdown

@fortnern fortnern marked this pull request as ready for review April 16, 2026 21:35
@bmribler
Copy link
Copy Markdown
Collaborator

996/3016 Test #30: H5TEST-dsets ...............................................................***Failed 3.59 sec

Testing filter returning incorrect data size after unfiltering with vlen type
CMake Error at /Users/runner/work/hdf5/hdf5/config/cmake/runExecute.cmake:178 (message):
Failed: Test program /Users/runner/work/hdf5/build/bin/dsets exited != 0.

@fortnern
Copy link
Copy Markdown
Member Author

996/3016 Test #30: H5TEST-dsets ...............................................................***Failed 3.59 sec

Testing filter returning incorrect data size after unfiltering with vlen type CMake Error at /Users/runner/work/hdf5/hdf5/config/cmake/runExecute.cmake:178 (message): Failed: Test program /Users/runner/work/hdf5/build/bin/dsets exited != 0.

Thanks, it only seems to happen on mac and sometimes windows. I might need help debugging this one if my last change doesn't fix it.

@bmribler
Copy link
Copy Markdown
Collaborator

bmribler commented Apr 17, 2026

Unfortunately, that test still failed.

@bmribler
Copy link
Copy Markdown
Collaborator

This should be brought up in SHINES.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Component - C Library Core C library issues (usually in the src directory)

Projects

Status: In progress

Development

Successfully merging this pull request may close these issues.

5 participants