Skip to content

Fix bilinear cache type decoding - #702

Open
Zaczero wants to merge 1 commit into
pytroll:mainfrom
Zaczero:zaczero/bilinear-cache
Open

Fix bilinear cache type decoding#702
Zaczero wants to merge 1 commit into
pytroll:mainfrom
Zaczero:zaczero/bilinear-cache

Conversation

@Zaczero

@Zaczero Zaczero commented Feb 28, 2026

Copy link
Copy Markdown
Contributor

Fixes bilinear cache type decoding:

mask_slices are boolean mask, and are stored as int8 via zarr.

boolean information is part of xarray metadata, so when skipping it we get back int8.

now mask_slices is not a mask but rather an array of 0, 1 indices making the mask useless.


I opted into chunks=auto automatic rechunking so Dask can pick the optimal chunk size, which gave a small performance boost on my data when using the cache:

  - raw zarr + astype(bool): load avg 0.006923s, compute avg 0.020060s
  - xarray open_zarr(chunks='auto'): load avg 0.004082s, compute avg 0.019647s
  • Tests added
  • Tests passed

@djhoese
djhoese requested a review from pnuu March 4, 2026 20:20
@djhoese djhoese added the bug label Mar 4, 2026
Comment thread pyresample/bilinear/xarr.py Outdated
@codecov

codecov Bot commented Mar 4, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 93.67%. Comparing base (4da28b0) to head (a5192ed).
⚠️ Report is 37 commits behind head on main.

Files with missing lines Patch % Lines
pyresample/bilinear/xarr.py 75.00% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #702   +/-   ##
=======================================
  Coverage   93.67%   93.67%           
=======================================
  Files          89       89           
  Lines       13621    13656   +35     
=======================================
+ Hits        12759    12792   +33     
- Misses        862      864    +2     
Flag Coverage Δ
unittests 93.67% <80.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Zaczero
Zaczero force-pushed the zaczero/bilinear-cache branch from 547f873 to a5192ed Compare March 14, 2026 12:30

@Manny7717 Manny7717 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — Approved

Verified locally (shapely 2.1.2, zarr installed): test_save_and_load_bil_info passes, including the new mask_slices.dtype == bool assertion.

Root cause is right: zarr stores Python bools as uint8 without dtype metadata, so da.array(fid[val]) re-exposed the raw uint8. open_zarr(...).data goes through xarray's dtype decoding and restores bool. The removed import zarr is safe — the only remaining zarr references in the module are the zarr_out variable and xarray's Dataset.to_zarr() / open_zarr, which lazy-import zarr themselves.

Non-blocking note (inline): caches written by the previous version will still decode as uint8 masks under the new load path (no dtype marker was stored). Harmless in practice since caches regenerate, but a comment in the changelog or a cache-format version bump would save someone a confusing debug session.

"""Load bilinear resampling look-up tables and initialize the resampler."""
try:
fid = zarr.open(filename, mode='r')
fid = open_zarr(filename, chunks="auto")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: caches written before this change (raw uint8 bools via zarr.open) will still come back as uint8 under open_zarr, since no dtype marker was persisted. Caches regenerate on the next save, so it's benign — a changelog note (or cache version bump) would prevent future confusion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants