Skip to content

GridPatchDataset caching can repeat, drop, or reject samples #9100

Description

@QiuLsG

Describe the bug

GridPatchDataset(cache=True) does not preserve dataset contents for several documented cache configurations:

  1. With a partial cache (0 < cache_rate < 1), an uncached item following a cached item reuses the previous item's cache index and emits the cached patches again.
  2. With with_coordinates=False, cached patches are zipped with an empty coordinate cache, so no cached patches are emitted.
  3. With cache_rate=0 (or cache_num=0), initialization tries to unpack an empty cache and raises ValueError.
  4. With a transform pipeline containing no random transform, cached iteration passes start=None to Compose and raises ValueError.

These are data-correctness issues: caching can silently replace or drop training samples, or reject valid cache configurations.

To Reproduce

from monai.data import GridPatchDataset
from monai.transforms import Lambda


def patches(image):
    for item in image:
        yield item, item * 10


partial = GridPatchDataset(
    [[1], [2]], patches, cache=True, cache_rate=0.5, progress=False
)
print(list(partial))
# Actual:   [(1, 10), (1, 10)]
# Expected: [(1, 10), (2, 20)]

no_coordinates = GridPatchDataset(
    [[1, 2]], patches, with_coordinates=False, cache=True, progress=False
)
print(list(no_coordinates))
# Actual:   []
# Expected: [1, 2]

zero_cache = GridPatchDataset(
    [[1], [2]], patches, cache=True, cache_rate=0, progress=False
)
# Actual: ValueError: not enough values to unpack (expected 2, got 0)
# Expected: construct successfully and iterate without using a cache

deterministic = GridPatchDataset(
    [[1]], patches, transform=Lambda(lambda x: x + 100), cache=True, progress=False
)
print(list(deterministic))
# Actual:   ValueError: 'start' (None) cannot be None
# Expected: [(101, 10)]

Expected behavior

Enabling caching must not change which patches are yielded. cache_rate should only select how many source items are cached, and with_coordinates should only control whether coordinates are included in each yielded item. Deterministic transforms should be computed while populating the cache, and a cache hit should resume at the end of that transform pipeline.

Environment

MONAI version: 0+untagged.3487.gd1306f6
MONAI rev id: d1306f6d1996cffeb9d10984dd1c056b7fe2ed1d
Python version: 3.12.0
NumPy version: 2.5.2
PyTorch version: 2.14.0+cpu
OS: Windows

Additional context

The partial-cache issue comes from cache_index being initialized before the image loop and not reset for cache misses. The coordinate-free path always calls zip(data, other) even though _cache_other is intentionally empty when with_coordinates=False. set_data() unconditionally unpacks zip(*self._fill_cache(...)), including when the configured cache size is zero. Finally, Compose.get_index_of_first(...) returns None when every transform is deterministic, but the cache-read path uses that value as the start index.

The cache implementation was introduced in #7180. Existing coverage exercises a full cache with coordinates enabled and a pipeline containing a random transform, so these paths are currently untested.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions