docs: add roadmap page outlining future plans by d-v-b · Pull Request #4149 · zarr-developers/zarr-python

d-v-b · 2026-07-15T20:08:18Z

Adds a Roadmap page to the documentation describing the goals for the
next major cycle of work ("v4"), the intended changes per theme, and
the three-stream release model (additive minors, deprecations, one
minimal removals-only major), with an option for a legacy module.

Assisted-by: ClaudeCode:claude-fable-5

Author attestation

I am a human, these are my changes, and I have reviewed and understood every change and can explain why each is correct.

Adds a Roadmap page to the documentation describing the goals for the next major cycle of work ("v4"), the intended changes per theme, and the three-stream release model (additive minors, deprecations, one minimal removals-only major). Assisted-by: ClaudeCode:claude-fable-5

d-v-b · 2026-07-15T20:08:46Z

cc @ilan-gold @maxrjones, opening this as a draft for visibility. lmk if there's anything more or less we should say here

ilan-gold · 2026-07-16T15:08:29Z

This doesn't lead me where I thought it would, somehow linking to an old page

ilan-gold · 2026-07-16T15:10:56Z

+Zarr-Python around the stack, so that each level is something you can depend
+on, conform to, or replace, without buying every other level:
+
+- **A focused package per level** — `zarr-metadata`, `zarr-store`,


Would maybe note the zarrs example here

ilan-gold · 2026-07-16T15:16:05Z

+complementary rather than competitive: Zarr-Python aims to be the best
+pure-Python Zarr implementation *and* the best wrapper around the
+compiled-language implementations, so that users who need native throughput can


I'm really conflicted on this, because I think baking fragmentation into the ecosystem is not great, but the flipside is that languages may have different features/advantages. It would be great to have a concrete reason (or list of reasons) as to why we would design around this level of extensibility

good points, I'd say we want the backend to be an extension point for a few reasons:

we always want a traditional python impl. IMO it's our job as the python implementation, and it keeps the ecosystem healthy to have multiple implementations

we can't guarantee uniform, complete feature support across all implementations. As long as we allow people to hack on stuff in python, there will be things the python impl does that e.g. a rust impl doesn't support, and that's OK. For now, a compiled backend is effectively an external dep, and we simply can't promise what external deps will do in perpetuity.

IMO a full backend is just an refined version of our current model, which requires extenders manage buffer prototypes, data type implementations, codec implementations, and store implementations. Consolidating all of this pluggability on a single surface should make extending Zarr Python easier even for other python backends (like a cupy backend, or a pysparse backend)

ilan-gold · 2026-07-16T15:18:05Z

+
+### Codecs
+
+The current codec API wraps every codec in an unnecessary async layer (a


Not anymore!

maxrjones

This is awesome! thanks so much for your work on this @d-v-b! 🚀

maxrjones · 2026-07-17T14:50:57Z

+- **Engine pluggability at the chunk-decoding level** — alternative engines
+  (zarrs, TensorStore) can take over IO without re-implementing hierarchy
+  traversal, indexing, or metadata handling.


this seems a bit misaligned with other statements that imply pluggabillity at each level

maxrjones · 2026-07-17T14:51:44Z

+Each theme below is backed by a detailed technical proposal; the summaries
+here describe the intended end state.


as we discussed at the dev meeting, if the idea is to reference zarr-python-planning for detailed technical proposals, we should move it to the zarr-developers github organization

maxrjones · 2026-07-17T14:57:56Z

+Make Zarr-Python's IO surfaces device-agnostic rather than adding GPU support
+as a bolted-on feature: stores and codecs grow APIs for writing into a
+caller-provided buffer (`read_into`, `decode_into`), and the `Array` facade
+returns array-like objects in the user's chosen Array API namespace. GPU
+support falls out once the assumption of CPU destinations is removed, and CPU
+paths get faster too, because pre-allocated output buffers eliminate per-chunk
+allocation.


@TomAugspurger it'd be great to get your thoughts on this component in particular.

maxrjones · 2026-07-17T15:11:28Z

+`Array.__getitem__` performs IO eagerly and returns NumPy arrays, which makes Zarr
+arrays the odd one out among modern array libraries and blocks compliance with
+the [Python Array API](https://data-apis.org/array-api/) standard. Add an
+opt-in `array.lazy[...]` accessor backed by a stable coordinate-mapping algebra
+(the `IndexTransform` work in
+[#3906](https://github.com/zarr-developers/zarr-python/pull/3906)), plus a
+small query planner that turns chained selections into a single IO plan before
+any chunks are fetched. No new array type is introduced. Whether the *default*
+of bare `array[...]` ever flips from eager to lazy is an explicit, separate
+decision — see [decision points](#decision-points) below.


Suggested change

`Array.__getitem__` performs IO eagerly and returns NumPy arrays, which makes Zarr

arrays the odd one out among modern array libraries and blocks compliance with

the [Python Array API](https://data-apis.org/array-api/) standard. Add an

opt-in `array.lazy[...]` accessor backed by a stable coordinate-mapping algebra

(the `IndexTransform` work in

[#3906](https://github.com/zarr-developers/zarr-python/pull/3906)), plus a

small query planner that turns chained selections into a single IO plan before

any chunks are fetched. No new array type is introduced. Whether the *default*

of bare `array[...]` ever flips from eager to lazy is an explicit, separate

decision — see [decision points](#decision-points) below.

The Zarr-Python Array API was initially designed to mirror NumPy, with eager

syntax. `Array.__getitem__` performs IO eagerly and returns a NumPy arrays.

That was helpful to the dominant use-case at the time of its creation, but it

means deferred I/O and computation currently require an external library

such as Dask. It means there is no build-in support for representing multi

step reads as a single deferred plan. Further, it means that every chained

selection round-trips to storage independently.

To solve this limitation, Add an opt-in `array.lazy[...]` accessor backed by a

stable coordinate-mapping algebra (the `IndexTransform` work in

[#3906](https://github.com/zarr-developers/zarr-python/pull/3906)), plus a

small query planner that turns chained selections into a single IO plan before

any chunks are fetched. No new array type is introduced. Whether the *default*

of bare `array[...]` ever flips from eager to lazy is an explicit, separate

decision — see [decision points](#decision-points) below.

I think the original description was a bit overstated with respect to the eagerness landscape and array API relationship. I suggest these revisions to clarify the motivations a bit.

It also may be worth separating the "how" out to the detailed document, only setting the "what" here.

maxrjones · 2026-07-17T15:13:55Z

+### Decision points
+
+Flipping the default of bare `array[...]` from eager to lazy is the single
+highest-migration-cost item in the plan, so it is handled as an explicit
+decision, not bundled into the additive work. The opt-in `array.lazy[...]`
+accessor ships first, with no default change. Whether the default ever flips
+hinges on whether Array API conformance at the bare-`__getitem__` surface turns
+out to be a hard requirement; if it does, the flip happens as a long-window
+deprecation with an explicit eager escape hatch and downstream coordination —
+never as a reason to adopt a major version.


To some extent, I think it would be better to separate this from the roadmap. Include the commitment to delivering lazy indexing, leave the how to the detailed documents and reference zarr-python-planning.

maxrjones · 2026-07-17T15:24:43Z

This doesn't lead me where I thought it would, somehow linking to an old page

fyi @d-v-b this can be fixed by removing this redirect:

zarr-python/mkdocs.yml

Line 204 in 50b7e01

'roadmap.md': 'https://zarr.readthedocs.io/en/v3.0.8/developers/roadmap.html'

Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com>

codecov · 2026-07-17T15:31:18Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.84%. Comparing base (d916f75) to head (649f60b).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4149   +/-   ##
=======================================
  Coverage   93.84%   93.84%           
=======================================
  Files          91       91           
  Lines       12570    12572    +2     
=======================================
+ Hits        11796    11798    +2     
  Misses        774      774

see 2 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

d-v-b added 2 commits July 15, 2026 09:56

docs: refine roadmap

9304a06

Merge branch 'main' into docs/roadmap

199b4e1

ilan-gold reviewed Jul 16, 2026

View reviewed changes

maxrjones reviewed Jul 17, 2026

View reviewed changes

d-v-b and others added 3 commits July 17, 2026 17:26

Update roadmap.md

bd0f1f1

Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com>

Update roadmap.md

641d5c0

Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com>

Update roadmap.md

649f60b

Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com>


		### Codecs

		The current codec API wraps every codec in an unnecessary async layer (a

		Each theme below is backed by a detailed technical proposal; the summaries
		here describe the intended end state.

Uh oh!

Uh oh!

Conversation

d-v-b commented Jul 15, 2026

Author attestation

Uh oh!

d-v-b commented Jul 15, 2026

Uh oh!

ilan-gold commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxrjones left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxrjones commented Jul 17, 2026

Uh oh!

codecov Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ilan-gold commented Jul 16, 2026 •

edited

Loading

codecov Bot commented Jul 17, 2026 •

edited

Loading