Fix list-of-struct nested projection

[godnight10061]

Summary

• Add a component-shredded vortex.list layout for List/FixedSizeList columns
• Enable nested projection/pushdown for list<struct> (e.g. projecting items.a)
• Extend vortex.get_item and the list reader to support list<struct>

Why

• Today list columns are written as opaque blobs, which prevents nested projections from being pushed down and breaks list-of-struct field selection.

Tests

• cargo +nightly fmt --all --check
• cargo clippy --locked --workspace --all-features --all-targets --exclude vortex-bench --exclude vortex-python --exclude vortex-duckdb --exclude vortex-fuzz --exclude duckdb-bench --exclude lance-bench --exclude datafusion-bench --exclude random-access-bench --exclude compress-bench -- -D warnings
• cargo nextest run --locked --workspace --all-features --no-fail-fast --exclude vortex-bench --exclude vortex-python --exclude vortex-duckdb --exclude vortex-fuzz --exclude duckdb-bench --exclude lance-bench --exclude datafusion-bench --exclude random-access-bench --exclude compress-bench (fails locally: vortex-cuda for_::tests::test_cuda_for_decompression requires a CUDA driver)
• cargo nextest run --locked --workspace --all-features --no-fail-fast --exclude vortex-cuda --exclude vortex-bench --exclude vortex-python --exclude vortex-duckdb --exclude vortex-fuzz --exclude duckdb-bench --exclude lance-bench --exclude datafusion-bench --exclude random-access-bench --exclude compress-bench

Compatibility

• New files may be written with a new list layout encoding id (vortex.list). Older readers that don't know this encoding won't be able to read those files; new readers continue to read existing files.

References

• Part of Better data layout for nested/repeated schemas #4889

[godnight10061]

I wasn't able to apply labels from a fork (permission denied). Could a maintainer please add the ix label for the changelog/CI gating?

[joseph-isaacs]

We don't really want to tie the get_item to have this for_each element behaviour.
I am not sure we want get_item to pierce thought lists, we would want a this to be map_list(get_item("a", _), get_item("items", root()) I am not sure we have designed the map_list or the _ expression yet?

[joseph-isaacs]

This is a great contribution, however we need to design the list expression more carefully

[codecov]

Codecov Report

❌ Patch coverage is 68.05556% with 276 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.83%. Comparing base (a30298b) to head (8a8803c).
⚠️ Report is 5 commits behind head on develop.

Files with missing lines	Patch %	Lines
vortex-array/src/expr/exprs/get_item.rs	43.69%	134 Missing ⚠️
vortex-layout/src/layouts/list/writer.rs	65.57%	63 Missing ⚠️
vortex-layout/src/layouts/list/reader.rs	80.86%	44 Missing ⚠️
vortex-layout/src/layouts/list/mod.rs	70.83%	35 Missing ⚠️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[joseph-isaacs]

Started a discussion: #6030, we can reopen once we find a good answer there.

Maybe you can expand on your use case there.

[a10y]

The one thing I struggled with the last time I tried to do ListLayout is that it's hard to split naturally.

Ideally you'd have a way to split based on the elements, then turn that back into row ranges...