[python][ray] read_by_row_id: dynamic_options + return a lazy Dataset#8468
Merged
JingsongLi merged 9 commits intoJul 6, 2026
Merged
Conversation
… overrides
Apply caller-supplied table options via table.copy before reading -- the
same mechanism a normal read uses -- so e.g. dynamic_options={"blob-as-
descriptor": "true"} returns small BlobDescriptor bytes instead of
materialising large blob payloads in the read task (resolve downstream with
map_with_blobs, bounded batch_size). The option rides on the table object and
flows through the existing scan-snapshot copy, so the distributed read path is
unchanged.
… the source read_by_row_id returns a Dataset, but two upfront checks forced the source to run at call time, so a bucket_join source executed twice (once here, once in the caller's action): - source_ds.schema() infers a lazy source's schema via an internal limit(1), so read it with fetch_if_missing=False and only validate the id column when the schema is already known; _project_rid re-checks per batch at run time otherwise. - the empty-source precheck (rid_ds.limit(1).count()) is gone; the count now runs only in the degenerate empty-target branch (mirrors update_by_row_id), and the empty-source schema is kept by unioning a typed-empty block into the result. Adds a test asserting the source is not executed until a downstream action.
XiaoHongbo-Hope
changed the title
…azy-read note - ray-data.md: add dynamic_options to the parameter list; the old 'prefer a materialized source' note is stale now the read no longer executes the source up front -- replaced with the lazy behaviour. - trim the dynamic_options docstring to two lines.
…me travel) Two issues with how dynamic_options was applied: - It was merged into the table before the capability checks, so a caller could override data-evolution.enabled / row-tracking.enabled / deletion-vectors.enabled and bypass them (failing later inside the planner with an internal error). The checks now run on the persisted table first, and dynamic_options that flip these invariants are rejected. - A time-travel option (scan.snapshot-id / scan.tag-name / ...) was silently ignored: the read resolved the latest snapshot and pinned it, returning the wrong snapshot. The read snapshot is now resolved from the option via TimeTravelUtil and used for both routing and reading; the key is reduced to a plain snapshot-id so the planner's own pin does not read as a conflicting second time-travel option. Tests: reject invariant overrides; time travel by snapshot-id and by tag.
…ghter test asserts - ray-data.md: 'current snapshot' -> 'resolved target snapshot (latest, or the one selected via dynamic_options)' now that time travel is supported. - narrow the time-travel foreign-id assertions to match the 'valid range' message. - add a test that two time-travel keys are rejected (TimeTravelUtil mutual exclusion).
…ow-tracking time travel - table.copy's _try_time_travel swallows TimeTravelUtil's 'more than one time-travel option' error (except Exception: return None), so the multi-key rejection was only working implicitly via the later _read_snapshot call. Reject >1 scan key explicitly at the entry so a refactor can't silently drop it. - time travel to a snapshot from before row-tracking was enabled has files without row ids and crashed deep in the planner (AttributeError). Check the resolved snapshot's own schema and raise a clear error instead. (Capability checks stay on the persisted table.)
Contributor
|
+1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Two follow-ups to #8465.
dynamic_options: a dict applied via table.copy (as a normal read does), so callers can override read options on a target passed by name — mainly{"blob-as-descriptor": "true"}to return small BlobDescriptor bytes instead ofmaterialising large blobs, resolved later with map_with_blobs. Distributed path unchanged.
Lazy Dataset: two upfront checks executed the source at call time (so a bucket_join source ran twice). Now the schema is read with fetch_if_missing=False and the empty-source precheck is dropped — limit(1).count() runs only in the empty-target branch (like update_by_row_id), schema preserved by unioning a typed-empty block.Tests
test_blob_as_descriptor_via_dynamic_optionstest_returns_lazy_without_executing_source