[SPARK-58980][SQL] Support GROUPS window frames - #58634
Open
mrk-andreev wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add SQL support for
GROUPSwindow frames. Offsets count peer groups: rows with equal values for all windowORDER BYexpressions within a partition.CURRENT ROWincludes the whole current peer group.The change adds parsing and validation, implements group boundaries in the existing window execution paths, and supports eligible aggregates through the segment-tree path. Group tracking retains only the current ordering keys as it advances through a partition. The SQL reference and keyword documentation are updated.
Why are the changes needed?
Some moving calculations need to include complete groups of tied rows. For example, a total across the current and previous batch should include every row in both batches, even when batch IDs have gaps.
GROUPSexpresses this directly.Does this PR introduce any user-facing change?
Yes. Previously, Spark rejected
GROUPSframe syntax. This query now sums each batch with the previous batch:For batch 9, the previous group is batch 3, so the total is
25 + 30 + 40 = 95.GROUPSrequiresORDER BY, supports multiple ordering expressions, and accepts constant, non-null, non-negative integer offsets.GROUPSremains usable as an identifier.How was this patch tested?
The commit adds parser and validation tests, SQL golden coverage, and execution tests for tied values, multiple ordering keys, nulls, descending order, partition boundaries, and spilling. It also enables previously disabled PostgreSQL
GROUPStest cases.Execution coverage includes randomized comparisons against
DENSE_RANKplusRANGE, segment-tree enabled/disabled comparisons, fallback and metrics checks, and a cluster-mode test inHiveSparkSubmitSuite.Was this patch authored or co-authored using generative AI tooling?
Yes. The code was co-authored using the following tools.
Generated-by: