Skip to content

[SPARK-58980][SQL] Support GROUPS window frames - #58634

Open
mrk-andreev wants to merge 1 commit into
apache:masterfrom
mrk-andreev:SPARK-58980
Open

[SPARK-58980][SQL] Support GROUPS window frames#58634
mrk-andreev wants to merge 1 commit into
apache:masterfrom
mrk-andreev:SPARK-58980

Conversation

@mrk-andreev

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Add SQL support for GROUPS window frames. Offsets count peer groups: rows with equal values for all window ORDER BY expressions within a partition. CURRENT ROW includes the whole current peer group.

The change adds parsing and validation, implements group boundaries in the existing window execution paths, and supports eligible aggregates through the segment-tree path. Group tracking retains only the current ordering keys as it advances through a partition. The SQL reference and keyword documentation are updated.

Why are the changes needed?

Some moving calculations need to include complete groups of tied rows. For example, a total across the current and previous batch should include every row in both batches, even when batch IDs have gaps. GROUPS expresses this directly.

Does this PR introduce any user-facing change?

Yes. Previously, Spark rejected GROUPS frame syntax. This query now sums each batch with the previous batch:

SELECT batch_id, amount,
       SUM(amount) OVER (
         ORDER BY batch_id GROUPS BETWEEN 1 PRECEDING AND CURRENT ROW
       ) AS moving_sum
FROM VALUES (1, 10), (1, 15), (2, 20), (3, 25), (3, 30), (9, 40)
  AS batches(batch_id, amount)
ORDER BY batch_id, amount;
batch_id  amount  moving_sum
1         10      25
1         15      25
2         20      45
3         25      75
3         30      75
9         40      95

For batch 9, the previous group is batch 3, so the total is 25 + 30 + 40 = 95.

GROUPS requires ORDER BY, supports multiple ordering expressions, and accepts constant, non-null, non-negative integer offsets. GROUPS remains usable as an identifier.

How was this patch tested?

The commit adds parser and validation tests, SQL golden coverage, and execution tests for tied values, multiple ordering keys, nulls, descending order, partition boundaries, and spilling. It also enables previously disabled PostgreSQL GROUPS test cases.

Execution coverage includes randomized comparisons against DENSE_RANK plus RANGE, segment-tree enabled/disabled comparisons, fallback and metrics checks, and a cluster-mode test in HiveSparkSubmitSuite.

Was this patch authored or co-authored using generative AI tooling?

Yes. The code was co-authored using the following tools.

Generated-by:

  • Codex (GPT-6 Astra)
  • Claude Code Opus 5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant