Skip to content

allow range to satisfy key distribution generally#23680

Open
gene-bordegaray wants to merge 1 commit into
apache:mainfrom
gene-bordegaray:gene.bordegaray/2026/07/allow_range_to_satisfy_key_distribution_generally
Open

allow range to satisfy key distribution generally#23680
gene-bordegaray wants to merge 1 commit into
apache:mainfrom
gene-bordegaray:gene.bordegaray/2026/07/allow_range_to_satisfy_key_distribution_generally

Conversation

@gene-bordegaray

@gene-bordegaray gene-bordegaray commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Partitioning::Range now satisfies Distribution::KeyPartitioned privately across all operators that require it: aggregates, windows, TopK, and co-partitioned joins. So now the temporary operator opt-ins can be consolidated.

What changes are included in this PR?

  • Allow compatible Partitioning::Range to satisfy Distribution::KeyPartitioned via Partitioning::satisfaction
  • Remove the temporary range-satisfaction helpers and operator-specific opt-ins

Are these changes tested?

Yes

Are there any user-facing changes?

No, this should not change any exisitng behavior just consolidation

@github-actions github-actions Bot added physical-expr Changes to the physical-expr crates optimizer Optimizer rules physical-plan Changes to the physical-plan crate labels Jul 17, 2026
@gene-bordegaray
gene-bordegaray force-pushed the gene.bordegaray/2026/07/allow_range_to_satisfy_key_distribution_generally branch from de0572d to 7df0261 Compare July 21, 2026 18:35
@codecov-commenter

codecov-commenter commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.02632% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.71%. Comparing base (eef1017) to head (03518df).
⚠️ Report is 13 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/physical-expr/src/partitioning.rs 97.84% 0 Missing and 3 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #23680      +/-   ##
==========================================
- Coverage   80.71%   80.71%   -0.01%     
==========================================
  Files        1089     1089              
  Lines      368748   368741       -7     
  Branches   368748   368741       -7     
==========================================
- Hits       297633   297614      -19     
- Misses      53372    53374       +2     
- Partials    17743    17753      +10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@gene-bordegaray
gene-bordegaray marked this pull request as ready for review July 21, 2026 19:55
@gene-bordegaray

Copy link
Copy Markdown
Contributor Author

cc: @gabotechs

@gene-bordegaray gene-bordegaray changed the title [WIP] allow range to satisfy key distribution generally allow range to satisfy key distribution generally Jul 21, 2026

@gabotechs gabotechs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice cleanup! 👍

}

#[test]
fn test_partitioning_partial_overlap() -> Result<()> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see some tests where deleted here? are they no longer relevant?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just consolidated these into the test_partitioning_key_satisfaction() test. Actually after you pointing this out I think I will rename this test_hash_partitioning_key_satisfaction() and then keep the test_range_partitioning_key_satisfaction()

I got rid of some case that were repearting testing the same code path such as:

"KeyPartitioned([a, b, c]) satisfied by Hash([b, a])"

and 

"KeyPartitioned([a, b, c]) satisfied by Hash([a, b])"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

were there any that popped out to you as bad to remove?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

None specifically, just pointing it out so that we don't loose coverage. If we are still covered, then it should be good 👍

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. I will double check when renaming test

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

renamed and went over tests to ensure we didn't lose coverage.

I added back: Hash([unknown]) vs KeyPartitioned([unknown,a]) to check an unknown cant match itself in subset

@gene-bordegaray
gene-bordegaray force-pushed the gene.bordegaray/2026/07/allow_range_to_satisfy_key_distribution_generally branch from 7df0261 to b173254 Compare July 22, 2026 14:34
@gene-bordegaray
gene-bordegaray force-pushed the gene.bordegaray/2026/07/allow_range_to_satisfy_key_distribution_generally branch from b173254 to 03518df Compare July 22, 2026 20:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Optimizer rules physical-expr Changes to the physical-expr crates physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow Partitioning::Range to satisfy Distribution::KeyPartitioned generally

3 participants