[SPARK-59354][SQL] Derive a length guard from LIKE patterns with '_' wildcards - #58641
Open
david-mollitor-db wants to merge 1 commit into
Open
[SPARK-59354][SQL] Derive a length guard from LIKE patterns with '_' wildcards#58641david-mollitor-db wants to merge 1 commit into
david-mollitor-db wants to merge 1 commit into
Conversation
…wildcards
A `LIKE` pattern containing the `_` wildcard (which matches exactly one code point) is not
simplified today -- `LikeSimplification` leaves it as a full per-row regex. Since `_` constrains
length, this derives a code-point length guard: a pattern with no `%` fixes the length
(`Length(col) = N`), and one with `%` gives a lower bound (`Length(col) >= N`), where N is the
number of non-`%` code points.
When the pattern has no literals (only `_`/`%`) the guard is exactly equivalent and replaces the
`LIKE`:
col LIKE '___' ==> Length(col) = 3
col LIKE '_%' ==> Length(col) >= 1
When the pattern also has literals, the guard is only a necessary condition, so the exact `LIKE`
is kept as the residual:
col LIKE 'a_c' ==> Length(col) = 3 && (col LIKE 'a_c')
col LIKE 'a_b%' ==> Length(col) >= 3 && (col LIKE 'a_b%')
`Length(col)` fails fast before the regex (and eliminates it entirely for the literal-free
cases). This is a CPU/short-circuit improvement; `Length(col)` is a function of the column, not
a pushable column reference, so it does not push down or prune I/O.
`Length` is a code-point count -- the right measure for `_`, which matches one code point
regardless of its UTF-8 byte width. The rewrite is valid in every context and needs no collation
gate: `Length(col) = N` agrees with the `LIKE` even on null (both null-intolerant), and for the
literal case `And(guard, LIKE)` is the `LIKE` conjoined with one of its necessary conditions.
Idempotency under the fixed-point batch is maintained with a `TreeNodeTag` on the residual `Like`.
Patterns with escape characters are skipped (as elsewhere in the rule). The anchored-exact
rewrite, positional `substring` rewrites, and `LikeAll`/`LikeAny` are out of scope.
Generated-by: Claude Opus 4.8
david-mollitor-db
force-pushed
the
like-underscore-length-guard
branch
from
September 9, 2026 13:25
7bddd91 to
0def0e9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
A
LIKEpattern containing the_wildcard (which matches exactly one code point) is notsimplified today —
LikeSimplificationleaves it as a full per-row regex. Since_constrainslength, this PR derives a code-point length guard: a pattern with no
%fixes the length(
Length(col) = N), and one with%gives a lower bound (Length(col) >= N), where N is thenumber of non-
%code points.When the pattern has no literals (only
_/%) the guard is exactly equivalent, so it replacesthe
LIKE:When the pattern also has literals, the guard is only a necessary condition, so the exact
LIKEis kept as the residual:
Patterns with escape characters are skipped (as elsewhere in the rule). The anchored-exact
rewrite (
'a__c'->Length && StartsWith && EndsWith), positionalsubstringrewrites, andLikeAll/LikeAnyare out of scope.Why are the changes needed?
Length(col)is a cheap check that fails fast before the regex, so short strings are rejected(and, for the literal-free cases, the regex is eliminated entirely). This is a CPU/short-circuit
improvement;
Length(col)is a function of the column rather than a pushable column reference,so it does not push down to the data source or prune I/O.
Correctness.
Lengthis a code-point count — the right measure for_, which matches onecode point regardless of its UTF-8 byte width (a byte length would be wrong here). The rewrite is
valid in every context (not just predicates) and needs no collation gate:
Length(col) = Nagrees with
col LIKE '...'even onnull(both are null-intolerant), and for the literal caseAnd(guard, LIKE)is just theLIKEconjoined with one of its necessary conditions. It assumeseach pattern token consumes exactly one input code point, which holds for Spark's
LIKE(Java-regex simple, 1:1 case folding). Idempotency under the fixed-point optimizer batch is
maintained via a
TreeNodeTagon the residualLike.Does this PR introduce any user-facing change?
No. Query results are identical; this is a performance improvement.
How was this patch tested?
New tests in
LikeSimplificationSuite:_-only patterns:'_','___'->Length =._with%:'_%','%_%','_%_'->Length >=._with literals:'a_c'->Length = 3 && LIKE;'a_b%'->Length >= 3 && LIKE._('a\_b') and a no-_pattern ('abc%'still ->StartsWith).build/sbt 'catalyst/testOnly *LikeSimplificationSuite'passes (24/24); scalastyle clean.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8