[SPARK-59355][SQL] Simplify LIKE patterns containing escaped wildcards - #58644
Open
david-mollitor-db wants to merge 1 commit into
Open
[SPARK-59355][SQL] Simplify LIKE patterns containing escaped wildcards#58644david-mollitor-db wants to merge 1 commit into
david-mollitor-db wants to merge 1 commit into
Conversation
uros-b
reviewed
Sep 9, 2026
uros-b
left a comment
Member
There was a problem hiding this comment.
Upon first look, this seems like a well tested and behaviour preserving optimization, thank you @david-mollitor-db! Adding @stevomitric to also review this PR
`LikeSimplification` gives up on a pattern the moment it contains the escape character
(`if (pattern.contains(escapeChar)) None`), so escaped-literal patterns stay a full per-row
regex even when they are trivially sargable once the escapes are decoded:
col LIKE 'ma\%ca' ==> col = 'ma%ca'
col LIKE 'abc\%def%' ==> StartsWith(col, 'abc%def')
col LIKE '%mn\%' ==> EndsWith(col, 'mn%')
col LIKE 'abbc' ESCAPE 'b' ==> col = 'abc'
This decodes valid escape sequences and applies the rule's existing shape simplifications
(EqualTo / StartsWith / EndsWith / Contains / StartsWith+EndsWith) to the decoded literals. The
resulting predicates push down to data sources and short-circuit the per-row regex.
How it works:
- Fast path (unchanged): a quick `pattern.contains(escapeChar)` scan -- when the escape
character is absent, the existing regexes run exactly as before.
- Guard: otherwise, if the escape character is itself `%` or `_` (a pathological `ESCAPE '%'` /
`ESCAPE '_'`), skip. This guard follows the fast-path scan on purpose: an `ESCAPE '%'` pattern
that contains no `%` is still simplifiable via the fast path.
- Decode: otherwise, walk the pattern turning `\%` / `\_` / `\\` into literal characters,
treating unescaped `%`/`_` as wildcards, and classify the decoded shape.
A valid-escape pattern's decoded literal is exactly the string the LIKE matches, so the rewrite
is behavior-preserving. An escape character followed by anything other than `%`, `_`, or itself,
or a trailing escape character, makes LIKE throw at runtime
(`INVALID_FORMAT.ESC_IN_THE_MIDDLE` / `ESC_AT_THE_END`); the rule returns None for these, so the
`Like` is kept and throws exactly as today. Every shape fully replaces the `Like` (no residual),
so no idempotency tag is needed. An unescaped `_` wildcard and `LikeAll`/`LikeAny` shapes beyond
what `simplifyLike` already covers remain out of scope.
Generated-by: Claude Opus 4.8
david-mollitor-db
force-pushed
the
like-escape-simplification
branch
from
September 9, 2026 13:27
007773a to
afc65f7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
LikeSimplificationgives up on a pattern the moment it contains the escape character(
if (pattern.contains(escapeChar)) None), so escaped-literal patterns stay a full per-rowregex even when they are trivially sargable once the escapes are decoded:
This PR decodes valid escape sequences and applies the rule's existing shape simplifications
(
EqualTo/StartsWith/EndsWith/Contains/StartsWith+EndsWith) to the decodedliterals.
How it works:
pattern.contains(escapeChar)scan — when the escapecharacter is absent, the existing five regexes run exactly as before, so the common case pays
nothing for the decode logic.
%or_(a pathologicalESCAPE '%'/ESCAPE '_'), skip — unchanged behavior. This guard follows the fast-path scanon purpose: an
ESCAPE '%'pattern that contains no%is still simplifiable via the fastpath.
\%/\_/\\into literal characters,treating unescaped
%/_as wildcards, and classify the decoded shape.An unescaped
_wildcard andLikeAll/LikeAnyshapes beyond whatsimplifyLikealreadycovers remain out of scope.
Why are the changes needed?
Escaped-literal patterns become sargable: the resulting
EqualTo/StartsWith/etc. push down todata sources (e.g. Parquet prunes on
StringStartsWith) and short-circuit the per-row regex,instead of always running the regex and pushing nothing.
Does this PR introduce any user-facing change?
No. Query results are identical, and error cases still error; this is a performance improvement.
A valid-escape pattern's decoded literal is exactly the string the
LIKEmatches, so the rewriteis behavior-preserving. An escape character followed by anything other than
%,_, or itself,or a trailing escape character, makes
LIKEthrow at runtime (INVALID_FORMAT.ESC_IN_THE_MIDDLE/
ESC_AT_THE_END); the rule returnsNonefor these, so theLikeis kept and throws exactlyas today. Every shape fully replaces the
Like(no residual), so the rewrite is idempotent.How was this patch tested?
LikeSimplificationSuite: updated the existing escape tests (they previously asserted thepattern was skipped and now assert the simplified form, including the
LikeAll/LikeAnymulti-pattern tests whose escaped entries now fold in), and added tests for valid-escape
simplification across all shapes and for the still-skipped cases (invalid escape, trailing
escape,
ESCAPE '%'/ESCAPE '_', and an unescaped_).build/sbt 'catalyst/testOnly *LikeSimplificationSuite'passes; scalastyle clean.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8