Skip to content

[SPARK-59354][SQL] Derive a length guard from LIKE patterns with '_' wildcards - #58641

Open
david-mollitor-db wants to merge 1 commit into
apache:masterfrom
david-mollitor-db:like-underscore-length-guard
Open

[SPARK-59354][SQL] Derive a length guard from LIKE patterns with '_' wildcards#58641
david-mollitor-db wants to merge 1 commit into
apache:masterfrom
david-mollitor-db:like-underscore-length-guard

Conversation

@david-mollitor-db

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

A LIKE pattern containing the _ wildcard (which matches exactly one code point) is not
simplified today — LikeSimplification leaves it as a full per-row regex. Since _ constrains
length, this PR derives a code-point length guard: a pattern with no % fixes the length
(Length(col) = N), and one with % gives a lower bound (Length(col) >= N), where N is the
number of non-% code points.

When the pattern has no literals (only _/%) the guard is exactly equivalent, so it replaces
the LIKE:

col LIKE '___'   ==>   Length(col) = 3
col LIKE '_%'    ==>   Length(col) >= 1

When the pattern also has literals, the guard is only a necessary condition, so the exact LIKE
is kept as the residual:

col LIKE 'a_c'   ==>   Length(col) = 3  && (col LIKE 'a_c')
col LIKE 'a_b%'  ==>   Length(col) >= 3 && (col LIKE 'a_b%')

Patterns with escape characters are skipped (as elsewhere in the rule). The anchored-exact
rewrite ('a__c' -> Length && StartsWith && EndsWith), positional substring rewrites, and
LikeAll/LikeAny are out of scope.

Why are the changes needed?

Length(col) is a cheap check that fails fast before the regex, so short strings are rejected
(and, for the literal-free cases, the regex is eliminated entirely). This is a CPU/short-circuit
improvement; Length(col) is a function of the column rather than a pushable column reference,
so it does not push down to the data source or prune I/O.

Correctness. Length is a code-point count — the right measure for _, which matches one
code point regardless of its UTF-8 byte width (a byte length would be wrong here). The rewrite is
valid in every context (not just predicates) and needs no collation gate: Length(col) = N
agrees with col LIKE '...' even on null (both are null-intolerant), and for the literal case
And(guard, LIKE) is just the LIKE conjoined with one of its necessary conditions. It assumes
each pattern token consumes exactly one input code point, which holds for Spark's LIKE
(Java-regex simple, 1:1 case folding). Idempotency under the fixed-point optimizer batch is
maintained via a TreeNodeTag on the residual Like.

Does this PR introduce any user-facing change?

No. Query results are identical; this is a performance improvement.

How was this patch tested?

New tests in LikeSimplificationSuite:

  • Exact replacement for _-only patterns: '_', '___' -> Length =.
  • Minimum-length replacement for _ with %: '_%', '%_%', '_%_' -> Length >=.
  • Additive guard for _ with literals: 'a_c' -> Length = 3 && LIKE; 'a_b%' -> Length >= 3 && LIKE.
  • Not rewritten: escaped _ ('a\_b') and a no-_ pattern ('abc%' still -> StartsWith).

build/sbt 'catalyst/testOnly *LikeSimplificationSuite' passes (24/24); scalastyle clean.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 4.8

…wildcards

A `LIKE` pattern containing the `_` wildcard (which matches exactly one code point) is not
simplified today -- `LikeSimplification` leaves it as a full per-row regex. Since `_` constrains
length, this derives a code-point length guard: a pattern with no `%` fixes the length
(`Length(col) = N`), and one with `%` gives a lower bound (`Length(col) >= N`), where N is the
number of non-`%` code points.

When the pattern has no literals (only `_`/`%`) the guard is exactly equivalent and replaces the
`LIKE`:

    col LIKE '___'   ==>   Length(col) = 3
    col LIKE '_%'    ==>   Length(col) >= 1

When the pattern also has literals, the guard is only a necessary condition, so the exact `LIKE`
is kept as the residual:

    col LIKE 'a_c'   ==>   Length(col) = 3  && (col LIKE 'a_c')
    col LIKE 'a_b%'  ==>   Length(col) >= 3 && (col LIKE 'a_b%')

`Length(col)` fails fast before the regex (and eliminates it entirely for the literal-free
cases). This is a CPU/short-circuit improvement; `Length(col)` is a function of the column, not
a pushable column reference, so it does not push down or prune I/O.

`Length` is a code-point count -- the right measure for `_`, which matches one code point
regardless of its UTF-8 byte width. The rewrite is valid in every context and needs no collation
gate: `Length(col) = N` agrees with the `LIKE` even on null (both null-intolerant), and for the
literal case `And(guard, LIKE)` is the `LIKE` conjoined with one of its necessary conditions.
Idempotency under the fixed-point batch is maintained with a `TreeNodeTag` on the residual `Like`.

Patterns with escape characters are skipped (as elsewhere in the rule). The anchored-exact
rewrite, positional `substring` rewrites, and `LikeAll`/`LikeAny` are out of scope.

Generated-by: Claude Opus 4.8
@david-mollitor-db
david-mollitor-db force-pushed the like-underscore-length-guard branch from 7bddd91 to 0def0e9 Compare September 9, 2026 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant