[SPARK-59319][SQL][FOLLOWUP] Enforce restricted mode in the single-pass analyzer and avoid eager class loading for rejected calls

[HyukjinKwon]

What changes were proposed in this pull request?

This is a follow-up to #58604, which added the opt-in restricted SQL execution mode (spark.sql.restrictedMode.enabled). It addresses three issues found in post-merge review, all scoped to when restricted mode is enabled (no change to default behavior):

1. Enforce the mode across both analyzer paths. CheckAnalysis.checkRestrictedMode runs only in the fixed-point analyzer. The single-pass resolver resolves reflect/java_method/try_reflect and marks the plan analyzed without calling checkAnalysis, so the mode was not enforced when the single-pass resolver was enabled. HybridAnalyzer.apply now routes a restricted-mode session to the fixed-point analyzer (the same branch used by default when the single-pass resolver is off) in every single-pass mode.

2. Do not load the referenced class before rejecting a reflect call. Type-checking a reflect/java_method/try_reflect call resolves the referenced class via Utils.classForName, which runs its static initializer. That happened during analysis, before checkRestrictedMode rejected the call. CallMethodViaReflection.checkInputDataTypes now short-circuits in restricted mode so a call that is going to be rejected never loads the class.

3. Traverse each subquery only once. checkRestrictedMode descended into subquery plans both through SubqueryExpression and through innerChildren (which already contains them), doubling the work at every nesting level and making validation exponential in subquery-nesting depth. It now recurses through innerChildren only for the non-subquery inner plans (analysis-only command bodies), keeping validation linear.

Why are the changes needed?

Restricted mode is a gate that blocks reflect/java_method/try_reflect and TRANSFORM ... USING. Without (1) the gate could be bypassed by enabling the single-pass resolver; without (2) a rejected call could still run a class's static initializer; (3) made the gate's cost blow up on deeply nested subqueries.

Does this PR introduce any user-facing change?

No. All changes only affect sessions that have opted into restricted mode; the default behavior is unchanged.

How was this patch tested?

New unit tests in RestrictedModeSuite (rejection without initializing the referenced class) and RestrictedModeCommandSuite (enforcement with the single-pass resolver enabled; a feature nested inside a scalar subquery is still reached; deeply nested subqueries with no restricted feature analyze in linear time). Existing CallMethodViaReflectionSuite still passes.

Was this patch authored or co-authored using generative AI tooling?

Yes, generated by Claude.

[sunchao]

Reviewed e344145. No actionable correctness findings.

Source tracing supports all three fixes: restricted-mode analyzer routing, avoiding reflection class initialization, and removing duplicate subquery traversal while retaining command-body checks. I reviewed the full diff, callers, tests, and prior discussion.

Validation: I did not run Spark tests locally because Scala/SBT dependencies are absent. CI precompilation and lint checks passed, and other checks are still running. The SparkR and pyspark-connect-old-client jobs currently fail on fillna with StringType is not matched at fillValue; these failures appear unrelated to the restricted-mode changes, but CI is not yet green.

[sunchao]

Thanks for the quick follow-up @HyukjinKwon !

[uros-b]

Thank you @HyukjinKwon and @sunchao!

[HyukjinKwon]

Merge Summary:

• merged into master 2e878ec
• merged into branch-4.x fa2ff07
• merged into branch-4.3 3b6ab0b

Posted by merge_spark_pr.py