[SPARK-36082][SQL][FOLLOWUP] Restore NAAJ broadcast hash join by default

[cloud-fan]

What changes were proposed in this pull request?

This PR follows #55678 and
#58404.

It restores the original planning behavior for the optimized single-column null-aware anti join
(NAAJ): by default, Spark selects the specialized null-aware BroadcastHashJoinExec and builds
the right side independently of spark.sql.autoBroadcastJoinThreshold.

It adds the internal spark.sql.nullAwareAntiJoinBroadcastThreshold configuration as an explicit
tuning option. The configuration defaults to Long.MaxValue, preserving the original behavior.
When the estimated right-side size exceeds the configured value, Spark skips the specialized hash
path and falls through to regular join planning. Setting it to -1 always skips the specialized
hash path.

The AQE broadcast-mode validation introduced in #55678 is retained. The PR removes the additional
NAAJ fallback decision and optimizer plumbing introduced in #58404.

Why are the changes needed?

#55678 made the NAAJ hash optimization conditional on the general automatic broadcast threshold,
and #58404 added planning logic to preserve the exact nested-loop fallback and its build side.
However, the automatic broadcast threshold and plan statistics are cost-planning heuristics; they
do not prove that either input can be materialized and broadcast safely.

Spark does not have a shuffle-capable NAAJ implementation. When regular join planning chooses the
right side for the fallback, BroadcastNestedLoopJoinExec still broadcasts that same input while
changing hash lookup from O(M + N) to nested-loop evaluation at O(M * N). This may replace a
broadcast failure with a much longer-running query without eliminating the memory risk.

When the right side exceeds the new threshold, regular join planning may choose a build-left
nested-loop join if the left side is estimated to be broadcastable. This can avoid broadcasting
the right side, but it is only best-effort: the estimate may be inaccurate, and the fallback may
still OOM or run in O(M * N) time. A generally scalable solution for large NAAJs still requires
a non-broadcast implementation.

Does this PR introduce any user-facing change?

Yes. Compared with the current behavior, the optimized single-column NAAJ uses the null-aware
broadcast hash join by default even when the estimated right side exceeds
spark.sql.autoBroadcastJoinThreshold.

Users who need the size-based fallback can set
spark.sql.nullAwareAntiJoinBroadcastThreshold. Its default value is Long.MaxValue, so the
default behavior is the same as before #55678.

How was this patch tested?

Added coverage to JoinSelectionHelperSuite showing that the default NAAJ threshold is independent
of spark.sql.autoBroadcastJoinThreshold and that an explicitly configured threshold rejects an
oversized right side.

Added coverage to JoinSuite showing that an explicit NAAJ threshold enables the build-left
nested-loop fallback for a small-left/large-right query.

The tests were not run locally.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-5)

[cloud-fan]

cc @sunchao