[SPARK-59339][ML][SQL] Add a SQL expression for ML vector dot products

[zhengruifeng]

What changes were proposed in this pull request?

This PR adds an internal VectorDotProduct Catalyst expression, registered as
ml_vector_dot_product, for MLlib dense and sparse vectors. The expression supports interpreted
evaluation and whole-stage code generation for all dense/sparse input combinations.

It also adds private ML helpers for vector-column and constant-coefficient inputs. This PR does not
integrate the expression into any ML algorithm yet.

Why are the changes needed?

The internal expression operates directly on the vector's SQL struct representation, preserves
sparse inputs without densifying them, and allows Catalyst code generation. It provides a focused
building block for future ML transform optimizations without changing an algorithm in this PR.

Does this PR introduce any user-facing change?

No. This changes the internal execution of ML model transforms.

How was this patch tested?

New tests cover interpreted and code-generated evaluation for dense/dense, dense/sparse,
sparse/dense, and sparse/sparse inputs, as well as nulls, empty vectors, infinite and NaN values,
and mismatched dimensions.

The following focused suites and cases were run:

build/sbt 'catalyst/testOnly *VectorDotProductSuite'
build/sbt 'mllib/testOnly *FunctionsSuite -- -z "vector_dot_product"'

dev/scalastyle also passed.

Benchmark

The expression was compared with a Scala UDF implementing (v1, v2) => v1.dot(v2). Inputs were
cached and consumed through a noop sink on Java 17 with local[1]. Each case had a two-second
warm-up followed by five timed iterations; the table reports the best iteration. Sparse vectors
had 10% density. Row counts were scaled inversely with dimensionality to keep approximately 10
million dense feature operations per iteration.

Features	Rows	Inputs	ml_vector_dot_product	Scala UDF	Speedup
10	1,000,000	dense/dense	105 ms	189 ms	1.81x
10	1,000,000	sparse/dense	92 ms	205 ms	2.22x
10	1,000,000	sparse/sparse	97 ms	238 ms	2.46x
100	100,000	dense/dense	45 ms	58 ms	1.30x
100	100,000	sparse/dense	29 ms	55 ms	1.90x
100	100,000	sparse/sparse	23 ms	52 ms	2.23x
1,000	10,000	dense/dense	39 ms	51 ms	1.31x
1,000	10,000	sparse/dense	21 ms	44 ms	2.10x
1,000	10,000	sparse/sparse	14 ms	35 ms	2.50x
10,000	1,000	dense/dense	36 ms	52 ms	1.44x
10,000	1,000	sparse/dense	22 ms	41 ms	1.82x
10,000	1,000	sparse/sparse	14 ms	32 ms	2.26x

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-5)

[zhengruifeng]

Merge Summary:

• merged into master 70d1a0d
• merged into branch-4.x 30c7084

Posted by merge_spark_pr.py