Spark: Restore 4.2 commit history

[manuzhang]

Summary

• Revert the squashed Spark 4.2 merge (031135677395a3fbc90a2e33f58a9fa3983335f5) and its follow-up cleanup (e76d63584d7f83b102026749e1ae0f91813cb78e).
• Replay the changes from Spark: Add support for 4.2.0 #14984 as three commits: move spark/v4.1 to spark/v4.2, copy it back, and add Spark 4.2 support. The support commit includes all follow-up fixes.
• Carry compatible Spark 4.1 changes that landed after the original branch point into Spark 4.2: executor-side delete-file caching for rewrite actions (Spark 4.1: Add rewrite option to enable executor cache for delete files #17868) and try-with-resources for StreamingOffset JSON generation (Core, Hive, Spark: Auto close JsonGenerator using try-with-resources #17931).
• Keep the Spark 4.1-only TableInfo API changes from Spark 4.1: Mark overrides of deprecated Spark API as deprecated #16028 out of Spark 4.2, where the required CatalogV2Util signatures are unavailable.

The cleanup from #17952 remains present because the replay starts from the current Spark 4.1 sources.

Merge Requirement

This PR must not be squash-merged because squashing would discard the commit and rename history it restores. Please use a merge commit or rebase merge.

Validation

• ./gradlew --no-daemon :iceberg-spark:iceberg-spark-4.2_2.13:assemble :iceberg-spark:iceberg-spark-extensions-4.2_2.13:assemble :iceberg-spark:iceberg-spark-runtime-4.2_2.13:assemble
• ./gradlew --no-daemon :iceberg-spark:iceberg-spark-4.2_2.13:test --tests org.apache.iceberg.spark.TestSparkExecutorCache --tests org.apache.iceberg.spark.actions.TestRewriteDataFilesAction
• git diff --check upstream/main...HEAD
• Verified spark/v4.1 is unchanged from upstream/main

---

AI Disclosure

• Model: GPT-5
• Platform/Tool: Codex
• Human Oversight: partially reviewed
• Prompt Summary: Revert the squashed Spark 4.2 commits and replay the original commits from PR Spark: Add support for 4.2.0 #14984 while preserving current Spark 4.2 compatibility.

[manuzhang]

@szehon-ho @huaxingao @nssalian This is to fix the Spark 4.2 commit history and should be Rebase and merge. Here are the detailed changes. We'd better move fast to avoid further conflicts and complexity. Please take a look. Thanks!

Summary

• Revert the squashed Spark 4.2 merge (031135677395a3fbc90a2e33f58a9fa3983335f5) and its follow-up cleanup (e76d63584d7f83b102026749e1ae0f91813cb78e).
• Replay the changes from Spark: Add support for 4.2.0 #14984 as three commits: move spark/v4.1 to spark/v4.2, copy it back, and add Spark 4.2 support. The support commit includes all follow-up fixes.
• Carry compatible Spark 4.1 changes that landed after the original branch point into Spark 4.2: executor-side delete-file caching for rewrite actions (Spark 4.1: Add rewrite option to enable executor cache for delete files #17868) and try-with-resources for StreamingOffset JSON generation (Core, Hive, Spark: Auto close JsonGenerator using try-with-resources #17931).
• Keep the Spark 4.1-only TableInfo API changes from Spark 4.1: Mark overrides of deprecated Spark API as deprecated #16028 out of Spark 4.2, where the required CatalogV2Util signatures are unavailable.

The cleanup from #17952 remains present because the replay starts from the current Spark 4.1 sources.