Spark: Stop async preload when either limit is reached

[iamrajatrana]

Async micro-batch initial preload currently combines its row and file safety limits with OR, so it keeps loading snapshots until both limits are reached. With skewed snapshots, this can substantially over-buffer files or rows during catch-up.

This changes the condition to AND so preload stops when either configured safety limit is reached across Spark 3.5, 4.0, 4.1, and 4.2. It also adds direct regression coverage for both asymmetric cases.

Fixes #17951

Tests:

• Focused TestAsyncSparkMicroBatchPlanner passes for Spark 3.5, 4.0, and 4.1
• Spark 4.2 focused test is currently blocked before test execution by an existing main compile failure in RewriteDataFilesSparkAction and RewritePositionDeleteFilesSparkAction
• ./gradlew -DsparkVersions=3.5,4.0,4.1,4.2 spotlessApply

---

AI Disclosure

• Model: GPT-5 (Codex)
• Platform/Tool: OpenAI Codex
• Human Oversight: unreviewed
• Prompt Summary: Diagnose and fix async Spark micro-batch initial preload exceeding either configured safety limit, add regression tests, and prepare the contribution.

[uros-b]

Makes sense overall, and seems correct and well scoped - thank you @iamrajatrana! Please make the CI green and ping committers for further review.

[iamrajatrana]

Thanks for the review, @uros-b. The branch is now rebased onto main, and all CI checks are green. @bryanck and @RjLi13, could you please take a look when you have a chance?