[SPARK-59183][SDP] SCD2 Ignore-null support; Populate version map per event row at ingest - #58645

Draft

AnishMahto wants to merge 16 commits into

apache:masterfrom

AnishMahto:SPARK-59183-populate-verison-map-at-ingest

Commits on Sep 8, 2026

[SPARK-59165][SQL] Add version map field to SCD2 CDC metadata schema
AnishMahto
committed
6eb834c
add test to demonstrate cdc_metadata schema evolution
AnishMahto
committed
e12068b
self-review
AnishMahto
committed
34fbb8e
introduce ignoreNullSelection in ChangeArgs with flow-level validation
AnishMahto
committed
3af99fd
self review
AnishMahto
committed
b3a6908
inline var
AnishMahto
committed
1686627
cleanup and unify helpers to Scd2VersionMap
AnishMahto
committed
f4e68e7
delete rows in microbatch receive null version maps
AnishMahto
committed
0806ed1
document risk for changing ignore-null selection between microbatch retries
AnishMahto
committed
a81f434
use withField
AnishMahto
committed
93af81e
use JSON serialization for version map keys
AnishMahto
committed
7f735db
link TODO for version map rewrite
AnishMahto
committed
725bce4
cleanup
AnishMahto
committed
0a9eac9
fix delete detection
AnishMahto
committed
ae528f0
self-review
AnishMahto
committed
891f9b5

Commits on Sep 9, 2026

refactor out generic extractLeafPaths
AnishMahto
committed
541bfcc