HDDS-15209. Stabilize TestRocksDBCheckpointDiffer#testDifferWithDB for RocksDB layout variance

[arunsarin85]

What changes were proposed in this pull request?

• Wait until inflight compactions are drained before assertions that depend on a complete compaction DAG.
• Stop using hard-coded SST id lists for diff expectations: derive the baseline diff per run via getSSTDiffList with the full column-family mask, then apply the same subset filtering as before; compare sorted file names to avoid ordering noise.
• Column-family resolution for filtering: resolve from the compaction DAG or snapshot metadata; ignore SST ids that are not present on this run when building expectations (so golden lists don’t fight varying RocksDB numbering).
• SST backup directory check: replace fixed .sst filenames with assertions on count, .sst suffix, and membership in getCompactionNodeMap() (and fix the reversed “expected vs actual” comparison that used the directory listing as “expected”).

Please describe your PR in detail:
TestRocksDBCheckpointDiffer#testDifferWithDB was flaky on CI because RocksDB does not always assign the same SST file numbers or compaction shape across runs and OSes. The test mixed timing (DAG not fully updated), hard-coded SST names for diff expectations and for SST backup links, and a misleading backup assertion where the filesystem listing was passed as expected and compared to a static list.

This change makes the test derive what it can from the same code under test (getSSTDiffList, compaction node map, backup dir contents) instead of pinning to one successful run’s numeric ids. It keeps the intent: compaction tracking runs, DAG-based diffs behave consistently across column-family subsets, and compaction-input backups exist and correspond to tracked SSTs.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15209

How was this patch tested?

flaky-test-check workflow with submodule=rocksdb-checkpoint-differ, test-class=org.apache.ozone.rocksdiff.TestRocksDBCheckpointDiffer, test-name=testDifferWithDB

https://github.com/arunsarin85/ozone/actions/runs/26268683979

[SaketaChalamchala]

Thanks for the patch @arunsarin85. Constructing the baselineDiffFileNames(from which expectedDiffFiles are derived) using the same method differ.getSSTDiffList() that the actualNames is constructed changes the intention of the test.
This test is meant to verify the correctness of differ.getSSTDiffList() so the expected diff list needs to be built independently of differ.getSSTDiffList()

Maybe a better approach would be:
Pseudocode:

setup:
  open active DB with auto-compactions disabled. cfOpts.setDisableAutoCompactions(true);

write_and_checkpoint():
    // First snapshot
    Insert keys
    snap-1 = create_checkpoint()
    snap_1_files = sst_file_list(snap_1) // grouped by key, file, directorytable column families

     // For the rest of the snapshots
     Insert keys
     // Before taking the snapshot flush all txns,  track all uncompacted SST files, run manual compaction then take a snapshot
     activeRocksDB.get().flush()
     snap_n_files = sst_file_list(activeRocksDB) // grouped by key, file, directorytable column families
     compactOptions.setBottommostLevelCompaction(kForce)
     activeRocksDB.get().compactRange(key/file/directoryTable, null, null, compactOptions)
     snap-n = create_checkpoint()
  
build_expected_diffs_between_adjacent_snaps():
  expected_diff_between_snap-2_snap-1 = snap_2_files - snap_1_files
  expected_diff_between_snap-3_snap-2 = snap_3_files - snap_2_files
  ...
  expected_diff_between_snap-n_snap-n-1 = snap_n_files - snap_n-1_files

diffAllSnapshots: 
    build_expected_diffs_between_adjacent_snaps()
    
    for each snap_n and snap_prev from (snap_1 .. snap_n-1);
         actual_diff = differ.getSSTDiffList(snap_n, snap_prev)
         expected_diff = expected_diff_between_snap-n_snap-n-1 + expected_diff_between_snap-n-1_snap-n-2 + .... + expected_diff_between_snap-prev+1_expected_diff_between_snap-prev
         assert expected_diff == actual_diff

[SaketaChalamchala]

Asserting the count here may still be flaky depending on how RocksDB compacts on different platforms. A more concrete check can be

• Get the diff SST files between the last snapshot and the first snapshot using differ.getSSTDiffList(). (The correctness of getSSTDiffList() should already be verified in diffAllSnapshots(). See comment below)
• Check whether these diff list of files are in the last snapshot.
• Otherwise they should be in SST backup directory (they should not be pruned as they will correspond to the leaf node in the compaction DAG).