[SPARK-59342][SQL] Allow configuring schema alignment behavior for DSv2 writes

[johanl-db]

What changes were proposed in this pull request?

This changes extends the DSv2 connector interface to allow connectors to configure schema alignment behavior and opt-out of the default behavior when required:

• spark.sql.storeAssignmentPolicy=LEGACY being rejected for DSv2 writes.
• Strict implicit casts check at analysis time for DSv2 writes under sparksql.storeAssignmentPolicy=ANSI

Extending the Table interface:

public interface Table {
 ...
  default SchemaAlignmentConfig schemaAlignmentConfig() {
     return SchemaAlignmentConfig.DEFAULT;   }
}

public interface SchemaAlignmentConfig {
  default boolean allowLegacyStoreAssignmentPolicy() {
     return false;
   }

  default boolean deferCastValidationToRuntime() {
     return false;
   }
}

Why are the changes needed?

Delta is migrating to DSv2, and historically has always allowed:

• writes under spark.sql.storeAssignmentPolicy=LEGACY
• Many implicit casts at analysis, relying instead on execution time checks to catch malformed values or overflows.

Without a way to configure this. behavior, this effectively blocks migrating Delta to DSv2.

How was this patch tested?

Add Spark suite SchemaAlignmentConfigSuite covering the configuration mechanism
Add Delta DSv2 variant of existing DeltaInsertIntoImplicitCastSuite, showing that casting behavior is preserved when running on DSv2.