Treat spark_env_vars as a backend default in config-remote-sync

[ilyakuz-db]

Changes

Treat spark_env_vars as a backend default in config-remote-sync, the same way custom_tags and cluster_log_conf already are. A remote-only value (e.g. injected by a cluster policy) is no longer synced back into config, while a value the user declares in config still syncs. Covers whole-block and per-key ([*]) injection for task, job-cluster, and standalone clusters.

Why

spark_env_vars is commonly populated remotely by cluster policies and can carry credentials. Syncing those back would write one environment's values into (often shared) config and break deploys in other environments.

Tests

Added an acceptance test: injected keys are skipped, while a user-declared key and an unrelated edit still sync.

[github-actions]

Approval status: pending

/acceptance/bundle/ - needs approval

5 files changed
Suggested: @denik
Also eligible: @andrewnester, @anton-107, @janniklasrose, @lennartkats-db, @pietern, @shreyas-goenka

/bundle/ - needs approval

Files: bundle/configsync/defaults.go
Suggested: @denik
Also eligible: @andrewnester, @anton-107, @janniklasrose, @lennartkats-db, @pietern, @shreyas-goenka

_{Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db, @rugpanov, @rclarey) can approve all areas.
See OWNERS for ownership rules.}

[eng-dev-ecosystem-bot]

Integration test report

Commit: b8b2fba

Run: 32131974667

	Env	🟨​KNOWN	🔄​flaky	💚​RECOVERED	🙈​SKIP	✅​pass	🙈​skip	Time
🟨​	aws linux	3		1	4	286	1152	8:16
🟨​	aws windows	3		1	4	288	1150	7:58
🟨​	azure linux	3		1	4	285	1152	7:33
🟨​	azure windows	3		1	4	287	1150	7:50
💚​	gcp linux			1	5	286	1152	7:58
🔄​	gcp windows		1	1	5	287	1150	6:32

9 interesting tests: 4 SKIP, 3 KNOWN, 1 RECOVERED, 1 flaky

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
💚​	TestAccept	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R
🙈​	TestAccept/bundle/invariant/no_drift	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/ssh/connection	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🔄​	TestSyncIncrementalFileSync	✅​p	✅​p	✅​p	✅​p	✅​p	🔄​f
🟨​	TestFetchRepositoryInfoAPI_FromRepo	🟨​K	🟨​K	🟨​K	🟨​K	🙈​S	🙈​S
🟨​	TestFetchRepositoryInfoAPI_FromRepo/root	🟨​K	🟨​K	🟨​K	🟨​K
🟨​	TestFetchRepositoryInfoAPI_FromRepo/subdir	🟨​K	🟨​K	🟨​K	🟨​K

Top 6 slowest tests (at least 2 minutes):

duration	env	testname
5:52	azure windows	TestAccept
5:42	gcp windows	TestAccept
5:42	aws windows	TestAccept
3:50	aws linux	TestAccept
3:46	gcp linux	TestAccept
3:44	azure linux	TestAccept