bundle: record and read deployment state via DMS - #6094
Open
shreyas-goenka wants to merge 166 commits into
Open
Conversation
…d IDs Wire the direct engine into the Deployment Metadata Service (DMS) so that a `record_deployment_history`-enabled bundle records each deploy/destroy as a version and can read its resource state back from DMS. The deployment ID is now assigned by the server: the first deploy calls CreateDeployment with an empty ID, reads the assigned ID back from the response, and persists it in the direct-engine state header (Header.DeploymentID). Later deploys pass the stored ID back, so a bundle maps one-to-one to a DMS deployment even after the local cache is deleted (the ID rides along in the workspace-synced state file). - libs/dms: Recorder creates the deployment (server-assigned ID) + version, heartbeats the lease, completes it, and deletes the deployment on destroy. - bundle/direct: operationRecorder reports each applied resource operation; the wire resource_key drops the CLI-internal "resources." prefix. - bundle/direct/dstate: Open takes a DMS client and overlays DMS resource state when DMS holds a successful version; deployment ID persisted in the header. - bundle/phases: create the version after plan approval, complete it under the lock, record operations during apply. - libs/testserver: stateful fake DMS (deployments/versions/operations/resources) with server-generated IDs; acceptance test covers deploy, cache-loss redeploy, and destroy. Co-authored-by: Isaac
The read overlay decided whether DMS owns a deployment's state by listing versions and scanning for a successful one. The deployment now exposes last_successful_version_id directly, so a single GetDeployment answers the same question — no version listing. The field is still stage:DEVELOPMENT in the proto and therefore stripped from the generated SDK, so this reads the deployment via a raw GET into a local struct as a temporary stub. Once the field is promoted to PRIVATE_PREVIEW and regenerated, the raw call collapses to client.GetDeployment(...). LastSuccessfulVersionId and the threaded config argument goes away (see the TODO in deploymentHasSuccessfulVersion). The testserver's GetDeployment now serves last_successful_version_id (tracked on version completion), and the now-unused ListVersions fake is removed. Co-authored-by: Isaac
Five fixes to the DMS state recording added in #6052: 1. The fake DMS server dropped last_successful_version_id. GetDeployment serialized the response through a struct embedding bundledeployments.Deployment, whose promoted MarshalJSON silently discards sibling fields. The CLI reads a missing value as "DMS does not own the state", so the entire read/overlay path (overlayDMSState, fetchDeploymentResources, deploymentHasSuccessfulVersion) never ran in any test. Serialize through a map instead, and unit-test the shape. 2. A bundle with no resources leaked a deployment record per deploy. Such a deploy writes no WAL entries, and the state file was only persisted when the WAL carried entries, so the server-assigned deployment ID was dropped and the next deploy created a second deployment. Track a dirty header so Finalize persists an ID change on its own. A header-only WAL that changed nothing still skips the write, keeping the serial in step (acceptance/bundle/deploy/wal/header-only-wal). 3. Deploy after destroy failed permanently. A successful destroy deletes the deployment record but leaves its ID in local state, so the next deploy's GetDeployment 404'd and the error was fatal — unrecoverable on retry. Treat a missing deployment as "create a new one"; any other read error stays fatal. 4. The overlay dropped depends_on. DMS does not record dependency edges, so replacing local state with DMS resources lost them, affecting delete ordering, the apply graph, and --select expansion. Carry depends_on over from the local entry. Masked by (1) until now. 5. Recording bypassed secret redaction. dstate.SaveState redacts bundle:"sensitive" fields before writing state, but the operation recorder marshalled raw, so a secret would be sent to DMS in plaintext and read back into local state. Route through structwalk.RedactSensitiveFields. Latent today: no resource state type carries a sensitive field yet. Adds acceptance coverage for deploy-destroy-deploy and for a bundle with no resources, both of which now exercise the read path (visible as GET .../resources in the recorded requests). Co-authored-by: Isaac
Recording an operation with the deployment metadata service used to happen inline on the apply worker, so every resource paid a CreateOperation round trip before the worker moved on to the next one. Queue the operations instead and upload them from a small pool of background workers (operationQueue, in the new opqueue.go). The queue holds resource keys rather than operations, so an operation recorded for a resource that is still waiting replaces the queued one: DMS keeps one state per resource key, so the later operation supersedes the earlier one and a single request records both. This is best effort - only operations that no worker has picked up yet are coalesced. Uploads are not fire-and-forget. Apply drains the queue before returning and reports the first failure, because a version that completes successfully makes DMS authoritative for resource state; dropping an operation would leave DMS with an incomplete resource set and the next deploy would plan to create resources that already exist. At most one upload per resource key runs at a time, so the last operation recorded for a resource is also the last one the service sees. Co-authored-by: Isaac
Recording deployment history is implemented end to end, but it cannot be exposed to users yet: enabling it makes the deployment metadata service the source of truth for resource state, and there is no upgrade path from an existing direct-engine state file to a DMS-owned one. Setting the flag is now an error. DATABRICKS_BUNDLE_ENABLE_RECORD_DEPLOYMENT_HISTORY lifts the error so the CLI's own acceptance tests and DMS development can exercise the feature until the direct state upgrade lands. Co-authored-by: Isaac
DATABRICKS_BUNDLE_ENABLE_RECORD_DEPLOYMENT_HISTORY read as if it were the switch that turns the feature on. It is not: the flag in databricks.yml does that, and this variable only permits the flag to be set while the feature is gated off. Name it DATABRICKS_BUNDLE_FORCE_ALLOW_RECORD_DEPLOYMENT_HISTORY. Co-authored-by: Isaac
Replace the blanket gate on experimental.record_deployment_history with a narrower check in dstate.DeploymentState.Open: recording is refused only when the state file already tracks deployed resources that DMS does not know about. Once DMS holds a successful version it is authoritative for resource state even when its resource set is empty, so adopting a state file written by a CLI that predates DMS would make already-owned resources look absent and create them a second time. A state file that DMS already owns is fine, and so is one with no resources (e.g. left behind by a destroy), which is what makes the error's destroy-and-redeploy advice work. The check keys off len(State) rather than the file existing because destroy leaves resources.json in place with an empty resource set. This drops validate.ValidateRecordDeploymentHistory and the DATABRICKS_BUNDLE_FORCE_ALLOW_RECORD_DEPLOYMENT_HISTORY escape hatch, which are no longer needed. Upgrading an existing state in place (state v3 with a feature flag plus per-resource tombstones so older clients refuse the state) is left as a TODO. Co-authored-by: Isaac
The concurrent-producer test tripped three linters: - modernize/revive want wg.Go instead of wg.Add + go func + defer wg.Done. - testifylint's go-require flags recordState, which calls require inside the spawned goroutines. testify assertions may only run on the goroutine running the test function. Record inline and send each error to a buffered channel that the test goroutine drains after wg.Wait, so the assertions stay on the test goroutine. Co-authored-by: Isaac
The operation queue collapses repeated writes to the same resource key by replacing the queued operation wholesale, so a create followed by an update was uploaded as an update. That tells DMS the resource already existed before this deploy, when in fact this deploy created it. Merge the actions instead: the state uploaded is still the later one, but a queued create or recreate wins over a subsequent update. A delete still wins over anything queued before it, since the resource is gone. This is not reachable from Apply today - each resource is recorded once per deploy, because there is one record call per graph node and dagrun visits each node exactly once - so this is about the queue being correct for any caller that records a resource more than once. Co-authored-by: Isaac
Co-authored-by: Isaac
The deployment ID was persisted in the local state file header, which made
the CLI the source of truth for a value the service mints. DMS registers
each deployment as a workspace node named resources.deployment.json under
initial_parent_path, and the node's ID *is* the deployment ID, so it can be
resolved from the workspace instead.
ResolveDeploymentID does a get-status on <state_path>/resources.deployment.json
and returns the node ID, or empty when the node is absent. Deploy, destroy,
and the read path all resolve the ID that way and pass it down; the read path
then constructs state from GetDeployment + ListResources as before.
Consequences:
- Header.DeploymentID, GetDeploymentID, and SetDeploymentID are gone,
along with the headerDirty machinery that only existed to persist the ID
on a resource-less deploy.
- Open takes a *DMSSource instead of a (client, config) pair, since the
resolved ID now has to be threaded in too.
- CreateDeployment sets initial_parent_path, which the service requires
and the CLI never set.
- createDeploymentVersion no longer recovers from a 404 on GetDeployment by
creating a second deployment. A destroy trashes the node, so a resolved
ID whose record is missing means the two are out of sync, and creating
another deployment would collide on the same node path.
The testserver models the real derivation: CreateDeployment creates the
workspace node and uses its object ID as the deployment ID, so the acceptance
tests exercise get-status resolution end to end. dms/record now wipes the
local cache before redeploying and records zero operations, which is the read
path reconstructing state entirely from DMS.
Co-authored-by: Isaac
- Limit recorded state to 64 KB, checked when the payload is built so an
oversized resource fails itself rather than the drain at close.
- Collapse the operation queue's pending/inflight pair into one `owned` set.
The two maps encoded a single question ("is this key already claimed?") and
had to be read together; `take` now only releases ownership.
- Only log "Coalescing" when an operation was actually merged. The old code
logged it for in-flight keys too, where nothing was coalesced.
- Restore mergeWalIntoState's `hasEntries` naming and comment from main. The
`persist` rename existed for the headerDirty case, which is gone.
- Trim the comments added by this PR.
Also note in fetchDeploymentResources that DMS has no field for dependency
edges and they cannot be recovered from the recorded state (references are
resolved to literals before serialization), so depends_on is carried over from
the local state file.
Co-authored-by: Isaac
The read path carried depends_on over from the local state file, which is empty exactly when it matters: a fresh checkout reconstructs state from DMS and got no dependency edges. Deletes are the one case that cannot recompute them, because the resource is gone from config, so two dropped resources with a real dependency could be deleted in the wrong order. DMS has no field for dependency edges, and they cannot be recovered from the recorded config either: references are resolved to literals before it is serialized. So Operation.State now carries an envelope, dstate.RecordedState, holding the config plus depends_on. Nesting depends_on inside the config would have collided with resource fields of the same name (jobs.Task.depends_on). The envelope mirrors the local ResourceEntry, so both sides of the round trip have the same shape. acceptance/bundle/dms/depends-on covers it: a job referencing another records its edge, and after wiping the local state a destroy still deletes the referencing job first. Co-authored-by: Isaac
Migrating an existing deployment is not supported: Open rejects a bundle that already has resources in state, so a deployment that exists in DMS was created by an opted-in CLI and DMS owns its resource set outright. The last_successful_version_id probe that decided whether to trust DMS was therefore always true by the time it ran. Removing it takes with it the raw GET that read the field (it is stage:DEVELOPMENT and stripped from the generated SDK) and DMSSource.Config, which existed only to make that call. overlayDMSState is now readDMSState, since it no longer overlays anything conditionally: it just reads. Recording stays opt-in via experimental.record_deployment_history; that flag is what makes the caller pass a DMSSource at all. acceptance/bundle/dms/existing-state also now covers wiping the local cache: deploy pulls the state file back from the workspace, so the resources stay tracked and opting in is still rejected. Co-authored-by: Isaac
Reading resource state from DMS now requires the state file to record a
"deployment_history" feature flag, rather than inferring eligibility from the
resource count.
The old check refused a state that had resources and no DMS deployment ID. That
happened to be right, but it inferred intent from a side effect: a state with
resources could equally be one DMS already owns. The flag says so directly.
Header.Features was already scaffolded for exactly this, so this fills it in:
- Open writes the flag when recording is enabled, and refuses a state that
has resources without it. Migrating such a target is not supported, so the
error names the target and the three ways out: use a new target, destroy
this one and redeploy, or unset the feature.
- A state recording any feature is written at featureStateVersion, so a CLI
that predates the flag refuses it (see migrateState) instead of deploying
against a resource set that lives in DMS and looks empty on disk.
- migrateState accepts features it implements and refuses only unknown ones,
naming just those in the error.
- WAL replay carries the features forward, so recovering a WAL from a
recording deploy still produces a state marked as DMS-owned.
The flag is per target, which matches how experimental.record_deployment_history
is set.
Co-authored-by: Isaac
Coalescing now keeps the newest operation outright instead of merging fields. Each operation carries the resource's full state rather than a delta, so a newer one entirely supersedes an older one - including its resource_id, which is the field that actually changes between two records (a create learns the ID only after the API call returns it). The previous code merged the action and overwrote the ID, which is backwards: the action cannot differ between records of the same resource, while the ID can. mergeAction is gone. Also renames `owned` to `queuedOrUploading`. Nothing is owned by a particular worker: a key can be handled by one worker, released, and picked up later by another. The mark only means "some worker will get to this", which is all record needs to know in order to not queue the key twice. The doc comments now lead with the two rules that shape the design (no overlapping uploads per resource; only the newest operation matters) so the mechanism reads as a consequence of them rather than as bookkeeping. Co-authored-by: Isaac
Collaborator
Integration test reportCommit: 435683c
Top 3 slowest tests (at least 2 minutes):
|
The dms tests pass workspace paths to $CLI (`workspace get-status /Workspace/...`). On Windows, Git Bash rewrites a leading-'/' argument into a Windows path before the CLI sees it, so the lookup goes to C:/Program Files/Git/Workspace/... and 404s, failing record, no-resources and redeploy-after-destroy on that platform only. Same fix and reason as acceptance/cmd/workspace/export-dir-*/test.toml. Co-authored-by: Isaac
Setting it in test.toml applied it to the whole test, which stopped Git Bash
converting the PATH too - so python3 could not find print_requests.py
("can't open file 'C:\\c\\a\\cli\\cli\\acceptance\\bin\\print_requests.py'")
and every dms test failed on Windows, including the three that were passing.
trace exports leading KEY=value pairs in a subshell, so setting it there fixes
the CLI's leading-'/' argument without reaching the helpers.
The precedent this was copied from
(acceptance/cmd/workspace/export-dir-*/test.toml) uses no python helpers, which
is why the test.toml form is safe there but not here.
Co-authored-by: Isaac
close runs on one goroutine after every apply worker has returned, so nothing else touches the queue by then and the closed flag needs no protection. The wg.Wait orders the upload workers' writes to err before it is read, so that read needs none either. Also tightens the coalescing test to count uploads for the resource instead of only checking the payload of the last one. It asserted the merged content but would have passed if the operations had been uploaded twice, which is the thing coalescing exists to prevent. Co-authored-by: Isaac
Drops the RedactSensitiveFields call, leaving a TODO: fields marked bundle:"sensitive" now reach DMS in plaintext, and the read path writes them back into the local state file unredacted. This has to be restored before the feature ships to users. Also documents why take leaves the key in queuedOrUploading when it hands an operation to a worker, and covers the case the comment describes: recording while that key's upload is in flight. The operation cannot join the in-flight request, so it is uploaded next by the same worker rather than being dropped or picked up concurrently by a second one. Co-authored-by: Isaac
Co-authored-by: Isaac
Drops the record_deployment_history annotation change (and the generated schema that followed from it), leaving both files as they are on main. CompleteVersion now keys its no-op on versionNum rather than on the heartbeat handle. Both are set together by CreateVersion, so the behaviour is the same - a deploy that was cancelled, or whose CreateVersion failed, does not complete a version that was never created - but the check now names the thing it is actually guarding. Callers defer CompleteVersion unconditionally, so this is the only thing standing between a failed CreateVersion and a CompleteVersion call against a nonexistent version. Co-authored-by: Isaac
Restores validate.ValidateRecordDeploymentHistory and the hidden DATABRICKS_BUNDLE_FORCE_ALLOW_RECORD_DEPLOYMENT_HISTORY escape hatch, so setting experimental.record_deployment_history is an error unless that variable is set. The service side is not ready for users: DMS is only deployed to dev and staging, and reading state back needs the workspace APIs to expose the deployment's tree node, which is still behind a flag. With the flag unreachable, the state feature flag added earlier is not needed yet, so dstate is back to the resource-count check in Open. The deployment_history feature, hasFeature/setFeature, the version bump on write and the WAL carry-over all come back with the state upgrade in a follow-up. The dms acceptance tests force allow the flag, and bundle/dms/not-supported covers the error users see. Operation requests in bundle/dms/depends-on print multi-line now: the state envelope nests two levels, which --oneline made unreadable. Co-authored-by: Isaac
An upload failure was only reported at close, so a DMS outage let apply deploy every remaining resource and fail at the end. That leaves resources in the workspace that DMS has no record of, and since a completed version makes DMS the source of truth for resource state, the next deploy would create them again. record now returns the first upload error, which the apply worker turns into a failed node, so the deploy stops shortly after the failure instead of running to completion. It refuses new work only: operations already recorded still upload, because close drains them, so the records DMS ends up with match the resources that were actually applied. Resources already mid-apply also finish. Also repeats the one test whose bug depends on a scheduler interleaving rather than on a forced handshake, so a single run gets many chances to hit the bad ordering. The other tests pin their interleaving with the started/block channels, so repetition would not add coverage; the new error tests use a `done` channel to wait for an upload to have finished rather than merely started. Co-authored-by: Isaac
The previous commit made record return the upload error, but record runs after the resource has already been created or updated, so every node that started before the failure was noticed still modified the workspace. Apply now checks for a recorded failure before it touches anything, right after the dependency check, so a node that has not started yet is refused rather than applied. Resources already mid-apply still finish - the check cannot unwind those - but the deploy no longer runs to completion against a service that is rejecting its records. acceptance/bundle/dms/operation-upload-fails covers it. Which resources get refused depends on how far apply got before a background upload failed, so the per-resource errors go to a LOG file and requests are not recorded; the test asserts the deploy fails and reports the upload error. Also drops --sort from the dms tests that deploy zero or one resource: their request order is already deterministic, and the unsorted output reads in chronological order. Co-authored-by: Isaac
shreyas-goenka
commented
Jul 29, 2026
| // of and the next deploy would create them a second time. Checked here rather | ||
| // than only where operations are recorded, which is after the resource has | ||
| // already been modified. | ||
| if err := opQueue.firstErr(); err != nil { |
Contributor
Author
There was a problem hiding this comment.
we could eventually extend this to record and return all multiple errors that happened.
shreyas-goenka
commented
Jul 29, 2026
| @@ -0,0 +1,4 @@ | |||
|
|
|||
| === An operation upload failure fails the deploy instead of reporting only at the end | |||
Contributor
Author
There was a problem hiding this comment.
it's hard to make a assert more here because we cannot control how many requests went through. We could harden this test by making the number of workers configurable and 1. Omitting for now.
CreateDeployment now only registers the workspace node whose ID names the
deployment; the record itself is created by the first CreateVersion. A client
that registers a deployment and then fails before recording a version leaves
just the node behind, not an empty deployment.
That state is reachable, so both sides of the CLI handle it:
- the recorder starts at version 1 under the ID the node already names,
instead of failing on the missing record or creating a second deployment
that would collide on the same node path
- the read path keeps the local (empty) state instead of surfacing the 404
from ListResources
acceptance/bundle/dms/version-never-created covers it end to end: the first
version fails, and the next deploy reuses the same deployment ID.
Also drops libs/testserver/bundle_test.go. The fake is exercised by every dms
acceptance test, so unit tests for it only duplicate that coverage.
Co-authored-by: Isaac
Prepare looked the deployment up a second time to work out which version to create. The id already comes from opening the state, so the last version comes with it now, and the next one is arithmetic the client does when it is built. Prepare and resolveNextVersion are gone; what remains of them is EnsureDeployment, which a deploy calls before the plan because the stamp has to be in the snapshot the plan takes. The same read serves 'bundle summary', so reporting the deployment costs no API call of its own. It runs with the other state-into-config mutators now rather than behind the option that existed to keep those calls off other commands. Two goldens move: the deployment read now precedes the resource read. Co-authored-by: Isaac
shreyas-goenka
force-pushed
the
isaac/pr6052-fixes
branch
from
August 25, 2026 10:12
7992110 to
cc178ff
Compare
shreyas-goenka
force-pushed
the
isaac/pr6052-fixes
branch
3 times, most recently
from
August 25, 2026 11:43
693083f to
59f96b5
Compare
display_name, target_name, deployment_mode and workspace_info belong to the deployment, not to each version that happened to carry them. CreateDeployment sets them, and a later run brings them up to date with UpdateDeployment, masking only what differs from the record already read when the state was opened - so a run that says the same thing writes nothing. Comparing workspace_info ignores ForceSendFields: the SDK fills it in from the response, so a record read back never deep-equals one built here, and every run would have looked like a change. CreateVersion keeps what a version owns: cli_version, version_type, previous_version_id, git_info and the staged operations. The update body is a map holding exactly the masked fields, as the operation body already is: the SDK struct's omitempty would drop deployment_mode when a target stops setting mode, and the service rejects a mask naming a field the body leaves out. The fake now refuses that too, so the next such drop fails locally instead of on a real workspace. Both callers - the deploy, before the plan is stamped, and Start - go through the same place, so what was written is cached and the second call has nothing left to do. UpdateDeployment is a third hand-written call; the generated client has none. The package also took "jobs.foo" while the rest of the CLI calls that resource "resources.jobs.foo", so every caller converted on the way in and back on the way out. Its exported names take the state key now, and the prefix comes off where the request is built - the only place the short form belongs. The requests on the wire are unchanged. The summary mutator reads the same deployment record the rest of this uses, so reporting it costs no API call of its own. It was still making one. Co-authored-by: Isaac <no-reply@databricks.com>
shreyas-goenka
force-pushed
the
isaac/pr6052-fixes
branch
from
August 25, 2026 11:50
59f96b5 to
5a808f3
Compare
denik
reviewed
Aug 25, 2026
| }) | ||
| if err != nil { | ||
| // The service caps how many operations one version may stage, so a bundle past the | ||
| // cap cannot be recorded at all. Say so rather than passing the raw API error on. |
Contributor
There was a problem hiding this comment.
why don't we make backend error understandable instead? We already have a nice formatter for SDK errors that includes code + message.
Contributor
Author
There was a problem hiding this comment.
We never expect this error to happen tbh. If this happens then something really went wrong with our implementation. The existing error is fine, just adding this context makes it a bit easier to grok.
| }, nil | ||
| } | ||
|
|
||
| // BufferedClient is everything one deploy or destroy records with DMS. Operations are buffered |
Contributor
There was a problem hiding this comment.
Can we trim it further to only deal with operations?
For other operations (CreateDeployment, CreateVersion etc) we already have a client and we can use that directly.
Contributor
Author
There was a problem hiding this comment.
I actually went in the other direction. Consolidating all interaction to DMS into the buffered client. There are a few advantages:
- Most (all?) interactions with DMS are gated behind this client.
- All DMS interactions and setup are co-located. Easier to reason about.
- Allows to idempotent close easily. We need two things during close - (1) wait for existing operations to drain, (2) Close out the completed version (call CompleteVersion). We include close behind a defer function and call Drain() separately, so it's useful to a single client.
Contributor
Author
There was a problem hiding this comment.
I asked claude the same question and it added some more reasons:
4. The version number is claimed before it exists, and only one thing can claim it. versionNum = last + 1 is
settled at construction, the plan is stamped with it via Version() before approval, and CreateVersion then
creates exactly that number with previous_version_id as the precondition — which is what makes claiming
it up front safe (a deploy that took it meanwhile gets a 409, reported, not overwritten). Split across two
clients, that invariant lives in two places.
5. The deployment record is read once and cached, which is what makes the masked update possible.
current is what staleFields compares against, so a run that says the same thing writes nothing. A caller
calling Client.UpdateDeployment itself would have to carry the record and the mask — and would hit the
double-PATCH I found on staging, where EnsureDeployment (before the plan) and the second call inside
CreateVersion both saw the record as stale.
7. Sequence ids only work if staging and updating are the same object's business. CreateVersion stages
operations at sequence_id "0"; the first update for each resource must send that as its precondition, and
each reply carries the next one. That's one protocol in two halves — stagedSequenceID is meaningless to a
caller that didn't stage.
9. The heartbeat's lifetime is bracketed by the two methods. CreateVersion starts it (line 314), Close stops
it (350). Direct Client use puts lease renewal on the caller — and an expired lease silently completes the
version as LEASE_EXPIRED after the 2-minute TTL.
11. Nil is the "recording off" null object, across the whole surface. 11 methods guard the nil receiver, so
deploy, destroy and the state DB call through unconditionally. Any call routed to Client instead needs its
own if recording guard back, since *dms.Client isn't nil-safe.
Also worth folding into your point 3: Close is where "a completed destroy deletes the deployment" lives
(VersionCompleteSuccess && versionType == Destroy). That's a policy decision, not a call — good to have
in one place rather than remembered by destroy.go.
|
|
||
| // Read the deployment once here: what follows needs its last version, to know | ||
| // which version to create, and its metadata, to know what to bring up to date. | ||
| dep, err := dms.ReadDeployment(ctx, dmsClient, deploymentID) |
Contributor
There was a problem hiding this comment.
So in cases where workspace path exists but database entry does not, this bundle becomes blocked?
Given that we already handle this on CreateDeployment side, we should also handle it here?
and I think should be abstracted away into backend (or temporarily into dms package), so you can do:
deploymentID, lastVersionID, err := ReadDeployment(ctx, state_path)
this returns non-empty deploymentID and lastVersionId only if deployment is complete - both file and DB entry exist. if either is missing, both are empty and caller must do
CreateDeployment(state_path) to create or finalize the deployment.
Contributor
Author
There was a problem hiding this comment.
So in cases where workspace path exists but database entry does not, this bundle becomes blocked?
Good point. Yeah we need to fix that. Lets to this as a separate PR? The code to recover from that state and recreate the deployment.
and I think should be abstracted away into backend (or temporarily into dms package), so you can do:
Yeah sounds good, eventually a method to lookup using path makes sense here.
Two tests added on main sit under acceptance/bundle, so they inherit the DMS matrix this branch adds there and their generated out.test.toml is now stale. Neither side is wrong on its own, so the staleness only appears once they meet - which is what CI checks, since it tests the merge rather than the branch. Co-authored-by: Isaac <no-reply@databricks.com>
shreyas-goenka
force-pushed
the
isaac/pr6052-fixes
branch
8 times, most recently
from
August 25, 2026 22:46
34e2ecf to
c609ce6
Compare
Open took a DMSSource - a client, a deployment id, and that deployment's record - so it could read the resources from the service and stash the other two for the summary mutator and the deploy phase to read back out of the state DB. All three already belong to the BufferedClient, so that is passed instead: the struct goes, and StateDB.DMSDeploymentID and .DMSDeployment go with it. NewBufferedClient now reads what it used to be handed - the deployment id under the state path, and the record naming the version to follow - so there is one constructor and it needs a workspace client rather than four resolved values. What is left on the bundle side is the mapping from config to Metadata, and the one condition that decides whether this run records at all. ReadDeployment and Client.GetDeployment were both wrappers with that constructor as their only caller, and between them they added a name, an early return and error text nothing asserted. The constructor returns early when there is no id - which reads better where the deployment is known not to exist yet - and otherwise makes the SDK call the wrappers stood in front of, passing its error through. Open makes the one read it needs and hands the resources to applyDMSState, which maps them and touches nothing else. That is what the state tests exercise now, with a slice instead of an SDK iterator fake standing in for a list call. The version type moves from construction to CreateVersion - Start until now, which said that something starts without saying what. It is the one input the client cannot have at construction: the place that opens the state serves deploy and destroy alike, and only the phase knows which run this is. Resolving the deployment id is nobody else's business now that the constructor is the only caller, so it is unexported. It keeps its name and its tests: reaching it through the constructor would mean faking a workspace and the service to cover what a path join and a not-found do. Open reads through the client and does not keep it. RegisterDmsClient still installs it for writing, once a version exists to write under. Co-authored-by: Isaac <no-reply@databricks.com>
shreyas-goenka
force-pushed
the
isaac/pr6052-fixes
branch
from
August 25, 2026 22:56
c609ce6 to
d3fee60
Compare
The package has a Client and a BufferedClient, so "dmsClient" left the reader to guess which one a field or parameter holds. Every one of them holds the buffered client, and now says so - the exported field on DeploymentBundle and the method that installs it included, since that is where the guessing started. Rename only. Co-authored-by: Isaac <no-reply@databricks.com>
Another test added on main sits under acceptance/bundle, so it inherits the DMS matrix this branch adds there and its generated out.test.toml went stale. This recurs on every merge until the branch lands. Co-authored-by: Isaac <no-reply@databricks.com>
The half-done state was recorded as OPERATION_STATUS_IN_PROGRESS, on the reading that the SDK enum trailed the service proto. It does not: the proto has UNSPECIFIED, SUCCEEDED, FAILED and PENDING, and nothing else. A real workspace parses the unknown name as UNSPECIFIED and refuses the write, so every recreate failed to record - and the local fake accepted it, so every test said otherwise. PENDING is what the service calls an operation it has recorded but not applied, which is exactly where a recreate sits between deleting the old resource and creating the new one. Verified against a real deployment: the delete half clears the state and reports PENDING, the create half then records the new resource, and the service ends up describing it. The service also exempts a pending operation from the state-vs-existence rules, which is what lets the delete half clear state before anything replaces it. The fake now refuses a status the enum does not have, rather than storing whatever string it is handed. That permissiveness is why an invented value passed every local run and failed only against a workspace. Co-authored-by: Isaac <no-reply@databricks.com>
shreyas-goenka
commented
Aug 26, 2026
| "sequence_id": "0", | ||
| "state": "", | ||
| "status": "OPERATION_STATUS_IN_PROGRESS" | ||
| "status": "OPERATION_STATUS_PENDING" |
Contributor
Author
There was a problem hiding this comment.
we can eventually make this an IN_PROGRESS status once we support this.
shreyas-goenka
commented
Aug 26, 2026
| @@ -52,6 +52,16 @@ func NewStateUpdate(resourceID string, state json.RawMessage, inProgress bool) ( | |||
| status = StatusPending | |||
Contributor
Author
There was a problem hiding this comment.
This will eventually become StatusInProgress once the backend supports it.
shreyas-goenka
force-pushed
the
isaac/pr6052-fixes
branch
from
August 26, 2026 00:33
8360fe4 to
d7ab4a2
Compare
Recording a delete sent the same mask as every other write, with an empty state and the id of what had just been removed. A real workspace refuses that: a delete that succeeded has no resource left to describe, so an empty state is a state it will not take, and an id is only accepted alongside one it was given. Destroy therefore failed for every bundle that records history - the resources went, the operation stayed pending, and the version completed as a failure. The service reads a masked state with no value as a clear, which is also what drops the resource from the deployment. So a delete now names state in the mask and leaves the value out, and does not name the id. Verified against a real workspace: deploy then destroy both complete VERSION_COMPLETE_SUCCESS, the deployment ends DELETED, and a delete of one resource among several drops only that one from the deployment. The fake took the old shape without complaint, which is why this only ever failed against a workspace. It now reads a masked state with no value as the clear the service reads it as, and counts only a state it was given when deciding whether an id may accompany it. Co-authored-by: Isaac <no-reply@databricks.com>
shreyas-goenka
force-pushed
the
isaac/pr6052-fixes
branch
from
August 26, 2026 00:53
d7ab4a2 to
435683c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds code to read and write state using DMS, behind
DATABRICKS_BUNDLE_RECORD_DEPLOYMENT_HISTORY.Design decisions:
CreateVersionstages one operation per planned resource, so the CLI only ever callsUpdateOperation— there is noCreateOperationcall. The service creates each staged operation asPENDINGatsequence_id = 0, which is the precondition the CLI uses for its first update of a resource. Needs databricks-eng/universe#2420238 (merged).Also adds an env var to toggle DMS, and records the API status and error code with a failure.
Testing Strategy
The whole acceptance suite runs a second time with recording on (
EnvMatrix.DATABRICKS_BUNDLE_RECORD_DEPLOYMENT_HISTORY), so every bundle test exercises DMS against the local test server and asserts the same golden files either way — thenostamphelper strips the deployment stamp for that. Focused coverage of the recorded calls themselves lives underacceptance/bundle/dms.Before private preview we will:
What is missing?