Migrate terraform state to direct before deploy

.nextchanges/bundles/migrate-before-deploy.md

@@ -0,0 +1 @@
+* Migrate existing Terraform deployment state to the direct engine before deploying (previously done after a Terraform deploy), so the deploy runs on the direct engine. ([#6749](https://github.com/databricks/cli/pull/6749))

acceptance/bundle/deploy/snapshot-comparison/output.txt

@@ -54,6 +54,8 @@ Destroy: 2 deleted
 
 === Cleanup bundle 2
 >>> [CLI] bundle destroy --auto-approve
+Notice: automatically migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765).
+Migrated 2 resources to direct deployment engine.
 The following resources will be deleted:
   delete resources.jobs.test_job
   delete resources.pipelines.test_pipeline

acceptance/bundle/migrate/auto-migrate-artifacts/databricks.yml

@@ -0,0 +1,15 @@
+bundle:
+  name: test-bundle
+
+resources:
+  jobs:
+    test_job:
+      name: "Test Auto-Migrate Artifacts Job"
+      tasks:
+        - task_key: TestTask
+          existing_cluster_id: "0717-132531-5opeqon1"
+          python_wheel_task:
+            package_name: "my_test_code"
+            entry_point: "run"
+          libraries:
+            - whl: ./dist/*.whl

acceptance/bundle/migrate/auto-migrate-artifacts/output.txt

@@ -0,0 +1,43 @@
+
+=== Initial deploy on terraform (expands the whl glob and uploads the wheel)
+>>> DATABRICKS_BUNDLE_ENGINE=terraform [CLI] bundle deploy
+Warn: unable to get spark version for cluster 0717-132531-5opeqon1, err: Not Found
+Uploading dist/my_test_code-0.0.1-py3-none-any.whl...
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Created jobs.test_job
+Files: 4 uploaded, 0 deleted
+Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
+
+=== Redeploy with direct default: auto-migration runs after the build, so library refs are resolved
+>>> [CLI] bundle deploy
+Warn: unable to get spark version for cluster 0717-132531-5opeqon1, err: Not Found
+Notice: automatically migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765).
+Migrated 1 resource to direct deployment engine.
+Uploading dist/my_test_code-0.0.1-py3-none-any.whl...
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Files: 0 uploaded, 0 deleted
+Resources: 0 created, 0 changed, 0 deleted, 1 unchanged
+
+=== Local state is now direct
+
+>>> find .databricks/bundle -name resources.json -type f
+.databricks/bundle/default/resources.json
+
+>>> find .databricks/bundle -name terraform.tfstate* -type f
+.databricks/bundle/default/terraform/terraform.tfstate.backup
+
+=== Migrated state's library path (should be the resolved remote path, not the local glob)
+>>> print_state.py
+[
+  {
+    "whl": "/Workspace/Users/[USERNAME]/.bundle/test-bundle/default/artifacts/.internal/my_test_code-0.0.1-py3-none-any.whl"
+  }
+]
+
+=== Subsequent deploy is a no-op on direct
+>>> [CLI] bundle deploy
+Warn: unable to get spark version for cluster 0717-132531-5opeqon1, err: Not Found
+Uploading dist/my_test_code-0.0.1-py3-none-any.whl...
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Files: 0 uploaded, 0 deleted
+Resources: 0 created, 0 changed, 0 deleted, 1 unchanged

acceptance/bundle/migrate/auto-migrate-artifacts/script

@@ -0,0 +1,17 @@
+export DATABRICKS_BUNDLE_ENGINE=
+
+title "Initial deploy on terraform (expands the whl glob and uploads the wheel)"
+trace DATABRICKS_BUNDLE_ENGINE=terraform $CLI bundle deploy
+
+title "Redeploy with direct default: auto-migration runs after the build, so library refs are resolved"
+trace $CLI bundle deploy
+
+title "Local state is now direct\n"
+trace find .databricks/bundle -name "resources.json" -type f
+trace find .databricks/bundle -name "terraform.tfstate*" -type f
+
+title "Migrated state's library path (should be the resolved remote path, not the local glob)"
+trace print_state.py | jq '.. | .libraries? | select(.)'
+
+title "Subsequent deploy is a no-op on direct"
+trace $CLI bundle deploy

acceptance/bundle/migrate/auto-migrate-artifacts/test.toml

@@ -0,0 +1 @@
+RecordRequests = false

acceptance/bundle/migrate/auto-migrate-clean/output.txt

@@ -15,12 +15,10 @@ direct_drymigrate_warnings false
   engine: direct
 
 >>> [CLI] bundle deploy
-Warn: Direct engine selected via bundle.engine setting at [TEST_TMP_DIR]/databricks.yml:3:11 but the existing state uses "terraform". Deploying on "terraform"; will attempt to migrate the state to the direct engine after this deploy.
+Migrated 1 resource to direct deployment engine.
 Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
 Files: 1 uploaded, 0 deleted
 Resources: 0 created, 0 changed, 0 deleted, 1 unchanged
-Migrating state to direct deployment engine (selected via bundle.engine setting at [TEST_TMP_DIR]/databricks.yml:3:11)...
-Migrated 1 resource to direct deployment engine.
 
 >>> print_migration_telemetry
 direct_migrated_via_config true

acceptance/bundle/migrate/auto-migrate-default/output.txt

@@ -12,17 +12,11 @@ direct_drymigrate_warnings false
 
 === Redeploy with nothing set: direct is the default, so auto-migration triggers
 >>> [CLI] bundle deploy
-Notice: the direct deployment engine is the default as of CLI v1.14.0.
-
-This bundle will be automatically migrated to use the direct deployment engine after this deployment.
-
-Learn more: https://docs.databricks.com/dev-tools/bundles/direct
-
+Notice: automatically migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765).
+Migrated 1 resource to direct deployment engine.
 Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
 Files: 0 uploaded, 0 deleted
 Resources: 0 created, 0 changed, 0 deleted, 1 unchanged
-Migrating state to direct deployment engine (selected via default)...
-Migrated 1 resource to direct deployment engine.
 
 >>> print_migration_telemetry
 direct_migrated_via_default true

acceptance/bundle/migrate/auto-migrate-delete-all/databricks.yml

@@ -0,0 +1,7 @@
+bundle:
+  name: test-bundle
+
+resources:
+  jobs:
+    job_a:
+      name: "Job A"

acceptance/bundle/migrate/auto-migrate-delete-all/output.txt

@@ -0,0 +1,20 @@
+
+=== Initial deploy on terraform
+>>> DATABRICKS_BUNDLE_ENGINE=terraform [CLI] bundle deploy
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Created jobs.job_a
+Files: 3 uploaded, 0 deleted
+Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
+
+=== Remove all resources, then deploy on direct: migrate and delete
+>>> [CLI] bundle deploy
+Notice: automatically migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765).
+Migrated 1 resource to direct deployment engine.
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Deleted jobs.job_a
+Files: 1 uploaded, 0 deleted
+Resources: 0 created, 0 changed, 1 deleted, 0 unchanged
+
+=== Follow-up plan is a clean no-op
+>>> [CLI] bundle plan
+Plan: 0 to add, 0 to change, 0 to delete, 0 unchanged

acceptance/bundle/migrate/auto-migrate-delete-all/script

@@ -0,0 +1,15 @@
+export DATABRICKS_BUNDLE_ENGINE=
+
+title "Initial deploy on terraform"
+trace DATABRICKS_BUNDLE_ENGINE=terraform $CLI bundle deploy
+
+# Removing every resource leaves the migration with no config-declared resources, so
+# BuildStateFromTF records no WAL entries. The migration must still persist the seeded
+# terraform-state resources so the direct plan can delete them (otherwise Finalize would
+# write no state file and the migration would fail with a missing resources.json).
+title "Remove all resources, then deploy on direct: migrate and delete"
+echo '{"bundle": {"name": "test-bundle"}}' > databricks.yml
+trace $CLI bundle deploy
+
+title "Follow-up plan is a clean no-op"
+trace $CLI bundle plan | contains.py "0 to delete, 0 unchanged"

acceptance/bundle/migrate/auto-migrate-delete-all/test.toml

@@ -0,0 +1 @@
+RecordRequests = false

acceptance/bundle/migrate/auto-migrate-delete/databricks.yml

@@ -0,0 +1,9 @@
+bundle:
+  name: test-bundle
+
+resources:
+  jobs:
+    job_keep:
+      name: "Keep Job"
+    job_drop: # TO_REMOVE
+      name: "Drop Job" # TO_REMOVE

acceptance/bundle/migrate/auto-migrate-delete/output.txt

@@ -0,0 +1,21 @@
+
+=== Initial deploy on terraform with two jobs
+>>> DATABRICKS_BUNDLE_ENGINE=terraform [CLI] bundle deploy
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Created jobs.job_drop
+Created jobs.job_keep
+Files: 3 uploaded, 0 deleted
+Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
+
+=== Drop one job from config, then deploy on direct: migrate two, delete the dropped one
+>>> [CLI] bundle deploy
+Notice: automatically migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765).
+Migrated 2 resources to direct deployment engine.
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Deleted jobs.job_drop
+Files: 1 uploaded, 0 deleted
+Resources: 0 created, 0 changed, 1 deleted, 1 unchanged
+
+=== Follow-up plan is a clean no-op
+>>> [CLI] bundle plan
+Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged

acceptance/bundle/migrate/auto-migrate-delete/script

@@ -0,0 +1,13 @@
+export DATABRICKS_BUNDLE_ENGINE=
+
+title "Initial deploy on terraform with two jobs"
+trace DATABRICKS_BUNDLE_ENGINE=terraform $CLI bundle deploy
+
+# The migration seeds every terraform-state resource into the direct state, so the
+# first direct plan can delete the ones the config no longer declares.
+title "Drop one job from config, then deploy on direct: migrate two, delete the dropped one"
+grep -v 'TO_REMOVE' databricks.yml > updated.yml && mv updated.yml databricks.yml
+trace $CLI bundle deploy
+
+title "Follow-up plan is a clean no-op"
+trace $CLI bundle plan | contains.py "1 unchanged"

acceptance/bundle/migrate/auto-migrate-delete/test.toml

@@ -0,0 +1 @@
+RecordRequests = false

acceptance/bundle/migrate/auto-migrate-destroy/databricks.yml

@@ -0,0 +1,7 @@
+bundle:
+  name: test-bundle
+
+resources:
+  jobs:
+    job_a:
+      name: "Job A"

acceptance/bundle/migrate/auto-migrate-destroy/output.txt

@@ -0,0 +1,24 @@
+
+=== Initial deploy on terraform
+>>> DATABRICKS_BUNDLE_ENGINE=terraform [CLI] bundle deploy
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Created jobs.job_a
+Files: 3 uploaded, 0 deleted
+Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
+
+=== Destroy on direct: migrate (commit) then destroy
+>>> [CLI] bundle destroy --auto-approve
+Notice: automatically migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765).
+Migrated 1 resource to direct deployment engine.
+The following resources will be deleted:
+  delete resources.jobs.job_a
+
+All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default
+
+Destroy: 1 deleted
+
+=== resources.json removed by destroy; terraform state backed up
+>>> find .databricks/bundle -name resources.json -type f
+
+>>> find .databricks/bundle -name terraform.tfstate* -type f
+.databricks/bundle/default/terraform/terraform.tfstate.backup

acceptance/bundle/migrate/auto-migrate-destroy/script

@@ -0,0 +1,14 @@
+export DATABRICKS_BUNDLE_ENGINE=
+
+title "Initial deploy on terraform"
+trace DATABRICKS_BUNDLE_ENGINE=terraform $CLI bundle deploy
+
+# Destroy behaves like deploy: it migrates the terraform state to direct in memory,
+# plan-checks it, commits the migration (resources.json pushed, terraform.tfstate backed
+# up), then destroys on the direct engine.
+title "Destroy on direct: migrate (commit) then destroy"
+trace $CLI bundle destroy --auto-approve
+
+title "resources.json removed by destroy; terraform state backed up"
+trace find .databricks/bundle -name "resources.json" -type f
+trace find .databricks/bundle -name "terraform.tfstate*" -type f

acceptance/bundle/migrate/auto-migrate-destroy/test.toml

@@ -0,0 +1 @@
+RecordRequests = false

acceptance/bundle/migrate/auto-migrate-direct-only-envvar/output.txt

@@ -9,45 +9,27 @@ Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
 === Add a direct-only resource to the config
 >>> update_file.py databricks.yml #instance_pools instance_pools
 
-=== Opt in via env var only: deploy migrates the state, ignoring the direct-only resource
+=== Opt in via env var only: deploy migrates the state and creates the direct-only resource
 >>> DATABRICKS_BUNDLE_ENGINE=direct [CLI] bundle deploy
-Warn: Direct engine selected via DATABRICKS_BUNDLE_ENGINE environment variable but the existing state uses "terraform". Deploying on "terraform"; will attempt to migrate the state to the direct engine after this deploy.
+Migrated 1 resource to direct deployment engine.
 Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Created instance_pools.pool
 Files: 1 uploaded, 0 deleted
-Resources: 0 created, 0 changed, 0 deleted, 2 unchanged
-Migrating state to direct deployment engine (selected via DATABRICKS_BUNDLE_ENGINE environment variable)...
-Migrated 1 resource to direct deployment engine.
+Resources: 1 created, 0 changed, 0 deleted, 1 unchanged
 
 >>> print_migration_telemetry
 direct_migrated_via_env true
 
-=== The direct-only resource is created on the next deploy
+=== Redeploy is a no-op: state is already on the direct engine
 >>> [CLI] bundle plan
-create instance_pools.pool
-
-Plan: 1 to add, 0 to change, 0 to delete, 1 unchanged
+Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged
 
 >>> [CLI] bundle deploy
 Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
-Created instance_pools.pool
 Files: 0 uploaded, 0 deleted
-Resources: 1 created, 0 changed, 0 deleted, 1 unchanged
+Resources: 0 created, 0 changed, 0 deleted, 2 unchanged
 
 >>> print_requests.py //instance-pools/create
-{
-  "headers": {
-    "User-Agent": [
-      "cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_deploy cmd-exec-id/[UUID] interactive/none engine/direct auth/pat"
-    ]
-  },
-  "method": "POST",
-  "path": "/api/2.0/instance-pools/create",
-  "body": {
-    "idle_instance_autotermination_minutes": 60,
-    "instance_pool_name": "My Pool",
-    "node_type_id": "Standard_DS3_v2"
-  }
-}
 
 === No further actions planned
 >>> [CLI] bundle plan

acceptance/bundle/migrate/auto-migrate-direct-only-envvar/script

@@ -7,12 +7,12 @@ rm -f "$OUT_REQUESTS"
 title "Add a direct-only resource to the config"
 trace update_file.py databricks.yml "#instance_pools" "instance_pools"
 
-title "Opt in via env var only: deploy migrates the state, ignoring the direct-only resource"
+title "Opt in via env var only: deploy migrates the state and creates the direct-only resource"
 trace DATABRICKS_BUNDLE_ENGINE=direct $CLI bundle deploy
 trace print_migration_telemetry
 rm -f "$OUT_REQUESTS"
 
-title "The direct-only resource is created on the next deploy"
+title "Redeploy is a no-op: state is already on the direct engine"
 trace $CLI bundle plan
 trace $CLI bundle deploy
 trace print_requests.py //instance-pools/create