Conversation
…edition case The backend stores a pipeline's channel and edition as canonical upper-case enums (and case-normalizes catalog), so a config value in a different case reads back changed and never converges. On a continuous pipeline the resulting phantom EditPipeline on every deploy cancels the running update and starts a new one (SETTINGS_CHANGE). The terraform provider suppressed the same diffs, so this only regressed after migrating to the direct engine. Add a normalize_case rule (sibling of normalize_slash) that skips a change when local and remote differ only by case, and apply it to pipeline channel, edition and catalog. Fixes #6745 Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Approval status: pending
|
…s config Handle a field listed under both normalize_slash and normalize_case: strip trailing slashes before the case-insensitive compare so a value differing on both axes at once still converges. Add a unit test for shouldSkipNormalized. Drop pipeline_continuous.yml.tmpl: pipeline_normalize_case.yml.tmpl is already a continuous pipeline, so it covers the same no-drift/migrate paths. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
Integration test reportCommit: 267e404
Top 4 slowest tests (at least 2 minutes):
|
The mock now lower-cases catalog like the backend, and the normalize_case fixture uses a mixed-case catalog, so the catalog rule has real regression coverage (removing it now fails the test instead of silently passing). Simplify shouldSkipNormalized to look up the slash rule once and trim the compared values once before the case-insensitive fold. Co-authored-by: Isaac <no-reply@databricks.com>
migrate deploys via Terraform first, and for a continuous pipeline the TF provider blocks until the pipeline update reaches RUNNING. The local mock never leaves IDLE, so the deploy hangs until the per-test timeout. This is the first continuous invariant config, so it is the first to hit this. Exclude it from migrate (same pattern as volume_uppercase); no_drift on the direct engine covers it. Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The backend stores a pipeline's
channelandeditionas canonical upper-case enums (and case-normalizescatalog), so a config value in a different case reads back changed and never converges. On a continuous pipeline the resulting phantomEditPipelineon every deploy cancels the running update and starts a new one (SETTINGS_CHANGE). The terraform provider suppressed the same diffs (EqualFoldDiffSuppress/SetSuppressDiff), so this only regressed after migrating to the direct engine.Changes
Add a
normalize_caserule (sibling ofnormalize_slash) that skips a change when local and remote differ only by case, and apply it to pipelinechannel,editionandcatalog.Tests
New invariant config
pipeline_normalize_case.yml.tmpl: a continuous pipeline with lower-casechannel/edition; the mock now upper-cases them like the backend. Verified on aws that it drifts without the fix and converges with it, including afterbundle deployment migrate. Unit test for the slash/case composition inshouldSkipNormalized.This pull request and its description were written by Isaac.