Skip to content

Default source: GIT on git_source job tasks - #6751

Merged
pietern merged 3 commits into
mainfrom
pn/direct-git-source-task-source
Sep 21, 2026
Merged

pietern merged 3 commits into
mainfrom
pn/direct-git-source-task-source

Conversation

@pietern

@pietern pietern commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Changes

Restore ApplyDefaultTaskSource as a shared resource mutator that sets source: GIT on the tasks of a job with a git_source (for the task types that support the field: dbt_task, gen_ai_compute_task, notebook_task, spark_python_task, sql_task.file, including for_each_task, and only when the user did not set it). It runs after PythonMutator, so it reflects any git_source/tasks that Python code added or removed, is not exposed to Python, and is still observable in bundle validate/summary. Non-git jobs are left untouched, and an explicit git_source: null is treated as absent (matching TranslatePaths). tfdyn is unchanged and becomes an idempotent no-op.

Why

Deploying a job with a git_source and a spark_python_task (repo-relative python_file) failed on the direct engine with Invalid python file reference: <path> (400 INVALID_PARAMETER_VALUE), while Terraform deployed it fine. Path translation is intentionally skipped for git_source jobs, so task file paths stay relative to the Git repo. The backend treats a task as git-sourced only when its source is explicitly GIT; it does not infer that from the job-level git_source. Terraform stamped source: GIT in tfdyn, but the direct engine sent no source, so the backend read the relative path as a workspace path and rejected it. This was latent since the defaulting moved from a shared mutator into tfdyn (#3528) and only became user-visible once the direct engine became the default. See #3359 and #3528 for the history.

Tests

Adds a direct-only acceptance test that reproduces the failure, plus testserver validation that mirrors the backend's python_file check so the test fails without the fix. Adds unit tests covering git/non-git/null/for_each and explicit-source cases. The only existing golden that changes is task-source/out.git_job.direct.txt, which now includes source: GIT. Verified end-to-end against e2-dogfood (direct deploy of a git-sourced spark_python_task now succeeds).

This pull request and its description were written by Isaac.

pietern and others added 3 commits September 17, 2026 09:19
A job with a git_source keeps repo-relative task file paths (path
translation is skipped for git_source jobs). The backend treats a task
as git-sourced only when its source is explicitly GIT, so it rejects the
relative path with "Invalid python file reference" when source is unset.
The direct engine never set it, so `bundle deploy` failed; terraform
worked because tfdyn stamps it.

Add an ApplyDefaultTaskSource mutator that runs after PythonMutator and
sets source: GIT on the tasks of git_source jobs. This covers both
engines from one place and is observable in `bundle validate`. Add
testserver validation mirroring the backend and a direct-only acceptance
test that reproduces the failure.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@pietern
pietern requested review from andrewnester and denik and removed request for denik September 18, 2026 11:23

@denik denik left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need another test for PyDABs? I vaguely remember there was some issue between this mutator and PyDABs resources. cc @andrewnester

@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 94fbafa

Run: 35339379831

Env ✅​pass 🙈​skip Time
✅​ aws linux 276 17 5:30
✅​ aws windows 278 15 5:32
✅​ azure linux 275 17 5:01
✅​ azure windows 277 15 4:31
✅​ gcp linux 276 17 5:15
✅​ gcp windows 278 15 5:53
Top 3 slowest tests (at least 2 minutes):
duration env testname
5:29 aws windows TestAccept
3:55 gcp windows TestAccept
3:41 azure windows TestAccept

@pietern pietern changed the title Set source: GIT on git_source job tasks so the direct engine can deploy them Set source: GIT on git_source job tasks so the direct engine can deploy them Sep 21, 2026
@pietern pietern changed the title Set source: GIT on git_source job tasks so the direct engine can deploy them Default source: GIT on git_source job tasks Sep 21, 2026
@pietern
pietern added this pull request to the merge queue Sep 21, 2026
Merged via the queue into main with commit 85b9f0c Sep 21, 2026
37 checks passed
@pietern
pietern deleted the pn/direct-git-source-task-source branch September 21, 2026 08:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants