air: add convert-to-dabs (run YAML -> Databricks Asset Bundle) - #6111
Open
vinchenzo-db wants to merge 9 commits into
Open
air: add convert-to-dabs (run YAML -> Databricks Asset Bundle)#6111vinchenzo-db wants to merge 9 commits into
vinchenzo-db wants to merge 9 commits into
Conversation
Collaborator
Integration test reportCommit: 2cf4f20
8 interesting tests: 4 RECOVERED, 4 SKIP
Top 6 slowest tests (at least 2 minutes):
|
vinchenzo-db
force-pushed
the
air-convert-to-dabs-main
branch
10 times, most recently
from
August 3, 2026 22:03
e74034f to
29248d4
Compare
Adds `air convert-to-dabs`, which translates an AIR CLI run YAML into a
deployable Databricks Asset Bundle so a workload authored for `air run` can be
managed and deployed with the standard DABs workflow (validate/deploy/run).
The emitted bundle is schema-valid: the ai_runtime_task maps to the SDK
jobs.AiRuntimeTask (experiment + deployments[].{command_path,compute} +
code_source_path), with framework fields (retries, timeout, budget policy) on
the surrounding task and the runtime environment in environments[].
Snapshotting is owned by the deploy-time aicode mutator, not by convert:
code_source_path points at a local *directory* staged inside the bundle, and
`bundle deploy` (aicode.PackageAndUpload) packages it into a content-addressed
tarball and uploads it. convert only lays down the source bytes — copying the
working tree (honoring .gitignore) or materializing a pinned git commit into the
directory. requirements.yaml is likewise not emitted: aicode.SynthesizeRequirements
regenerates it from the environments[] spec, so convert folds the whole dependency
set (inline or requirements-file) into that spec instead.
env_variables / secrets / parameters have no native ai_runtime_task field, so they
ride as env_vars.json / secret_env_vars.json / hyperparameters.yaml sidecars (same
as `air run`), and a "Notes:" section tells a migrating user what was transformed
or staged out-of-band.
This is the top of a 2-PR stack: it builds on the aicode deploy-time packaging
mutator so the two compose end-to-end.
Co-authored-by: Isaac
The Windows CI job failed the convert-to-dabs acceptance test with "tar (child): Cannot connect to C: resolve failed": the system tar reads the `C:` in an absolute archive path as a remote host:path. Two fixes: - createPlainTarball now passes the archive as a bare basename with cmd.Dir set to the output directory (and an absolute parent), so no `C:\...` path reaches tar's -f argument. Mirrors how git archive is invoked; safe on GNU tar and bsdtar. (This helper is shared with the `air run` snapshot path.) - extractTarball is rewritten in pure Go (archive/tar + compress/gzip) instead of shelling out to `tar -xzf`, eliminating the same drive-letter hazard on the extract side and dropping the external-tar dependency for extraction. It rejects entries that would escape the destination (path traversal, absolute/escaping symlinks) and bounds each file copy to its header size. Adds unit tests for extractTarball (happy path incl. nested dirs + in-tree symlink; traversal + escaping-symlink rejection). Co-authored-by: Isaac
convert-to-dabs is now a purely local, syntactic translation. It no longer copies, snapshots, or git-archives the code source — the deploy-time aicode mutator (from the base PR) packages the source in place at `bundle deploy`. - code_source_path is emitted as the source directory relative to the bundle (bundle root defaults to the input YAML's directory, which contains it), rather than a copied ./code_source dir. - Removed the materialize/extract-tarball machinery and the git-pinned-commit path. code_source.snapshot.git is now rejected (deploy packages the working tree); a code_source outside the bundle directory is rejected with guidance. - writeBundle now only writes databricks.yml + command.sh + the env/secret/param sidecars. Dependencies stay folded into the environments[] spec. Tests updated: assert code_source_path points at ./src with no copy, reject git-pin and out-of-bundle sources; the acceptance test converts in place. Co-authored-by: Isaac
git and remote_volume stay unsupported, but the errors now explain why and what to do: for git, check out the revision before converting (deploy packages the working tree as-is); for remote_volume, set workspace.artifact_path (the bundle artifact location is bundle-wide, not per-code-source). Co-authored-by: Isaac
Re-running a conversion previously failed with no way to overwrite, and the suggested remedy (--output-dir) is a dead end in place: code_source must live inside the bundle dir. Add --force, and point the error at it. The next-steps message said "databricks bundle ...". A CLI without ai_runtime_task support only warns on the unknown field and then deploys a job with no AI task, so print the invoked binary instead and call out the risk. Co-authored-by: Isaac
The bundle is written next to the input YAML, so `cd .` was a no-op step; it is now printed only when the bundle lands elsewhere. Drop the notes block and the stale-CLI warning, and point at `bundle summary` for what was deployed. Co-authored-by: Isaac
vinchenzo-db
force-pushed
the
air-convert-to-dabs-main
branch
from
August 4, 2026 22:44
7a2b086 to
c7698c9
Compare
include_paths narrows the archive to a subset of root_path, which a bundle can't
express per code source — deploy packages the whole directory. Converting was
silently dropping it and uploading files the user meant to exclude, so reject it
and point at sync.exclude / .gitignore.
Emit a job resource key that YAML would type as a non-string scalar ("12345" ->
!!int) as a quoted key instead of prefixing it with "job_". yamlsaver already had
this logic for scalar *values* (isScalarValueInString); apply it to map keys too,
so the resource key keeps the experiment name.
Co-authored-by: Isaac
ben-hansen-db
approved these changes
Aug 5, 2026
ben-hansen-db
left a comment
ben-hansen-db
left a comment
Contributor
There was a problem hiding this comment.
Thanks for the updates!
vinchenzo-db
enabled auto-merge
August 5, 2026 16:54
Keep the change inside experimental/air/ instead of teaching the shared
yamlsaver to quote map keys. quoteJobKey rewrites the emitted job resource key
when the experiment name would otherwise load as a non-string scalar ("12345" ->
!!int), which the bundle loader rejects.
NewSaverWithStyle can't do this: its style map applies to a key's whole subtree,
so quoting the job key also quotes every nested key and turns
accelerator_count: 1 into a string.
Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
air convert-to-dabs, which translates an AIR CLI run YAML into a deployable Databricks Asset Bundle so a workload authored forair runcan be managed and deployed with the standard DABs workflow (validate/deploy/run).The emitted bundle is schema-valid: the ai_runtime_task maps to the SDK jobs.AiRuntimeTask (experiment + deployments[].{command_path,compute} + code_source_path), with framework fields (retries, timeout, budget policy) on the surrounding task and the runtime environment in environments[].
Snapshotting is owned by the deploy-time aicode mutator, not by convert: code_source_path points at a local directory staged inside the bundle, and
bundle deploy(aicode.PackageAndUpload) packages it into a content-addressed tarball and uploads it. convert only lays down the source bytes — copying the working tree (honoring .gitignore) or materializing a pinned git commit into the directory. requirements.yaml is likewise not emitted: aicode.SynthesizeRequirements regenerates it from the environments[] spec, so convert folds the whole dependency set (inline or requirements-file) into that spec instead.env_variables / secrets / parameters have no native ai_runtime_task field, so they ride as env_vars.json / secret_env_vars.json / hyperparameters.yaml sidecars (same as
air run), and a "Notes:" section tells a migrating user what was transformed or staged out-of-band.This is the top of a 2-PR stack: it builds on the aicode deploy-time packaging mutator so the two compose end-to-end.
Co-authored-by: Isaac
Changes
Why
Tests