Skip to content

fix versioning on main #600

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
evertlammerts merged 1 commit into from
Aug 20, 2026
Merged

fix versioning on main #600

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 27 additions & 9 deletions .github/actions/build-wheel/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,11 @@ inputs:
required: false
default: ''
set-version:
description: 'Force version, passed to the build as OVERRIDE_GIT_DESCRIBE'
description: 'Force the package version, passed to the build as OVERRIDE_GIT_DESCRIBE'
required: false
default: ''
set-duckdb-version:
description: 'Force the DuckDB version, passed to the build as OVERRIDE_DUCKDB_GIT_DESCRIBE'
required: false
default: ''
ccache-save:
Expand All @@ -46,7 +50,11 @@ runs:

- name: Set build environment
shell: bash
env:
SET_VERSION: ${{ inputs.set-version }}
SET_DUCKDB_VERSION: ${{ inputs.set-duckdb-version }}
run: |
set -eo pipefail
echo "CCACHE_DIR=${{ github.workspace }}/.ccache" >> $GITHUB_ENV
# The Visual Studio generator (the CMake default on windows) ignores
# CMAKE_<LANG>_COMPILER_LAUNCHER, so ccache never sees the build.
Expand All @@ -55,19 +63,29 @@ runs:
if [[ "${{ inputs.cibw-system }}" == "win" ]]; then
echo "CMAKE_GENERATOR=Ninja" >> $GITHUB_ENV
fi
# Version overrides the build has to see, collected as NAME=value words.
# OVERRIDE_GIT_DESCRIBE forces the package version, OVERRIDE_DUCKDB_GIT_DESCRIBE
# forces the version DuckDB is built with. They are independent.
overrides=""
if [[ -n "${SET_VERSION}" ]]; then
overrides="${overrides} OVERRIDE_GIT_DESCRIBE=${SET_VERSION}"
fi
if [[ -n "${SET_DUCKDB_VERSION}" ]]; then
overrides="${overrides} OVERRIDE_DUCKDB_GIT_DESCRIBE=${SET_DUCKDB_VERSION}"
fi
# CIBW_ENVIRONMENT is set ONLY for manylinux, where env must cross into
# the container. The env var REPLACES any [tool.cibuildwheel.<os>]
# environment table from pyproject; setting it on macos silently dropped
# MACOSX_DEPLOYMENT_TARGET and shipped wheels tagged macosx_10_9. On
# macos and windows the build runs on the host, so plain job env works.
if [[ -n "${{ inputs.set-version }}" ]]; then
if [[ "${{ inputs.cibw-system }}" == "manylinux" ]]; then
echo "CIBW_ENVIRONMENT=CCACHE_DIR=/host${{ github.workspace }}/.ccache OVERRIDE_GIT_DESCRIBE=${{ inputs.set-version }}" >> $GITHUB_ENV
else
echo "OVERRIDE_GIT_DESCRIBE=${{ inputs.set-version }}" >> $GITHUB_ENV
fi
elif [[ "${{ inputs.cibw-system }}" == "manylinux" ]]; then
echo "CIBW_ENVIRONMENT=CCACHE_DIR=/host${{ github.workspace }}/.ccache" >> $GITHUB_ENV
# Every override has to ride along on that single CIBW_ENVIRONMENT line,
# or it never reaches the build inside the container.
if [[ "${{ inputs.cibw-system }}" == "manylinux" ]]; then
echo "CIBW_ENVIRONMENT=CCACHE_DIR=/host${{ github.workspace }}/.ccache${overrides}" >> $GITHUB_ENV
else
for override in ${overrides}; do
echo "${override}" >> $GITHUB_ENV
done
fi

# Ninja needs cl.exe on PATH. The Visual Studio generator activated the
Expand Down
14 changes: 12 additions & 2 deletions .github/workflows/packaging.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,11 @@ on:
required: false
set-version:
type: string
description: Force version (vX.Y.Z, optionally -postN or a PEP440 pre-release suffix like -a1/-alpha1/-b2/-rc3)
description: Force the package version (vX.Y.Z, optionally -postN or a PEP440 pre-release suffix like -a1/-alpha1/-b2/-rc3)
required: false
set-duckdb-version:
type: string
description: Force the version DuckDB is built with. Handed to DuckDB's build verbatim, which validates it.
required: false
workflow_call:
inputs:
Expand All @@ -48,7 +52,11 @@ on:
description: Override the DuckDB submodule commit or ref to build against
required: false
set-version:
description: Force version (vX.Y.Z, optionally -postN or a PEP440 pre-release suffix like -a1/-alpha1/-b2/-rc3)
description: Force the package version (vX.Y.Z, optionally -postN or a PEP440 pre-release suffix like -a1/-alpha1/-b2/-rc3)
required: false
type: string
set-duckdb-version:
description: Force the version DuckDB is built with. Handed to DuckDB's build verbatim, which validates it.
required: false
type: string

Expand All @@ -69,6 +77,7 @@ jobs:
duckdb-python-sha: ${{ inputs.duckdb-python-sha != '' && inputs.duckdb-python-sha || github.sha }}
duckdb-sha: ${{ inputs.duckdb-sha }}
set-version: ${{ inputs.set-version }}
set-duckdb-version: ${{ inputs.set-duckdb-version }}

build_wheels:
name: Build and test releases
Expand All @@ -79,3 +88,4 @@ jobs:
duckdb-python-sha: ${{ inputs.duckdb-python-sha != '' && inputs.duckdb-python-sha || github.sha }}
duckdb-sha: ${{ inputs.duckdb-sha }}
set-version: ${{ inputs.set-version }}
set-duckdb-version: ${{ inputs.set-duckdb-version }}
70 changes: 64 additions & 6 deletions .github/workflows/packaging_sdist.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,11 @@ on:
description: Override the DuckDB submodule commit or ref to build against
required: false
set-version:
description: Force version (vX.Y.Z, optionally -postN or a PEP440 pre-release suffix like -a1/-alpha1/-b2/-rc3)
description: Force the package version (vX.Y.Z, optionally -postN or a PEP440 pre-release suffix like -a1/-alpha1/-b2/-rc3)
required: false
type: string
set-duckdb-version:
description: Force the version DuckDB is built with. Handed to DuckDB's build verbatim, which validates it.
required: false
type: string
outputs:
Expand Down Expand Up @@ -51,9 +55,19 @@ jobs:
git fetch origin
git checkout ${{ inputs.duckdb-sha }}

- name: Set OVERRIDE_GIT_DESCRIBE
if: ${{ inputs.set-version != '' }}
run: echo "OVERRIDE_GIT_DESCRIBE=${{ inputs.set-version }}" >> $GITHUB_ENV
# Inputs go through env, never interpolated into the script body.
- name: Set the version overrides
env:
SET_VERSION: ${{ inputs.set-version }}
SET_DUCKDB_VERSION: ${{ inputs.set-duckdb-version }}
run: |
set -euo pipefail
if [[ -n "${SET_VERSION}" ]]; then
echo "OVERRIDE_GIT_DESCRIBE=${SET_VERSION}" >> $GITHUB_ENV
fi
if [[ -n "${SET_DUCKDB_VERSION}" ]]; then
echo "OVERRIDE_DUCKDB_GIT_DESCRIBE=${SET_DUCKDB_VERSION}" >> $GITHUB_ENV
fi

- name: Install Astral UV
uses: astral-sh/setup-uv@v7
Expand Down Expand Up @@ -82,11 +96,55 @@ jobs:
tests_dir="${tests_root}${{ inputs.testsuite == 'fast' && '/fast' || '/' }}"
uv run --verbose pytest -c ${{ github.workspace }}/pyproject.toml $tests_dir

# Reads both versions back out of the installed sdist and fails if they are
# not what was asked for. This is the only place that proves a forced version
# actually reached the build. It has silently not reached it before: the env
# var never crossed into the manylinux container, and this step itself read a
# duckdb.duckdb_version attribute that does not exist and published an empty
# string for months.
- id: versioning
env:
SET_VERSION: ${{ inputs.set-version }}
SET_DUCKDB_VERSION: ${{ inputs.set-duckdb-version }}
run: |
set -euo pipefail
cd ${{ runner.temp }}
echo "pkg_version=$( .venv/bin/python -c 'import duckdb; print(duckdb.__version__)' )" >> $GITHUB_OUTPUT
echo "duckdb_version=$( .venv/bin/python -c 'import duckdb; print(duckdb.duckdb_version)' )" >> $GITHUB_OUTPUT
pkg_version=$( .venv/bin/python -c 'import duckdb; print(duckdb.__version__)' )
duckdb_version=$( .venv/bin/python -c 'import duckdb; print(duckdb.__duckdb_version__)' )
echo "package version: '${pkg_version}'"
echo "duckdb version: '${duckdb_version}'"

if [[ -z "${pkg_version}" || -z "${duckdb_version}" ]]; then
echo "::error::Version introspection produced an empty string"
exit 1
fi

if [[ -n "${SET_DUCKDB_VERSION}" ]]; then
if [[ "${SET_DUCKDB_VERSION}" == *-g* ]]; then
# A full git describe. DuckDB rewrites those to vX.Y.Z-devN, so there
# is nothing to compare against.
echo "::notice::set-duckdb-version is a git describe string, skipping the exact match"
elif [[ "v${duckdb_version}" != "${SET_DUCKDB_VERSION}" ]]; then
echo "::error::DuckDB was asked to build as ${SET_DUCKDB_VERSION} but reports v${duckdb_version}. The override did not reach the build."
exit 1
fi
fi

if [[ -n "${SET_VERSION}" ]]; then
if [[ "${SET_VERSION}" == *-g* ]]; then
echo "::notice::set-version is a git describe string, skipping the exact match"
else
expected=$(PYTHONPATH=${{ github.workspace }} python3 -c \
'import os; from duckdb_packaging._versioning import git_tag_to_pep440; print(git_tag_to_pep440(os.environ["SET_VERSION"]))')
if [[ "${pkg_version}" != "${expected}" ]]; then
echo "::error::The package was asked to build as ${SET_VERSION} (${expected}) but reports ${pkg_version}. The override did not reach the build."
exit 1
fi
fi
fi

echo "pkg_version=${pkg_version}" >> $GITHUB_OUTPUT
echo "duckdb_version=${duckdb_version}" >> $GITHUB_OUTPUT

- uses: actions/upload-artifact@v4
with:
Expand Down
9 changes: 8 additions & 1 deletion .github/workflows/packaging_wheels.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,11 @@ on:
description: Override the DuckDB submodule commit or ref to build against
required: false
set-version:
description: Force version (vX.Y.Z, optionally -postN or a PEP440 pre-release suffix like -a1/-alpha1/-b2/-rc3)
description: Force the package version (vX.Y.Z, optionally -postN or a PEP440 pre-release suffix like -a1/-alpha1/-b2/-rc3)
required: false
type: string
set-duckdb-version:
description: Force the version DuckDB is built with. Handed to DuckDB's build verbatim, which validates it.
required: false
type: string

Expand Down Expand Up @@ -82,6 +86,7 @@ jobs:
testsuite: ${{ inputs.testsuite }}
duckdb-sha: ${{ inputs.duckdb-sha }}
set-version: ${{ inputs.set-version }}
set-duckdb-version: ${{ inputs.set-duckdb-version }}
ccache-save: 'true'

sanity_wheels:
Expand Down Expand Up @@ -112,6 +117,7 @@ jobs:
testsuite: ${{ inputs.testsuite }}
duckdb-sha: ${{ inputs.duckdb-sha }}
set-version: ${{ inputs.set-version }}
set-duckdb-version: ${{ inputs.set-duckdb-version }}
ccache-save: 'false'

build_wheels:
Expand Down Expand Up @@ -148,4 +154,5 @@ jobs:
testsuite: ${{ inputs.testsuite }}
duckdb-sha: ${{ inputs.duckdb-sha }}
set-version: ${{ inputs.set-version }}
set-duckdb-version: ${{ inputs.set-duckdb-version }}
ccache-save: 'false'
92 changes: 76 additions & 16 deletions .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
# Release is called by duckdb's InvokeCI -> NotifyExternalRepositories job
# Called by DuckDB core when its release artifacts are ready. On main that is
# Main.yml -> duckdb-workflow-trigger (core_ready), which sends duckdb-version.
# On release branches it is still InvokeCI -> NotifyExternalRepositories, which
# sends only duckdb-sha. We also dispatch it by hand for stable releases.
name: Release
on:
workflow_dispatch:
Expand All @@ -13,11 +16,18 @@ on:
required: true
duckdb-version:
type: string
description: The DuckDB version associated with duckdb-sha
description: >-
Build DuckDB with this exact version identifier (vX.Y.Z, or a prerelease like
vX.Y.Z-alphaN or vX.Y.Z-rcN). Handed to DuckDB's build verbatim, which is the
only thing that validates it. Empty means derive it from the submodule.
required: false
stable-version:
duckdb-python-version:
type: string
description: Release a tagged version (vX.Y.Z, optionally -postN or a PEP440 pre-release suffix like -a1/-alpha1/-b2/-rc3)
description: >-
Release this version of the Python package (vX.Y.Z, optionally -postN or a
PEP440 pre-release suffix like -a1/-alpha1/-b2/-rc3). Setting it makes this a
stable release: the submodule pin must equal duckdb-sha and the package goes
to the prod index. Requires duckdb-version. Empty means a nightly.
required: false
pypi-index:
type: choice
Expand All @@ -41,28 +51,76 @@ defaults:
shell: bash

jobs:
# Two independent version identifiers reach this workflow. duckdb-version sets
# the version DuckDB is built with, and nothing else. duckdb-python-version sets
# the version of the package we publish, and setting it is what makes this run a
# stable release instead of a nightly.
#
# A stable release must be built from a commit whose submodule pin equals the
# duckdb-sha input. Otherwise the build is dirty and the released artifacts
# correspond to no commit in this repository.
#
# The gate is applied per step, not on the job, on purpose: a skipped job
# The pin gate is applied per step, not on the job, on purpose: a skipped job
# propagates the skip down the needs chain, and the downstream jobs
# (workflow_state, submodule_pr) have no !failure() && !cancelled() guard, so
# a job-level skip here silently skips the whole nightly (no wheels, no
# publish, no bump PR) while the run still reports success. With the gate on
# the steps, the job always completes success and the steps no-op on
# nightlies (empty stable-version).
verify_stable_pin:
name: Verify submodule pin for stable release
# nightlies (empty duckdb-python-version).
verify_release_inputs:
name: Verify release inputs
runs-on: ubuntu-latest
steps:
- name: Validate the version inputs
env:
DUCKDB_VERSION: ${{ inputs.duckdb-version }}
DUCKDB_PYTHON_VERSION: ${{ inputs.duckdb-python-version }}
run: |
set -euo pipefail
version_re='^v([0-9]+\.[0-9]+\.[0-9]+)'

duckdb_base=""
if [[ -n "${DUCKDB_VERSION}" ]]; then
if [[ ! "${DUCKDB_VERSION}" =~ ${version_re} ]]; then
echo "::error::duckdb-version (${DUCKDB_VERSION}) must start with a vX.Y.Z version"
exit 1
fi
duckdb_base="${BASH_REMATCH[1]}"
fi

if [[ -z "${DUCKDB_PYTHON_VERSION}" ]]; then
echo "::notice::Nightly release${DUCKDB_VERSION:+, building DuckDB as ${DUCKDB_VERSION}}"
exit 0
fi

if [[ -z "${DUCKDB_VERSION}" ]]; then
echo "::error::duckdb-python-version is set but duckdb-version is not. A stable release states the DuckDB version it ships, we never infer that."
exit 1
fi

if [[ ! "${DUCKDB_PYTHON_VERSION}" =~ ${version_re} ]]; then
echo "::error::duckdb-python-version (${DUCKDB_PYTHON_VERSION}) must start with a vX.Y.Z version"
exit 1
fi
python_base="${BASH_REMATCH[1]}"

# duckdb-python follows DuckDB's major, minor and patch. Only the suffix
# may differ, for example DuckDB v1.5.4 republished as v1.5.4-post1. A
# mismatch on the rest is a typo in one of the two inputs.
if [[ "${duckdb_base}" != "${python_base}" ]]; then
echo "::error::duckdb-version (${DUCKDB_VERSION}) and duckdb-python-version (${DUCKDB_PYTHON_VERSION}) disagree on major.minor.patch: ${duckdb_base} vs ${python_base}"
exit 1
fi

echo "::notice::Stable release ${DUCKDB_PYTHON_VERSION}, shipping DuckDB ${DUCKDB_VERSION}"

- uses: actions/checkout@v4
if: ${{ inputs.stable-version != '' }}
if: ${{ inputs.duckdb-python-version != '' }}
with:
ref: ${{ inputs.duckdb-python-sha != '' && inputs.duckdb-python-sha || github.sha }}

- name: Compare the pin to duckdb-sha
if: ${{ inputs.stable-version != '' }}
if: ${{ inputs.duckdb-python-version != '' }}
env:
GH_TOKEN: ${{ github.token }}
run: |
Expand All @@ -78,14 +136,15 @@ jobs:

build_sdist:
name: Build an sdist and determine versions
needs: [verify_stable_pin]
needs: [verify_release_inputs]
if: ${{ !failure() && !cancelled() }}
uses: ./.github/workflows/packaging_sdist.yml
with:
testsuite: all
duckdb-python-sha: ${{ inputs.duckdb-python-sha != '' && inputs.duckdb-python-sha || github.sha }}
duckdb-sha: ${{ inputs.duckdb-sha }}
set-version: ${{ inputs.stable-version }}
set-version: ${{ inputs.duckdb-python-version }}
set-duckdb-version: ${{ inputs.duckdb-version }}

submodule_pr:
name: Create or update PR to bump submodule to given SHA
Expand Down Expand Up @@ -135,7 +194,7 @@ jobs:
else
pypi_state=VERSION_FOUND
fi
if [[ -z "${{ inputs.stable-version }}" ]]; then
if [[ -z "${{ inputs.duckdb-python-version }}" ]]; then
age=${result#age = }
if [ "${age}" -ge "${{ inputs.nightly-stale-after-days }}" ]; then
echo "::warning title=Stale nightly for ${{ github.ref_name }}::Nightly is ${age} days old (max=${{ inputs.nightly-stale-after-days }})"
Expand All @@ -150,7 +209,7 @@ jobs:
if [[ test == "${{ inputs.pypi-index }}" ]]; then
ci_env=pypi-test
elif [[ prod == "${{ inputs.pypi-index }}" ]]; then
ci_env=pypi-prod${{ inputs.stable-version == '' && '-nightly' || '' }}
ci_env=pypi-prod${{ inputs.duckdb-python-version == '' && '-nightly' || '' }}
else
echo "::error::Invalid value for inputs.pypi-index: ${{ inputs.pypi-index }}"
exit 1
Expand Down Expand Up @@ -189,7 +248,8 @@ jobs:
testsuite: all
duckdb-python-sha: ${{ inputs.duckdb-python-sha != '' && inputs.duckdb-python-sha || github.sha }}
duckdb-sha: ${{ inputs.duckdb-sha }}
set-version: ${{ inputs.stable-version }}
set-version: ${{ inputs.duckdb-python-version }}
set-duckdb-version: ${{ inputs.duckdb-version }}

# Renders the built artifact set as a matrix in the run summary, so the
# maintainer can check completeness at a glance and link one summary from the
Expand Down Expand Up @@ -338,7 +398,7 @@ jobs:
fi
echo "* Package index: ${pypi_host}" >> $GITHUB_STEP_SUMMARY
echo "* Vendored DuckDB Version: ${{ needs.build_sdist.outputs.duckdb-version }} (${dsha:0:10})" >> $GITHUB_STEP_SUMMARY
echo "* Dispatch DuckDB Version: ${{ inputs.duckdb-version }} (${{ inputs.duckdb-sha }})" >> $GITHUB_STEP_SUMMARY
echo "* Requested DuckDB Version: ${{ inputs.duckdb-version != '' && inputs.duckdb-version || '(derived from the submodule)' }} (${{ inputs.duckdb-sha }})" >> $GITHUB_STEP_SUMMARY
echo "* S3 upload status: ${{ needs.upload_s3.result == 'success' && needs.workflow_state.outputs.s3_url || needs.upload_s3.result }}" >> $GITHUB_STEP_SUMMARY
echo "* CI Environment: ${{ needs.workflow_state.outputs.ci_env }}" >> $GITHUB_STEP_SUMMARY

Expand Down
Loading
Loading