Skip to content

Create one master baseline and compare against that #826

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cvanelteren wants to merge 5 commits into from
Closed

Create one master baseline and compare against that #826

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c1252ca
Create one master baseline and compare against that
cvanelteren Sep 10, 2026
41af484
attempt fix
cvanelteren Sep 10, 2026
283d6d0
pack artefact
cvanelteren Sep 10, 2026
fdfb0ba
attempt 3
cvanelteren Sep 10, 2026
ec5c7db
attempt 4
cvanelteren Sep 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
140 changes: 27 additions & 113 deletions .github/workflows/build-ultraplot.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,6 @@ jobs:
compare-baseline:
name: Compare baseline Python ${{ inputs.python-version }} with MPL ${{ inputs.matplotlib-version }}
runs-on: ubuntu-latest
continue-on-error: true
env:
IS_PR: ${{ github.event_name == 'pull_request' }}
TEST_MODE: ${{ inputs.test-mode }}
Expand Down Expand Up @@ -133,128 +132,43 @@ jobs:
cache-environment: true
cache-downloads: false

- name: Resolve baseline reference
id: baseline-ref
run: |
if [ "${IS_PR}" = "true" ]; then
BASE_REF="${{ github.event.pull_request.base.ref }}"
git fetch origin "${BASE_REF}"
BASE_SHA="$(git rev-parse "origin/${BASE_REF}")"
else
BASE_REF="${GITHUB_REF_NAME:-main}"
BASE_SHA="${GITHUB_SHA}"
fi
echo "base_ref=${BASE_REF}" >> "${GITHUB_OUTPUT}"
echo "base_sha=${BASE_SHA}" >> "${GITHUB_OUTPUT}"
echo "Resolved baseline ref=${BASE_REF} sha=${BASE_SHA}"
- name: Download canonical visual baseline
uses: actions/download-artifact@v7
with:
name: visual-baseline
path: ${{ runner.temp }}/visual-baseline

- name: Check baseline version support
id: baseline-support
- name: Extract canonical visual baseline
run: |
git show '${{ steps.baseline-ref.outputs.base_sha }}:pyproject.toml' > '${{ runner.temp }}/baseline-pyproject.toml'
python tools/ci/version_support.py \
--baseline-pyproject '${{ runner.temp }}/baseline-pyproject.toml' \
--python-version '${{ inputs.python-version }}' \
--matplotlib-version '${{ inputs.matplotlib-version }}' >> "$GITHUB_OUTPUT"
rm -rf ultraplot/tests/baseline
mkdir -p ultraplot/tests

- name: Explain unavailable baseline
if: steps.baseline-support.outputs.baseline-supported != 'true'
run: |
echo '::notice::Image comparison skipped: the base commit does not support this Python/Matplotlib pair. PR tests still run in the coverage job.'
echo 'Image comparison skipped for Python ${{ inputs.python-version }} / Matplotlib ${{ inputs.matplotlib-version }}: base commit ${{ steps.baseline-ref.outputs.base_sha }} does not support this pair. The coverage job still tests the PR on this pair.' >> "$GITHUB_STEP_SUMMARY"
tar -C ultraplot/tests \
-xf "${RUNNER_TEMP}/visual-baseline/visual-baseline.tar"

# Cache Baseline Figures (Restore step)
- name: Cache Baseline Figures
id: cache-baseline
uses: actions/cache@v6
if: env.IS_PR == 'true' && steps.baseline-support.outputs.baseline-supported == 'true'
with:
path: ./ultraplot/tests/baseline # The directory to cache
# Key is based on OS, Python/Matplotlib versions, and the base commit SHA
key: ${{ runner.os }}-baseline-base-v5-hs${{ env.PYTHONHASHSEED }}-${{ steps.baseline-ref.outputs.base_sha }}-${{ inputs.python-version }}-${{ inputs.matplotlib-version }}
restore-keys: |
${{ runner.os }}-baseline-base-v5-hs${{ env.PYTHONHASHSEED }}-${{ steps.baseline-ref.outputs.base_sha }}-${{ inputs.python-version }}-${{ inputs.matplotlib-version }}-
COUNT=$(find ultraplot/tests/baseline -type f -name '*.png' | wc -l)
echo "Extracted ${COUNT} baseline images"
if [ "${COUNT}" -eq 0 ]; then
echo "::error::Downloaded baseline contains no PNG files"
exit 1
fi

# Conditional Baseline Generation (Only runs on cache miss)
- name: Generate baseline from main
# Skip this step if the cache was found (cache-hit is true)
if: steps.baseline-support.outputs.baseline-supported == 'true' && (steps.cache-baseline.outputs.cache-hit != 'true' || env.IS_PR != 'true')
- name: Prepare selected test nodeids
run: |
mkdir -p ultraplot/tests/baseline
echo "TEST_MODE=${TEST_MODE}"
echo "IS_PR=${IS_PR}"
echo "PR_BASE_REF=${{ steps.baseline-ref.outputs.base_ref }}"
echo "PR_BASE_SHA=${{ steps.baseline-ref.outputs.base_sha }}"
echo "TEST_NODEIDS=${TEST_NODEIDS}"
# Save PR-selected nodeids for reuse after checkout (if provided)
: > /tmp/pr_selected_nodeids.txt
if [ "${TEST_MODE}" = "selected" ] && [ -n "${TEST_NODEIDS}" ]; then
python -c 'import json,os; raw=os.environ.get("TEST_NODEIDS","").strip(); parsed=json.loads(raw) if raw and raw!="[]" else []; parsed=[parsed] if isinstance(parsed,str) else parsed; nodeids=[item for item in parsed if isinstance(item,str) and item]; open("/tmp/pr_selected_nodeids.txt","w",encoding="utf-8").write("".join(f"{nodeid}\n" for nodeid in nodeids)); print(f"Selected nodeids parsed: {len(nodeids)}")'
else
: > /tmp/pr_selected_nodeids.txt
fi
# Checkout the resolved base-branch tip for PR baseline generation.
if [ "${IS_PR}" = "true" ]; then
git checkout "${{ steps.baseline-ref.outputs.base_sha }}"
fi

# Install the Ultraplot version from the base branch's code
pip install --no-build-isolation --no-deps .
python - <<'PY'
import json
import os

# Generate the baseline images and hash library
python -c "import ultraplot as plt; plt.config.Configurator()._save_yaml('ultraplot.yml')"
if [ "${TEST_MODE}" = "selected" ] && [ -s /tmp/pr_selected_nodeids.txt ]; then
status=0
FILTERED_NODEIDS=()
while IFS= read -r nodeid; do
if [ -z "$nodeid" ]; then
continue
fi
path="${nodeid%%::*}"
if [ -f "$path" ]; then
FILTERED_NODEIDS+=("$nodeid")
fi
done < /tmp/pr_selected_nodeids.txt
echo "FILTERED_NODEIDS_BASE_COUNT=${#FILTERED_NODEIDS[@]}"
if [ "${#FILTERED_NODEIDS[@]}" -eq 0 ]; then
echo "No valid nodeids found on base; skipping baseline generation."
else
echo "=== Memory before baseline generation ===" && free -h
pytest -n ${PYTEST_WORKERS} --dist loadfile --tb=short --disable-warnings -W ignore \
--mpl-generate-path=./ultraplot/tests/baseline/ \
--mpl-default-style="./ultraplot.yml" \
"${FILTERED_NODEIDS[@]}" || status=$?
echo "=== Memory after baseline generation ===" && free -h
if [ "$status" -eq 4 ] || [ "$status" -eq 5 ]; then
echo "No tests collected from selected nodeids on base; skipping baseline generation."
status=0
fi
fi
# Return to the PR branch before continuing
if [ "${IS_PR}" = "true" ]; then
echo "Checking out PR branch: ${{ github.sha }}"
git checkout ${{ github.sha }} || echo "Warning: git checkout failed, but continuing"
fi
if [ "$status" -ne 0 ]; then
echo "Baseline generation failed with status $status"
exit "$status"
fi
else
echo "=== Memory before baseline generation ===" && free -h
pytest -n ${PYTEST_WORKERS} --dist loadfile --tb=short --disable-warnings -W ignore \
--mpl-generate-path=./ultraplot/tests/baseline/ \
--mpl-default-style="./ultraplot.yml" \
ultraplot/tests
echo "=== Memory after baseline generation ===" && free -h
# Return to the PR branch for the rest of the job
if [ "${IS_PR}" = "true" ]; then
echo "Checking out PR branch: ${{ github.sha }}"
git checkout ${{ github.sha }} || echo "Warning: git checkout failed, but continuing"
fi
nodeids = json.loads(os.environ.get("TEST_NODEIDS") or "[]")
with open("/tmp/pr_selected_nodeids.txt", "w", encoding="utf-8") as fh:
for nodeid in nodeids:
fh.write(f"{nodeid}\n")
print(f"Prepared {len(nodeids)} selected nodeids")
PY
fi

# Image Comparison (Uses cached or newly generated baseline)
- name: Image Comparison Ultraplot
if: steps.baseline-support.outputs.baseline-supported == 'true'
run: |
set -uo pipefail
# This workflow runs in a login shell (bash -el), which executes
Expand Down
Loading
Loading