From 430b8111d542e9006bd5e2abbb33592c28412003 Mon Sep 17 00:00:00 2001 From: Colin Walters Date: Thu, 20 Aug 2026 14:06:31 -0400 Subject: [PATCH 1/2] ci: Stop leaking host systemd/dbus into install-tests, drop systemd-run The "Test install" job's `priv-integration.sh` step hangs intermittently at the chunkah build (a nested `podman run` invoked via `systemd-run -dP --wait`). The root cause is not fuse-overlayfs performance or the chunkah build itself: it's that the outer `sudo podman run --privileged --pid=host ...` invocation bind-mounts the HOST's real systemd and dbus sockets in (`-v /run/dbus:/run/dbus -v /run/systemd:/run/systemd`), so every `systemd-run` call inside the container talks to the host's actual systemd manager rather than anything scoped to the container. `systemd-run --wait` asks systemd to fork and supervise the wrapped command, and with the host's real systemd manager on the other end of that dbus call, the forked process ends up living in the HOST's mount and IPC namespaces instead of the container's -- despite having been launched from inside the privileged container. When that escaped process is itself a `podman run` (as with the chunkah build), it starts contending on host-level podman locks (e.g. `/dev/shm/libpod_lock`) against the *actual* host-level `sudo podman run --privileged` process that is itself still running the CI job, which deadlocks forever. This was tracked down empirically: comparing `/proc//ns/mnt` and `/proc//ns/ipc` of the hung process against host PID 1 confirmed the escape, and reproduced the hang reliably in isolation by stripping priv-integration.sh down to just the chunkah step. None of the four `systemd-run -dP --wait` call sites here actually need systemd supervision: they're synchronous builds/copies (`podman build`, two `skopeo copy` calls, and the chunkah `podman run`) that work fine run directly, and were apparently only wrapped in systemd-run to get a cgroup for the nested container. Fixing this isn't just a matter of deleting those wrappers, though. `/run/dbus` can go entirely -- nothing here ever dials it once systemd-run is gone -- but `/run/systemd` has to stay mounted, because the `nsenter -m -t 1 journalctl _COMM=bootc` check a few lines above this depends on it: ostree-ext's structured logging silently skips writing to the journal at all unless `libsystemd::daemon::booted()` (a check for `/run/systemd/system`) is true, and this test's own "layers already present" assertion relies on that write happening. But leaving /run/systemd mounted for the whole script also breaks cgroup delegation for the nested podman invocations below (podman's cgroup-manager auto-detection only produces a working, fully delegated `cgroupfs` setup when it can't see a live systemd socket at all) -- and that turns out to hold regardless of `--cgroup-manager` flags at any level, since it's decided dynamically per invocation rather than once at container-creation time. So instead: keep /run/systemd mounted at container creation for the journal check, then have priv-integration.sh itself unmount it right after that check, before any of the systemd-run-free podman/build work that follows. Verified with the same namespace comparison that the nested chunkah process's mnt/ipc namespaces match the outer container's own entrypoint rather than the host's, and with a full, unmodified end-to-end run of priv-integration.sh that both the journal check and the final "ok privileged integration" line pass. Assisted-by: AI Signed-off-by: Colin Walters Signed-off-by: Pragyan Poudyal --- .github/workflows/ci.yml | 2 +- crates/ostree-ext/ci/priv-integration.sh | 23 +++++++++++++++-------- 2 files changed, 16 insertions(+), 9 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 65b4abf86..f4151b0d2 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -207,7 +207,7 @@ jobs: # The ostree-container tests sudo podman run --privileged --pid=host -v /:/run/host -v $(pwd):/src:ro -v /var/tmp:/var/tmp \ --tmpfs /var/lib/containers \ - -v /run/dbus:/run/dbus -v /run/systemd:/run/systemd localhost/bootc /src/crates/ostree-ext/ci/priv-integration.sh + -v /run/systemd:/run/systemd localhost/bootc /src/crates/ostree-ext/ci/priv-integration.sh # Nondestructive but privileged tests sudo bootc-integration-tests host-privileged localhost/bootc-install # Install tests diff --git a/crates/ostree-ext/ci/priv-integration.sh b/crates/ostree-ext/ci/priv-integration.sh index b263561c3..de3156db9 100755 --- a/crates/ostree-ext/ci/priv-integration.sh +++ b/crates/ostree-ext/ci/priv-integration.sh @@ -91,6 +91,16 @@ if ! grep 'layers already present: ' logs.txt; then exit 1 fi +# The check above requires /run/systemd (bind-mounted from the host) to be +# present: ostree_ext's structured logging silently skips writing to the +# journal otherwise. But leaving it mounted for the rest of the script +# breaks cgroup delegation for the nested `podman run`/`podman build` +# invocations below (each of which creates its own container) -- podman's +# cgroup-manager auto-detection only produces a working, fully delegated +# cgroupfs setup when it can't see a live systemd socket at all. Drop it +# now that the journal check is done; nothing after this point needs it. +umount /run/systemd + podman pull ${image} ostree --repo="${sysroot}/ostree/repo" init --mode=bare-user ostree container image pull ${sysroot}/ostree/repo ostree-unverified-image:containers-storage:${image} @@ -106,11 +116,11 @@ cat >Dockerfile << EOF FROM ${image} RUN touch /usr/share/somefile EOF -systemd-run -dP --wait podman build -t localhost/fcos-derived . +podman build -t localhost/fcos-derived . derived_img=oci:/var/tmp/derived.oci derived_img_dir=dir:/var/tmp/derived.dir -systemd-run -dP --wait skopeo copy containers-storage:localhost/fcos-derived "${derived_img}" -systemd-run -dP --wait skopeo copy "${derived_img}" "${derived_img_dir}" +skopeo copy containers-storage:localhost/fcos-derived "${derived_img}" +skopeo copy "${derived_img}" "${derived_img_dir}" # Prune to reset state ostree --repo="${repo}" refs ostree/container/image --delete @@ -150,10 +160,8 @@ ostree container image prune-images --full --sysroot="${sysroot}" # See also https://github.com/coreos/chunkah?tab=readme-ov-file#compatibility-with-bootable-bootc-images nonostree_archive=/var/tmp/nonostree.ociarchive chunkah_config="$(podman inspect ${image})" -systemd-run -dP --wait podman info -systemd-run -dP --wait skopeo copy containers-storage:${image} oci:/var/tmp/fcos-oci:latest -systemd-run -dP --wait podman --log-level=debug run --rm --network=none \ - -v /var/tmp/fcos-oci:/chunkah:ro \ +podman run --rm \ + --mount=type=image,src=${image},dst=/chunkah \ -v /var/tmp:/output:z \ -e CHUNKAH_CONFIG_STR="${chunkah_config}" \ -e RUST_LOG=chunkah=debug \ @@ -162,7 +170,6 @@ systemd-run -dP --wait podman --log-level=debug run --rm --network=none \ --label ostree.commit- \ --label ostree.final-diffid- \ -o /output/nonostree.ociarchive -rm -rf /var/tmp/fcos-oci # Deploy the non-ostree image with debug logging to capture relabeling messages RUST_LOG=ostree_ext=debug ostree container image deploy \ From 3f2bbe901e8d4d0a2b183b2c78181911a5668b32 Mon Sep 17 00:00:00 2001 From: Colin Walters Date: Thu, 20 Aug 2026 14:06:43 -0400 Subject: [PATCH 2/2] test-install: Enable debug logging for chunkah build step Turn on `podman --log-level=debug` and `RUST_LOG=chunkah=debug` for the chunkah build invocation in priv-integration.sh. This step converts the FCOS test image into a non-ostree image and previously had no visibility into podman or chunkah internals when something went wrong here (as with the host-namespace-escape hang fixed in the prior commit). Cheap to leave on permanently since chunkah's own progress output is otherwise sparse, and useful for any future regressions in this step regardless of cause. This was originally proposed as part of PR #2392 alongside a (different, since superseded) fix attempt for the same underlying hang; keeping it here since it's independently useful. Assisted-by: AI Signed-off-by: Colin Walters --- crates/ostree-ext/ci/priv-integration.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/ostree-ext/ci/priv-integration.sh b/crates/ostree-ext/ci/priv-integration.sh index de3156db9..9c3dc0bbf 100755 --- a/crates/ostree-ext/ci/priv-integration.sh +++ b/crates/ostree-ext/ci/priv-integration.sh @@ -160,7 +160,7 @@ ostree container image prune-images --full --sysroot="${sysroot}" # See also https://github.com/coreos/chunkah?tab=readme-ov-file#compatibility-with-bootable-bootc-images nonostree_archive=/var/tmp/nonostree.ociarchive chunkah_config="$(podman inspect ${image})" -podman run --rm \ +podman --log-level=debug run --rm \ --mount=type=image,src=${image},dst=/chunkah \ -v /var/tmp:/output:z \ -e CHUNKAH_CONFIG_STR="${chunkah_config}" \