Skip to content

systemd: serialize initrd root fsck with a drop-in - #18639

Draft
Pawel Winogrodzki (PawelWMS) wants to merge 1 commit into
microsoft:3.0-devfrom
PawelWMS:pawelwi/fsck-device-lock-dropins
Draft

systemd: serialize initrd root fsck with a drop-in#18639
Pawel Winogrodzki (PawelWMS) wants to merge 1 commit into
microsoft:3.0-devfrom
PawelWMS:pawelwi/fsck-device-lock-dropins

Conversation

@PawelWMS

@PawelWMS Pawel Winogrodzki (PawelWMS) commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

What

  • Add one drop-in that replaces systemd-fsck-root.service execution with a root-only lock helper.
  • Add the helper and a dracut configuration that copies the two custom files into every rebuilt initramfs.
  • Add source signatures and prevent the systemd package from claiming dracut-owned parent directories.

Why

The root filesystem check can update ext4 metadata while systemd-udevd probes the same disk. If udev observes metadata during that update, its filesystem probe can temporarily lose the filesystem UUID and remove the corresponding /dev/disk/by-uuid link. The generated root mount then cannot resolve its device, and boot stops in the initramfs.

systemd-udevd already takes a shared whole-disk lock while probing. Running the root filesystem check under an exclusive udevadm lock prevents those reads from overlapping the metadata update.

How it works

Inside the initramfs, systemd-fstab-generator creates systemd-fsck-root.service for the exact root device. The generated unit includes a dependency such as:

BindsTo=dev-disk-by\x2duuid-....device

The drop-in replaces the generated ExecStart with systemd-fsck-root-device-lock. That single-purpose helper:

  1. Reads the BindsTo= property of systemd-fsck-root.service.
  2. Requires exactly one dev-*.device dependency.
  3. Converts the escaped device-unit name back to its /dev/... path.
  4. Acquires the exclusive whole-disk lock with udevadm lock --device=<path>.
  5. Runs systemd-fsck <path> while the lock is held.

The same unit name also has a static non-initrd form. When no generated device dependency exists, the helper preserves that path by resolving the mounted root through udevadm lock --backing=/ and invoking argument-free systemd-fsck.

Initramfs integration

The systemd RPM installs 50-systemd-fsck-root-device-lock.conf under /usr/lib/dracut/dracut.conf.d. Azure Linux's initramfs package already watches systemd updates and runs dracut --regenerate-all.

During regeneration, dracut reads this configuration from the host filesystem and copies only these custom files into each initramfs:

/usr/lib/systemd/systemd-fsck-root-device-lock
/usr/lib/systemd/system/systemd-fsck-root.service.d/systemd-fsck-root-device-lock.conf

Standard dracut modules already provide systemctl, systemd-escape, udevadm, and systemd-fsck.

split-files.py generates file lists for systemd's RPM subpackages. Its one-line change excludes only the dracut-owned parent directory entries. The configuration file itself remains in systemd-udev and is installed on the final image.

Scope

This change targets only the generated root filesystem check involved in the boot failure. It does not change separate /usr checks or instantiated checks for other filesystems. Draft #18640 demonstrates the alternative generator-layer implementation.

Risk

  • Early boot now depends on the helper, systemctl show, systemd-escape, and dracut file inclusion.
  • The generated root unit must retain exactly one block-device BindsTo= dependency.
  • Lock acquisition has a 180-second ceiling. A timeout prevents fsck from starting and fails the root check.
  • This drop-in approach has more packaging and runtime indirection than systemd: serialize generated initrd fsck device probes #18640.

Verification

  • Completed a 20-attempt, 10-VM resize campaign with the root check under the whole-disk lock: 390 measured boot transitions, zero UUID removals, and zero root lookup failures.
  • Verified device-unit names round-trip through systemd-escape --unescape --path.
  • Validated helper shell syntax after Git LF normalization.
  • Validated dracut install_items entries and exact staged source hashes.
  • Confirmed no custom %check changes remain.
  • Passed repository commit and push hooks.
  • Completed adversarial Advocate, Skeptic, Architect, and focused helper reviews.
  • Full package build was not available locally; draft CI and review are required.

@microsoft-github-policy-service microsoft-github-policy-service Bot added Packaging 3.0-dev PRs Destined for AzureLinux 3.0 labels Aug 26, 2026
@PawelWMS Pawel Winogrodzki (PawelWMS) changed the title systemd: serialize static fsck device probes systemd: serialize fsck device probes with initrd drop-ins Aug 27, 2026
@PawelWMS Pawel Winogrodzki (PawelWMS) changed the title systemd: serialize fsck device probes with initrd drop-ins systemd: serialize initrd root fsck with a drop-in Aug 27, 2026
@PawelWMS
Pawel Winogrodzki (PawelWMS) force-pushed the pawelwi/fsck-device-lock-dropins branch from 5e559b8 to 838b384 Compare August 27, 2026 20:07
Prevent udev from probing root filesystem metadata while fsck updates it.

Ship a root-only systemd drop-in and helper in the initramfs. The helper
derives the generated root device from its BindsTo dependency, acquires the
whole-disk lock, and then runs systemd-fsck against that exact device.

Keep the static non-initrd root path working by falling back to --backing=/
when the generated device dependency is absent.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: bb0e6716-4886-4e95-9efe-71dcde8687d6
@PawelWMS
Pawel Winogrodzki (PawelWMS) force-pushed the pawelwi/fsck-device-lock-dropins branch from 838b384 to ef51c07 Compare August 27, 2026 21:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3.0-dev PRs Destined for AzureLinux 3.0 Packaging

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant