Skip to content

fix: add systemd-networkd ordering and wait-online timeout - #169

Open
rgoltz wants to merge 1 commit into
amazonlinux:mainfrom
rgoltz:fix/policy-routes-wait-online-timeout
Open

fix: add systemd-networkd ordering and wait-online timeout#169
rgoltz wants to merge 1 commit into
amazonlinux:mainfrom
rgoltz:fix/policy-routes-wait-online-timeout

Conversation

@rgoltz

@rgoltz rgoltz commented Aug 17, 2026

Copy link
Copy Markdown

Fixes #168

When a DHCP lease is lost during boot (e.g. another service calls networkctl reload), the systemd-networkd DHCP client may stop requesting rather than retrying. policy-routes@.service then loops on systemd-networkd-wait-online with no explicit timeout, blocking indefinitely.

This PR:

  • Adds After=systemd-networkd.service to reduce the race window on boot
  • Adds --timeout=60 for bounded wait and faster retry cycle
  • Increases StartLimitIntervalSec from 10 to 60 to prevent permanent failure when wait-online fails quickly in a transient state

Journal evidence and full analysis in #168.

Update 1 of PR: Revised based on review feedback from Joe and some more insights from AWS Support: Removed Wants=systemd-networkd.service (only ordering needed, not dependency pull). Changed StartLimitIntervalSec from 300 to 60 to avoid delaying recovery after transient IMDS failures. Corrected root cause description: DHCP client stops requesting, not server failing to respond.

Comment thread systemd/system/policy-routes@.service Outdated
StartLimitIntervalSec=10
After=systemd-networkd.service
Wants=systemd-networkd.service
StartLimitIntervalSec=300

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am worried that changing StartLimitIntervalSec might affect transient failure retires such as IMDS failures. Ie it exhausts all 5 retries and has to wait another 300s to start again in which time it suceeds.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Changed to 60: Might be enough to prevent permanent failure from rapid retries while not delaying recovery after transient IMDS failures. I updated the code here in the PR.

Comment thread systemd/system/policy-routes@.service Outdated
Description=Set up policy routes for %I
StartLimitIntervalSec=10
After=systemd-networkd.service
Wants=systemd-networkd.service

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wants is a little different from After it will try to restart systemd-networkd.service even if it fails. Do you want this?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, dropped it. The intent is to ensure policy-routes doesn't start before networkd has loaded its .network files, not to pull networkd into the transaction or restart it. After= alone achieves that. PR updated.

@rgoltz
rgoltz force-pushed the fix/policy-routes-wait-online-timeout branch from ed10586 to ebbc5f4 Compare August 18, 2026 01:57
policy-routes@.service can enter an endless retry loop when a DHCP
lease is lost during boot (e.g. due to networkctl reload by another
service). The systemd-networkd DHCP client may stop requesting rather
than retrying, and wait-online blocks indefinitely with no explicit
timeout.

Changes:
- Add After=systemd-networkd.service to ensure networkd is running
  before policy-routes attempts to wait for the interface
- Increase StartLimitIntervalSec from 10 to 60 to prevent permanent
  failure when wait-online fails quickly in a transient state
- Add --timeout=60 to systemd-networkd-wait-online for bounded wait
  and faster retry cycle

Fixes: amazonlinux#168
@rgoltz
rgoltz force-pushed the fix/policy-routes-wait-online-timeout branch from ebbc5f4 to 4fce925 Compare August 18, 2026 02:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

policy-routes@.service loops indefinitely when DHCP lease is lost: no timeout, no active recovery

2 participants