Skip to content

[medium] fix(ci): poll the healthcheck instead of sleeping a fixed 10 seconds - #897

Open
elhoim wants to merge 1 commit into
MISP:mainfrom
elhoim:ci-fix-healthcheck-race
Open

[medium] fix(ci): poll the healthcheck instead of sleeping a fixed 10 seconds#897
elhoim wants to merge 1 commit into
MISP:mainfrom
elhoim:ci-fix-healthcheck-race

Conversation

@elhoim

@elhoim elhoim commented Sep 3, 2026

Copy link
Copy Markdown
Member

BLUF — A fixed 10-second sleep before the CI healthcheck races module imports; a bounded poll replaces it.

  • Problem — The test job in .github/workflows/test-package.yml starts misp-modules in the background, sleeps exactly 10 seconds, then runs a single curl against /healthcheck; when module imports have not finished in time the curl exits 7 and the matrix leg fails, so unrelated pull requests go red intermittently.
  • Fix — Replaces the fixed sleep and one-shot curl with a bounded poll that succeeds as soon as the server answers and prints error.log if it never does.
  • Effect — Removes a recurring source of false CI failures and lets the common case finish sooner than the old unconditional 10-second wait.

The failure

Affected pull requests fail with:

##[error]Process completed with exit code 7

Exit code 7 is curl's "failed to connect to host". Reading up the job log, misp-modules is still importing when the check fires:

2026-08-31 01:52:44,718 - misp-modules - INFO - MISP module convert_markdown_to_pdf (type=expansion) imported
Post job cleanup.

Why it is a race

- name: Run server in background
  run: poetry run misp-modules -l 127.0.0.1 2>error.log &

- name: Sleep for 10 seconds
  run: sleep 10s

- name: Check if server is running
  run: curl -sS localhost:6666/healthcheck

Ten seconds is a guess. Start-up time depends on the runner and on the Python version in the matrix, and misp-modules imports a large number of expansion modules before it listens. When the guess is wrong there is no retry, and the single failure takes the leg down. This shows up on pull requests that do not touch anything related — it has been observed on two changes whose diffs are a SPARQL escaping fix and an exception guard.

A second, smaller problem: the old step used curl -sS without -f, so an HTTP error status would have been reported as success.

The change

- name: Wait for the server to answer its healthcheck
  shell: bash
  run: |
    for attempt in $(seq 1 60); do
      if curl -sSf --max-time 5 localhost:6666/healthcheck; then
        echo "healthcheck answered after ${attempt}s"
        exit 0
      fi
      sleep 1
    done
    echo "misp-modules did not answer /healthcheck within 60s" >&2
    cat error.log >&2 || true
    exit 1
  • Returns as soon as the server is up, so a healthy run no longer pays a flat 10 seconds.
  • Tolerates a slow start up to 60 seconds instead of failing at 10.
  • -f makes an HTTP error status a failure rather than a pass.
  • On a genuine start-up failure the job now prints error.log, so the cause is visible instead of a bare exit code.

CI-configuration change only; no module or test code is touched. Validated with a YAML parse of the workflow (12 steps, correctly ordered) and bash -n on the script body.

🤖 Generated with Claude Code

The test job starts misp-modules in the background, sleeps exactly 10
seconds, then runs a single "curl -sS localhost:6666/healthcheck". When
the server has not finished importing every module within those 10
seconds, curl exits 7 (failed to connect) and the whole matrix leg fails.
The job log shows modules still being imported at the moment the check
runs, so this is a startup race and not a defect in the change under test.

Because start-up time varies with the runner and the Python version, this
fires intermittently and on unrelated pull requests.

Replace the fixed sleep and one-shot curl with a bounded poll: retry once
a second for up to 60 seconds, succeed as soon as the healthcheck answers,
and on timeout print error.log before failing so the real start-up error
is visible in the job output. The fast path is now faster than the old
unconditional 10-second sleep.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VpjGT8zb2wRNqCjQwFugLC
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant