ci: make a CI failure legible - crash frames, budget overruns, shared memory, and a coverage job that fits its cap

.github/actions/test-margin-and-environment/action.yml

@@ -0,0 +1,43 @@
+name: Record the test margin and the DDS environment
+description: >-
+  Says plainly whether a test step ran out of time, and records what /dev/shm and
+  the installed middleware looked like afterwards. Both are invisible once a run
+  has finished, and a step killed by its cap is reported by the runner as an
+  ordinary failure naming whichever test was in flight.
+
+inputs:
+  start:
+    description: Epoch seconds recorded immediately before the test step.
+    required: true
+  cap-minutes:
+    description: >-
+      The cap the guard measures against. Pass the step's timeout-minutes where
+      the step has one; where only the job is capped, pass the job's - the
+      overrun branch cannot fire there, because a job-level kill stops every
+      remaining step, but the margin warning still arrives in time to act on.
+    required: true
+
+runs:
+  using: composite
+  steps:
+    - shell: bash
+      run: |
+        # The start is missing only when an earlier step failed before it ran.
+        # Say so and carry on to the census, rather than dying on a usage error
+        # and taking the environment evidence down with it.
+        if [ -n "${{ inputs.start }}" ]; then
+          ./scripts/ci_test_completeness.sh "${{ inputs.start }}" "${{ inputs.cap-minutes }}" || guard_rc=$?
+        else
+          echo "::notice::no test step start was recorded - the run failed before the tests"
+        fi
+        # A DDS participant killed rather than shut down leaves its segments
+        # behind for the rest of the job, and the installed middleware decides
+        # which transport those segments belong to.
+        df -h /dev/shm
+        echo "shm segments: $(ls /dev/shm | wc -l)"
+        # No `head` in this pipeline: it closes the pipe early, SIGPIPEs `ls`, and
+        # the step's `pipefail` would then kill the census exactly when there are
+        # enough segments to be worth listing.
+        ls -la /dev/shm | tail -n +1
+        dpkg -l | grep -E 'rmw-fastrtps|fastdds|cyclonedds' || true
+        exit "${guard_rc:-0}"

CONTRIBUTING.md

@@ -89,6 +89,14 @@ is the tally of the run you just started and not of everything tested since the
 rather than matching a name. Run it on its own with `./scripts/drop_stale_results.sh` if you
 invoke `colcon test` directly.
 
+Every run also reclaims shared memory left behind by DDS participants that were killed
+rather than shut down. The aggregation suites kill peers on purpose, and nothing gives
+those segments back: about 0.65 MB each, and a container's `/dev/shm` is 64 MB by default,
+so they accumulate across runs until something unrelated fails for lack of space. The
+reclaiming is `fastdds shm clean`, the vendor's own tool, which removes a segment only when
+no process holds its lock. Run it on its own with `./scripts/sweep_shm.sh`; it refuses
+while ROS processes are alive, and `--force` overrides that.
+
 #### Pre-commit and Pre-push Hooks
 
 ```bash
@@ -174,6 +182,11 @@ genhtml coverage.info --output-directory coverage_html --ignore-errors source
 ./scripts/check_coverage_packages.sh coverage.info --skip ros2_medkit_opcua
 ```
 
+In CI this gate runs in the `coverage (merge)` job rather than alongside the tests: the
+coverage run is sharded by package, so each shard's tracefile is legitimately partial and
+the merged report is the first place the question "did every package reach the report"
+can be asked.
+
 Open `coverage_html/index.html` in your browser.
 
 `ros2_medkit_opcua` is skipped because it pulls `open62541pp` over the network

scripts/ci_test_completeness.sh

@@ -0,0 +1,105 @@
+#!/usr/bin/env bash
+# Say plainly when a test step ran out of time, instead of letting it read as a
+# test failure.
+#
+# A step killed by its timeout is marked FAILED by the runner, not cancelled, and
+# the in-flight test leaves no result file. `colcon test-result` then reports that
+# as one erroring test named `<test>.xunit.missing_result`: a test that never
+# failed, among thousands that never ran. On its own that is indistinguishable
+# from a single flake, which is how a budget overrun gets filed as a test problem
+# while every package queued behind it disappears without trace.
+#
+# The step's own log is not readable from inside the job, so the overrun is
+# detected by the clock instead: the caller records a start timestamp before the
+# test step and passes the step's cap here. Elapsed at or above the cap means the
+# runner killed it.
+#
+# What this deliberately does NOT do is count registered tests against result
+# files. Some tests legitimately write none - `test_dds_domain_allocation` is
+# registered by ros2_medkit_cmake in every package and produces no xunit - so
+# that comparison reports missing results on a run where everything passed.
+#
+# Usage: ci_test_completeness.sh <start-epoch-seconds> <cap-minutes>
+set -euo pipefail
+
+START_EPOCH="${1:-}"
+CAP_MINUTES="${2:-}"
+
+usage() {
+  echo "usage: ci_test_completeness.sh <start-epoch-seconds> <cap-minutes>" >&2
+  echo "  both are positive integers; the start is seconds since the epoch, as" >&2
+  echo "  written by date +%s before the step being measured" >&2
+}
+
+# Validated rather than trusted: a start value from a different clock, or a cap
+# of zero, otherwise turns this from a diagnostic into a source of false
+# verdicts - a monotonic start reads as an enormous overrun, and a zero cap
+# divides by zero while still printing an overrun.
+if [[ ! "${START_EPOCH}" =~ ^[0-9]+$ || ! "${CAP_MINUTES}" =~ ^[0-9]+$ ]]; then
+  usage
+  exit 2
+fi
+if [[ "${START_EPOCH}" -eq 0 || "${CAP_MINUTES}" -eq 0 ]]; then
+  usage
+  exit 2
+fi
+
+now=$(date +%s)
+if [[ "${START_EPOCH}" -gt "${now}" ]]; then
+  echo "::error::the recorded start (${START_EPOCH}) is in the future - the clock moved, so no margin can be computed"
+  exit 2
+fi
+
+elapsed=$(( now - START_EPOCH ))
+cap_seconds=$(( CAP_MINUTES * 60 ))
+
+empty=0
+found=0
+while IFS= read -r xml; do
+  found=$((found + 1))
+  if [[ ! -s "${xml}" ]]; then
+    echo "::error::empty result file: ${xml}"
+    empty=$((empty + 1))
+  fi
+done < <(find build -path '*/test_results/*' -name '*.xml' 2>/dev/null)
+
+# `missing_result` is not a filename. ament writes it as the name attribute of a
+# <testcase> INSIDE the ordinary xunit file, before running the command, so the
+# file exists and is not empty. Measured: across five build trees, zero files are
+# named that, while one tree carried two genuine missing results in its XML.
+# `|| true` inside the braces, not after the pipe: grep exits 1 when it matches
+# nothing, which is the healthy case, and under `set -e` with `pipefail` that
+# killed this script before it printed anything at all - so a clean run reported
+# nothing and failed the step.
+missing=$( { grep -l 'missing_result' -r build --include='*.xml' 2>/dev/null || true; } | wc -l)
+
+printf 'test step: %ds elapsed of a %ds cap (%d%%); results: %d written, %d empty, %d missing_result\n' \
+  "${elapsed}" "${cap_seconds}" "$(( elapsed * 100 / cap_seconds ))" "${found}" "${empty}" "${missing}"
+
+status=0
+
+if [[ "${found}" -eq 0 ]]; then
+  echo "::error::no test result files at all - the run never reached the tests"
+  status=1
+fi
+
+if [[ "${empty}" -gt 0 ]]; then
+  status=1
+fi
+
+# The runner kills the step AT its cap and this check runs afterwards, so an
+# overrun is simply elapsed at or past the cap. An earlier version subtracted a
+# fixed minute, which made every cap below five minutes report an overrun on a
+# run that had barely started.
+if [[ "${elapsed}" -ge "${cap_seconds}" ]]; then
+  echo "::error::the test step reached its ${CAP_MINUTES} minute cap. Any missing_result above belongs to the test that was interrupted, and the packages queued behind it did not run at all. This is a budget overrun, not a test verdict."
+  status=1
+elif [[ "${elapsed}" -ge $(( cap_seconds * 8 / 10 )) ]]; then
+  echo "::warning::the test step used $(( elapsed * 100 / cap_seconds ))% of its ${CAP_MINUTES} minute cap - it will start being killed before anyone decides to raise it"
+fi
+
+if [[ "${missing}" -gt 0 && "${elapsed}" -lt "${cap_seconds}" ]]; then
+  echo "::warning::${missing} test(s) produced no result file without the step running out of time - investigate the test, not the budget"
+fi
+
+exit "${status}"

scripts/sweep_shm.sh

@@ -0,0 +1,83 @@
+#!/usr/bin/env bash
+# Reclaim shared-memory segments left behind by DDS participants that were killed
+# rather than shut down.
+#
+# A participant that exits cleanly frees its segments; one that dies on SIGKILL
+# does not, and nothing reclaims them afterwards. Measured on Fast DDS: about
+# 0.65 MB per participant, and 125 killed participants filled 63 MB of a 64 MB
+# /dev/shm. Test runs that kill nodes on purpose - the aggregation suites do -
+# accumulate this across every run on a developer machine, and the failure it
+# eventually produces looks like anything but a full tmpfs.
+#
+# The reclaiming is done by `fastdds shm clean`, which is the vendor's own tool
+# and ships with every distro we support. It decides a segment is stale by taking
+# an exclusive non-blocking flock on the segment's lock file; the kernel drops
+# that lock when a process dies, SIGKILL included, so a segment is removed only
+# when nothing holds it. Each distro's copy matches its own file naming
+# (fastrtps_* on 2.x, fastdds_* on 3.x), so the tool from the sourced
+# distribution is the one to call - there is nothing for us to match on.
+#
+# Refuses to run while ROS processes are alive: the flock test protects live
+# segments, but a sweep in the middle of a test run is still a lie about what the
+# run measured.
+set -euo pipefail
+
+usage() {
+  echo "usage: sweep_shm.sh [--force]" >&2
+  echo "  --force  sweep even if ROS processes are running" >&2
+}
+
+FORCE=0
+case "${1:-}" in
+  --force) FORCE=1 ;;
+  "") ;;
+  *) usage; exit 2 ;;
+esac
+
+shm_bytes() {
+  # One value or nothing. An earlier form piped du into cut with `|| echo 0`,
+  # which under pipefail could emit cut's partial output AND the fallback, so the
+  # arithmetic below got a two-line operand and died with a bad math expression.
+  local out
+  if ! out=$(du -sb /dev/shm 2>/dev/null); then
+    echo unknown
+    return
+  fi
+  printf '%s' "${out}" | head -n 1 | cut -f1
+}
+
+if [[ "${FORCE}" -eq 0 ]]; then
+  # Match on the command line rather than the process name: colcon runs as
+  # python3, and a pattern that also matches this script would match itself.
+  # demo_ covers the launch fixtures the documented demo workflow starts, which
+  # an earlier pattern missed entirely - a developer running only demo nodes got
+  # a sweep in the middle of live work.
+  #
+  # Our own process and our caller are excluded, because the caller's command
+  # line is often a match by itself: `./scripts/test.sh test_demo_lifecycle`
+  # contains demo_, so every single-test preset refused to sweep and the only
+  # symptom was a puzzling message.
+  if pgrep -f '[g]ateway_node|[f]ault_manager_node|[d]emo_|[c]test|[l]aunch_test|[c]olcon test' 2>/dev/null |
+      grep -qvE "^($$|${PPID})$"; then
+    echo "sweep_shm: ROS processes are running - refusing to sweep (use --force if you mean it)" >&2
+    exit 1
+  fi
+fi
+
+before="$(shm_bytes)"
+
+if command -v fastdds >/dev/null 2>&1; then
+  fastdds shm clean || echo "sweep_shm: 'fastdds shm clean' reported a problem, continuing" >&2
+else
+  echo "sweep_shm: no 'fastdds' on PATH - source a ROS distribution first" >&2
+  exit 1
+fi
+
+after="$(shm_bytes)"
+
+if [[ "${before}" == unknown || "${after}" == unknown ]]; then
+  echo "sweep_shm: cleaned, but /dev/shm could not be measured (du declined)" >&2
+  exit 0
+fi
+
+printf 'sweep_shm: /dev/shm %s -> %s bytes (reclaimed %s)\n' "${before}" "${after}" "$((before - after))"

scripts/test.sh

@@ -57,6 +57,12 @@ COMMON_ARGS=(--event-handlers console_direct+ --parallel-workers "$(nproc)" --re
 # being able to ask for on its own. The reasoning lives there.
 "$HERE/drop_stale_results.sh" build
 
+# Reclaim shared memory stranded by participants that were killed rather than shut
+# down - the aggregation suites kill peers on purpose, and nothing gives those
+# segments back. Tolerated when it declines: it refuses to sweep while ROS
+# processes are alive, and that refusal must not stop a test run.
+"$HERE/sweep_shm.sh" || true
+
 # Run tests, capture exit code so we always show results even on failure.
 set +e
 case "$PRESET" in

src/ros2_medkit_fault_reporter/test/test_domain_launch.test.py

@@ -43,8 +43,17 @@
 
 TOPIC = '/medkit_launch_domain_probe'
 
+# The child answers the shutdown signal instead of dying from it, so its exit
+# status is a fact the test can pin rather than a range to tolerate.
+#
+# What this does NOT buy, despite the obvious guess: it strands no shared memory
+# either way. Measured on jazzy, /dev/shm bytes before and after - a bare
+# `rclpy.spin` killed by SIGINT returns to baseline exactly, because CPython
+# finalises the interpreter before re-raising, and the participant is destroyed
+# on the way out. Only SIGKILL strands segments (+0.65 MB, measured), and no
+# handler in this child can affect that.
 CHILD = (
-    'import os, sys, rclpy;'
+    'import os, signal, sys, rclpy;'
     'from rclpy.node import Node;'
     'from std_msgs.msg import String;'
     "print('CHILD_ROS_DOMAIN_ID=' + str(os.environ.get('ROS_DOMAIN_ID')), flush=True);"
@@ -53,7 +62,18 @@
     f"pub = node.create_publisher(String, '{TOPIC}', 10);"
     "msg = String(data='from-the-child');"
     'timer = node.create_timer(0.1, lambda: pub.publish(msg));'
-    'rclpy.spin(node)'
+    # launch escalates to SIGTERM when SIGINT is not answered in time, and
+    # Python's default action for that one is to die on the spot. Routing it
+    # through the same handler SIGINT uses keeps both paths ending in the
+    # shutdown below.
+    'signal.signal(signal.SIGTERM, signal.default_int_handler)\n'
+    'try:\n'
+    '    rclpy.spin(node)\n'
+    'except KeyboardInterrupt:\n'
+    '    pass\n'
+    'finally:\n'
+    '    node.destroy_node()\n'
+    '    rclpy.try_shutdown()\n'
 )
 
 
@@ -111,5 +131,11 @@ def test_the_child_node_is_reachable_on_that_domain(self):
 class TestChildShutdown(unittest.TestCase):
 
     def test_the_child_was_stopped(self, proc_info, child):
-        # SIGINT/SIGTERM during launch shutdown, not a crash.
-        self.assertIn(proc_info[child].returncode, (0, -2, -15, 130, 143))
+        # Zero, not merely "not a crash". A fixture that dies from its shutdown
+        # signal reports the same status whether it shut down or was cut off
+        # mid-flight, so the old range accepted both. Zero distinguishes them.
+        self.assertEqual(
+            proc_info[child].returncode,
+            0,
+            'the child did not shut down cleanly, so its DDS participant leaked',
+        )

src/ros2_medkit_integration_tests/CMakeLists.txt

@@ -104,6 +104,7 @@ target_include_directories(demo_param_beacon_node PRIVATE ${_demo_include_dir})
 medkit_target_dependencies(demo_param_beacon_node rclcpp)
 
 add_executable(managed_lifecycle demo_nodes/managed_lifecycle_node.cpp)
+target_include_directories(managed_lifecycle PRIVATE ${_demo_include_dir})
 medkit_target_dependencies(managed_lifecycle rclcpp rclcpp_lifecycle)
 
 # Proof fixture for GRAPH_NODE_UNREADABLE (see the file doc): a plain rclcpp::Node that
@@ -206,6 +207,15 @@ if(BUILD_TESTING)
   set_tests_properties(test_test_utils_constants PROPERTIES LABELS "unit")
   medkit_test_needs_no_domain(test_test_utils_constants)
 
+  # The crash reporter every demo binary installs. Death tests, because the
+  # behaviour under test only exists in a process that is being killed - a
+  # handler that is merely installed proves nothing about what a crash leaves
+  # behind. ROS-free: it creates no node, only signals.
+  find_package(ament_cmake_gmock REQUIRED)
+  medkit_add_gmock(test_crash_backtrace test/test_crash_backtrace.cpp)
+  target_include_directories(test_crash_backtrace PRIVATE ${_demo_include_dir})
+  medkit_test_needs_no_domain(test_crash_backtrace)
+
   # Configures every launch-test-registering package this file can reach for
   # real, twice each (ordinary workspace conditions and system-package
   # conditions), and compares what each run actually registered. Lives here
@@ -439,6 +449,10 @@ if(BUILD_TESTING)
       endif()
     endforeach()
   endif()
+
+  # gtest/gmock vendor sources are built as subdirectory targets and inherit
+  # this package's promoted warnings, which they do not compile clean under.
+  ros2_medkit_relax_vendor_warnings()
 endif()
 
 ament_package()

src/ros2_medkit_integration_tests/README.md

@@ -78,6 +78,32 @@ ros2 launch ros2_medkit_integration_tests demo_nodes.launch.py
 | `long_calibration` | `/powertrain/engine` | Action | Fibonacci-based long-running action |
 | `dual_calibration` | `/testrig/dual` | Services + Actions | `left/calibrate` and `right/calibrate`, `left/sweep` and `right/sweep` - one provider carrying each operation short name twice, so the ROS path is the only id that separates the copies |
 
+### Crash reports
+
+Every demo binary installs a fatal-signal handler before anything else runs.
+Most get it from `run_demo_node()` in
+`include/ros2_medkit_integration_tests/demo_node_main.hpp`; the two that cannot
+use that helper - `unresponsive_param_node` and `managed_lifecycle_node`, which
+own their shutdown sequence - call `install_crash_backtrace()` themselves. A node killed by `SIGSEGV`,
+`SIGBUS` or `SIGABRT` writes its stack to stderr, so launch_testing captures it
+with the rest of the process output:
+
+```
+MEDKIT-CRASH signal=SIGSEGV
+/path/to/demo_brake_pressure_sensor(+0x129cb) [0x5603199349cb]
+/opt/ros/<distro>/lib/libfastdds.so.3(...) [0x...]
+MEDKIT-CRASH end
+```
+
+Grep a CI log for `MEDKIT-CRASH` to find one. Frames from our own binaries come
+back as `binary(+0xOFFSET)` because the release build has no exported symbols;
+resolve them with `addr2line -e <binary> <offset>`. The process still dies from
+the original signal, so the exit status a test asserts on is unchanged.
+
+This exists because a node that dies during startup otherwise leaves nothing at
+all: no output, and no core file, since a container cannot set the host's
+`core_pattern`.
+
 ## Writing New Tests
 
 ### Feature Test Template