fault_manager: make a time-based confirmation as visible as a reported one

docs/config/fault-manager.rst

@@ -65,16 +65,48 @@ The fault manager uses AUTOSAR DEM-style debounce filtering to prevent fault fla
      - When true, PASSED events can heal confirmed faults.
    * - ``healing_threshold``
      - ``3``
-     - Number of PASSED events to transition from CONFIRMED to HEALED.
+     - Counter value at which a fault heals, not a number of events. Healing costs
+       ``healing_threshold`` minus the counter the fault confirmed at, so with the
+       default ``-1`` it takes four PASSED events.
    * - ``auto_confirm_after_sec``
      - ``0.0``
      - Auto-confirm prefailed faults after this duration. Set to 0 to disable.
 
 .. tip::
 
-   For immediate fault confirmation (no debounce), set ``confirmation_threshold: 0``.
+   For immediate fault confirmation (no debounce), set ``confirmation_threshold: -1``,
+   which is also the default. ``0`` is rejected: the threshold must be strictly
+   negative, and the node falls back to ``-1`` with a warning.
    Faults with ``SEVERITY_CRITICAL`` always bypass debounce regardless of this setting.
 
+.. important::
+
+   The counter only moves when an event arrives, so ``confirmation_threshold`` and
+   ``healing_threshold`` are tunable only for a reporter that keeps sending events while a
+   condition holds. Confirmation needs repeated FAILED; healing needs repeated PASSED.
+
+   A reporter that sends one FAILED when a condition appears and one clear when it goes away
+   never sends the second event. ``confirmation_threshold: -3`` then leaves the fault in
+   PREFAILED, and the default fault list returns CONFIRMED only, so the fault is never seen.
+   Healing has the same shape: it needs ``healing_threshold - confirmation_threshold``
+   consecutive PASSED events counted from where the fault confirmed, and only one is sent.
+
+   For such a reporter, leave the confirmation threshold alone and make healing reachable:
+
+   .. code-block:: yaml
+
+      fault_manager:
+        ros__parameters:
+          confirmation_threshold: -1    # the default; the single FAILED confirms
+          healing_enabled: true
+          healing_threshold: 0          # heal on the single PASSED
+
+   ``auto_confirm_after_sec`` promotes a fault that stayed PREFAILED for that long and looks
+   like it would allow a deeper threshold. It does not: HEALED is latched, leaving that latch
+   costs ``healing_threshold - confirmation_threshold`` FAILED events, and a one-event reporter
+   sends one. The second occurrence of a fault code would then never confirm again. Filter
+   noisy samples in the reporter instead, where the samples are.
+
 Near-Miss Retention
 ~~~~~~~~~~~~~~~~~~~
 
@@ -530,7 +562,7 @@ by default: with it off there is no table, no file and no write cost.
      - Turn the audit log on.
    * - ``audit_log.transitions``
      - ``"all"``
-     - Which transitions are recorded: ``all`` (occurred, confirmed, cleared) or
+     - Which transitions are recorded: ``all`` (occurred, confirmed, healed, cleared) or
        ``confirmed_only``. Any other value falls back to ``all`` with a warning.
    * - ``audit_log.retention_max_records``
      - ``0``
@@ -593,11 +625,13 @@ Complete Example
        storage_type: "sqlite"
        database_path: "/var/lib/ros2_medkit/faults.db"
 
-       # Debounce (require 3 FAILED events to confirm)
+       # Debounce for a reporter that repeats its events while a condition holds:
+       # three FAILED events confirm, and four PASSED events heal from there.
+       # For a reporter that sends one event per transition, use -1 with
+       # healing_threshold 0 instead - see the note under Debounce Settings.
        confirmation_threshold: -3
        healing_enabled: true
        healing_threshold: 3
-       auto_confirm_after_sec: 30.0
 
        # Per-entity debounce overrides
        entity_thresholds:

src/ros2_medkit_fault_manager/CMakeLists.txt

@@ -192,6 +192,19 @@ if(BUILD_TESTING)
   medkit_add_launch_test(test_entity_thresholds_integration test/integration/test_entity_thresholds_integration.test.py
     TIMEOUT 60 LABELS "integration")
 
+  # Drives healing with the event counts a one-event-per-transition reporter
+  # actually sends: one FAILED per raise, one PASSED per clear. Parametrized over
+  # healing_threshold, so the node launches twice, and one case holds a settled
+  # fault for a quiet period - hence the timeout.
+  medkit_add_launch_test(test_debounce_and_healing test/integration/test_debounce_and_healing.test.py
+    TIMEOUT 120 LABELS "integration")
+
+  # Time-based confirmation has to reach the event stream, not just the store:
+  # the SSE feed and the trigger subscribers key off the published event, and
+  # black-box capture is enqueued alongside it.
+  medkit_add_launch_test(test_auto_confirm_visibility test/integration/test_auto_confirm_visibility.test.py
+    TIMEOUT 120 LABELS "integration")
+
   medkit_add_launch_test(test_rosbag_entity_scope test/integration/test_rosbag_entity_scope.test.py TIMEOUT 120
     LABELS "integration")
 

src/ros2_medkit_fault_manager/README.md

@@ -47,7 +47,8 @@ ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault \
 ## Features
 
 - **Multi-source aggregation**: Same `fault_code` from different sources creates a single fault
-- **Occurrence tracking**: Counts total reports and tracks all reporting sources
+- **Occurrence tracking**: Counts outages, not reports - the count starts at one and rises only
+  when a cleared fault is raised again - and tracks all reporting sources
 - **Severity escalation**: Fault severity is updated if a higher severity is reported
 - **Persistent storage**: SQLite backend ensures faults survive node restarts
 - **Debounce filtering** (optional): AUTOSAR DEM-style counter-based fault confirmation with per-entity threshold overrides
@@ -236,6 +237,8 @@ For systems that need to filter transient faults, enable debounce filtering by s
 ### Configuration
 
 ```bash
+# For a reporter that repeats its events while a condition holds.
+# See "Choosing the right lever for your reporter" below before copying this.
 ros2 run ros2_medkit_fault_manager fault_manager_node --ros-args \
   -p confirmation_threshold:=-3 \
   -p healing_enabled:=true \
@@ -254,7 +257,9 @@ The fault manager uses an AUTOSAR DEM-style debounce model:
 The counter is always clamped to `[confirmation_threshold, healing_threshold]`, so a long run of
 one-sided events cannot push it out to the integer limits and delay the opposite transition.
 `confirmation_threshold < 0 <= healing_threshold` is required (`healing_threshold = 0` heals on a
-single PASSED event); invalid thresholds fall back to safe defaults with a warning.
+single PASSED event). A positive confirmation threshold or a negative healing threshold is
+sign-flipped with a warning, so `5` becomes `-5`; a confirmation threshold of `0` is then
+rejected and falls back to `-1`.
 
 `CONFIRMED` and `HEALED` are **latched** (hysteresis): once reached, the status holds until the
 counter reaches the opposite threshold, so a single opposite-direction event cannot flip it. As a
@@ -263,6 +268,40 @@ events to return to the default (CONFIRMED-only) list. During that window `last_
 reflects the activity; `occurrence_count` does not, because it counts the edge that started the
 occurrence, not every report within it.
 
+### Choosing the right lever for your reporter
+
+The counter only moves when an event arrives, so the count-based settings above work only for a
+reporter that keeps sending FAILED while the condition is still there. A reporter that samples a
+value on a timer and reports on every sample is of that kind.
+
+Many reporters do not work that way. They send one FAILED when the condition appears and one clear
+when it goes away, and nothing in between. For such a reporter the second FAILED never arrives, so
+`confirmation_threshold: -3` means the fault stays PREFAILED and never confirms. The default fault
+list returns CONFIRMED only, so the fault is invisible. Healing has the same problem in reverse:
+`healing_threshold: 3` needs four consecutive PASSED events after a fault confirmed at `-1`, and
+only one PASSED is ever sent, so the fault stays CONFIRMED until someone calls `~/clear_fault`.
+
+Pick by how your reporter behaves:
+
+| Reporter repeats FAILED while the condition holds | Reporter sends one event per transition |
+|---|---|
+| `confirmation_threshold: -N` confirms on the Nth FAILED, so it rides out N-1 noisy samples | `confirmation_threshold` cannot filter here; see below |
+| `healing_threshold: N` needs `N - confirmation_threshold` clean samples | `healing_threshold: 0` heals on the single PASSED |
+
+For the second column, `auto_confirm_after_sec` looks like it fills the gap: it promotes a fault
+that has stayed PREFAILED for that long, without changing the counter. Pairing it with
+`confirmation_threshold: -2` does keep a single FAILED out of CONFIRMED. It also has a trap. HEALED
+is latched, and leaving that latch costs `healing_threshold - confirmation_threshold` FAILED
+events, which at `-2` is two. A reporter that sends one means the SECOND occurrence of a fault code
+never confirms again, and `occurrence_count` does not move either. Prefer leaving
+`confirmation_threshold` at `-1` for such a reporter, and filter noisy samples in the reporter
+itself, where the samples are.
+
+Two things ignore the counter. `SEVERITY_CRITICAL` confirms at once unless
+`critical_immediate_confirm` is turned off in the debounce config, and that field is not exposed as
+a ROS parameter. `auto_confirm_after_sec` promotes on elapsed time since the last FAILED, which is
+not the same as observing that the condition is still there.
+
 ### Fault Lifecycle with Debounce
 
 ```
@@ -284,7 +323,7 @@ PREFAILED -----> CONFIRMED -----> HEALED (retained)
 
 | Status | Description |
 |--------|-------------|
-| `PREFAILED` | Debounce counter < 0, not yet confirmed |
+| `PREFAILED` | Not yet confirmed. Usually a negative counter, but a fault that returns to 0 keeps the status it had |
 | `CONFIRMED` | Fault is active and verified |
 | `HEALED` | Resolved via PASSED events (if healing enabled) |
 | `CLEARED` | Manually acknowledged via `~/clear_fault` |
@@ -468,9 +507,10 @@ names automatically via the `fault_manager.namespace` parameter:
 
 ```yaml
 # gateway_params.yaml
-fault_manager:
-  namespace: "robot1"          # -> /robot1/fault_manager/list_faults
-  service_timeout_sec: 5.0
+ros2_medkit_gateway:
+  ros__parameters:
+    fault_manager.namespace: "robot1"      # -> /robot1/fault_manager/list_faults
+    fault_manager.service_timeout_sec: 5.0
 ```
 
 Launch the fault manager in a namespace:

src/ros2_medkit_fault_manager/config/fault_manager.yaml

@@ -18,8 +18,14 @@ fault_manager:
 
     # Healing OFF by default: a recovery signal (e.g. action SUCCEEDED) does not
     # auto-clear the fault until this is enabled.
+    #
+    # Threshold 0, so that turning healing on here works. Healing costs
+    # healing_threshold minus the counter the fault confirmed at, and a reporter
+    # that signals recovery once - the action bridge sends one PASSED - cannot
+    # move the counter further than that. Any higher value leaves the fault
+    # CONFIRMED after it has recovered.
     healing_enabled: false
-    healing_threshold: 3
+    healing_threshold: 0
 
     # Black-box rosbag capture OFF by default (opt-in). When enabled it defaults
     # to entity-scoped capture and is crash-safe (falls back / self-disables if no

src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/correlation/correlation_engine.hpp

@@ -115,6 +115,11 @@ class CorrelationEngine {
   /// Get count of muted faults
   uint32_t get_muted_count() const;
 
+  /// Whether a fault code is currently muted as a symptom.
+  /// @param fault_code Code to test
+  /// @return True while the code is suppressed by a root cause
+  bool is_muted(const std::string & fault_code) const;
+
   /// Get all active clusters
   /// @return List of cluster data
   std::vector<ClusterData> get_clusters() const;

src/ros2_medkit_fault_manager/include/ros2_medkit_fault_manager/fault_manager_node.hpp

@@ -175,6 +175,12 @@ class FaultManagerNode : public rclcpp::Node {
   void publish_fault_event(const std::string & event_type, const ros2_medkit_msgs::msg::Fault & fault,
                            const std::vector<std::string> & auto_cleared_codes = {});
 
+  /// Enqueue snapshot + rosbag capture for a fault that has just confirmed.
+  /// Shared by the report path and the time-based confirmation timer so a
+  /// confirmation produces the same evidence whichever one produced it.
+  /// @param fault_code Code of the fault that reached CONFIRMED
+  void capture_on_confirm(const std::string & fault_code);
+
   /// Validate severity value
   static bool is_valid_severity(uint8_t severity);
 

src/ros2_medkit_fault_manager/src/correlation/correlation_engine.cpp

@@ -251,6 +251,11 @@ uint32_t CorrelationEngine::get_muted_count() const {
   return static_cast<uint32_t>(muted_faults_.size());
 }
 
+bool CorrelationEngine::is_muted(const std::string & fault_code) const {
+  std::lock_guard<std::mutex> lock(mutex_);
+  return muted_faults_.find(fault_code) != muted_faults_.end();
+}
+
 std::vector<ClusterData> CorrelationEngine::get_clusters() const {
   std::lock_guard<std::mutex> lock(mutex_);