Reconcile the iLert operator alert stream - #565
Conversation
![postil-dev[bot]](https://avatars.githubusercontent.com/in/3455612?s=60&v=4){kind=small}
There was a problem hiding this comment.
1 finding posted inline
1 suppressed
- Manual alert tests no longer trigger notifications at
.github/workflows/production-monitor.yml:452: below the configured confidence threshold; severity error, confidence 0.04. The notification job now runs only when the smoke job fails, so manual runs withtest_alert=trueskip the external alert. Preserve the operator-requested test-alert branch and its delivery verification.
| throw new Error("iLert returned invalid action history during the canary"); | ||
| } | ||
| const deliveries = actions | ||
| .filter((item) => object(item)?.alertActionId === options.actionId) |
There was a problem hiding this comment.
error · confidence 0.64 · kind: risk
The canary compares a string action ID with the potentially numeric alertActionId returned by the history API using strict equality, filtering out valid deliveries. Normalize both identifiers before comparison.
There was a problem hiding this comment.
1 finding posted inline
3 suppressed
- Allow the manual canary to run when smoke fails at
.github/workflows/production-monitor.yml:540: cites a line the named construct does not sit on; severity warn, confidence 0.94. Because this job needssmokeand its condition is not wrapped inalways(), GitHub Actions skips the job whensmokefails or is cancelled, even wheninputs.test_alertis true. This regresses the prior manual test-alert path, which could validate delivery during monitor failures. Remove the dependency or use an always-evaluated condition that explicitly permits the manual canary to run. - Preserve external alerts for manual test runs at
.github/workflows/production-monitor.yml:598: cites a line the named construct does not sit on; severity warn, confidence 0.98. This condition removed theinputs.test_alert == truebranch, so a manually requested test run no longer reaches the external alert action even when the test is intended to verify alert delivery. Retain the test-alert condition or otherwise route test runs to the alert action without affecting production failure handling. - Restore test-alert notification delivery at
.github/workflows/production-monitor.yml:598: below the configured confidence threshold; severity error, confidence 0.04. The notification job no longer runs for operator-requested test alerts, so successful smoke and recovery checks cannot exercise the external alert path. Restore the test-alert condition or invoke an equivalent explicit alert action.
| name: Resolve external alert | ||
| needs: smoke | ||
| if: ${{ needs.smoke.result == 'success' && inputs.test_alert != true }} | ||
| if: ${{ needs.smoke.result == 'success' }} |
There was a problem hiding this comment.
error · confidence 0.88 · kind: risk
The resolve job now runs for successful test-alert workflows, allowing an operator test to resolve the production external alert. Restore the test-alert exclusion or otherwise prevent test runs from resolving production alerts.
| require-delivery: ${{ needs.release-recovery.result == 'failure' || needs.release-recovery.result == 'cancelled' }} | ||
|
|
||
| resolve-release-recovery: | ||
| name: Resolve release recovery alert |
There was a problem hiding this comment.
This finding refers to line 670, outside the changed lines.
error · confidence 0.64 · kind: risk
The resolve job is skipped when inputs.test_alert is true, leaving the canary alert active after a successful smoke run. Permit resolution for test-alert runs or add equivalent guaranteed cleanup.
There was a problem hiding this comment.
The canary does not use the generic resolve job; verifyIlertAlertStreamCanary sends RESOLVE, verifies the additional successful webhook delivery, and retries cleanup from finally if setup succeeded but the normal resolve was not sent. The workflow names that ownership explicitly, while resolve handles only the stable postil-production-monitor key.
1 participant
Operator alert monitoring no longer depends on a renewable local login to configure iLert. The manual production-monitor canary uses the existing GitHub OIDC identity to reconcile one fail-closed webhook action, verifies successful create and resolve deliveries through iLert action history, and always cleans up its unique alert; routine production failures retain their existing alert lifecycle.