Context
Operational alarms currently remain inside container logs and SQLite. For example, three consecutive all-LLM-failure cycles emit an ERROR, set pipeline_runs.status='failed', and record llm_fail_count=N, but no active notification is delivered to an operator.
This is acceptable for now because the event is expected to be rare. If active notification is added later, it should not be coupled directly to Bluesky or implemented as a one-off LLM alarm sender.
Direction
Research a generic, transport-agnostic notification layer for structured operational events. The eventual design should allow event producers to report facts without knowing whether delivery uses a webhook, Telegram, Discord, email, or another sink.
Potential concerns to evaluate when this is picked up:
- structured event schema, severity, source, timestamp, and stable deduplication identity
- configurable delivery adapters without hard-coding a platform into pipeline logic
- rate limiting, one-shot versus repeated alerts, recovery notifications, and alert fatigue
- durable delivery state and retry behavior when the notifier itself is unavailable
- failure isolation so notification problems cannot crash collection or publishing
- secret-safe configuration and bounded log output
- which existing events should use the layer beyond the three-cycle LLM alarm
Do not select a transport or lock an architecture in this backlog issue. Reassess available infrastructure and operator needs when implementation is scheduled.
Initial candidate event
- Three consecutive cycles in which all attempted LLM batches fail (
llm_fail_count >= threshold).
The existing log and database records remain the source of truth until a notification design is approved.
Priority
Low-priority architectural backlog. No immediate implementation required.
Context
Operational alarms currently remain inside container logs and SQLite. For example, three consecutive all-LLM-failure cycles emit an
ERROR, setpipeline_runs.status='failed', and recordllm_fail_count=N, but no active notification is delivered to an operator.This is acceptable for now because the event is expected to be rare. If active notification is added later, it should not be coupled directly to Bluesky or implemented as a one-off LLM alarm sender.
Direction
Research a generic, transport-agnostic notification layer for structured operational events. The eventual design should allow event producers to report facts without knowing whether delivery uses a webhook, Telegram, Discord, email, or another sink.
Potential concerns to evaluate when this is picked up:
Do not select a transport or lock an architecture in this backlog issue. Reassess available infrastructure and operator needs when implementation is scheduled.
Initial candidate event
llm_fail_count >= threshold).The existing log and database records remain the source of truth until a notification design is approved.
Priority
Low-priority architectural backlog. No immediate implementation required.