From 51d13a841071bde240832ab84d9cbb286c2e9d04 Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 04:57:48 -0600
Subject: [PATCH 01/25] Revise the error-alerting plan against the current code

The July draft predates the supervisor, boot start, both retention sweeps,
sheets-link delivery and non-blocking delivery. The revision adds their
alert sites, persists throttle state so a crash-loop can't bypass it, and
reports crashes from the restarted process.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 ERROR-ALERTING-PLAN.md | 371 ++++++++++++++++++++++++++---------------
 1 file changed, 237 insertions(+), 134 deletions(-)

diff --git a/ERROR-ALERTING-PLAN.md b/ERROR-ALERTING-PLAN.md
index 299bda0..993b45f 100644
--- a/ERROR-ALERTING-PLAN.md
+++ b/ERROR-ALERTING-PLAN.md
@@ -1,171 +1,274 @@
-# FileMill Error Alerting — Plan
+# FileMill Error Alerting — Implementation Plan (issue #7)
 
-Goal: when something goes **wrong** in FileMill, email an operator alert to
-`support@example.com`, so an unattended background service (it runs at logon on
-a personal laptop) doesn't fail silently.
+**Status:** plan only, not implemented. First drafted 2026-07-19; **revised
+2026-09-11** against the current code. Since the first draft, FileMill gained the
+supervisor loop (#4), boot start, the two retention sweeps, sheets-link delivery,
+and non-blocking delivery. Each adds alert sites, and the supervisor changes how a
+crash can be reported at all.
 
-Status: **plan only, not implemented.** Written 2026-07-20.
+Goal: when FileMill fails *systemically*, email the operator (the address in
+`config/email.yaml`, which is gitignored; examples here use `support@example.com`),
+so an unattended worker doesn't fail silently.
 
-The *sending* is trivial — FileMill already has a working Mailgun Send path
-(`Service.send` in `internal/mailgun/outbound.go`). The work is deciding
-**which** failures alert, **not** drowning `support@` in noise, and handling the
-cases email structurally can't cover.
+Sending is the easy part: `Service.send` in `internal/mailgun/outbound.go`
+already works. The real work is deciding **which** failures alert, **throttling**
+them, and being honest about what email **cannot** report.
 
 ---
 
-## 1. Guiding principles
-
-1. **Alert on systemic failures, stay silent on expected ones.** A malformed
-   PDF that a transformer cleanly rejected is a *normal outcome the sender
-   already hears about* — it must not page an operator. A transformer that
-   crashed, timed out, or produced no valid result is *systemic* — alert.
-2. **Throttle everything.** Loops and bots can generate thousands of identical
-   errors. Deduplication + rate limiting is mandatory, not optional.
-3. **Never let alerting recurse or crash the app.** An alert-send that fails is
-   logged and dropped; it must never trigger another alert or panic.
-4. **Email can't report that the app is gone.** Total-crash / machine-asleep
-   detection needs an external heartbeat, not self-sent email. In scope as a
-   complementary Phase 4, called out honestly.
+## 1. Principles
+
+1. **Alert on systemic failures, stay silent on expected ones.** A PDF that a
+   transformer cleanly rejected is a normal outcome the sender already hears about.
+   A transformer that crashed, timed out, or broke the contract is systemic.
+2. **Throttle everything, and keep the throttle state across restarts.** The
+   delivery loop ticks every second, and the supervisor restarts a crashing worker
+   every ≤120s. An in-memory throttle resets on every restart, so a crash-loop
+   would still send an alert per restart. Throttle state lives in SQLite.
+3. **Alerting never blocks, recurses, or crashes.** `Report` enqueues and returns.
+   A failed alert send is logged and dropped, never re-reported.
+4. **A process can't report its own death.** A crash is reported by the *next*
+   process, the one the supervisor restarts. "Never came back" or "machine offline"
+   is the heartbeat's job (#5), not this issue's.
 
 ---
 
-## 2. What alerts, and what doesn't
-
-| Failure | Code site | Alert `support@`? | Why |
-|---|---|---|---|
-| Forged / replayed webhook (401) | `webhook.go` `receive` | **No** | Bots scan public URLs; would flood |
-| Malformed / oversize body (400) | `receive` | **No** | Client noise |
-| Unrouted recipient / disallowed sender (silent 200) | `receive` | **No** | Benign |
-| **Intake failure (500)** — storage / filesystem / Submit | `receive` → `intake` | **Yes** | Systemic |
-| **Job: transformer missing from config** | `app.go` `execute` | **Yes** | Systemic (misconfig) |
-| **Job: transformer timed out** | `execute` | **Yes** | Systemic |
-| **Job: missing / corrupt `result.json`** | `execute` → `readResult` | **Yes** | Systemic (contract violation) |
-| **Job: nonzero exit with no usable result** | `execute` | **Yes** | Systemic (crash) |
-| Job: transformer cleanly reported `success:false` (bad input) | `execute` | **No** | Expected; sender already told in the reply |
-| **Delivery failure** — Mailgun send non-2xx | `outbound.go` `deliverPending`/`send` | **Yes (throttled)** | Replies aren't going out |
-| **Worker loop dies** (`store.Next` error → `fatal`) | `app.go` `Run` → `main` | **Yes** | Catastrophic |
-| **Panic** in worker or delivery goroutine | `Run`, `Deliver` | **Yes** | Would otherwise crash silently |
-
-The subtle one is the two kinds of "job failed." Today `a.finish(id,"failed",msg)`
-lumps them. The split hinges on **whether the transformer honored the contract**:
-a valid `result.json` with `success:false` = *handled* (no alert); a crash /
-timeout / missing / corrupt result = *systemic* (alert).
+## 2. What alerts
+
+| Failure | Code site | Alert? | Category | Notes |
+|---|---|---|---|---|
+| Forged/stale webhook (401), malformed/oversize body (400) | `webhook.go` `receive` | No | — | Bot/client noise |
+| Unrouted recipient, disallowed sender, no attachments, rejected file type | `receive` | No | — | Benign or sender's fault |
+| **Intake failure (500)** — storage/filesystem/Submit | `receive` → `intake` | **Yes** | `intake` | Mailgun retries ~8h; the throttle absorbs the retry burst |
+| `store(notify=)` route misconfiguration warning | `receive` | **Yes** | `route-config` | Today it's a log WARNING only; every real submission is being lost |
+| **Transformer missing from config** | `app.go` `execute` | **Yes** | `job-systemic` | Misconfiguration |
+| **Transformer timed out** | `execute` | **Yes** | `job-systemic` | |
+| **Missing/corrupt/invalid `result.json`** | `execute` → `readResult` | **Yes** | `job-systemic` | Contract violation |
+| **Nonzero exit without a valid `success:false` result** | `execute` | **Yes** | `job-systemic` | Crash |
+| Valid `result.json` with `success:false` | `execute` | No | — | Sender told in the reply |
+| **Panic while running a job** | `execute` (new `recover`) | **Yes** | `panic` | Job marked failed; worker continues |
+| **Reply send failing** (non-2xx, timeout) for ≥5 min | `outbound.go` `deliver` | **Yes** | `delivery` | Time-based, not per tick, to ride out a blip (§3.5) |
+| **`MarkEmailDelivered` failing after a successful send** | `deliver` | **Yes, no grace period** | `delivery-mark` | Resends the reply **every second** (#6). Most urgent alert in the table |
+| **Sheets-link publish failing** (token expired, quota, Drive outage) | `deliver` → `publish` | **Yes** | `publish` | Can persist for hours |
+| **Drive file orphaned** (`PutDelivery` failed after upload) | `publish` | **Yes** | `publish-orphan` | Names the file ID for manual cleanup |
+| **Job claim error** (`store.Next`) persisting ≥1 min | `app.go` `Run` | **Yes** | `worker-claim` | Today it retries silently forever; a stuck DB halts all work |
+| Retention sweep: Drive delete failures | `mailgun/retention.go` | **Yes** | `sweep-drive` | World-editable files outliving the 30-day promise |
+| Retention sweep: workspace delete failures | `app/retention.go` | **Yes** | `sweep-jobs` | Low urgency; throttled daily anyway |
+| **Worker restarted after a crash** | startup (via supervisor, §3.6) | **Yes** | `restart` | Includes exit code and rapid-restart count |
+| **Jobs interrupted** (running at the last shutdown) | `store.Open` marks them `interrupted` | **Yes, if N > 0** | `restart` | Second crash signal that needs no supervisor |
+| Startup `fatal()` (bad config, DB, incomplete env, bind failure) | `main.go` | **No** | — | No reporter exists yet; covered by heartbeat #5 |
+| Mailgun send itself failing | `send` | **Can't** | — | The alert channel is the broken thing; heartbeat #5 |
+
+**The job-failure split** (`execute`), decided by whether the transformer honored
+the contract:
+
+- `result.json` valid and `success:false` → handled. No alert, whatever the exit code.
+- `result.json` valid, `success:true`, but nonzero exit → systemic (the result contradicts the exit code).
+- Anything else that fails (timeout, missing/invalid result, missing transformer) → systemic.
 
 ---
 
 ## 3. Design
 
-### 3.1 A `Reporter` dependency (DI, testable)
+### 3.1 New package `internal/alert`
 
-Introduce an interface, following the `Engine` pattern already in the codebase:
+Neither `app` nor `mailgun` can own this: `mailgun` imports `app`, and both need to
+report. A leaf package both import:
 
 ```go
-type Reporter interface {
-    // Report records a systemic failure. Implementations must be non-blocking
-    // enough for hot paths, must throttle, and must never panic or recurse.
-    Report(category string, detail string, err error)
+type Alert struct {
+    Category string // throttle key, e.g. "job-systemic"
+    Summary  string // one line; becomes the subject
+    Detail   string // job id, operation, input name, error, truncated output
 }
-```
 
-- A **no-op reporter** is the default (alerting disabled).
-- A **Mailgun-backed reporter** emails `support@example.com` via the existing
-  send path, with throttling in front.
-- Both `*app.App` and the mailgun `Service` take a `Reporter` (constructor
-  injection), so job-side and email-side systemic failures funnel to one place.
-- Tests use a `fakeReporter` that records calls — per the project's
-  isolation-testing preference.
+// Reporter records a systemic failure. Report must not block or panic.
+type Reporter interface{ Report(Alert) }
 
-### 3.2 Throttle / dedupe (the mandatory middle layer)
+type Nop struct{}          // the default: alerting disabled
 
-A small stateful wrapper in front of the Mailgun reporter:
+// Mailer sends one plain-text message. *mailgun.Service satisfies it.
+type Mailer interface {
+    SendAlert(ctx context.Context, to, subject, text string) error
+}
 
-- **Per-category cooldown:** at most one email per `category` per window
-  (default ~15 min). The next email for a suppressed category includes a
-  "N more occurrences since last alert" count.
-- **Global cap:** a hard ceiling (e.g. ≤ N alert emails/hour) as a backstop.
-- Rationale: the delivery loop ticks every second, so an unthrottled delivery
-  failure = one email/second; a bot spraying 401s (already excluded) would be
-  worse still.
+// Ledger is the persisted throttle state. *store.Store satisfies it.
+type Ledger interface {
+    LastAlert(category string) (sentAt time.Time, suppressed int, ok bool, err error)
+    RecordAlertSent(category string, at time.Time) error // resets suppressed
+    RecordAlertSuppressed(category string) error         // suppressed++
+    AlertsSentSince(t time.Time) (int, error)            // for the global cap
+}
 
-### 3.3 The feedback loop
+// Emailer is the real Reporter: a throttle in front of a Mailer, draining a
+// buffered channel on its own goroutine so Report never blocks a hot path.
+func NewEmailer(m Mailer, l Ledger, to string, cfg Config, now func() time.Time, log *log.Logger) *Emailer
+func (e *Emailer) Run(ctx context.Context) // started by main; drains the queue
+```
 
-- The Mailgun reporter's own send failures are **log-only** — never re-reported.
-- "Mailgun send is failing" therefore may not reach `support@` at all (it needs
-  the very channel that's down). This is the structural gap that Phase 4's
-  heartbeat covers.
+- `Report` does a non-blocking send on a buffered channel (e.g. 64). If the buffer
+  is full, it logs and drops the alert. A flood that fills it would be throttled
+  anyway.
+- The throttle is **per-category cooldown** (default 15 min), plus a **global cap**
+  (default 10 emails/hour). A suppressed alert increments the count, and the next
+  email for that category says "N more since the last alert".
+- Clock, Mailer and Ledger are all injected, so the throttle is tested without
+  sleeping, SQLite or the network.
 
-### 3.4 Configuration
+### 3.2 Persistence (`internal/store`)
 
-Add to `config/email.yaml` (non-secret; secrets stay in env):
+Two small tables, created like the existing ones in `store.Open`:
 
-```yaml
-alert_recipient: support@example.com     # empty/absent => alerting disabled
-# alert_from: filemill@mill.example.com  # optional; defaults to REPLY_FROM
-# alert_cooldown_minutes: 15
+```sql
+CREATE TABLE IF NOT EXISTS alert_state (
+  category TEXT PRIMARY KEY, last_sent_at TEXT NOT NULL, suppressed INTEGER NOT NULL DEFAULT 0);
+CREATE TABLE IF NOT EXISTS alert_sends (sent_at TEXT NOT NULL); -- pruned to 24h on write
 ```
 
-Alerting is enabled only when `alert_recipient` is set and Mailgun is configured.
-
-### 3.5 Alert content
+`alert_sends` exists only for the global cap. `alert_state` alone can't count sends
+per hour.
+
+### 3.3 Mailgun changes
+
+- `send` currently applies `replySubject`, which adds a "Re:" prefix, inside the
+  function. Move that to the caller (`deliver`) so `send` takes its subject
+  verbatim. Then add `SendAlert(ctx, to, subject, text)` as a thin call to `send`
+  with no attachments and no threading headers.
+- Alerts come from `REPLY_FROM`. Subject: `[FileMill] <summary>`.
+- `Service` gets a `reporter alert.Reporter` field that defaults to `alert.Nop{}`,
+  plus `SetReporter`.
+- `fileConfig` gets `alert_recipient` (empty means disabled),
+  `alert_cooldown_minutes` and `alert_max_per_hour`.
+
+### 3.4 App changes
+
+- `App` gets `reporter alert.Reporter` (default `Nop`) and `SetReporter`. It's a
+  setter, not a constructor argument, because `app.Open` runs before the Mailgun
+  service that the real reporter needs exists.
+- `execute` is reshaped around the split in §2. Systemic branches call `Report` with
+  the job ID, operation, input name, message and the last 2 KB of transformer
+  output (already captured by `CombinedOutput`).
+- A `defer`/`recover` around each job marks the job failed, reports `panic` with
+  the stack, and lets `Run` continue.
+- `Run` tracks the time of the first consecutive claim error and reports
+  `worker-claim` once it has persisted for ≥1 min.
+
+### 3.5 Delivery failure sensitivity
+
+A 3-tick grace period is 3 seconds, which isn't a blip ride-out. Instead
+`Service` keeps an in-memory `firstFailure map[int64]time.Time` keyed by
+submission ID. It is set on the first failure, cleared on success, and alerts
+(`delivery` or `publish`) once a submission has failed for ≥5 min.
+`MarkEmailDelivered` failures skip the grace period, because each second of delay
+is another duplicate reply.
+
+This map is the natural first step toward #6's per-submission failure count and
+dead-letter state. Building #6's retry cap at the same time would remove the
+resend storm that `delivery-mark` exists to report.
+
+### 3.6 Reporting crashes across a restart
+
+The supervisor can't send email, and a crashed process can't report itself. So:
+
+1. `Supervise-FileMill.ps1` sets `FILEMILL_PREVIOUS_EXIT` (the last exit code;
+   unset on the first launch) and `FILEMILL_RAPID_RESTARTS` in the child's
+   environment before each launch.
+2. At startup, once the reporter is wired, `run` reports `restart` if
+   `FILEMILL_PREVIOUS_EXIT` is set. The summary is "restarted after exit code N",
+   plus "(crash-loop: K rapid restarts)" when K ≥ 4, the supervisor's existing
+   threshold.
+3. `store.Open` already marks any `running` job `interrupted`. Return the count
+   from `Open` (or expose it) and fold it into the same `restart` alert.
+
+Because the throttle is persisted (§3.2), a crash-loop sends **one** `restart`
+alert per cooldown with a suppressed count, not one per restart. The case this
+can't cover is a worker that dies before the reporter is wired (startup `fatal`),
+which is left to heartbeat #5.
+
+### 3.7 Wiring in `main.go` (`run`, continuous mode only)
 
-Subject: `[FileMill] <category>`. Body: timestamp, category, job ID / operation /
-input filename where relevant, the error text, and the transformer's captured
-stderr for job crashes. **Open decision:** stderr can contain fragments of the
-input file — decide whether to include it or a truncated/redacted form.
-
----
+```
+app.Open → mailgun.Load → if alert_recipient set:
+    emailer := alert.NewEmailer(mail, store, …); go emailer.Run(ctx)
+    application.SetReporter(emailer); mail.SetReporter(emailer)
+    report restart / interrupted jobs (§3.6)
+```
 
-## 4. Phasing
-
-Each phase builds and tests independently.
-
-- **Phase 0 — `Reporter` interface + no-op + `fakeReporter` + isolation tests.**
-  Inject into `App` and `Service`; default no-op; nothing emails yet.
-- **Phase 1 — split systemic vs handled in `execute`.** Call `reporter.Report`
-  at the systemic branches only (transformer-missing, timeout, missing/corrupt
-  result, crash-without-result). No email yet — verify via `fakeReporter` that
-  *only* systemic paths report and handled `success:false` does not. This is
-  good design independent of alerting.
-- **Phase 2 — Mailgun-backed reporter + config + throttle/dedupe.** Reuse the
-  `send` path (with the `sendBase` injection already present for testing).
-  Isolation-test the throttle: N rapid identical reports → one email with a
-  suppressed-count.
-- **Phase 3 — wire the remaining sites:** delivery failures (throttled),
-  `recover()` in the `Run` and `Deliver` goroutines (report then continue/exit
-  cleanly instead of crashing), and the worker-loop fatal in `main`.
-- **Phase 4 (adjacent, optional) — external heartbeat.** FileMill pings a
-  dead-man's-switch (e.g. healthchecks.io) every few minutes; if the ping stops,
-  *that* service emails you. This is the only thing that catches "the process is
-  dead / the laptop slept," which self-sent email cannot. Mostly external setup
-  plus a small ticker in `run`.
-
-Partial coverage to accept: **startup `fatal()`s** (bad config, can't open the
-DB, incomplete Mailgun env) happen *before* the reporter exists, so they won't
-email unless we special-case loading just enough config early to send one
-message before exit. Deferred; the heartbeat covers "never started" anyway.
+`--once` mode and a Mailgun-less configuration keep `Nop`. `emailer.Run` should
+drain the queue on shutdown within the existing 10s window, so an alert raised
+during shutdown still goes out.
 
 ---
 
-## 5. Scope estimate
-
-- Phase 0–1: ~0.5–1 day (interface, split, tests).
-- Phase 2: ~1 day (reporter, config, throttle — the throttle is the bulk).
-- Phase 3: ~0.5 day (wiring + panic recovery).
-- Phase 4: mostly external config + a small ticker.
-
-Roughly 2–3 days for something trustworthy. The sending is minutes; the
-taxonomy split, throttle, and panic-recovery are the real work.
+## 4. Phases (one PR each; tests with fakes written first, per project convention)
+
+**Phase 1 — `internal/alert` core.**
+- Write `fakeMailer`, `fakeLedger` and a fake clock, then the tests, then
+  `Emailer`.
+- Tests:
+  - N identical reports inside the cooldown → 1 email; the next email after the cooldown carries a suppressed count of N-1.
+  - Different categories don't suppress each other.
+  - The global cap holds across categories.
+  - A failed send is logged, not retried or re-reported.
+  - `Report` never blocks when the queue is full.
+  - Throttle state survives a new `Emailer` built over the same ledger, simulating a restart.
+- Real `Ledger` in `store`, with its own tests against a temp DB.
+- **Nothing is wired yet**, so there is no behavior change.
+
+**Phase 2 — job taxonomy split.**
+- `App.SetReporter` plus a `fakeReporter` in the `app` tests.
+- Table-driven `execute` tests use tiny fake transformer scripts: clean
+  `success:false`, `success:false` with nonzero exit, timeout, missing
+  `result.json`, invalid contract version, `success:true` with nonzero exit,
+  missing transformer, and panic recovery.
+- The assertion is that *only* the systemic cases report, and job statuses and
+  messages are unchanged from today.
+
+**Phase 3 — Mailgun sites and wiring.**
+- Move `replySubject` to the caller and add `SendAlert`.
+- Config fields, `Service.SetReporter`, and the 5-minute delivery and publish
+  grace period.
+- Immediate `delivery-mark`, `publish-orphan`, `intake`, `route-config`, and both
+  sweeps.
+- Wire everything in `main.go`, and add `alert_recipient` to
+  `email.yaml.example`.
+- Tests go through the existing `fakeEngine` plus a `fakeReporter`.
+- **Alerting goes live here.**
+
+**Phase 4 — crash reporting across restarts.**
+- Supervisor env vars, the startup `restart` report, and the interrupted-job count
+  from `store.Open`.
+- Add `recover` to the `Deliver` tick and both sweep loops; each reports `panic`
+  and keeps looping.
+- Remove the "log-only; alerting tracked in issue #7" wording from the supervisor
+  script and README.
+- Verify by hand: kill the worker from an elevated shell and confirm that one
+  `restart` email arrives, and that a forced crash-loop still produces one.
+
+**Phase 5 — live verification.**
+- Set the real `alert_recipient` in the gitignored `config/email.yaml`.
+- Trigger one alert per sink: a transformer that exits 1 with no result, a bad
+  Mailgun domain for delivery, and a restart.
+- Confirm the throttle by forcing a repeated failure for 20 minutes and expecting
+  two emails, the second carrying a suppressed count.
+
+**Estimate:** about 3 days. Phase 1 (the throttle and persistence) is the bulk.
+Phases 2 and 3 are about 1 day together. Phase 4 is half a day.
 
 ---
 
-## 6. Open decisions (resolve before Phase 2)
-
-1. **Cooldown/digest style:** per-category cooldown with suppressed-count
-   (recommended) vs a periodic digest email vs alert-on-every-occurrence
-   (rejected — spam).
-2. **Transformer stderr in alert bodies:** include (best for diagnosis) vs
-   omit/redact (may echo input-file content).
-3. **From address:** reuse `REPLY_FROM` (`filemill@`) vs a dedicated `alerts@`.
-4. **Heartbeat (Phase 4):** in scope now, or a separate later effort?
-5. **Delivery-failure sensitivity:** how many consecutive failures before the
-   first alert (immediate vs after 2–3 ticks, to ride out a blip)?
+## 5. Decisions (recommended defaults; change before Phase 3 if needed)
+
+1. **Throttle style:** per-category cooldown (15 min) with suppressed counts, plus a
+   global cap (10/hour). No digest emails.
+2. **Transformer output in alerts:** include the last 2 KB. The recipient is the
+   operator, who already holds the input files under `data/jobs/`, so this reveals
+   nothing new to them. It is also the most useful diagnostic.
+3. **From address:** reuse `REPLY_FROM`. No new Mailgun sender to set up.
+4. **Delivery sensitivity:** 5 min of continuous failure per submission; immediate
+   for `delivery-mark`.
+5. **Heartbeat:** out of scope here, tracked as #5. It is the only cover for
+   startup fatals, a Mailgun outage, and a machine that is off.
+6. **Ordering with #6:** do #6's retry cap before or alongside Phase 3. Otherwise
+   the first real `delivery-mark` alert arrives *after* the sender has already had
+   dozens of duplicate replies.

From 339d1a32bc277872ebce289a4fed6f42d94a2d3e Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 05:14:32 -0600
Subject: [PATCH 02/25] Cap operator alerts at 20 a day in the alerting plan

The account is on Mailgun's Free plan: 100 sends a day, shared with
replies, hard-rejected past the limit. The hourly cap alone would allow
240 alerts a day, enough to lock out every reply to senders. A persisted
rolling 24h cap of 20 keeps alerts to a fifth of the daily budget, even
across a crash-loop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 ERROR-ALERTING-PLAN.md | 36 +++++++++++++++++++++++++++---------
 1 file changed, 27 insertions(+), 9 deletions(-)

diff --git a/ERROR-ALERTING-PLAN.md b/ERROR-ALERTING-PLAN.md
index 993b45f..3b3b28d 100644
--- a/ERROR-ALERTING-PLAN.md
+++ b/ERROR-ALERTING-PLAN.md
@@ -109,9 +109,16 @@ func (e *Emailer) Run(ctx context.Context) // started by main; drains the queue
 - `Report` does a non-blocking send on a buffered channel (e.g. 64). If the buffer
   is full, it logs and drops the alert. A flood that fills it would be throttled
   anyway.
-- The throttle is **per-category cooldown** (default 15 min), plus a **global cap**
-  (default 10 emails/hour). A suppressed alert increments the count, and the next
-  email for that category says "N more since the last alert".
+- The throttle is **per-category cooldown** (default 15 min), plus two **global
+  caps**: 10 emails/hour and **20 emails/day** (a rolling 24h window). A
+  suppressed alert increments the count, and the next email for that category
+  says "N more since the last alert".
+- **The daily cap is the binding one.** The Mailgun Free plan allows 100 sends a
+  day, *shared with replies*, and hard-rejects past that until the next day. The
+  hourly cap alone would allow 240/day, enough to lock out every reply to senders.
+  20/day keeps alerts to at most a fifth of the day's budget. When the daily cap
+  is hit, one last alert says so ("daily alert cap reached; further alerts are
+  logged only until <time>"); it counts within the 20, so the cap stays exact.
 - Clock, Mailer and Ledger are all injected, so the throttle is tested without
   sleeping, SQLite or the network.
 
@@ -125,8 +132,10 @@ CREATE TABLE IF NOT EXISTS alert_state (
 CREATE TABLE IF NOT EXISTS alert_sends (sent_at TEXT NOT NULL); -- pruned to 24h on write
 ```
 
-`alert_sends` exists only for the global cap. `alert_state` alone can't count sends
-per hour.
+`alert_sends` exists only for the global caps. `alert_state` alone can't count sends
+per hour or per day. Keeping 24h of rows covers both windows. Persisting matters
+most for the daily cap: a crash-looping worker that reset it on every restart
+could spend the whole Free-plan budget.
 
 ### 3.3 Mailgun changes
 
@@ -138,7 +147,8 @@ per hour.
 - `Service` gets a `reporter alert.Reporter` field that defaults to `alert.Nop{}`,
   plus `SetReporter`.
 - `fileConfig` gets `alert_recipient` (empty means disabled),
-  `alert_cooldown_minutes` and `alert_max_per_hour`.
+  `alert_cooldown_minutes`, `alert_max_per_hour` (default 10) and
+  `alert_max_per_day` (default 20).
 
 ### 3.4 App changes
 
@@ -208,7 +218,12 @@ during shutdown still goes out.
 - Tests:
   - N identical reports inside the cooldown → 1 email; the next email after the cooldown carries a suppressed count of N-1.
   - Different categories don't suppress each other.
-  - The global cap holds across categories.
+  - The hourly cap holds across categories.
+  - The daily cap holds across categories and across hours: the 21st alert in a
+    rolling 24h is suppressed even when the hourly cap would allow it. The 20th
+    is the "daily cap reached" notice. The cap reopens as the oldest send leaves
+    the window.
+  - The daily cap survives a restart (a new `Emailer` over the same ledger).
   - A failed send is logged, not retried or re-reported.
   - `Report` never blocks when the queue is full.
   - Throttle state survives a new `Emailer` built over the same ledger, simulating a restart.
@@ -259,8 +274,11 @@ Phases 2 and 3 are about 1 day together. Phase 4 is half a day.
 
 ## 5. Decisions (recommended defaults; change before Phase 3 if needed)
 
-1. **Throttle style:** per-category cooldown (15 min) with suppressed counts, plus a
-   global cap (10/hour). No digest emails.
+1. **Throttle style:** per-category cooldown (15 min) with suppressed counts, plus
+   global caps of 10/hour and **20/day**. No digest emails. The daily cap is
+   set by the Mailgun **Free plan** (100 sends/day shared with replies, hard
+   stop at the limit). Revisit it if the account moves to a paid plan, where
+   alert volume is a rounding error in the bill.
 2. **Transformer output in alerts:** include the last 2 KB. The recipient is the
    operator, who already holds the input files under `data/jobs/`, so this reveals
    nothing new to them. It is also the most useful diagnostic.

From e8568f7555e810cb0f37599be8dc81f455efd6e6 Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 05:28:57 -0600
Subject: [PATCH 03/25] Add the internal/alert core: reporter, throttle and
 queue

Phase 1 of the error-alerting plan (#7). Emailer is a Reporter that queues
alerts on a buffered channel drained by Run, and throttles them before they
reach a Mailer: a 15-minute cooldown per category, 10 sends an hour and 20 in
a rolling 24h across categories. The 20th send in the window says the daily
cap was reached. Suppressed alerts are logged and counted into the next email
in their category.

Throttle state lives behind a Ledger so it survives a worker restart. Nothing
is wired in yet, so there is no behavior change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 internal/alert/alert.go        |  50 +++++
 internal/alert/emailer.go      | 221 ++++++++++++++++++++++
 internal/alert/emailer_test.go | 330 +++++++++++++++++++++++++++++++++
 internal/alert/fake_test.go    | 177 ++++++++++++++++++
 4 files changed, 778 insertions(+)
 create mode 100644 internal/alert/alert.go
 create mode 100644 internal/alert/emailer.go
 create mode 100644 internal/alert/emailer_test.go
 create mode 100644 internal/alert/fake_test.go

diff --git a/internal/alert/alert.go b/internal/alert/alert.go
new file mode 100644
index 0000000..5c854a1
--- /dev/null
+++ b/internal/alert/alert.go
@@ -0,0 +1,50 @@
+// Package alert emails the operator when FileMill fails systemically.
+//
+// It is a leaf package because both app and mailgun report, and mailgun
+// imports app, so neither can own it. See ERROR-ALERTING-PLAN.md.
+package alert
+
+import (
+	"context"
+	"time"
+)
+
+// Alert is one systemic failure.
+type Alert struct {
+	Category string // throttle key, e.g. "job-systemic"
+	Summary  string // one line; becomes the subject
+	Detail   string // job id, operation, input name, error, truncated output
+}
+
+// Reporter records a systemic failure. Report must not block or panic.
+type Reporter interface{ Report(Alert) }
+
+// Nop is the default Reporter: alerting disabled.
+type Nop struct{}
+
+func (Nop) Report(Alert) {}
+
+// Mailer sends one plain-text message with its subject verbatim.
+// *mailgun.Service satisfies it.
+type Mailer interface {
+	SendAlert(ctx context.Context, to, subject, text string) error
+}
+
+// Ledger is the persisted throttle state. *store.Store satisfies it.
+//
+// It lives in the database, not in memory, because the supervisor restarts a
+// crashing worker every ≤120s and an in-memory throttle would reset each time.
+type Ledger interface {
+	// LastAlert returns when category last sent and how many of its alerts
+	// have been suppressed since. A zero sentAt means it has never sent; it
+	// can still have a suppressed count, held back by a global cap.
+	LastAlert(category string) (sentAt time.Time, suppressed int, err error)
+	// RecordAlertSent records a send at at and resets category's suppressed
+	// count.
+	RecordAlertSent(category string, at time.Time) error
+	// RecordAlertSuppressed increments category's suppressed count.
+	RecordAlertSuppressed(category string) error
+	// AlertSendsSince returns the time of every send after t, in any
+	// category, oldest first. It must cover at least the last 24h.
+	AlertSendsSince(t time.Time) ([]time.Time, error)
+}
diff --git a/internal/alert/emailer.go b/internal/alert/emailer.go
new file mode 100644
index 0000000..1980e0d
--- /dev/null
+++ b/internal/alert/emailer.go
@@ -0,0 +1,221 @@
+package alert
+
+import (
+	"context"
+	"fmt"
+	"log"
+	"slices"
+	"strings"
+	"time"
+)
+
+// Throttle defaults. The daily cap is the binding one: the Mailgun Free plan
+// allows 100 sends a day, shared with replies, and rejects every send past
+// that until the next day. The hourly cap alone would allow 240. 20 keeps
+// alerts to a fifth of the budget.
+const (
+	DefaultCooldown   = 15 * time.Minute
+	DefaultMaxPerHour = 10
+	DefaultMaxPerDay  = 20
+)
+
+const (
+	// day is the rolling window of the daily cap.
+	day = 24 * time.Hour
+	// queueSize bounds the alerts waiting for Run. A flood that fills it
+	// would be throttled anyway, so Report drops rather than waits.
+	queueSize = 64
+	// drainTimeout bounds how long Run keeps sending after ctx is cancelled,
+	// inside main's 10s shutdown window.
+	drainTimeout  = 5 * time.Second
+	subjectPrefix = "[FileMill] "
+)
+
+// Config tunes the throttle. A zero or negative field takes its default.
+type Config struct {
+	Cooldown   time.Duration // per category
+	MaxPerHour int           // across categories
+	MaxPerDay  int           // across categories, rolling 24h
+}
+
+func (c Config) withDefaults() Config {
+	if c.Cooldown <= 0 {
+		c.Cooldown = DefaultCooldown
+	}
+	if c.MaxPerHour <= 0 {
+		c.MaxPerHour = DefaultMaxPerHour
+	}
+	if c.MaxPerDay <= 0 {
+		c.MaxPerDay = DefaultMaxPerDay
+	}
+	return c
+}
+
+// Emailer is the real Reporter: a throttle in front of a Mailer, draining a
+// buffered channel on its own goroutine so Report never blocks a hot path.
+//
+// Every alert passes a per-category cooldown and two global caps, all read
+// from the Ledger. A suppressed alert is logged and counted, and the next
+// email in its category says how many were held back.
+type Emailer struct {
+	mailer Mailer
+	ledger Ledger
+	to     string
+	cfg    Config
+	now    func() time.Time
+	log    *log.Logger
+	queue  chan Alert
+}
+
+func NewEmailer(m Mailer, l Ledger, to string, cfg Config, now func() time.Time, log *log.Logger) *Emailer {
+	return &Emailer{
+		mailer: m, ledger: l, to: to, cfg: cfg.withDefaults(), now: now, log: log,
+		queue: make(chan Alert, queueSize),
+	}
+}
+
+// Report queues a for Run and returns at once. A full queue drops the alert.
+func (e *Emailer) Report(a Alert) {
+	select {
+	case e.queue <- a:
+	default:
+		e.log.Printf("alert %s: queue full; dropped: %s", a.Category, a.Summary)
+	}
+}
+
+// Run sends queued alerts until ctx is cancelled, then sends whatever is still
+// queued, for up to drainTimeout, so an alert raised during shutdown still
+// goes out. It returns when the drain is done.
+func (e *Emailer) Run(ctx context.Context) {
+	for {
+		select {
+		case <-ctx.Done():
+			e.drain(ctx)
+			return
+		case a := <-e.queue:
+			e.handle(ctx, a)
+		}
+	}
+}
+
+func (e *Emailer) drain(ctx context.Context) {
+	ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), drainTimeout)
+	defer cancel()
+	for {
+		select {
+		case a := <-e.queue:
+			if ctx.Err() != nil {
+				e.log.Printf("alert %s: shutdown drain timed out; dropped: %s", a.Category, a.Summary)
+				continue
+			}
+			e.handle(ctx, a)
+		default:
+			return
+		}
+	}
+}
+
+// handle throttles one alert and sends it if it passes.
+//
+// The send is recorded before it is attempted, and a failed send is logged
+// and dropped, never retried or re-reported. It still counts against the
+// caps: a send that timed out may have reached Mailgun and been charged. For
+// the same reason the alert is dropped when the Ledger can't be read or
+// written: without it the caps can't be enforced.
+func (e *Emailer) handle(ctx context.Context, a Alert) {
+	defer func() {
+		if r := recover(); r != nil {
+			e.log.Printf("alert %s: panic while sending: %v; dropped: %s", a.Category, r, a.Summary)
+		}
+	}()
+
+	now := e.now()
+	sentAt, suppressed, err := e.ledger.LastAlert(a.Category)
+	if err != nil {
+		e.log.Printf("alert %s: read throttle state: %v; dropped: %s", a.Category, err, a.Summary)
+		return
+	}
+	if !sentAt.IsZero() && now.Sub(sentAt) < e.cfg.Cooldown {
+		e.suppress(a, "cooldown")
+		return
+	}
+	sends, err := e.ledger.AlertSendsSince(now.Add(-day))
+	if err != nil {
+		e.log.Printf("alert %s: read send history: %v; dropped: %s", a.Category, err, a.Summary)
+		return
+	}
+	if sentAfter(sends, now.Add(-time.Hour)) >= e.cfg.MaxPerHour {
+		e.suppress(a, "hourly cap")
+		return
+	}
+	if len(sends) >= e.cfg.MaxPerDay {
+		e.suppress(a, "daily cap")
+		return
+	}
+
+	// The send that fills the daily cap says so. It counts within the cap, so
+	// the cap stays exact, and the slot reopens when the oldest send in the
+	// window, possibly this one, leaves it.
+	capReopens := time.Time{}
+	if len(sends) == e.cfg.MaxPerDay-1 {
+		capReopens = slices.MinFunc(append(sends, now), time.Time.Compare).Add(day)
+	}
+	subject, text := e.compose(a, suppressed, capReopens)
+
+	if err := e.ledger.RecordAlertSent(a.Category, now); err != nil {
+		e.log.Printf("alert %s: record send: %v; dropped: %s", a.Category, err, a.Summary)
+		return
+	}
+	if err := e.mailer.SendAlert(ctx, e.to, subject, text); err != nil {
+		e.log.Printf("alert %s: send failed: %v; dropped: %s", a.Category, err, a.Summary)
+		return
+	}
+	e.log.Printf("alert %s: sent: %s", a.Category, a.Summary)
+}
+
+// suppress logs a held-back alert and counts it against its category.
+func (e *Emailer) suppress(a Alert, reason string) {
+	e.log.Printf("alert %s: suppressed (%s): %s", a.Category, reason, a.Summary)
+	if err := e.ledger.RecordAlertSuppressed(a.Category); err != nil {
+		e.log.Printf("alert %s: record suppression: %v", a.Category, err)
+	}
+}
+
+// compose builds the email. A non-zero capReopens marks it as the cap notice.
+func (e *Emailer) compose(a Alert, suppressed int, capReopens time.Time) (subject, text string) {
+	summary := oneLine(a.Summary)
+	if summary == "" {
+		summary = a.Category
+	}
+	subject = subjectPrefix + summary
+
+	var b strings.Builder
+	if !capReopens.IsZero() {
+		subject += " (daily alert cap reached)"
+		fmt.Fprintf(&b, "Daily alert cap reached: this is alert %d of %d in 24 hours. Further alerts are logged only until %s.\n\n",
+			e.cfg.MaxPerDay, e.cfg.MaxPerDay, capReopens.Format("2006-01-02 15:04 MST"))
+	}
+	fmt.Fprintf(&b, "%s\n\nCategory: %s\n", summary, a.Category)
+	if a.Detail != "" {
+		fmt.Fprintf(&b, "\n%s\n", strings.TrimRight(a.Detail, "\n"))
+	}
+	if suppressed > 0 {
+		fmt.Fprintf(&b, "\n%d more since the last alert in this category, suppressed by the throttle.\n", suppressed)
+	}
+	return subject, b.String()
+}
+
+// sentAfter counts the sends after t.
+func sentAfter(sends []time.Time, t time.Time) int {
+	n := 0
+	for _, at := range sends {
+		if at.After(t) {
+			n++
+		}
+	}
+	return n
+}
+
+// oneLine collapses whitespace, newlines included, so a summary carrying an
+// error's text still makes a single subject line.
+func oneLine(s string) string { return strings.Join(strings.Fields(s), " ") }
diff --git a/internal/alert/emailer_test.go b/internal/alert/emailer_test.go
new file mode 100644
index 0000000..3282036
--- /dev/null
+++ b/internal/alert/emailer_test.go
@@ -0,0 +1,330 @@
+package alert
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"strings"
+	"testing"
+	"time"
+)
+
+var (
+	_ Reporter = Nop{}
+	_ Reporter = (*Emailer)(nil)
+)
+
+// capNotice marks the one email that says the daily cap was reached.
+const capNotice = "daily alert cap reached"
+
+// Repeats inside the cooldown send nothing, but are counted, and the next
+// email for the category says how many were held back.
+func TestCooldownSuppressesRepeatsAndCountsThem(t *testing.T) {
+	h := newHarness(t)
+
+	for i := range 5 {
+		h.report("job-systemic", "transformer timed out")
+		if i < 4 {
+			h.clock.Advance(time.Minute)
+		}
+	}
+	sent := h.mailer.sends()
+	if len(sent) != 1 {
+		t.Fatalf("5 reports inside the cooldown sent %d emails, want 1", len(sent))
+	}
+	if sent[0].to != testRecipient || sent[0].subject != "[FileMill] transformer timed out" {
+		t.Fatalf("first email = to %q, subject %q", sent[0].to, sent[0].subject)
+	}
+	if strings.Contains(sent[0].text, "more since") {
+		t.Errorf("first email claims suppressed alerts:\n%s", sent[0].text)
+	}
+
+	// The cooldown runs from the first send; 11 more minutes reaches it exactly.
+	h.clock.Advance(11 * time.Minute)
+	h.report("job-systemic", "transformer timed out")
+	sent = h.mailer.sends()
+	if len(sent) != 2 {
+		t.Fatalf("a report after the cooldown sent %d emails in total, want 2", len(sent))
+	}
+	if !strings.Contains(sent[1].text, "4 more since the last alert") {
+		t.Errorf("second email should carry the 4 suppressed reports:\n%s", sent[1].text)
+	}
+
+	// The send reset the count.
+	h.clock.Advance(15 * time.Minute)
+	h.report("job-systemic", "transformer timed out")
+	if sent = h.mailer.sends(); strings.Contains(sent[2].text, "more since") {
+		t.Errorf("third email repeats a count already reported:\n%s", sent[2].text)
+	}
+}
+
+func TestCategoriesDoNotSuppressEachOther(t *testing.T) {
+	h := newHarness(t)
+
+	h.report("job-systemic", "transformer crashed")
+	h.report("delivery", "reply send failing")
+	h.report("restart", "restarted after exit code 1")
+
+	if sent := h.mailer.sends(); len(sent) != 3 {
+		t.Fatalf("3 categories at once sent %d emails, want 3", len(sent))
+	}
+}
+
+// The hourly cap counts sends in every category together, and an alert it
+// holds back is counted against its category like a cooldown suppression,
+// even though that category has never sent.
+func TestHourlyCapHoldsAcrossCategories(t *testing.T) {
+	h := newHarness(t)
+
+	for i := range 11 {
+		h.report(fmt.Sprintf("cat-%02d", i), "failure")
+	}
+	if sent := h.mailer.sends(); len(sent) != 10 {
+		t.Fatalf("11 categories in one hour sent %d emails, want 10", len(sent))
+	}
+	if !strings.Contains(h.logs.String(), "hourly cap") {
+		t.Errorf("the capped alert should be logged; log:\n%s", h.logs)
+	}
+
+	h.clock.Advance(time.Hour)
+	h.report("cat-10", "failure")
+	sent := h.mailer.sends()
+	if len(sent) != 11 {
+		t.Fatalf("an hour later the cap should reopen; %d emails in total, want 11", len(sent))
+	}
+	if !strings.Contains(sent[10].text, "1 more since the last alert") {
+		t.Errorf("the reopened category should carry its capped alert:\n%s", sent[10].text)
+	}
+}
+
+// The daily cap binds even when the hourly one would allow a send. The 20th
+// email in a rolling 24h is the cap notice, and as the oldest send leaves the
+// window one slot reopens, whose send is again the notice.
+func TestDailyCapHoldsAcrossCategoriesAndHours(t *testing.T) {
+	h := newHarness(t)
+	start := h.clock.Now()
+
+	// 20 categories, 10 minutes apart: 6 an hour, well under the hourly cap.
+	for i := range 20 {
+		h.report(fmt.Sprintf("cat-%02d", i), fmt.Sprintf("failure %d", i))
+		h.clock.Advance(10 * time.Minute)
+	}
+	sent := h.mailer.sends()
+	if len(sent) != 20 {
+		t.Fatalf("20 alerts across 3+ hours sent %d emails, want 20", len(sent))
+	}
+	for i, s := range sent[:19] {
+		if strings.Contains(s.subject, capNotice) {
+			t.Fatalf("email %d is marked as the cap notice: %q", i+1, s.subject)
+		}
+	}
+	notice := sent[19]
+	if notice.subject != "[FileMill] failure 19 ("+capNotice+")" {
+		t.Errorf("20th subject = %q, want the alert marked as the cap notice", notice.subject)
+	}
+	// The oldest send leaves the window 24h after it went out.
+	until := start.Add(24 * time.Hour).Format("2006-01-02 15:04 MST")
+	if !strings.Contains(notice.text, "logged only until "+until) {
+		t.Errorf("notice should say alerts are logged until %s:\n%s", until, notice.text)
+	}
+	if !strings.Contains(notice.text, "detail for failure 19") {
+		t.Errorf("the notice must still carry the alert it replaced:\n%s", notice.text)
+	}
+
+	h.report("cat-20", "failure 20")
+	if n := len(h.mailer.sends()); n != 20 {
+		t.Fatalf("the 21st alert in 24h was sent (%d emails); the daily cap must hold", n)
+	}
+	if !strings.Contains(h.logs.String(), "daily cap") {
+		t.Errorf("the capped alert should be logged; log:\n%s", h.logs)
+	}
+
+	// Exactly 24h after the first send it has left the window: one slot.
+	h.clock.Advance(start.Add(24 * time.Hour).Sub(h.clock.Now()))
+	h.report("cat-21", "failure 21")
+	h.report("cat-22", "failure 22")
+	sent = h.mailer.sends()
+	if len(sent) != 21 {
+		t.Fatalf("with one slot reopened, %d emails in total, want 21", len(sent))
+	}
+	if !strings.Contains(sent[20].subject, capNotice) {
+		t.Errorf("the send filling the reopened slot is the 20th in the window, so the notice: %q", sent[20].subject)
+	}
+}
+
+// A crash-looping worker gets a new Emailer on every restart. Reading the cap
+// from the ledger, not from memory, is what keeps it from spending the whole
+// Mailgun budget.
+func TestDailyCapSurvivesRestart(t *testing.T) {
+	h := newHarness(t)
+	for i := range 20 {
+		h.report(fmt.Sprintf("cat-%02d", i), "failure")
+		h.clock.Advance(10 * time.Minute)
+	}
+
+	h.restart()
+	h.report("restart", "restarted after exit code 2")
+
+	if n := len(h.mailer.sends()); n != 20 {
+		t.Fatalf("a restarted Emailer sent past the daily cap: %d emails, want 20", n)
+	}
+}
+
+func TestCooldownSurvivesRestart(t *testing.T) {
+	h := newHarness(t)
+	h.report("restart", "restarted after exit code 1")
+
+	for range 3 {
+		h.clock.Advance(2 * time.Minute)
+		h.restart()
+		h.report("restart", "restarted after exit code 1")
+	}
+	if n := len(h.mailer.sends()); n != 1 {
+		t.Fatalf("restarts inside the cooldown sent %d emails, want 1", n)
+	}
+
+	h.clock.Advance(15 * time.Minute)
+	h.restart()
+	h.report("restart", "restarted after exit code 1")
+	sent := h.mailer.sends()
+	if len(sent) != 2 || !strings.Contains(sent[1].text, "3 more since the last alert") {
+		t.Fatalf("after the cooldown, want a 2nd email carrying 3 suppressed restarts; got %d emails:\n%v", len(sent), sent)
+	}
+}
+
+// A failed send is logged and dropped. It still counts against the caps: a
+// send that timed out may have been delivered and charged by Mailgun anyway.
+func TestFailedSendIsLoggedNotRetried(t *testing.T) {
+	h := newHarness(t)
+	h.mailer.err = errors.New("mailgun: 500 Internal Server Error")
+
+	h.report("delivery", "reply send failing")
+
+	if n := len(h.mailer.sends()); n != 1 {
+		t.Fatalf("send attempts = %d, want exactly 1", n)
+	}
+	if !strings.Contains(h.logs.String(), "500 Internal Server Error") {
+		t.Errorf("the failure should be logged; log:\n%s", h.logs)
+	}
+	if n := len(h.e.queue); n != 0 {
+		t.Errorf("a failed send was re-reported: %d alerts queued", n)
+	}
+	if n := len(h.ledger.sent); n != 1 {
+		t.Errorf("recorded sends = %d, want 1 (the attempt counts)", n)
+	}
+
+	h.report("delivery", "reply send failing")
+	if n := len(h.mailer.sends()); n != 1 {
+		t.Fatalf("a failed alert was retried: %d attempts", n)
+	}
+}
+
+// Without throttle state the caps can't be enforced, so the alert is dropped
+// rather than sent blind.
+func TestLedgerErrorDropsAlert(t *testing.T) {
+	h := newHarness(t)
+	h.ledger.err = errors.New("database is locked")
+
+	h.report("worker-claim", "job claim failing")
+
+	if n := len(h.mailer.sends()); n != 0 {
+		t.Fatalf("sent %d emails without throttle state, want 0", n)
+	}
+	if !strings.Contains(h.logs.String(), "database is locked") {
+		t.Errorf("the ledger error should be logged; log:\n%s", h.logs)
+	}
+}
+
+func TestMailerPanicIsContained(t *testing.T) {
+	h := newHarness(t)
+	h.mailer.panicWith = "boom"
+
+	h.report("publish", "sheets publish failing")
+
+	if !strings.Contains(h.logs.String(), "boom") {
+		t.Errorf("the panic should be logged; log:\n%s", h.logs)
+	}
+	h.mailer.panicWith = nil
+	h.report("intake", "intake failing")
+	if n := len(h.mailer.sends()); n != 2 {
+		t.Fatalf("the Emailer should keep working after a panic; %d sends, want 2", n)
+	}
+}
+
+func TestReportNeverBlocksWhenQueueIsFull(t *testing.T) {
+	h := newHarness(t) // Run is never started, so nothing drains the queue
+	extra := 5
+
+	done := make(chan struct{})
+	go func() {
+		defer close(done)
+		for i := range cap(h.e.queue) + extra {
+			h.e.Report(Alert{Category: "intake", Summary: fmt.Sprintf("intake failing %d", i)})
+		}
+	}()
+	select {
+	case <-done:
+	case <-time.After(5 * time.Second):
+		t.Fatal("Report blocked on a full queue")
+	}
+
+	if n := len(h.e.queue); n != cap(h.e.queue) {
+		t.Errorf("queue holds %d alerts, want %d", n, cap(h.e.queue))
+	}
+	if n := strings.Count(h.logs.String(), "queue full"); n != extra {
+		t.Errorf("logged %d drops, want %d; log:\n%s", n, extra, h.logs)
+	}
+}
+
+func TestRunSendsWhileRunning(t *testing.T) {
+	h := newHarness(t)
+	h.mailer.notify = make(chan struct{}, 1)
+	ctx, cancel := context.WithCancel(context.Background())
+	done := make(chan struct{})
+	go func() {
+		defer close(done)
+		h.e.Run(ctx)
+	}()
+
+	h.e.Report(Alert{Category: "panic", Summary: "job panicked"})
+	select {
+	case <-h.mailer.notify:
+	case <-time.After(5 * time.Second):
+		t.Fatal("Run did not send a reported alert")
+	}
+	cancel()
+	<-done
+}
+
+// An alert raised during shutdown still goes out: Run drains what is queued
+// before it returns.
+func TestRunDrainsQueueOnShutdown(t *testing.T) {
+	h := newHarness(t)
+	for _, category := range []string{"intake", "delivery", "sweep-jobs"} {
+		h.e.Report(Alert{Category: category, Summary: category + " failing"})
+	}
+
+	ctx, cancel := context.WithCancel(context.Background())
+	cancel()
+	h.e.Run(ctx)
+
+	if n := len(h.mailer.sends()); n != 3 {
+		t.Fatalf("shutdown drain sent %d emails, want 3", n)
+	}
+}
+
+// A summary is the subject line; one carrying an error's newlines must not
+// break it.
+func TestSubjectIsOneLine(t *testing.T) {
+	h := newHarness(t)
+	h.report("intake", "intake failed:\nopen data/inbox: access denied\r\n")
+
+	subject := h.mailer.sends()[0].subject
+	if subject != "[FileMill] intake failed: open data/inbox: access denied" {
+		t.Errorf("subject = %q", subject)
+	}
+}
+
+func TestNopReportDoesNothing(t *testing.T) {
+	Nop{}.Report(Alert{Category: "panic", Summary: "ignored"})
+}
diff --git a/internal/alert/fake_test.go b/internal/alert/fake_test.go
new file mode 100644
index 0000000..de7497f
--- /dev/null
+++ b/internal/alert/fake_test.go
@@ -0,0 +1,177 @@
+package alert
+
+import (
+	"bytes"
+	"context"
+	"log"
+	"slices"
+	"sync"
+	"testing"
+	"time"
+)
+
+// sentAlert is one message the fake mailer was asked to send.
+type sentAlert struct{ to, subject, text string }
+
+// fakeMailer records every send. Tests read it through sends(), which copies
+// under the lock, because Run calls SendAlert from its own goroutine.
+type fakeMailer struct {
+	mu        sync.Mutex
+	sent      []sentAlert
+	err       error         // when set, every send is recorded and then fails with it
+	panicWith any           // when set, every send is recorded and then panics with it
+	notify    chan struct{} // when set, receives one value per send
+}
+
+func (m *fakeMailer) SendAlert(ctx context.Context, to, subject, text string) error {
+	m.mu.Lock()
+	m.sent = append(m.sent, sentAlert{to, subject, text})
+	err, panicWith := m.err, m.panicWith
+	m.mu.Unlock()
+	if m.notify != nil {
+		m.notify <- struct{}{}
+	}
+	if panicWith != nil {
+		panic(panicWith)
+	}
+	return err
+}
+
+func (m *fakeMailer) sends() []sentAlert {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	return slices.Clone(m.sent)
+}
+
+// fakeLedger is an in-memory Ledger with the store's semantics: a category can
+// hold a suppressed count before it has ever sent, and a send resets it.
+type fakeLedger struct {
+	mu         sync.Mutex
+	lastSent   map[string]time.Time
+	suppressed map[string]int
+	sent       []time.Time // every recorded send, oldest first
+	err        error       // when set, every method fails with it
+}
+
+func newFakeLedger() *fakeLedger {
+	return &fakeLedger{lastSent: map[string]time.Time{}, suppressed: map[string]int{}}
+}
+
+func (l *fakeLedger) LastAlert(category string) (time.Time, int, error) {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	if l.err != nil {
+		return time.Time{}, 0, l.err
+	}
+	return l.lastSent[category], l.suppressed[category], nil
+}
+
+func (l *fakeLedger) RecordAlertSent(category string, at time.Time) error {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	if l.err != nil {
+		return l.err
+	}
+	l.lastSent[category] = at
+	l.suppressed[category] = 0
+	l.sent = append(l.sent, at)
+	return nil
+}
+
+func (l *fakeLedger) RecordAlertSuppressed(category string) error {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	if l.err != nil {
+		return l.err
+	}
+	l.suppressed[category]++
+	return nil
+}
+
+func (l *fakeLedger) AlertSendsSince(t time.Time) ([]time.Time, error) {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	if l.err != nil {
+		return nil, l.err
+	}
+	var out []time.Time
+	for _, at := range l.sent {
+		if at.After(t) {
+			out = append(out, at)
+		}
+	}
+	return out, nil
+}
+
+// fakeClock is the Emailer's injected clock. It only moves when a test
+// advances it, so the throttle windows are crossed without sleeping.
+type fakeClock struct {
+	mu sync.Mutex
+	t  time.Time
+}
+
+func newFakeClock() *fakeClock {
+	return &fakeClock{t: time.Date(2026, 9, 11, 9, 0, 0, 0, time.UTC)}
+}
+
+func (c *fakeClock) Now() time.Time {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	return c.t
+}
+
+func (c *fakeClock) Advance(d time.Duration) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	c.t = c.t.Add(d)
+}
+
+// syncBuffer is a log sink that is safe to read while Run may still write.
+type syncBuffer struct {
+	mu  sync.Mutex
+	buf bytes.Buffer
+}
+
+func (b *syncBuffer) Write(p []byte) (int, error) {
+	b.mu.Lock()
+	defer b.mu.Unlock()
+	return b.buf.Write(p)
+}
+
+func (b *syncBuffer) String() string {
+	b.mu.Lock()
+	defer b.mu.Unlock()
+	return b.buf.String()
+}
+
+const testRecipient = "support@example.com"
+
+// harness bundles an Emailer with the fakes behind it. Its ledger, mailer and
+// clock outlive the Emailer, so restart can model a new process over the same
+// database.
+type harness struct {
+	e      *Emailer
+	mailer *fakeMailer
+	ledger *fakeLedger
+	clock  *fakeClock
+	logs   *syncBuffer
+}
+
+func newHarness(t *testing.T) *harness {
+	t.Helper()
+	h := &harness{mailer: &fakeMailer{}, ledger: newFakeLedger(), clock: newFakeClock(), logs: &syncBuffer{}}
+	h.restart()
+	return h
+}
+
+// restart replaces the Emailer, as a worker restart would, keeping the
+// ledger. Only persisted state survives it.
+func (h *harness) restart() {
+	h.e = NewEmailer(h.mailer, h.ledger, testRecipient, Config{}, h.clock.Now, log.New(h.logs, "", 0))
+}
+
+// report runs one alert through the throttle synchronously, the way Run does
+// for each alert it takes off the queue.
+func (h *harness) report(category, summary string) {
+	h.e.handle(context.Background(), Alert{Category: category, Summary: summary, Detail: "detail for " + summary})
+}

From 53ca9bcf6f05d66e8a20a72ea9b6aa2fcb75c99a Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 05:28:57 -0600
Subject: [PATCH 04/25] Persist the alert throttle ledger in the store

store.Open creates alert_state and alert_sends, and Store implements
alert.Ledger over them. alert_sends is pruned to 24h on each write. Times use
a fixed-width format so the window comparisons, done as strings in SQL, order
correctly at sub-second precision.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 internal/store/alert.go      |  91 ++++++++++++++++++++
 internal/store/alert_test.go | 159 +++++++++++++++++++++++++++++++++++
 internal/store/store.go      |  11 +++
 3 files changed, 261 insertions(+)
 create mode 100644 internal/store/alert.go
 create mode 100644 internal/store/alert_test.go

diff --git a/internal/store/alert.go b/internal/store/alert.go
new file mode 100644
index 0000000..f1ff626
--- /dev/null
+++ b/internal/store/alert.go
@@ -0,0 +1,91 @@
+package store
+
+import (
+	"database/sql"
+	"fmt"
+	"time"
+)
+
+// alertTimeFormat is fixed-width, unlike RFC3339Nano, which trims trailing
+// zeros from the fraction: "10:00:00Z" sorts after "10:00:00.5Z". The alert
+// windows are compared as strings in SQL, so the width matters.
+const alertTimeFormat = "2006-01-02T15:04:05.000000000Z07:00"
+
+// alertSendsRetention is the longest window the alert caps read, the rolling
+// 24h of the daily cap. Older rows are pruned on each write.
+const alertSendsRetention = 24 * time.Hour
+
+func formatAlertTime(t time.Time) string { return t.UTC().Format(alertTimeFormat) }
+
+// LastAlert returns when category last sent an alert and how many have been
+// suppressed since. Zero values mean it has no record; a zero sentAt with a
+// count means a global cap held it back before it ever sent.
+func (s *Store) LastAlert(category string) (time.Time, int, error) {
+	var sentAt sql.NullString
+	var suppressed int
+	err := s.db.QueryRow("SELECT last_sent_at,suppressed FROM alert_state WHERE category=?", category).Scan(&sentAt, &suppressed)
+	if err == sql.ErrNoRows {
+		return time.Time{}, 0, nil
+	}
+	if err != nil {
+		return time.Time{}, 0, err
+	}
+	if !sentAt.Valid {
+		return time.Time{}, suppressed, nil
+	}
+	t, err := time.Parse(alertTimeFormat, sentAt.String)
+	if err != nil {
+		return time.Time{}, 0, fmt.Errorf("alert_state %s: %w", category, err)
+	}
+	return t, suppressed, nil
+}
+
+// RecordAlertSent records a send in category at at, resets its suppressed
+// count, and prunes sends too old for any cap window.
+func (s *Store) RecordAlertSent(category string, at time.Time) error {
+	tx, err := s.db.Begin()
+	if err != nil {
+		return err
+	}
+	defer tx.Rollback()
+	if _, err := tx.Exec(`INSERT INTO alert_state(category,last_sent_at,suppressed) VALUES(?,?,0)
+ ON CONFLICT(category) DO UPDATE SET last_sent_at=excluded.last_sent_at, suppressed=0`, category, formatAlertTime(at)); err != nil {
+		return err
+	}
+	if _, err := tx.Exec("INSERT INTO alert_sends(sent_at) VALUES(?)", formatAlertTime(at)); err != nil {
+		return err
+	}
+	if _, err := tx.Exec("DELETE FROM alert_sends WHERE sent_at<=?", formatAlertTime(at.Add(-alertSendsRetention))); err != nil {
+		return err
+	}
+	return tx.Commit()
+}
+
+// RecordAlertSuppressed counts one held-back alert against category.
+func (s *Store) RecordAlertSuppressed(category string) error {
+	_, err := s.db.Exec(`INSERT INTO alert_state(category,suppressed) VALUES(?,1)
+ ON CONFLICT(category) DO UPDATE SET suppressed=suppressed+1`, category)
+	return err
+}
+
+// AlertSendsSince returns every alert send after t, oldest first.
+func (s *Store) AlertSendsSince(t time.Time) ([]time.Time, error) {
+	rows, err := s.db.Query("SELECT sent_at FROM alert_sends WHERE sent_at>? ORDER BY sent_at", formatAlertTime(t))
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+	var out []time.Time
+	for rows.Next() {
+		var raw string
+		if err := rows.Scan(&raw); err != nil {
+			return nil, err
+		}
+		at, err := time.Parse(alertTimeFormat, raw)
+		if err != nil {
+			return nil, fmt.Errorf("alert_sends: %w", err)
+		}
+		out = append(out, at)
+	}
+	return out, rows.Err()
+}
diff --git a/internal/store/alert_test.go b/internal/store/alert_test.go
new file mode 100644
index 0000000..351adbf
--- /dev/null
+++ b/internal/store/alert_test.go
@@ -0,0 +1,159 @@
+package store
+
+import (
+	"path/filepath"
+	"testing"
+	"time"
+
+	"filemill/internal/alert"
+)
+
+var _ alert.Ledger = (*Store)(nil)
+
+// countAlertSends reads the raw row count, so a test can see pruning that
+// AlertSendsSince's window would hide. Test-only.
+func (s *Store) countAlertSends() (int, error) {
+	var n int
+	err := s.db.QueryRow("SELECT COUNT(*) FROM alert_sends").Scan(&n)
+	return n, err
+}
+
+func openTestStore(t *testing.T) (*Store, string) {
+	t.Helper()
+	path := filepath.Join(t.TempDir(), "test.db")
+	s, err := Open(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Cleanup(func() { s.Close() })
+	return s, path
+}
+
+func TestLastAlertUnknownCategoryIsZero(t *testing.T) {
+	s, _ := openTestStore(t)
+
+	sentAt, suppressed, err := s.LastAlert("job-systemic")
+	if err != nil || !sentAt.IsZero() || suppressed != 0 {
+		t.Fatalf("LastAlert = %v, %d, %v; want zero values", sentAt, suppressed, err)
+	}
+}
+
+// A global cap can hold back a category that has never sent, so the count
+// must be storable before there is any send time.
+func TestAlertSuppressedBeforeAnySend(t *testing.T) {
+	s, _ := openTestStore(t)
+
+	for range 3 {
+		if err := s.RecordAlertSuppressed("publish"); err != nil {
+			t.Fatal(err)
+		}
+	}
+	sentAt, suppressed, err := s.LastAlert("publish")
+	if err != nil || !sentAt.IsZero() || suppressed != 3 {
+		t.Fatalf("LastAlert = %v, %d, %v; want no send time and 3 suppressed", sentAt, suppressed, err)
+	}
+}
+
+func TestRecordAlertSentStoresTimeAndResetsSuppressed(t *testing.T) {
+	s, _ := openTestStore(t)
+	at := time.Date(2026, 9, 11, 9, 30, 15, 123456789, time.UTC)
+
+	if err := s.RecordAlertSuppressed("delivery"); err != nil {
+		t.Fatal(err)
+	}
+	if err := s.RecordAlertSent("delivery", at); err != nil {
+		t.Fatal(err)
+	}
+	sentAt, suppressed, err := s.LastAlert("delivery")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !sentAt.Equal(at) || suppressed != 0 {
+		t.Fatalf("LastAlert = %v, %d; want %v, 0", sentAt, suppressed, at)
+	}
+	// Other categories are untouched.
+	if other, _, _ := s.LastAlert("intake"); !other.IsZero() {
+		t.Errorf("an unrelated category picked up a send time: %v", other)
+	}
+}
+
+// The window is exclusive of its start, so a send leaves it exactly 24h (or
+// 1h) after it went out. Times differing only in fractional seconds must
+// still order correctly, which RFC3339Nano's trimmed zeros would break.
+func TestAlertSendsSinceIsExclusiveAndOrdered(t *testing.T) {
+	s, _ := openTestStore(t)
+	whole := time.Date(2026, 9, 11, 10, 0, 0, 0, time.UTC)
+	half := whole.Add(500 * time.Millisecond)
+	later := whole.Add(time.Minute)
+
+	for _, at := range []time.Time{later, whole, half} {
+		if err := s.RecordAlertSent("intake", at); err != nil {
+			t.Fatal(err)
+		}
+	}
+
+	got, err := s.AlertSendsSince(whole)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(got) != 2 || !got[0].Equal(half) || !got[1].Equal(later) {
+		t.Fatalf("AlertSendsSince(%v) = %v; want [%v %v]", whole, got, half, later)
+	}
+}
+
+// Rows are kept for 24h, the longest window the caps read, and pruned on the
+// next write.
+func TestAlertSendsArePrunedAfter24h(t *testing.T) {
+	s, _ := openTestStore(t)
+	t0 := time.Date(2026, 9, 11, 9, 0, 0, 0, time.UTC)
+
+	for _, at := range []time.Time{t0, t0.Add(time.Hour), t0.Add(24 * time.Hour)} {
+		if err := s.RecordAlertSent("restart", at); err != nil {
+			t.Fatal(err)
+		}
+	}
+
+	n, err := s.countAlertSends()
+	if err != nil {
+		t.Fatal(err)
+	}
+	if n != 2 {
+		t.Fatalf("alert_sends holds %d rows, want 2 (the send 24h old pruned)", n)
+	}
+	got, err := s.AlertSendsSince(t0.Add(-time.Hour))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(got) != 2 || !got[0].Equal(t0.Add(time.Hour)) {
+		t.Fatalf("kept sends = %v, want the two newest", got)
+	}
+}
+
+// Surviving a restart is the whole reason the throttle lives in SQLite.
+func TestAlertStateSurvivesReopen(t *testing.T) {
+	s, path := openTestStore(t)
+	at := time.Date(2026, 9, 11, 9, 0, 0, 0, time.UTC)
+
+	if err := s.RecordAlertSent("restart", at); err != nil {
+		t.Fatal(err)
+	}
+	if err := s.RecordAlertSuppressed("restart"); err != nil {
+		t.Fatal(err)
+	}
+	s.Close()
+
+	reopened, err := Open(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer reopened.Close()
+
+	sentAt, suppressed, err := reopened.LastAlert("restart")
+	if err != nil || !sentAt.Equal(at) || suppressed != 1 {
+		t.Fatalf("after reopen LastAlert = %v, %d, %v; want %v, 1", sentAt, suppressed, err, at)
+	}
+	sends, err := reopened.AlertSendsSince(at.Add(-time.Minute))
+	if err != nil || len(sends) != 1 {
+		t.Fatalf("after reopen AlertSendsSince = %v, %v; want the one send", sends, err)
+	}
+}
diff --git a/internal/store/store.go b/internal/store/store.go
index cefaee2..94f9640 100644
--- a/internal/store/store.go
+++ b/internal/store/store.go
@@ -111,6 +111,17 @@ func Open(path string) (*Store, error) {
  file_id TEXT NOT NULL, link TEXT NOT NULL, created_at TEXT NOT NULL, deleted_at TEXT,
  PRIMARY KEY(submission_id, output_index), FOREIGN KEY(submission_id) REFERENCES email_submissions(id)
 ); CREATE INDEX IF NOT EXISTS email_deliveries_sweep ON email_deliveries(deleted_at, created_at);`)
+	if err != nil {
+		db.Close()
+		return nil, err
+	}
+	// Operator-alert throttle state (internal/alert), kept here so a restarted
+	// worker inherits it. last_sent_at is NULL for a category a global cap held
+	// back before it ever sent. alert_sends exists only for those caps and is
+	// pruned to 24h on write.
+	_, err = db.Exec(`CREATE TABLE IF NOT EXISTS alert_state (
+ category TEXT PRIMARY KEY, last_sent_at TEXT, suppressed INTEGER NOT NULL DEFAULT 0
+); CREATE TABLE IF NOT EXISTS alert_sends (sent_at TEXT NOT NULL);`)
 	if err != nil {
 		db.Close()
 		return nil, err

From 6d6cca1604721018ccefcbd4121a78695cf9bb24 Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 05:28:57 -0600
Subject: [PATCH 05/25] Update the alerting plan for the phase 1 Ledger as
 built

LastAlert drops its redundant ok flag, AlertSendsSince returns send times
instead of a count, and alert_state.last_sent_at is nullable so a category
capped before its first send can still hold a suppressed count.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 ERROR-ALERTING-PLAN.md | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/ERROR-ALERTING-PLAN.md b/ERROR-ALERTING-PLAN.md
index 3b3b28d..b4388f7 100644
--- a/ERROR-ALERTING-PLAN.md
+++ b/ERROR-ALERTING-PLAN.md
@@ -1,6 +1,7 @@
 # FileMill Error Alerting — Implementation Plan (issue #7)
 
-**Status:** plan only, not implemented. First drafted 2026-07-19; **revised
+**Status:** Phase 1 (`internal/alert` core and the store `Ledger`) implemented
+2026-09-11; phases 2–5 not started. First drafted 2026-07-19; **revised
 2026-09-11** against the current code. Since the first draft, FileMill gained the
 supervisor loop (#4), boot start, the two retention sweeps, sheets-link delivery,
 and non-blocking delivery. Each adds alert sites, and the supervisor changes how a
@@ -94,10 +95,10 @@ type Mailer interface {
 
 // Ledger is the persisted throttle state. *store.Store satisfies it.
 type Ledger interface {
-    LastAlert(category string) (sentAt time.Time, suppressed int, ok bool, err error)
+    LastAlert(category string) (sentAt time.Time, suppressed int, err error) // zero sentAt: never sent
     RecordAlertSent(category string, at time.Time) error // resets suppressed
     RecordAlertSuppressed(category string) error         // suppressed++
-    AlertsSentSince(t time.Time) (int, error)            // for the global cap
+    AlertSendsSince(t time.Time) ([]time.Time, error)    // sends after t, oldest first
 }
 
 // Emailer is the real Reporter: a throttle in front of a Mailer, draining a
@@ -128,12 +129,20 @@ Two small tables, created like the existing ones in `store.Open`:
 
 ```sql
 CREATE TABLE IF NOT EXISTS alert_state (
-  category TEXT PRIMARY KEY, last_sent_at TEXT NOT NULL, suppressed INTEGER NOT NULL DEFAULT 0);
+  category TEXT PRIMARY KEY, last_sent_at TEXT, suppressed INTEGER NOT NULL DEFAULT 0);
 CREATE TABLE IF NOT EXISTS alert_sends (sent_at TEXT NOT NULL); -- pruned to 24h on write
 ```
 
 `alert_sends` exists only for the global caps. `alert_state` alone can't count sends
-per hour or per day. Keeping 24h of rows covers both windows. Persisting matters
+per hour or per day. Keeping 24h of rows covers both windows. `AlertSendsSince`
+returns the send times, not just a count, so one read serves both caps and gives
+the cap notice its "until <time>" (the oldest send in the window, plus 24h).
+
+`last_sent_at` is nullable because a global cap can suppress a category that has
+never sent, and its count still has to be stored. Times are stored in a
+fixed-width UTC format, not `RFC3339Nano`: that trims trailing zeros, so
+`10:00:00Z` sorts after `10:00:00.5Z`, and these windows are compared as strings
+in SQL. Persisting matters
 most for the daily cap: a crash-looping worker that reset it on every restart
 could spend the whole Free-plan budget.
 

From f1b66b883ee8559b5c3a1f9f8e75262c2a8658ff Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 06:08:40 -0600
Subject: [PATCH 06/25] Throttle alerts in memory when the ledger fails instead
 of dropping them

A sick database is the likeliest cause of the failures worth alerting on
(worker-claim, intake), so dropping alerts when the Ledger failed silenced
exactly those. The Emailer now keeps an in-memory copy of the throttle state,
synced from the Ledger on each good read, and after the first Ledger error
throttles from it for the rest of the process. It never switches back: a
Ledger whose writes failed would read back too few sends. Each email sent in
this mode says the throttle's state failed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 ERROR-ALERTING-PLAN.md         |  11 ++++
 internal/alert/emailer.go      | 107 +++++++++++++++++++++++++++------
 internal/alert/emailer_test.go |  78 +++++++++++++++++++++---
 internal/alert/fake_test.go    |  19 +++---
 4 files changed, 181 insertions(+), 34 deletions(-)

diff --git a/ERROR-ALERTING-PLAN.md b/ERROR-ALERTING-PLAN.md
index b4388f7..ecebbba 100644
--- a/ERROR-ALERTING-PLAN.md
+++ b/ERROR-ALERTING-PLAN.md
@@ -120,6 +120,17 @@ func (e *Emailer) Run(ctx context.Context) // started by main; drains the queue
   20/day keeps alerts to at most a fifth of the day's budget. When the daily cap
   is hit, one last alert says so ("daily alert cap reached; further alerts are
   logged only until <time>"); it counts within the 20, so the cap stays exact.
+- **A failing Ledger doesn't silence alerts.** A sick database is the likeliest
+  cause of `worker-claim` and `intake`, so dropping alerts when the Ledger fails
+  would silence exactly those. The `Emailer` keeps an in-memory copy of the
+  throttle state, synced from the Ledger on every good read. After the first
+  Ledger error, it throttles from that copy for the rest of the process, and
+  each email says so. It never switches back: if writes failed but reads still
+  worked, the Ledger would read back too few sends. The risk that remains is a
+  worker that crash-loops while its database opens but can't hold the ledger,
+  since the in-memory copy resets on every restart. A database that can't be
+  written usually fails `store.Open` instead, which is a startup `fatal` with no
+  reporter at all.
 - Clock, Mailer and Ledger are all injected, so the throttle is tested without
   sleeping, SQLite or the network.
 
diff --git a/internal/alert/emailer.go b/internal/alert/emailer.go
index 1980e0d..5baf94a 100644
--- a/internal/alert/emailer.go
+++ b/internal/alert/emailer.go
@@ -57,6 +57,13 @@ func (c Config) withDefaults() Config {
 // Every alert passes a per-category cooldown and two global caps, all read
 // from the Ledger. A suppressed alert is logged and counted, and the next
 // email in its category says how many were held back.
+//
+// A failing Ledger does not silence alerts: a sick database is the likeliest
+// cause of the failures worth alerting on (worker-claim, intake). The Emailer
+// keeps an in-memory copy of the throttle state, in step with the Ledger while
+// it works, and after the first Ledger error it throttles from that copy for
+// the rest of the process. It never goes back: a Ledger whose writes failed
+// would read back too few sends, and the caps would leak.
 type Emailer struct {
 	mailer Mailer
 	ledger Ledger
@@ -65,12 +72,17 @@ type Emailer struct {
 	now    func() time.Time
 	log    *log.Logger
 	queue  chan Alert
+
+	// mem and ledgerErr are touched only by the goroutine running handle.
+	mem       *memLedger
+	ledgerErr error // the first Ledger error; non-nil means throttling from mem
 }
 
 func NewEmailer(m Mailer, l Ledger, to string, cfg Config, now func() time.Time, log *log.Logger) *Emailer {
 	return &Emailer{
 		mailer: m, ledger: l, to: to, cfg: cfg.withDefaults(), now: now, log: log,
 		queue: make(chan Alert, queueSize),
+		mem:   newMemLedger(),
 	}
 }
 
@@ -119,9 +131,7 @@ func (e *Emailer) drain(ctx context.Context) {
 //
 // The send is recorded before it is attempted, and a failed send is logged
 // and dropped, never retried or re-reported. It still counts against the
-// caps: a send that timed out may have reached Mailgun and been charged. For
-// the same reason the alert is dropped when the Ledger can't be read or
-// written: without it the caps can't be enforced.
+// caps: a send that timed out may have reached Mailgun and been charged.
 func (e *Emailer) handle(ctx context.Context, a Alert) {
 	defer func() {
 		if r := recover(); r != nil {
@@ -130,20 +140,11 @@ func (e *Emailer) handle(ctx context.Context, a Alert) {
 	}()
 
 	now := e.now()
-	sentAt, suppressed, err := e.ledger.LastAlert(a.Category)
-	if err != nil {
-		e.log.Printf("alert %s: read throttle state: %v; dropped: %s", a.Category, err, a.Summary)
-		return
-	}
+	sentAt, suppressed, sends := e.readState(a.Category, now.Add(-day))
 	if !sentAt.IsZero() && now.Sub(sentAt) < e.cfg.Cooldown {
 		e.suppress(a, "cooldown")
 		return
 	}
-	sends, err := e.ledger.AlertSendsSince(now.Add(-day))
-	if err != nil {
-		e.log.Printf("alert %s: read send history: %v; dropped: %s", a.Category, err, a.Summary)
-		return
-	}
 	if sentAfter(sends, now.Add(-time.Hour)) >= e.cfg.MaxPerHour {
 		e.suppress(a, "hourly cap")
 		return
@@ -160,12 +161,13 @@ func (e *Emailer) handle(ctx context.Context, a Alert) {
 	if len(sends) == e.cfg.MaxPerDay-1 {
 		capReopens = slices.MinFunc(append(sends, now), time.Time.Compare).Add(day)
 	}
-	subject, text := e.compose(a, suppressed, capReopens)
 
+	e.mem.recordSent(a.Category, now)
 	if err := e.ledger.RecordAlertSent(a.Category, now); err != nil {
-		e.log.Printf("alert %s: record send: %v; dropped: %s", a.Category, err, a.Summary)
-		return
+		e.ledgerFailed(err)
 	}
+	// Composed after recording, so an email whose own record failed says so.
+	subject, text := e.compose(a, suppressed, capReopens)
 	if err := e.mailer.SendAlert(ctx, e.to, subject, text); err != nil {
 		e.log.Printf("alert %s: send failed: %v; dropped: %s", a.Category, err, a.Summary)
 		return
@@ -173,11 +175,40 @@ func (e *Emailer) handle(ctx context.Context, a Alert) {
 	e.log.Printf("alert %s: sent: %s", a.Category, a.Summary)
 }
 
+// readState returns category's last send and suppressed count, and every send
+// after since, from the Ledger while it works and from mem after it fails.
+func (e *Emailer) readState(category string, since time.Time) (time.Time, int, []time.Time) {
+	if e.ledgerErr == nil {
+		sentAt, suppressed, err := e.ledger.LastAlert(category)
+		if err == nil {
+			var sends []time.Time
+			if sends, err = e.ledger.AlertSendsSince(since); err == nil {
+				e.mem.sync(category, sentAt, suppressed, sends)
+				return sentAt, suppressed, sends
+			}
+		}
+		e.ledgerFailed(err)
+	}
+	return e.mem.read(category, since)
+}
+
+// ledgerFailed switches the throttle to mem for the rest of the process. Only
+// the first error is logged; later writes are still attempted, so the next
+// process inherits as much history as the Ledger will take.
+func (e *Emailer) ledgerFailed(err error) {
+	if e.ledgerErr != nil {
+		return
+	}
+	e.ledgerErr = err
+	e.log.Printf("alert: throttle ledger failed: %v; throttling in memory until restart", err)
+}
+
 // suppress logs a held-back alert and counts it against its category.
 func (e *Emailer) suppress(a Alert, reason string) {
 	e.log.Printf("alert %s: suppressed (%s): %s", a.Category, reason, a.Summary)
+	e.mem.recordSuppressed(a.Category)
 	if err := e.ledger.RecordAlertSuppressed(a.Category); err != nil {
-		e.log.Printf("alert %s: record suppression: %v", a.Category, err)
+		e.ledgerFailed(err)
 	}
 }
 
@@ -202,6 +233,9 @@ func (e *Emailer) compose(a Alert, suppressed int, capReopens time.Time) (subjec
 	if suppressed > 0 {
 		fmt.Fprintf(&b, "\n%d more since the last alert in this category, suppressed by the throttle.\n", suppressed)
 	}
+	if e.ledgerErr != nil {
+		fmt.Fprintf(&b, "\nThe alert throttle's database state failed (%v), so it is throttling in memory until the worker restarts.\n", e.ledgerErr)
+	}
 	return subject, b.String()
 }
 
@@ -219,3 +253,42 @@ func sentAfter(sends []time.Time, t time.Time) int {
 // oneLine collapses whitespace, newlines included, so a summary carrying an
 // error's text still makes a single subject line.
 func oneLine(s string) string { return strings.Join(strings.Fields(s), " ") }
+
+// memLedger is the Emailer's in-memory copy of the throttle state. While the
+// Ledger works, every read syncs it; afterwards it is the throttle. Its send
+// history comes from the last good read plus every send since, so the global
+// caps carry over a Ledger failure. A category the Ledger was never read for
+// starts with no cooldown, but the caps still bound it.
+type memLedger struct {
+	lastSent   map[string]time.Time
+	suppressed map[string]int
+	sends      []time.Time
+}
+
+func newMemLedger() *memLedger {
+	return &memLedger{lastSent: map[string]time.Time{}, suppressed: map[string]int{}}
+}
+
+func (m *memLedger) sync(category string, sentAt time.Time, suppressed int, sends []time.Time) {
+	m.lastSent[category] = sentAt
+	m.suppressed[category] = suppressed
+	m.sends = slices.Clone(sends)
+}
+
+func (m *memLedger) read(category string, since time.Time) (time.Time, int, []time.Time) {
+	var sends []time.Time
+	for _, at := range m.sends {
+		if at.After(since) {
+			sends = append(sends, at)
+		}
+	}
+	return m.lastSent[category], m.suppressed[category], sends
+}
+
+func (m *memLedger) recordSent(category string, at time.Time) {
+	m.lastSent[category] = at
+	m.suppressed[category] = 0
+	m.sends = append(slices.DeleteFunc(m.sends, func(t time.Time) bool { return !t.After(at.Add(-day)) }), at)
+}
+
+func (m *memLedger) recordSuppressed(category string) { m.suppressed[category]++ }
diff --git a/internal/alert/emailer_test.go b/internal/alert/emailer_test.go
index 3282036..82b7e17 100644
--- a/internal/alert/emailer_test.go
+++ b/internal/alert/emailer_test.go
@@ -219,19 +219,81 @@ func TestFailedSendIsLoggedNotRetried(t *testing.T) {
 	}
 }
 
-// Without throttle state the caps can't be enforced, so the alert is dropped
-// rather than sent blind.
-func TestLedgerErrorDropsAlert(t *testing.T) {
+// A sick database is the likeliest cause of the failures worth alerting on, so
+// a Ledger that can't be read must not silence alerts. The throttle carries on
+// in memory, cooldown and caps alike.
+func TestLedgerReadErrorThrottlesInMemory(t *testing.T) {
 	h := newHarness(t)
-	h.ledger.err = errors.New("database is locked")
+	h.ledger.readErr = errors.New("database is locked")
 
 	h.report("worker-claim", "job claim failing")
+	sent := h.mailer.sends()
+	if len(sent) != 1 {
+		t.Fatalf("sent %d emails with an unreadable ledger, want 1", len(sent))
+	}
+	if !strings.Contains(sent[0].text, "database is locked") {
+		t.Errorf("the email should say the throttle's state failed:\n%s", sent[0].text)
+	}
+
+	h.report("worker-claim", "job claim failing")
+	if n := len(h.mailer.sends()); n != 1 {
+		t.Fatalf("the in-memory cooldown let a repeat through: %d emails", n)
+	}
+
+	for i := range 12 {
+		h.report(fmt.Sprintf("cat-%02d", i), "failure")
+	}
+	if n := len(h.mailer.sends()); n != 10 {
+		t.Fatalf("the in-memory hourly cap allowed %d emails, want 10", n)
+	}
+	if n := strings.Count(h.logs.String(), "throttle ledger failed"); n != 1 {
+		t.Errorf("the ledger failure was logged %d times, want once; log:\n%s", n, h.logs)
+	}
+}
+
+// When writes fail but reads still work, the Ledger reads back too few sends.
+// Once it has failed at all, the Emailer must not trust it again.
+func TestLedgerWriteErrorStillThrottles(t *testing.T) {
+	h := newHarness(t)
+	h.ledger.writeErr = errors.New("disk full")
+
+	h.report("intake", "intake failing")
+	h.report("intake", "intake failing")
+	sent := h.mailer.sends()
+	if len(sent) != 1 {
+		t.Fatalf("an unrecorded send lifted the cooldown: %d emails, want 1", len(sent))
+	}
+	if !strings.Contains(sent[0].text, "disk full") {
+		t.Errorf("the email whose record failed should say so:\n%s", sent[0].text)
+	}
+
+	h.clock.Advance(15 * time.Minute)
+	h.report("intake", "intake failing")
+	sent = h.mailer.sends()
+	if len(sent) != 2 || !strings.Contains(sent[1].text, "1 more since the last alert") {
+		t.Fatalf("after the cooldown, want a 2nd email carrying 1 suppressed; got %d emails:\n%v", len(sent), sent)
+	}
+}
 
-	if n := len(h.mailer.sends()); n != 0 {
-		t.Fatalf("sent %d emails without throttle state, want 0", n)
+// The in-memory copy starts from the Ledger's history, so a failure late in
+// the day doesn't reopen the daily cap.
+func TestLedgerFailureKeepsSendHistory(t *testing.T) {
+	h := newHarness(t)
+	for i := range 19 {
+		h.report(fmt.Sprintf("cat-%02d", i), "failure")
+		h.clock.Advance(10 * time.Minute)
+	}
+
+	h.ledger.readErr = errors.New("database is locked")
+	h.report("cat-19", "failure 19")
+	h.report("cat-20", "failure 20")
+
+	sent := h.mailer.sends()
+	if len(sent) != 20 {
+		t.Fatalf("after the ledger failed, %d emails in total, want 20", len(sent))
 	}
-	if !strings.Contains(h.logs.String(), "database is locked") {
-		t.Errorf("the ledger error should be logged; log:\n%s", h.logs)
+	if !strings.Contains(sent[19].subject, capNotice) {
+		t.Errorf("the 20th send should still be the cap notice: %q", sent[19].subject)
 	}
 }
 
diff --git a/internal/alert/fake_test.go b/internal/alert/fake_test.go
index de7497f..a2e3323 100644
--- a/internal/alert/fake_test.go
+++ b/internal/alert/fake_test.go
@@ -50,7 +50,8 @@ type fakeLedger struct {
 	lastSent   map[string]time.Time
 	suppressed map[string]int
 	sent       []time.Time // every recorded send, oldest first
-	err        error       // when set, every method fails with it
+	readErr    error       // when set, LastAlert and AlertSendsSince fail with it
+	writeErr   error       // when set, the Record methods fail with it
 }
 
 func newFakeLedger() *fakeLedger {
@@ -60,8 +61,8 @@ func newFakeLedger() *fakeLedger {
 func (l *fakeLedger) LastAlert(category string) (time.Time, int, error) {
 	l.mu.Lock()
 	defer l.mu.Unlock()
-	if l.err != nil {
-		return time.Time{}, 0, l.err
+	if l.readErr != nil {
+		return time.Time{}, 0, l.readErr
 	}
 	return l.lastSent[category], l.suppressed[category], nil
 }
@@ -69,8 +70,8 @@ func (l *fakeLedger) LastAlert(category string) (time.Time, int, error) {
 func (l *fakeLedger) RecordAlertSent(category string, at time.Time) error {
 	l.mu.Lock()
 	defer l.mu.Unlock()
-	if l.err != nil {
-		return l.err
+	if l.writeErr != nil {
+		return l.writeErr
 	}
 	l.lastSent[category] = at
 	l.suppressed[category] = 0
@@ -81,8 +82,8 @@ func (l *fakeLedger) RecordAlertSent(category string, at time.Time) error {
 func (l *fakeLedger) RecordAlertSuppressed(category string) error {
 	l.mu.Lock()
 	defer l.mu.Unlock()
-	if l.err != nil {
-		return l.err
+	if l.writeErr != nil {
+		return l.writeErr
 	}
 	l.suppressed[category]++
 	return nil
@@ -91,8 +92,8 @@ func (l *fakeLedger) RecordAlertSuppressed(category string) error {
 func (l *fakeLedger) AlertSendsSince(t time.Time) ([]time.Time, error) {
 	l.mu.Lock()
 	defer l.mu.Unlock()
-	if l.err != nil {
-		return nil, l.err
+	if l.readErr != nil {
+		return nil, l.readErr
 	}
 	var out []time.Time
 	for _, at := range l.sent {

From 8cd990680b4746f4c022abd95ed47c0aa020c3eb Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 06:48:31 -0600
Subject: [PATCH 07/25] Split systemic job failures from sender rejections and
 report them

Phase 2 of the error-alerting plan (#7). execute now reports job-systemic for
any failure the transformer didn't make through the contract: a timeout, a
crash, a missing or invalid result.json, a command that won't start, a
transformer no longer configured, or success:true with a nonzero exit. A valid
success:false stays silent whatever the exit code, and so does a transformer
killed because the worker itself is shutting down, which would otherwise read
as a crash on every restart that lands mid-job. Job statuses and messages are
unchanged.

A recover around each job fails it and reports panic with the stack, and Run
reports worker-claim once claims have failed for a minute, once per outage.
App.SetReporter takes the reporter; it defaults to alert.Nop, and nothing
calls it yet, so there is no behavior change.

The tests run the test binary itself as the transformer, so they need no
Python or scripts.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 internal/app/app.go          | 139 ++++++++++++++++++++--
 internal/app/execute_test.go | 225 +++++++++++++++++++++++++++++++++++
 internal/app/fake_test.go    | 153 ++++++++++++++++++++++++
 3 files changed, 509 insertions(+), 8 deletions(-)
 create mode 100644 internal/app/execute_test.go
 create mode 100644 internal/app/fake_test.go

diff --git a/internal/app/app.go b/internal/app/app.go
index 9915104..b4a56fe 100644
--- a/internal/app/app.go
+++ b/internal/app/app.go
@@ -10,16 +10,32 @@ import (
 	"os"
 	"os/exec"
 	"path/filepath"
+	"runtime/debug"
 	"strings"
 	"time"
 
+	"filemill/internal/alert"
 	"filemill/internal/config"
 	"filemill/internal/contract"
 	"filemill/internal/store"
 	"github.com/google/uuid"
 )
 
-const timeout = 10 * time.Minute
+// jobTimeout bounds one transformer run. A var, not a const, so the timeout
+// test can shrink it rather than wait 10 minutes.
+var jobTimeout = 10 * time.Minute
+
+// claimAlertAfter is how long job claims must keep failing before Run reports
+// it: a blip clears itself, a stuck database halts all work. A var so Run's
+// test need not wait a minute.
+var claimAlertAfter = time.Minute
+
+// outputTailSize is how much of a transformer's output an alert carries.
+const outputTailSize = 2048
+
+// panicMessage is a job's message after a panic. The sender reads it in the
+// reply, so it says no more than that the fault was ours.
+const panicMessage = "internal error while running the job"
 
 // ErrRejected marks a Submit failure caused by unacceptable input — a wrong
 // file type, a directory, or an unknown operation. The input itself is the
@@ -34,6 +50,7 @@ type App struct {
 	store      *store.Store
 	log        *log.Logger
 	logFile    *os.File
+	reporter   alert.Reporter
 }
 type OutputFile struct{ Name, Path string }
 
@@ -58,10 +75,21 @@ func Open(root string) (*App, error) {
 		s.Close()
 		return nil, err
 	}
-	return &App{root: root, data: data, cfg: cfg, store: s, log: log.New(lf, "", log.LstdFlags|log.LUTC), logFile: lf}, nil
+	return &App{root: root, data: data, cfg: cfg, store: s, log: log.New(lf, "", log.LstdFlags|log.LUTC), logFile: lf, reporter: alert.Nop{}}, nil
 }
 func (a *App) Close() error { a.logFile.Close(); return a.store.Close() }
 
+// SetReporter routes the worker's systemic failures to r; nil restores the
+// no-op default. It is a setter, not an Open argument, because the real
+// reporter needs the Mailgun service, which is built after the App. Call it
+// before Run.
+func (a *App) SetReporter(r alert.Reporter) {
+	if r == nil {
+		r = alert.Nop{}
+	}
+	a.reporter = r
+}
+
 // LogWriter exposes the application log sink so adapters (e.g. the Mailgun
 // webhook) can write to the same filemill.log the worker uses.
 func (a *App) LogWriter() io.Writer { return a.logFile }
@@ -180,6 +208,7 @@ func (a *App) MarkDeliveryDeleted(submissionID int64, outputIndex int) error {
 }
 func (a *App) Run(ctx context.Context, once bool) error {
 	a.log.Printf("worker started once=%t", once)
+	var claims claimFailures
 	for {
 		j, err := a.store.Next()
 		if err != nil {
@@ -187,8 +216,17 @@ func (a *App) Run(ctx context.Context, once bool) error {
 				return err
 			}
 			// A transient store error (e.g. SQLITE_BUSY under load) must not
-			// take down the worker: log it, back off, and retry.
+			// take down the worker: log it, back off, and retry. One that
+			// persists halts all work, so it is reported.
 			a.log.Printf("worker claim error: %v", err)
+			if claims.failed(time.Now()) {
+				a.reporter.Report(alert.Alert{
+					Category: "worker-claim",
+					Summary:  "worker cannot claim jobs",
+					Detail: fmt.Sprintf("Claiming the next job has failed continuously since %s, so no job can run.\n\nLatest error: %v\n",
+						claims.since.Format(time.RFC3339), err),
+				})
+			}
 			select {
 			case <-ctx.Done():
 				return nil
@@ -196,6 +234,7 @@ func (a *App) Run(ctx context.Context, once bool) error {
 			}
 			continue
 		}
+		claims.succeeded()
 		if j != nil {
 			a.execute(ctx, *j)
 			if once {
@@ -214,14 +253,50 @@ func (a *App) Run(ctx context.Context, once bool) error {
 		}
 	}
 }
+
+// claimFailures tracks an unbroken run of failed job claims, so Run reports
+// one that has lasted claimAlertAfter, and only once per run.
+type claimFailures struct {
+	since    time.Time // the run's first failure; zero while claims work
+	reported bool
+}
+
+// failed records a failed claim at now and reports whether to alert.
+func (c *claimFailures) failed(now time.Time) bool {
+	if c.since.IsZero() {
+		c.since = now
+	}
+	if c.reported || now.Sub(c.since) < claimAlertAfter {
+		return false
+	}
+	c.reported = true
+	return true
+}
+
+func (c *claimFailures) succeeded() { *c = claimFailures{} }
+
+// execute runs one job and records how it ended. A failure the transformer
+// reported through the contract (a valid result.json with success:false) is
+// the sender's problem and the reply tells them. Any other failure is
+// FileMill's problem and is reported as well.
 func (a *App) execute(parent context.Context, j store.Job) {
+	// A panic fails this job, not the worker: execute returns normally and Run
+	// moves on to the next job.
+	defer func() {
+		if r := recover(); r != nil {
+			a.log.Printf("job=%s panic=%v", j.ID, r)
+			a.finish(j.ID, store.StatusFailed, panicMessage)
+			a.reportJob("panic", j, fmt.Sprintf("%s job panicked: %v", j.Operation, r), panicMessage,
+				fmt.Sprintf("Panic: %v\n\n%s", r, debug.Stack()))
+		}
+	}()
 	t, ok := a.cfg.Find(j.Operation)
 	if !ok {
-		a.finish(j.ID, store.StatusFailed, "registered transformer no longer exists")
+		a.failSystemic(j, "registered transformer no longer exists", nil, nil)
 		return
 	}
 	workspace := filepath.Join(a.data, "jobs", j.ID)
-	ctx, cancel := context.WithTimeout(parent, timeout)
+	ctx, cancel := context.WithTimeout(parent, jobTimeout)
 	defer cancel()
 	args := append(append([]string{}, t.Command[1:]...), "job.json")
 	cmd := exec.CommandContext(ctx, t.Command[0], args...)
@@ -231,7 +306,7 @@ func (a *App) execute(parent context.Context, j store.Job) {
 		a.log.Printf("job=%s transformer_output=%s", j.ID, strings.TrimSpace(string(output)))
 	}
 	if ctx.Err() == context.DeadlineExceeded {
-		a.finish(j.ID, store.StatusFailed, "transformer timed out after 10 minutes")
+		a.failSystemic(j, "transformer timed out after 10 minutes", nil, output)
 		return
 	}
 	result, readErr := readResult(filepath.Join(workspace, "result.json"), workspace)
@@ -240,11 +315,23 @@ func (a *App) execute(parent context.Context, j store.Job) {
 		if readErr == nil && result.Message != "" {
 			msg = result.Message
 		}
-		a.finish(j.ID, store.StatusFailed, msg)
+		switch {
+		case readErr == nil && !result.Success:
+			// Turned down through the contract, whatever the exit code.
+			a.finish(j.ID, store.StatusFailed, msg)
+		case parent.Err() != nil:
+			// The worker is shutting down and killed the transformer. That is
+			// FileMill stopping, not the transformer failing.
+			a.finish(j.ID, store.StatusFailed, msg)
+		case readErr != nil:
+			a.failSystemic(j, msg, fmt.Errorf("%w; %w", err, readErr), output)
+		default:
+			a.failSystemic(j, msg, fmt.Errorf("result.json reports success, but the transformer ended with %w", err), output)
+		}
 		return
 	}
 	if readErr != nil {
-		a.finish(j.ID, store.StatusFailed, readErr.Error())
+		a.failSystemic(j, readErr.Error(), nil, output)
 		return
 	}
 	if !result.Success {
@@ -260,6 +347,42 @@ func (a *App) finish(id, status, message string) {
 	}
 	a.log.Printf("job=%s status=%s message=%q", id, status, message)
 }
+
+// failSystemic fails a job the transformer didn't handle through the contract
+// and reports it. cause adds what the job's message leaves out, such as the
+// exit status.
+func (a *App) failSystemic(j store.Job, message string, cause error, output []byte) {
+	a.finish(j.ID, store.StatusFailed, message)
+	var extra strings.Builder
+	if cause != nil {
+		fmt.Fprintf(&extra, "Cause: %v\n", cause)
+	}
+	if tail := outputTail(output); tail != "" {
+		fmt.Fprintf(&extra, "\nTransformer output (last 2 KB):\n%s\n", tail)
+	}
+	a.reportJob("job-systemic", j, fmt.Sprintf("%s job failed: %s", j.Operation, message), message, extra.String())
+}
+
+// reportJob reports a failed job, identified the same way in every alert.
+func (a *App) reportJob(category string, j store.Job, summary, message, extra string) {
+	detail := fmt.Sprintf("Job: %s\nOperation: %s\nInput: %s\nMessage: %s\n", j.ID, j.Operation, j.InputName, message)
+	if extra != "" {
+		detail += "\n" + extra
+	}
+	a.reporter.Report(alert.Alert{Category: category, Summary: strings.Join(strings.Fields(summary), " "), Detail: detail})
+}
+
+// outputTail returns the end of a transformer's output, where a crash's
+// traceback lands, capped at outputTailSize bytes.
+func outputTail(output []byte) string {
+	s := strings.TrimSpace(string(output))
+	if len(s) <= outputTailSize {
+		return s
+	}
+	// The cut may split a multi-byte character; drop what is left of it.
+	return "…" + strings.ToValidUTF8(s[len(s)-outputTailSize:], "")
+}
+
 func readResult(path, workspace string) (contract.Result, error) {
 	b, err := os.ReadFile(path)
 	if err != nil {
diff --git a/internal/app/execute_test.go b/internal/app/execute_test.go
new file mode 100644
index 0000000..f4371ce
--- /dev/null
+++ b/internal/app/execute_test.go
@@ -0,0 +1,225 @@
+package app
+
+import (
+	"context"
+	"strings"
+	"testing"
+	"time"
+
+	"filemill/internal/config"
+	"filemill/internal/store"
+)
+
+// Only failures that are FileMill's problem report. A transformer that turned
+// its input down through the contract is the sender's problem, and they hear
+// about it in the reply. Job statuses and messages are the same either way:
+// alerting adds a report, it changes nothing the sender sees.
+func TestExecuteReportsOnlySystemicFailures(t *testing.T) {
+	for _, tc := range []struct {
+		name       string
+		command    []string      // nil: the fake transformer acting out mode
+		mode       string        // the fake transformer's behavior
+		setup      func(*App)    // runs after the job is claimed
+		timeout    time.Duration // 0: the production timeout
+		cancelled  bool          // the worker is shutting down
+		wantStatus string
+		wantMsg    string // prefix of the job's message
+		wantReport bool
+		wantDetail string // also in the alert's detail, besides the job's identity
+	}{
+		{name: "success", mode: "succeed",
+			wantStatus: store.StatusSucceeded, wantMsg: "done"},
+		{name: "clean success:false", mode: "reject",
+			wantStatus: store.StatusFailed, wantMsg: "not a worker list"},
+		{name: "success:false with nonzero exit", mode: "reject-exit",
+			wantStatus: store.StatusFailed, wantMsg: "not a worker list"},
+		{name: "crash without result", mode: "crash",
+			wantStatus: store.StatusFailed, wantMsg: "transformer exited unsuccessfully",
+			wantReport: true, wantDetail: "Traceback: boom"},
+		{name: "missing result.json", mode: "no-result",
+			wantStatus: store.StatusFailed, wantMsg: "result.json missing", wantReport: true},
+		{name: "invalid contract version", mode: "bad-version",
+			wantStatus: store.StatusFailed, wantMsg: `unsupported result contract version "0"`, wantReport: true},
+		{name: "success:true with nonzero exit", mode: "succeed-exit",
+			wantStatus: store.StatusFailed, wantMsg: "done",
+			wantReport: true, wantDetail: "exit status 3"},
+		{name: "timeout", mode: "hang", timeout: time.Second,
+			wantStatus: store.StatusFailed, wantMsg: "transformer timed out after 10 minutes", wantReport: true},
+		{name: "transformer no longer configured", mode: "succeed",
+			setup:      func(a *App) { a.cfg = config.Config{} },
+			wantStatus: store.StatusFailed, wantMsg: "registered transformer no longer exists", wantReport: true},
+		{name: "transformer cannot start", command: []string{"no-such-transformer.exe"},
+			wantStatus: store.StatusFailed, wantMsg: "transformer exited unsuccessfully",
+			wantReport: true, wantDetail: "no-such-transformer"},
+		// Shutdown cancels the job's context, which kills the transformer (here
+		// before it starts). That is FileMill stopping, not the transformer
+		// failing, so it must not alert on every restart that lands mid-job.
+		{name: "killed by shutdown", mode: "hang", cancelled: true,
+			wantStatus: store.StatusFailed, wantMsg: "transformer exited unsuccessfully"},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			command := tc.command
+			if command == nil {
+				command = fakeTransformer(t, tc.mode)
+			}
+			a, rep := newTestApp(t, command...)
+			j := claimJob(t, a)
+			if tc.setup != nil {
+				tc.setup(a)
+			}
+			if tc.timeout > 0 {
+				saved := jobTimeout
+				jobTimeout = tc.timeout
+				t.Cleanup(func() { jobTimeout = saved })
+			}
+			ctx, cancel := context.WithCancel(context.Background())
+			defer cancel()
+			if tc.cancelled {
+				cancel()
+			}
+
+			a.execute(ctx, j)
+
+			got, err := a.Job(j.ID)
+			if err != nil {
+				t.Fatal(err)
+			}
+			if got.Status != tc.wantStatus || !strings.HasPrefix(got.Message, tc.wantMsg) {
+				t.Errorf("job = %s %q, want %s %q…", got.Status, got.Message, tc.wantStatus, tc.wantMsg)
+			}
+
+			alerts := rep.reports()
+			if !tc.wantReport {
+				if len(alerts) != 0 {
+					t.Fatalf("a non-systemic failure reported: %+v", alerts)
+				}
+				return
+			}
+			if len(alerts) != 1 || alerts[0].Category != "job-systemic" {
+				t.Fatalf("alerts = %+v, want one job-systemic", alerts)
+			}
+			for _, want := range []string{j.ID, "Operation: fake", "Input: input.txt", tc.wantDetail} {
+				if !strings.Contains(alerts[0].Detail, want) {
+					t.Errorf("detail lacks %q:\n%s", want, alerts[0].Detail)
+				}
+			}
+			if strings.ContainsAny(alerts[0].Summary, "\r\n") {
+				t.Errorf("summary is not one line: %q", alerts[0].Summary)
+			}
+		})
+	}
+}
+
+// A panic fails the job it happened in and is reported with its stack, and
+// execute returns normally, so Run carries on with the next job. A transformer
+// with no command is a real way in: execute slices the command's arguments
+// before it runs anything. config.Load rejects one, but the App's own guard
+// shouldn't depend on that.
+func TestExecuteRecoversFromPanic(t *testing.T) {
+	a, rep := newTestApp(t, fakeTransformer(t, "succeed")...)
+	j := claimJob(t, a)
+	a.cfg = config.Config{Transformers: []config.Transformer{{Operation: "fake"}}}
+
+	a.execute(context.Background(), j)
+
+	got, err := a.Job(j.ID)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got.Status != store.StatusFailed {
+		t.Errorf("status after a panic = %s, want %s", got.Status, store.StatusFailed)
+	}
+	alerts := rep.reports()
+	if len(alerts) != 1 || alerts[0].Category != "panic" {
+		t.Fatalf("alerts = %+v, want one panic", alerts)
+	}
+	for _, want := range []string{j.ID, "slice bounds out of range", "goroutine"} {
+		if !strings.Contains(alerts[0].Detail, want) {
+			t.Errorf("panic detail lacks %q:\n%s", want, alerts[0].Detail)
+		}
+	}
+}
+
+// The traceback of a crash is at the end of its output, so that is the part
+// an alert keeps.
+func TestOutputTailKeepsTheEnd(t *testing.T) {
+	if got := outputTail([]byte("  short output\n")); got != "short output" {
+		t.Errorf("short output = %q, want it whole and trimmed", got)
+	}
+
+	long := strings.Repeat("a", 3*outputTailSize) + "END"
+	got := outputTail([]byte(long))
+	if !strings.HasSuffix(got, "END") || !strings.HasPrefix(got, "…") {
+		t.Errorf("long output tail = %q…, want the marked end", got[:20])
+	}
+	if n := len(got) - len("…"); n != outputTailSize {
+		t.Errorf("tail keeps %d bytes, want %d", n, outputTailSize)
+	}
+
+	// A cut through a multi-byte character must not leave half of it behind.
+	accented := "é" + strings.Repeat("b", outputTailSize-1)
+	if got := outputTail([]byte(accented)); !strings.HasPrefix(got, "…b") {
+		t.Errorf("tail starts %q, want the broken character dropped", got[:8])
+	}
+}
+
+// A claim error is reported once it has lasted a minute, once per unbroken
+// run of failures. A blip that clears within the minute stays silent.
+func TestClaimFailuresReportOnceAfterAMinute(t *testing.T) {
+	var c claimFailures
+	t0 := time.Date(2026, 9, 11, 9, 0, 0, 0, time.UTC)
+
+	for _, step := range []struct {
+		at   time.Duration
+		want bool
+	}{
+		{0, false},
+		{59 * time.Second, false},
+		{time.Minute, true},
+		{2 * time.Minute, false}, // already reported this run
+	} {
+		if got := c.failed(t0.Add(step.at)); got != step.want {
+			t.Fatalf("failed at +%v = %t, want %t", step.at, got, step.want)
+		}
+	}
+
+	c.succeeded()
+	if c.failed(t0.Add(3 * time.Minute)) {
+		t.Fatal("a new run of failures reported at its first failure")
+	}
+	if !c.failed(t0.Add(4 * time.Minute)) {
+		t.Fatal("a new run of failures was not reported after a minute")
+	}
+}
+
+// Run wires claimFailures to the reporter and keeps retrying. Closing the
+// store makes every claim fail, as a stuck database would.
+func TestRunReportsPersistentClaimFailure(t *testing.T) {
+	a, rep := newTestApp(t, fakeTransformer(t, "succeed")...)
+	rep.notify = make(chan struct{}, 1)
+	saved := claimAlertAfter
+	claimAlertAfter = 0
+	t.Cleanup(func() { claimAlertAfter = saved })
+	a.store.Close()
+
+	ctx, cancel := context.WithCancel(context.Background())
+	done := make(chan error, 1)
+	go func() { done <- a.Run(ctx, false) }()
+	select {
+	case <-rep.notify:
+	case <-time.After(10 * time.Second):
+		t.Fatal("Run did not report failing claims")
+	}
+	cancel()
+	if err := <-done; err != nil {
+		t.Fatalf("Run returned %v; claim failures must not stop the worker", err)
+	}
+
+	alerts := rep.reports()
+	if len(alerts) != 1 || alerts[0].Category != "worker-claim" {
+		t.Fatalf("alerts = %+v, want one worker-claim", alerts)
+	}
+	if !strings.Contains(alerts[0].Detail, "database is closed") {
+		t.Errorf("detail should carry the claim error:\n%s", alerts[0].Detail)
+	}
+}
diff --git a/internal/app/fake_test.go b/internal/app/fake_test.go
new file mode 100644
index 0000000..4123e75
--- /dev/null
+++ b/internal/app/fake_test.go
@@ -0,0 +1,153 @@
+package app
+
+import (
+	"encoding/json"
+	"fmt"
+	"os"
+	"path/filepath"
+	"slices"
+	"strings"
+	"sync"
+	"testing"
+	"time"
+
+	"filemill/internal/alert"
+	"filemill/internal/contract"
+	"filemill/internal/store"
+)
+
+// fakeTransformerArg, as the test binary's first argument, makes it act as a
+// transformer instead of running tests. See TestMain.
+const fakeTransformerArg = "-filemill-fake-transformer"
+
+// TestMain lets execute's tests run real transformer processes without Python
+// or scripts: fakeTransformer points a transformer's command back at this test
+// binary, which acts out the behavior named after fakeTransformerArg. The
+// check comes before m.Run, so a normal test run never reaches it.
+func TestMain(m *testing.M) {
+	if len(os.Args) > 2 && os.Args[1] == fakeTransformerArg {
+		os.Exit(actOut(os.Args[2]))
+	}
+	os.Exit(m.Run())
+}
+
+// actOut is the fake transformer. It runs in the job's workspace, as execute
+// starts it there, and returns its exit code.
+func actOut(mode string) int {
+	switch mode {
+	case "succeed":
+		writeResult(contract.Version, true, "done")
+		return 0
+	case "reject": // the contract's way to turn an input down
+		writeResult(contract.Version, false, "not a worker list")
+		return 0
+	case "reject-exit": // turned down, and exits nonzero as well
+		writeResult(contract.Version, false, "not a worker list")
+		return 1
+	case "crash":
+		fmt.Fprintln(os.Stderr, "Traceback: boom")
+		return 1
+	case "no-result":
+		return 0
+	case "bad-version":
+		writeResult("0", true, "done")
+		return 0
+	case "succeed-exit": // result and exit code contradict each other
+		writeResult(contract.Version, true, "done")
+		return 3
+	case "hang":
+		time.Sleep(time.Minute) // killed by the job timeout long before this
+		return 0
+	}
+	fmt.Fprintf(os.Stderr, "unknown fake transformer mode %q\n", mode)
+	return 2
+}
+
+func writeResult(version string, success bool, message string) {
+	b, err := json.Marshal(contract.Result{ContractVersion: version, Success: success, Message: message})
+	if err == nil {
+		err = os.WriteFile("result.json", b, 0644)
+	}
+	if err != nil {
+		fmt.Fprintln(os.Stderr, err)
+	}
+}
+
+// fakeTransformer is a transformer command that runs this test binary as the
+// fake transformer acting out mode.
+func fakeTransformer(t *testing.T, mode string) []string {
+	t.Helper()
+	exe, err := os.Executable()
+	if err != nil {
+		t.Fatal(err)
+	}
+	return []string{exe, fakeTransformerArg, mode}
+}
+
+// fakeReporter records every alert, so a test can assert which failures
+// report and which stay silent.
+type fakeReporter struct {
+	mu     sync.Mutex
+	alerts []alert.Alert
+	notify chan struct{} // when set, signalled (without blocking) on each report
+}
+
+func (r *fakeReporter) Report(a alert.Alert) {
+	r.mu.Lock()
+	r.alerts = append(r.alerts, a)
+	r.mu.Unlock()
+	if r.notify != nil {
+		select {
+		case r.notify <- struct{}{}:
+		default:
+		}
+	}
+}
+
+func (r *fakeReporter) reports() []alert.Alert {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	return slices.Clone(r.alerts)
+}
+
+// newTestApp opens an App with one operation, "fake", that runs command on
+// .txt inputs, and gives it a fakeReporter.
+func newTestApp(t *testing.T, command ...string) (*App, *fakeReporter) {
+	t.Helper()
+	root := t.TempDir()
+	quoted := make([]string, len(command))
+	for i, c := range command {
+		quoted[i] = yamlQuote(c)
+	}
+	writeTransformers(t, root, fmt.Sprintf("transformers:\n  - operation: fake\n    command: [%s]\n    extensions: [txt]\n", strings.Join(quoted, ", ")))
+	a, err := Open(root)
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Cleanup(func() { a.Close() })
+	rep := &fakeReporter{}
+	a.SetReporter(rep)
+	return a, rep
+}
+
+// yamlQuote single-quotes s, where YAML has no backslash escapes, so a
+// Windows path survives as written.
+func yamlQuote(s string) string { return "'" + strings.ReplaceAll(s, "'", "''") + "'" }
+
+// claimJob submits input.txt to "fake" and claims it, as Run would, leaving
+// the job running and ready for execute.
+func claimJob(t *testing.T, a *App) store.Job {
+	t.Helper()
+	src := filepath.Join(t.TempDir(), "input.txt")
+	if err := os.WriteFile(src, []byte("hello"), 0644); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := a.Submit("fake", src); err != nil {
+		t.Fatal(err)
+	}
+	j, err := a.store.Next()
+	if err != nil || j == nil {
+		t.Fatalf("claim: job %v, err %v", j, err)
+	}
+	return *j
+}

From 9db5af9951912328a9363d7412d7fd1c61d926e7 Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 06:48:31 -0600
Subject: [PATCH 08/25] Record phase 2 and its shutdown exception in the
 alerting plan

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 ERROR-ALERTING-PLAN.md | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/ERROR-ALERTING-PLAN.md b/ERROR-ALERTING-PLAN.md
index ecebbba..ba55bfe 100644
--- a/ERROR-ALERTING-PLAN.md
+++ b/ERROR-ALERTING-PLAN.md
@@ -1,7 +1,7 @@
 # FileMill Error Alerting — Implementation Plan (issue #7)
 
-**Status:** Phase 1 (`internal/alert` core and the store `Ledger`) implemented
-2026-09-11; phases 2–5 not started. First drafted 2026-07-19; **revised
+**Status:** Phase 1 (`internal/alert` core and the store `Ledger`) and phase 2
+(job taxonomy split) implemented 2026-09-11; phases 3–5 not started. First drafted 2026-07-19; **revised
 2026-09-11** against the current code. Since the first draft, FileMill gained the
 supervisor loop (#4), boot start, the two retention sweeps, sheets-link delivery,
 and non-blocking delivery. Each adds alert sites, and the supervisor changes how a
@@ -65,7 +65,12 @@ the contract:
 
 - `result.json` valid and `success:false` → handled. No alert, whatever the exit code.
 - `result.json` valid, `success:true`, but nonzero exit → systemic (the result contradicts the exit code).
-- Anything else that fails (timeout, missing/invalid result, missing transformer) → systemic.
+- Anything else that fails (timeout, missing/invalid result, missing transformer,
+  a command that won't start) → systemic.
+- Killed because the worker itself is shutting down (Ctrl+C, a failed webhook
+  listener: the job's context is cancelled) → not the transformer's failure, so
+  no alert. Otherwise every restart that lands mid-job would read as a crash.
+  The job is still marked failed, as before.
 
 ---
 

From dccf590b4fce797ed5c7458099a7809d80eae1cd Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 09:08:45 -0600
Subject: [PATCH 09/25] Say so when a transformer reports success but exits
 with an error

A valid result.json with success:true and a nonzero exit is failed as
systemic, but the job kept the transformer's own message. The reply then told
the sender their report was ready ("Report ready") with nothing attached, and
the alert's subject read "job failed: Report ready". The job's message now
says what happened; the transformer's text moves to the alert's detail.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 ERROR-ALERTING-PLAN.md       | 2 ++
 internal/app/app.go          | 7 ++++++-
 internal/app/execute_test.go | 9 +++++----
 3 files changed, 13 insertions(+), 5 deletions(-)

diff --git a/ERROR-ALERTING-PLAN.md b/ERROR-ALERTING-PLAN.md
index ba55bfe..95f28bc 100644
--- a/ERROR-ALERTING-PLAN.md
+++ b/ERROR-ALERTING-PLAN.md
@@ -65,6 +65,8 @@ the contract:
 
 - `result.json` valid and `success:false` → handled. No alert, whatever the exit code.
 - `result.json` valid, `success:true`, but nonzero exit → systemic (the result contradicts the exit code).
+  The job's message says so rather than repeating the transformer's success text,
+  which would tell the sender their report is ready in a reply with nothing attached.
 - Anything else that fails (timeout, missing/invalid result, missing transformer,
   a command that won't start) → systemic.
 - Killed because the worker itself is shutting down (Ctrl+C, a failed webhook
diff --git a/internal/app/app.go b/internal/app/app.go
index b4a56fe..48de75b 100644
--- a/internal/app/app.go
+++ b/internal/app/app.go
@@ -326,7 +326,12 @@ func (a *App) execute(parent context.Context, j store.Job) {
 		case readErr != nil:
 			a.failSystemic(j, msg, fmt.Errorf("%w; %w", err, readErr), output)
 		default:
-			a.failSystemic(j, msg, fmt.Errorf("result.json reports success, but the transformer ended with %w", err), output)
+			// The result claims success but the exit code contradicts it. The
+			// transformer's own message would tell the sender their report is
+			// ready in a reply with nothing attached, so say what happened
+			// instead, and keep its message for the alert.
+			a.failSystemic(j, "transformer reported success but then exited with an error",
+				fmt.Errorf("the transformer ended with %w; its result.json said %q", err, result.Message), output)
 		}
 		return
 	}
diff --git a/internal/app/execute_test.go b/internal/app/execute_test.go
index f4371ce..732f2b4 100644
--- a/internal/app/execute_test.go
+++ b/internal/app/execute_test.go
@@ -12,8 +12,9 @@ import (
 
 // Only failures that are FileMill's problem report. A transformer that turned
 // its input down through the contract is the sender's problem, and they hear
-// about it in the reply. Job statuses and messages are the same either way:
-// alerting adds a report, it changes nothing the sender sees.
+// about it in the reply. Reporting changes nothing the sender sees: statuses
+// and messages are as they were before alerting, except that a result
+// contradicting its exit code now says so instead of claiming success.
 func TestExecuteReportsOnlySystemicFailures(t *testing.T) {
 	for _, tc := range []struct {
 		name       string
@@ -41,8 +42,8 @@ func TestExecuteReportsOnlySystemicFailures(t *testing.T) {
 		{name: "invalid contract version", mode: "bad-version",
 			wantStatus: store.StatusFailed, wantMsg: `unsupported result contract version "0"`, wantReport: true},
 		{name: "success:true with nonzero exit", mode: "succeed-exit",
-			wantStatus: store.StatusFailed, wantMsg: "done",
-			wantReport: true, wantDetail: "exit status 3"},
+			wantStatus: store.StatusFailed, wantMsg: "transformer reported success but then exited with an error",
+			wantReport: true, wantDetail: `exit status 3; its result.json said "done"`},
 		{name: "timeout", mode: "hang", timeout: time.Second,
 			wantStatus: store.StatusFailed, wantMsg: "transformer timed out after 10 minutes", wantReport: true},
 		{name: "transformer no longer configured", mode: "succeed",

From b25e0b3f775d4e7e70c7d943b9f1e2581a715e3c Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 09:13:47 -0600
Subject: [PATCH 10/25] Retry only the mark when a sent reply cannot be marked
 delivered

When Mailgun accepted a reply but MarkEmailDelivered failed, the submission
stayed pending and the delivery loop sent the reply again on every 1-second
tick. That floods the sender with duplicates and spends the Mailgun Free
plan's 100 sends a day in under two minutes. Service now remembers such a
submission in memory, and later ticks retry the mark alone. The memory dies
with the process, so each restart before the mark succeeds sends one
duplicate.

The remaining at-least-once window from #6 (a crash between send and mark, or
a send that timed out on our side after Mailgun accepted it) is accepted and
documented on deliver.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 internal/mailgun/delivery_test.go | 44 +++++++++++++++++++++++++++++++
 internal/mailgun/fake_test.go     |  6 +++++
 internal/mailgun/outbound.go      | 34 +++++++++++++++++++++++-
 internal/mailgun/service.go       |  5 ++++
 4 files changed, 88 insertions(+), 1 deletion(-)

diff --git a/internal/mailgun/delivery_test.go b/internal/mailgun/delivery_test.go
index 199f9d6..537687d 100644
--- a/internal/mailgun/delivery_test.go
+++ b/internal/mailgun/delivery_test.go
@@ -228,6 +228,50 @@ func TestDeliverRetryAfterSendFailureDoesNotRepublish(t *testing.T) {
 	}
 }
 
+// Once Mailgun has accepted a reply, the send is done even if marking it
+// delivered fails. The submission stays pending, and sending again on every
+// tick would flood the sender with duplicates and spend the Mailgun Free
+// plan's 100 sends a day in under two minutes. Only the mark is retried.
+func TestDeliverRetriesOnlyTheMarkAfterASuccessfulSend(t *testing.T) {
+	f := newDeliveryFixture(t)
+	f.addSubmission(t, 1, "iwk@mill.test", "schedule.xlsx")
+	f.engine.markErr = fmt.Errorf("disk I/O error")
+
+	for range 3 {
+		if err := f.service.deliverPending(context.Background()); err != nil {
+			t.Fatalf("a failed mark must be skipped, not returned: %v", err)
+		}
+	}
+	if len(f.mailgun.sent) != 1 {
+		t.Fatalf("a reply Mailgun accepted was sent %d times, want 1", len(f.mailgun.sent))
+	}
+	if len(f.publisher.published) != 1 {
+		t.Fatalf("retrying the mark re-published: %d uploads, want 1", len(f.publisher.published))
+	}
+	if f.engine.markCalls != 3 {
+		t.Errorf("mark attempts = %d, want one per tick", f.engine.markCalls)
+	}
+	if f.engine.delivered[1] {
+		t.Fatal("submission marked delivered while the mark was failing")
+	}
+	if !strings.Contains(f.logs.String(), "disk I/O error") {
+		t.Errorf("the failed mark must be logged; got %q", f.logs.String())
+	}
+
+	// The database recovers; the next tick records the delivery, still
+	// without sending again.
+	f.engine.markErr = nil
+	if err := f.service.deliverPending(context.Background()); err != nil {
+		t.Fatalf("recovery tick: %v", err)
+	}
+	if !f.engine.delivered[1] {
+		t.Error("submission must be marked delivered once the mark succeeds")
+	}
+	if len(f.mailgun.sent) != 1 {
+		t.Errorf("recovery sent the reply again: %d sends, want 1", len(f.mailgun.sent))
+	}
+}
+
 // Idempotency is per output file, not per submission — which is why the record
 // is keyed on both. A job declaring two outputs that fails partway must resume
 // where it stopped: the file already in Drive is reused, only the missing one
diff --git a/internal/mailgun/fake_test.go b/internal/mailgun/fake_test.go
index d2eab77..56e4f31 100644
--- a/internal/mailgun/fake_test.go
+++ b/internal/mailgun/fake_test.go
@@ -37,6 +37,8 @@ type fakeEngine struct {
 	jobs      map[int64]map[int]string // sid -> attachment index -> job id
 	expected  map[int64]int
 	delivered map[int64]bool
+	markErr   error // when set, MarkEmailDelivered fails with it
+	markCalls int   // MarkEmailDelivered attempts, failed ones included
 
 	submitCount     int
 	sources         []string // source paths passed to Submit, in order
@@ -149,6 +151,10 @@ func (f *fakeEngine) AddEmailJob(id int64, index int, jobID string) error {
 func (f *fakeEngine) PendingEmails() ([]store.EmailSubmission, error) { return f.pending, nil }
 
 func (f *fakeEngine) MarkEmailDelivered(id int64) error {
+	f.markCalls++
+	if f.markErr != nil {
+		return f.markErr
+	}
 	f.delivered[id] = true
 	return nil
 }
diff --git a/internal/mailgun/outbound.go b/internal/mailgun/outbound.go
index f8334f9..cb345c9 100644
--- a/internal/mailgun/outbound.go
+++ b/internal/mailgun/outbound.go
@@ -83,7 +83,23 @@ func finished(sub store.EmailSubmission) bool {
 // Mailgun accepts the reply, so a transient send failure is retried on the next
 // tick; under sheets-link delivery the upload is already recorded by then and
 // is not repeated.
+//
+// Delivery is therefore at-least-once (#6), and two windows can still send a
+// reply twice: a crash between Mailgun accepting the reply and the mark, and a
+// send that times out on our side after Mailgun accepted it. Each costs one
+// duplicate, sent after the restart or on the next tick. That is accepted: a
+// duplicate reply is harmless, and closing the window would take either a
+// "delivering" state, which only trades a rare duplicate for a rare lost
+// reply, or a provider-side idempotency key.
+//
+// A mark that fails while the process runs is not accepted: the submission
+// stays pending and would be re-sent on every tick. markDelivered remembers
+// it, so later ticks retry the mark alone. That memory dies with the process,
+// so each restart before the mark succeeds sends one more duplicate.
 func (s *Service) deliver(ctx context.Context, sub store.EmailSubmission) error {
+	if s.sentUnmarked[sub.ID] {
+		return s.markDelivered(sub.ID)
+	}
 	var lines []string
 	var outputs []app.OutputFile
 	var labels []string
@@ -123,7 +139,23 @@ func (s *Service) deliver(ctx context.Context, sub store.EmailSubmission) error
 	if err := s.send(ctx, sub.Sender, sub.Subject, threadingID(sub.MessageID), text, attachments); err != nil {
 		return err
 	}
-	return s.engine.MarkEmailDelivered(sub.ID)
+	return s.markDelivered(sub.ID)
+}
+
+// markDelivered records a sent reply as delivered. On failure it remembers the
+// submission, so deliver retries only the mark and never the send: re-sending
+// on every 1-second tick would flood the sender with duplicates and spend the
+// Mailgun Free plan's 100 sends a day in under two minutes.
+func (s *Service) markDelivered(id int64) error {
+	if err := s.engine.MarkEmailDelivered(id); err != nil {
+		if s.sentUnmarked == nil {
+			s.sentUnmarked = map[int64]bool{}
+		}
+		s.sentUnmarked[id] = true
+		return fmt.Errorf("reply sent, but marking it delivered failed (retrying the mark only): %w", err)
+	}
+	delete(s.sentUnmarked, id)
+	return nil
 }
 
 // deliveryMode returns how replies to a recipient address are delivered.
diff --git a/internal/mailgun/service.go b/internal/mailgun/service.go
index 6863933..711fcbd 100644
--- a/internal/mailgun/service.go
+++ b/internal/mailgun/service.go
@@ -95,6 +95,11 @@ type Service struct {
 	sendBase string       // Mailgun Send API base URL; overridable in tests
 	client   *http.Client // outbound HTTP client (carries the send timeout)
 	log      *log.Logger
+
+	// sentUnmarked holds submissions whose reply Mailgun accepted but whose
+	// delivered mark failed, so the delivery loop retries only the mark (see
+	// markDelivered). Only the delivery goroutine touches it; nil until needed.
+	sentUnmarked map[int64]bool
 }
 
 // Handler returns the webhook HTTP handler. It is mounted by cmd/filemill run

From 97125ab48fdeddccb166a7f78031e0effcccda8d Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 09:13:47 -0600
Subject: [PATCH 11/25] Record the resend-storm fix and #6's accepted window in
 the alerting plan

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 ERROR-ALERTING-PLAN.md | 25 +++++++++++++++----------
 1 file changed, 15 insertions(+), 10 deletions(-)

diff --git a/ERROR-ALERTING-PLAN.md b/ERROR-ALERTING-PLAN.md
index 95f28bc..483ca7a 100644
--- a/ERROR-ALERTING-PLAN.md
+++ b/ERROR-ALERTING-PLAN.md
@@ -49,7 +49,7 @@ them, and being honest about what email **cannot** report.
 | Valid `result.json` with `success:false` | `execute` | No | — | Sender told in the reply |
 | **Panic while running a job** | `execute` (new `recover`) | **Yes** | `panic` | Job marked failed; worker continues |
 | **Reply send failing** (non-2xx, timeout) for ≥5 min | `outbound.go` `deliver` | **Yes** | `delivery` | Time-based, not per tick, to ride out a blip (§3.5) |
-| **`MarkEmailDelivered` failing after a successful send** | `deliver` | **Yes, no grace period** | `delivery-mark` | Resends the reply **every second** (#6). Most urgent alert in the table |
+| **`MarkEmailDelivered` failing after a successful send** | `deliver` | **Yes, no grace period** | `delivery-mark` | The database is failing. Only the mark is retried, not the send (§3.5), but every restart until it recovers sends one duplicate reply |
 | **Sheets-link publish failing** (token expired, quota, Drive outage) | `deliver` → `publish` | **Yes** | `publish` | Can persist for hours |
 | **Drive file orphaned** (`PutDelivery` failed after upload) | `publish` | **Yes** | `publish-orphan` | Names the file ID for manual cleanup |
 | **Job claim error** (`store.Next`) persisting ≥1 min | `app.go` `Run` | **Yes** | `worker-claim` | Today it retries silently forever; a stuck DB halts all work |
@@ -196,12 +196,17 @@ A 3-tick grace period is 3 seconds, which isn't a blip ride-out. Instead
 `Service` keeps an in-memory `firstFailure map[int64]time.Time` keyed by
 submission ID. It is set on the first failure, cleared on success, and alerts
 (`delivery` or `publish`) once a submission has failed for ≥5 min.
-`MarkEmailDelivered` failures skip the grace period, because each second of delay
-is another duplicate reply.
-
-This map is the natural first step toward #6's per-submission failure count and
-dead-letter state. Building #6's retry cap at the same time would remove the
-resend storm that `delivery-mark` exists to report.
+`MarkEmailDelivered` failures skip the grace period: the database is failing, and
+every restart until it recovers sends the sender one more duplicate.
+
+The resend storm is fixed ahead of phase 3. It used to happen when the send
+succeeded but the mark failed, so the reply went out again on every 1-second
+tick. Now `Service.sentUnmarked` remembers a submission whose reply Mailgun
+accepted, and later ticks retry only the mark. #6's crash window (a crash between
+send and mark, or a send that timed out on our side after Mailgun accepted it) is
+accepted and documented on `deliver`: each costs one duplicate. A per-submission
+retry cap and dead-letter state is a separate future issue, and this map is its
+natural first step.
 
 ### 3.6 Reporting crashes across a restart
 
@@ -314,6 +319,6 @@ Phases 2 and 3 are about 1 day together. Phase 4 is half a day.
    for `delivery-mark`.
 5. **Heartbeat:** out of scope here, tracked as #5. It is the only cover for
    startup fatals, a Mailgun outage, and a machine that is off.
-6. **Ordering with #6:** do #6's retry cap before or alongside Phase 3. Otherwise
-   the first real `delivery-mark` alert arrives *after* the sender has already had
-   dozens of duplicate replies.
+6. **Ordering with #6:** done before Phase 3. A failed mark no longer resends the
+   reply every second (§3.5). #6's crash window is accepted and documented. The
+   retry cap and dead-letter state for stuck submissions will be their own issue.

From 8a5addf81951fca41e041df58e7c646a3f7d2411 Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 09:26:02 -0600
Subject: [PATCH 12/25] Report job-sweep delete failures and serve the alert
 ledger from App

Phase 3 of the error-alerting plan (#7), app side. The job sweep reports
sweep-jobs once per sweep when workspaces past retention can't be deleted,
naming each job. App hands the alert Ledger calls to its store, as it does
for the other store methods, so main can give the Emailer the App.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 internal/app/app.go              | 11 +++++
 internal/app/fake_test.go        |  3 ++
 internal/app/retention.go        | 29 ++++++++++---
 internal/app/sweep_alert_test.go | 73 ++++++++++++++++++++++++++++++++
 4 files changed, 111 insertions(+), 5 deletions(-)
 create mode 100644 internal/app/sweep_alert_test.go

diff --git a/internal/app/app.go b/internal/app/app.go
index 48de75b..ea00eec 100644
--- a/internal/app/app.go
+++ b/internal/app/app.go
@@ -206,6 +206,17 @@ func (a *App) ExpiredDeliveries(cutoff time.Time) ([]store.Delivery, error) {
 func (a *App) MarkDeliveryDeleted(submissionID int64, outputIndex int) error {
 	return a.store.MarkDeliveryDeleted(submissionID, outputIndex)
 }
+
+// The alert throttle's state lives in the job database, so a restarted worker
+// inherits it. main hands the App to alert.NewEmailer as its Ledger.
+func (a *App) LastAlert(category string) (time.Time, int, error) { return a.store.LastAlert(category) }
+func (a *App) RecordAlertSent(category string, at time.Time) error {
+	return a.store.RecordAlertSent(category, at)
+}
+func (a *App) RecordAlertSuppressed(category string) error {
+	return a.store.RecordAlertSuppressed(category)
+}
+func (a *App) AlertSendsSince(t time.Time) ([]time.Time, error) { return a.store.AlertSendsSince(t) }
 func (a *App) Run(ctx context.Context, once bool) error {
 	a.log.Printf("worker started once=%t", once)
 	var claims claimFailures
diff --git a/internal/app/fake_test.go b/internal/app/fake_test.go
index 4123e75..3a64a79 100644
--- a/internal/app/fake_test.go
+++ b/internal/app/fake_test.go
@@ -16,6 +16,9 @@ import (
 	"filemill/internal/store"
 )
 
+// main.go hands the App to the alert Emailer as its throttle ledger.
+var _ alert.Ledger = (*App)(nil)
+
 // fakeTransformerArg, as the test binary's first argument, makes it act as a
 // transformer instead of running tests. See TestMain.
 const fakeTransformerArg = "-filemill-fake-transformer"
diff --git a/internal/app/retention.go b/internal/app/retention.go
index 921f92f..e6bfacb 100644
--- a/internal/app/retention.go
+++ b/internal/app/retention.go
@@ -2,9 +2,13 @@ package app
 
 import (
 	"context"
+	"fmt"
 	"os"
 	"path/filepath"
+	"strings"
 	"time"
+
+	"filemill/internal/alert"
 )
 
 // jobRetentionPeriod is how long a completed job's workspace stays on disk
@@ -17,6 +21,10 @@ import (
 // own test shrinks this instead of waiting 30 real days.
 var jobRetentionPeriod = 30 * 24 * time.Hour
 
+// removeWorkspace deletes one job's workspace. A var so the sweep's test can
+// make a delete fail, which the filesystem won't do on request.
+var removeWorkspace = os.RemoveAll
+
 const (
 	// jobSweepInterval mirrors mailgun's retention sweep: retention is measured
 	// in days, so checking daily is ample.
@@ -55,9 +63,9 @@ func (a *App) SweepExpiredJobs(ctx context.Context) {
 //
 // A workspace that fails to delete is logged and left unmarked, so the next
 // sweep retries it; one stuck directory must not strand the rest of the
-// batch. RemoveAll is idempotent — a workspace already gone counts as
-// deleted — so marking a job after a crash-interrupted sweep is never a
-// problem.
+// batch. The failures are reported together, once per sweep. RemoveAll is
+// idempotent — a workspace already gone counts as deleted — so marking a job
+// after a crash-interrupted sweep is never a problem.
 //
 // A sweep that deleted something says so; an idle one stays silent, so the
 // line means something when it does appear — the same reasoning as
@@ -68,9 +76,11 @@ func (a *App) sweepExpiredJobs() error {
 		return err
 	}
 	deleted := 0
+	var failed []string
 	for _, id := range ids {
-		if err := os.RemoveAll(filepath.Join(a.data, "jobs", id)); err != nil {
+		if err := removeWorkspace(filepath.Join(a.data, "jobs", id)); err != nil {
 			a.log.Printf("job sweep: delete %s: %v", id, err)
+			failed = append(failed, fmt.Sprintf("%s: %v", id, err))
 			continue
 		}
 		deleted++
@@ -78,8 +88,17 @@ func (a *App) sweepExpiredJobs() error {
 			a.log.Printf("job sweep: record deletion of %s: %v", id, err)
 		}
 	}
+	days := int(jobRetentionPeriod.Hours() / 24)
 	if deleted > 0 {
-		a.log.Printf("job sweep: deleted %d job workspace(s) older than %d days", deleted, int(jobRetentionPeriod.Hours()/24))
+		a.log.Printf("job sweep: deleted %d job workspace(s) older than %d days", deleted, days)
+	}
+	if len(failed) > 0 {
+		a.reporter.Report(alert.Alert{
+			Category: "sweep-jobs",
+			Summary:  fmt.Sprintf("job sweep could not delete %d workspace(s)", len(failed)),
+			Detail: fmt.Sprintf("These completed jobs are past the %d-day retention period, but their workspaces under data/jobs could not be deleted. The next sweep retries them.\n\n%s\n",
+				days, strings.Join(failed, "\n")),
+		})
 	}
 	return nil
 }
diff --git a/internal/app/sweep_alert_test.go b/internal/app/sweep_alert_test.go
new file mode 100644
index 0000000..cf6a484
--- /dev/null
+++ b/internal/app/sweep_alert_test.go
@@ -0,0 +1,73 @@
+package app
+
+import (
+	"errors"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"filemill/internal/store"
+)
+
+// expireEverything makes the job sweep treat every completed job as past
+// retention, with no waiting. The horizon is a whole hour in the future, not a
+// nanosecond: the store compares RFC3339Nano text, which trims trailing zeros,
+// so times a fraction of a second apart can sort the wrong way.
+func expireEverything(t *testing.T) {
+	t.Helper()
+	saved := jobRetentionPeriod
+	jobRetentionPeriod = -time.Hour
+	t.Cleanup(func() { jobRetentionPeriod = saved })
+}
+
+func completedJob(t *testing.T, a *App) store.Job {
+	t.Helper()
+	j := claimJob(t, a)
+	if err := a.store.Complete(j.ID, store.StatusSucceeded, "ok"); err != nil {
+		t.Fatal(err)
+	}
+	return j
+}
+
+// A workspace that won't delete keeps the sender's files on disk past the
+// retention promise. The sweep reports once for the batch, naming each job.
+func TestSweepExpiredJobsReportsDeleteFailures(t *testing.T) {
+	a, rep := newTestApp(t, fakeTransformer(t, "succeed")...)
+	j := completedJob(t, a)
+	expireEverything(t)
+	saved := removeWorkspace
+	removeWorkspace = func(string) error { return errors.New("access is denied") }
+	t.Cleanup(func() { removeWorkspace = saved })
+
+	if err := a.sweepExpiredJobs(); err != nil {
+		t.Fatalf("sweepExpiredJobs: %v", err)
+	}
+
+	alerts := rep.reports()
+	if len(alerts) != 1 || alerts[0].Category != "sweep-jobs" {
+		t.Fatalf("alerts = %+v, want one sweep-jobs", alerts)
+	}
+	for _, want := range []string{j.ID, "access is denied"} {
+		if !strings.Contains(alerts[0].Detail, want) {
+			t.Errorf("detail lacks %q:\n%s", want, alerts[0].Detail)
+		}
+	}
+}
+
+func TestCleanJobSweepDoesNotReport(t *testing.T) {
+	a, rep := newTestApp(t, fakeTransformer(t, "succeed")...)
+	j := completedJob(t, a)
+	expireEverything(t)
+
+	if err := a.sweepExpiredJobs(); err != nil {
+		t.Fatalf("sweepExpiredJobs: %v", err)
+	}
+	if _, err := os.Stat(filepath.Join(a.data, "jobs", j.ID)); !os.IsNotExist(err) {
+		t.Fatalf("the job was not swept (stat err = %v), so this test proves nothing", err)
+	}
+	if alerts := rep.reports(); len(alerts) != 0 {
+		t.Errorf("a clean sweep reported: %+v", alerts)
+	}
+}

From 51f57850ac3045d34e844611273f7d5680eeb6af Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 09:26:02 -0600
Subject: [PATCH 13/25] Report Mailgun-side failures and add SendAlert

Phase 3 of the error-alerting plan (#7), Mailgun side.
- send takes its subject verbatim; deliver adds "Re:". SendAlert sends a plain
  message from REPLY_FROM with no threading or attachments, and satisfies
  alert.Mailer.
- Delivery errors carry their alert category. A submission reports delivery or
  publish once it has failed for 5 minutes, delivery-mark at once, and each
  only once per outage. Every orphaned Drive file reports publish-orphan with
  its file ID.
- The webhook reports intake on a 500 and route-config on the store(notify=)
  warning. The Drive sweep reports sweep-drive for files it can't delete.
- email.yaml gains alert_recipient and the throttle settings. A Service with
  no reporter stays silent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 internal/mailgun/alerts_test.go   | 390 ++++++++++++++++++++++++++++++
 internal/mailgun/config.go        |  25 ++
 internal/mailgun/delivery_test.go |   6 +-
 internal/mailgun/fake_test.go     |   4 +
 internal/mailgun/outbound.go      | 124 +++++++++-
 internal/mailgun/retention.go     |  23 +-
 internal/mailgun/service.go       |  38 +++
 internal/mailgun/webhook.go       |  15 +-
 8 files changed, 608 insertions(+), 17 deletions(-)
 create mode 100644 internal/mailgun/alerts_test.go

diff --git a/internal/mailgun/alerts_test.go b/internal/mailgun/alerts_test.go
new file mode 100644
index 0000000..7f9f92d
--- /dev/null
+++ b/internal/mailgun/alerts_test.go
@@ -0,0 +1,390 @@
+package mailgun
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+	"time"
+
+	"filemill/internal/alert"
+	"filemill/internal/app"
+)
+
+var _ alert.Mailer = (*Service)(nil)
+
+// fakeReporter records every alert. The Service reports from the goroutine
+// that calls it, so these tests need no locking.
+type fakeReporter struct{ alerts []alert.Alert }
+
+func (r *fakeReporter) Report(a alert.Alert) { r.alerts = append(r.alerts, a) }
+
+// only returns the alerts in one category.
+func (r *fakeReporter) only(category string) []alert.Alert {
+	var out []alert.Alert
+	for _, a := range r.alerts {
+		if a.Category == category {
+			out = append(out, a)
+		}
+	}
+	return out
+}
+
+// fakeClock stands in for the Service's clock, so a grace period is crossed
+// without sleeping.
+type fakeClock struct{ t time.Time }
+
+func (c *fakeClock) now() time.Time          { return c.t }
+func (c *fakeClock) advance(d time.Duration) { c.t = c.t.Add(d) }
+
+// newAlertingFixture is a deliveryFixture whose Service reports to a
+// fakeReporter and reads a fakeClock.
+func newAlertingFixture(t *testing.T) (*deliveryFixture, *fakeReporter, *fakeClock) {
+	t.Helper()
+	f := newDeliveryFixture(t)
+	rep := &fakeReporter{}
+	clock := &fakeClock{t: time.Date(2026, 9, 11, 9, 0, 0, 0, time.UTC)}
+	f.service.SetReporter(rep)
+	f.service.now = clock.now
+	return f, rep, clock
+}
+
+func tick(t *testing.T, f *deliveryFixture) {
+	t.Helper()
+	if err := f.service.deliverPending(context.Background()); err != nil {
+		t.Fatalf("deliverPending: %v", err)
+	}
+}
+
+// A failing reply is retried every second, and a Mailgun blip clears on its
+// own, so it reports only once a submission has failed for five minutes, and
+// then once, not on every tick after.
+func TestDeliveryFailureReportsAfterFiveMinutes(t *testing.T) {
+	f, rep, clock := newAlertingFixture(t)
+	f.addSubmission(t, 1, "excel@mill.test", "schedule.xlsx")
+	f.mailgun.failTo["sender1@example.com"] = true
+
+	tick(t, f)
+	clock.advance(5*time.Minute - time.Second)
+	tick(t, f)
+	if len(rep.alerts) != 0 {
+		t.Fatalf("reported inside the grace period: %+v", rep.alerts)
+	}
+
+	clock.advance(time.Second)
+	tick(t, f)
+	got := rep.only("delivery")
+	if len(got) != 1 || len(rep.alerts) != 1 {
+		t.Fatalf("after 5 minutes of failing, alerts = %+v; want one delivery", rep.alerts)
+	}
+	for _, want := range []string{"sender1@example.com", "simulated mailgun failure"} {
+		if !strings.Contains(got[0].Detail, want) {
+			t.Errorf("detail lacks %q:\n%s", want, got[0].Detail)
+		}
+	}
+
+	clock.advance(time.Minute)
+	tick(t, f)
+	if len(rep.alerts) != 1 {
+		t.Errorf("reported again in the same outage: %d alerts", len(rep.alerts))
+	}
+}
+
+// A failure that clears inside the grace period is the blip the grace period
+// exists for.
+func TestDeliveryBlipStaysSilent(t *testing.T) {
+	f, rep, clock := newAlertingFixture(t)
+	f.addSubmission(t, 1, "excel@mill.test", "schedule.xlsx")
+	f.mailgun.failTo["sender1@example.com"] = true
+
+	tick(t, f)
+	clock.advance(4 * time.Minute)
+	f.mailgun.failTo = map[string]bool{}
+	tick(t, f)
+	clock.advance(10 * time.Minute)
+	tick(t, f)
+
+	if !f.engine.delivered[1] {
+		t.Fatal("the submission should have been delivered once Mailgun recovered")
+	}
+	if len(rep.alerts) != 0 {
+		t.Errorf("a failure that cleared within the grace period reported: %+v", rep.alerts)
+	}
+}
+
+func TestPublishFailureReportsAfterFiveMinutes(t *testing.T) {
+	f, rep, clock := newAlertingFixture(t)
+	f.addSubmission(t, 1, "iwk@mill.test", "schedule.xlsx")
+	f.publisher.err = fmt.Errorf("google token expired")
+
+	tick(t, f)
+	clock.advance(5 * time.Minute)
+	tick(t, f)
+
+	got := rep.only("publish")
+	if len(got) != 1 || len(rep.alerts) != 1 {
+		t.Fatalf("alerts = %+v; want one publish", rep.alerts)
+	}
+	if !strings.Contains(got[0].Detail, "google token expired") {
+		t.Errorf("detail lacks the publish error:\n%s", got[0].Detail)
+	}
+}
+
+// A reply that went out but can't be marked delivered means the database is
+// failing, and every restart until it recovers sends another duplicate, so it
+// skips the grace period.
+func TestMarkFailureReportsAtOnce(t *testing.T) {
+	f, rep, clock := newAlertingFixture(t)
+	f.addSubmission(t, 1, "excel@mill.test", "schedule.xlsx")
+	f.engine.markErr = fmt.Errorf("disk I/O error")
+
+	tick(t, f)
+	if got := rep.only("delivery-mark"); len(got) != 1 || len(rep.alerts) != 1 {
+		t.Fatalf("alerts = %+v; want one delivery-mark at the first failure", rep.alerts)
+	}
+
+	clock.advance(10 * time.Minute)
+	tick(t, f)
+	if len(rep.alerts) != 1 {
+		t.Errorf("retrying the mark reported again: %d alerts", len(rep.alerts))
+	}
+}
+
+// An upload whose record failed leaves a world-editable file in Drive that the
+// retention sweep will never delete. Each one is reported at once, naming the
+// file, since only a person can clean it up.
+func TestOrphanedDriveFileReportsAtOnce(t *testing.T) {
+	f, rep, _ := newAlertingFixture(t)
+	f.addSubmission(t, 1, "iwk@mill.test", "schedule.xlsx")
+	f.engine.putDeliveryErr = fmt.Errorf("database is locked")
+
+	tick(t, f)
+	tick(t, f) // the retry uploads again, orphaning a second file
+
+	got := rep.only("publish-orphan")
+	if len(got) != 2 {
+		t.Fatalf("orphan alerts = %+v; want one per orphaned file", got)
+	}
+	for i, id := range []string{"drive-file-1", "drive-file-2"} {
+		if !strings.Contains(got[i].Detail, id) {
+			t.Errorf("orphan alert %d does not name %s:\n%s", i+1, id, got[i].Detail)
+		}
+	}
+	if len(rep.only("publish")) != 0 {
+		t.Error("the publish grace period still applies to the submission itself")
+	}
+}
+
+// A 500 from intake is our fault. Mailgun retries it for hours, and the
+// throttle absorbs the burst.
+func TestIntakeFailureReports(t *testing.T) {
+	engine := newFakeEngine()
+	engine.submitErr = fmt.Errorf("disk full")
+	s := routedService(engine)
+	rep := &fakeReporter{}
+	s.SetReporter(rep)
+
+	r := signedMultipart(t, "k",
+		map[string]string{"recipient": "workerlist@mill.example.com", "sender": "kevin@example.com", "Message-Id": "<orig-3@example.com>"},
+		map[string][]byte{"attachment-1": []byte("pdf")})
+	if status, _ := s.receive(r); status != http.StatusInternalServerError {
+		t.Fatalf("status = %d, want 500", status)
+	}
+
+	got := rep.only("intake")
+	if len(got) != 1 || len(rep.alerts) != 1 {
+		t.Fatalf("alerts = %+v; want one intake", rep.alerts)
+	}
+	for _, want := range []string{"disk full", "kevin@example.com"} {
+		if !strings.Contains(got[0].Detail, want) {
+			t.Errorf("detail lacks %q:\n%s", want, got[0].Detail)
+		}
+	}
+}
+
+// Webhook noise and the sender's own mistakes are not FileMill failures.
+func TestRoutineWebhookOutcomesDoNotReport(t *testing.T) {
+	for _, tc := range []struct {
+		name    string
+		prepare func(*fakeEngine)
+		request func(t *testing.T) *http.Request
+	}{
+		{"rejected attachment", func(e *fakeEngine) {
+			e.submitErr = fmt.Errorf(`workerlist does not accept ".html": %w`, app.ErrRejected)
+		}, func(t *testing.T) *http.Request {
+			return signedMultipart(t, "k", map[string]string{"recipient": "workerlist@mill.example.com", "sender": "kevin@example.com"},
+				map[string][]byte{"attachment-1": []byte("<html>")})
+		}},
+		{"unrouted recipient", nil, func(t *testing.T) *http.Request {
+			return signedMultipart(t, "k", map[string]string{"recipient": "nobody@mill.example.com", "sender": "kevin@example.com"},
+				map[string][]byte{"attachment-1": []byte("pdf")})
+		}},
+		{"forged signature", nil, func(t *testing.T) *http.Request {
+			r := httptest.NewRequest(http.MethodPost, "/mailgun-webhook", strings.NewReader("timestamp=1&token=x&signature=forged"))
+			r.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+			return r
+		}},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			engine := newFakeEngine()
+			if tc.prepare != nil {
+				tc.prepare(engine)
+			}
+			s := routedService(engine)
+			rep := &fakeReporter{}
+			s.SetReporter(rep)
+
+			s.receive(tc.request(t))
+
+			if len(rep.alerts) != 0 {
+				t.Errorf("reported: %+v", rep.alerts)
+			}
+		})
+	}
+}
+
+// Attachments declared but not posted mean the Mailgun route stores messages
+// instead of forwarding them, and every real submission is being lost.
+func TestStoredRouteWarningReports(t *testing.T) {
+	s := routedService(newFakeEngine())
+	rep := &fakeReporter{}
+	s.SetReporter(rep)
+	ts, token, sig := signedFields(s.signKey)
+	body := "timestamp=" + ts + "&token=" + token + "&signature=" + sig +
+		"&recipient=workerlist%40mill.example.com&sender=someone%40example.com&attachment-count=2"
+	r := httptest.NewRequest(http.MethodPost, "/mailgun-webhook", strings.NewReader(body))
+	r.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+
+	s.receive(r)
+
+	got := rep.only("route-config")
+	if len(got) != 1 || !strings.Contains(got[0].Detail, "store(notify") {
+		t.Fatalf("alerts = %+v; want one route-config naming the likely cause", rep.alerts)
+	}
+}
+
+// A published file that won't delete is world-editable data outliving the
+// 30-day promise. The sweep reports once for the batch, naming every file.
+func TestSweepReportsDriveDeleteFailures(t *testing.T) {
+	f, rep, _ := newAlertingFixture(t)
+	old := time.Now().UTC().Add(-31 * 24 * time.Hour)
+	f.age(1, 0, "stuck-file", old)
+	f.age(2, 0, "other-file", old)
+	f.publisher.deleteErr = fmt.Errorf("drive unavailable")
+
+	if err := f.service.sweepExpired(context.Background()); err != nil {
+		t.Fatalf("sweepExpired: %v", err)
+	}
+
+	got := rep.only("sweep-drive")
+	if len(got) != 1 {
+		t.Fatalf("alerts = %+v; want one sweep-drive for the batch", rep.alerts)
+	}
+	for _, want := range []string{"stuck-file", "other-file", "drive unavailable"} {
+		if !strings.Contains(got[0].Detail, want) {
+			t.Errorf("detail lacks %q:\n%s", want, got[0].Detail)
+		}
+	}
+}
+
+func TestCleanSweepDoesNotReport(t *testing.T) {
+	f, rep, _ := newAlertingFixture(t)
+	f.age(1, 0, "old-file", time.Now().UTC().Add(-31*24*time.Hour))
+
+	if err := f.service.sweepExpired(context.Background()); err != nil {
+		t.Fatalf("sweepExpired: %v", err)
+	}
+	if len(rep.alerts) != 0 {
+		t.Errorf("a clean sweep reported: %+v", rep.alerts)
+	}
+}
+
+// The reply adds "Re:" itself, so send can carry an alert's subject verbatim.
+func TestReplySubjectCarriesRe(t *testing.T) {
+	f := newDeliveryFixture(t)
+	f.addSubmission(t, 1, "excel@mill.test", "schedule.xlsx")
+
+	deliverOne(t, f)
+
+	if got := f.mailgun.sent[0].subject; got != "Re: Schedule" {
+		t.Errorf("reply subject = %q, want %q", got, "Re: Schedule")
+	}
+}
+
+// An alert is a plain message: its subject as given, from REPLY_FROM, with no
+// threading headers and no attachments.
+func TestSendAlertPostsAPlainMessage(t *testing.T) {
+	var fields map[string][]string
+	var files int
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if err := r.ParseMultipartForm(1 << 20); err != nil {
+			t.Errorf("parse form: %v", err)
+			return
+		}
+		fields, files = r.MultipartForm.Value, len(r.MultipartForm.File)
+		w.WriteHeader(http.StatusOK)
+	}))
+	defer server.Close()
+	s := sendTestService(server.URL, &http.Client{Timeout: time.Second})
+
+	if err := s.SendAlert(context.Background(), "support@example.com", "[FileMill] intake failing", "the detail"); err != nil {
+		t.Fatalf("SendAlert: %v", err)
+	}
+
+	for field, want := range map[string]string{
+		"to":      "support@example.com",
+		"from":    "filemill@mill.test",
+		"subject": "[FileMill] intake failing",
+		"text":    "the detail",
+	} {
+		if got := fields[field]; len(got) != 1 || got[0] != want {
+			t.Errorf("%s = %v, want %q", field, got, want)
+		}
+	}
+	if _, ok := fields["h:In-Reply-To"]; ok {
+		t.Error("an alert must not thread into a conversation")
+	}
+	if files != 0 {
+		t.Errorf("an alert carried %d attachments, want none", files)
+	}
+}
+
+func TestAlertSettings(t *testing.T) {
+	to, cfg, err := alertSettings(fileConfig{})
+	if err != nil || to != "" || cfg != (alert.Config{}) {
+		t.Fatalf("no alert keys = %q, %+v, %v; want alerting off and the defaults", to, cfg, err)
+	}
+
+	to, cfg, err = alertSettings(fileConfig{
+		AlertRecipient: " support@example.com ", AlertCooldownMinutes: 20, AlertMaxPerHour: 5, AlertMaxPerDay: 12,
+	})
+	if err != nil {
+		t.Fatal(err)
+	}
+	want := alert.Config{Cooldown: 20 * time.Minute, MaxPerHour: 5, MaxPerDay: 12}
+	if to != "support@example.com" || cfg != want {
+		t.Errorf("settings = %q, %+v; want support@example.com, %+v", to, cfg, want)
+	}
+
+	if _, _, err := alertSettings(fileConfig{AlertRecipient: "support"}); err == nil {
+		t.Error("an alert_recipient that is not an address must fail at load")
+	}
+}
+
+// A Service built without SetReporter, as most tests build it, must not panic
+// on a path that reports.
+func TestServiceWithoutReporterDoesNotPanic(t *testing.T) {
+	engine := newFakeEngine()
+	engine.submitErr = errors.New("disk full")
+	s := routedService(engine)
+	r := signedMultipart(t, "k",
+		map[string]string{"recipient": "workerlist@mill.example.com", "sender": "kevin@example.com"},
+		map[string][]byte{"attachment-1": []byte("pdf")})
+
+	if status, _ := s.receive(r); status != http.StatusInternalServerError {
+		t.Fatalf("status = %d, want 500", status)
+	}
+}
diff --git a/internal/mailgun/config.go b/internal/mailgun/config.go
index 1c7b626..2e6bf48 100644
--- a/internal/mailgun/config.go
+++ b/internal/mailgun/config.go
@@ -9,6 +9,7 @@ import (
 	"strings"
 	"time"
 
+	"filemill/internal/alert"
 	"filemill/internal/app"
 	"filemill/internal/gsheets"
 	"gopkg.in/yaml.v3"
@@ -32,6 +33,12 @@ type fileConfig struct {
 	Allowed     []string          `yaml:"allowed_senders"`
 	Max         int64             `yaml:"max_attachment_bytes"`
 	SendTimeout int               `yaml:"send_timeout_seconds"`
+
+	// Operator alerts; see alertSettings.
+	AlertRecipient       string `yaml:"alert_recipient"`
+	AlertCooldownMinutes int    `yaml:"alert_cooldown_minutes"`
+	AlertMaxPerHour      int    `yaml:"alert_max_per_hour"`
+	AlertMaxPerDay       int    `yaml:"alert_max_per_day"`
 }
 
 // credentials holds the Google OAuth secrets the sheets-link delivery mode
@@ -93,6 +100,9 @@ func Load(root string, engine *app.App, logger *log.Logger) (*Service, error) {
 		return nil, err
 	}
 	s.delivery = delivery
+	if s.alertTo, s.alertCfg, err = alertSettings(cfg); err != nil {
+		return nil, err
+	}
 
 	switch {
 	case s.apiKey == "" && s.signKey == "" && s.domain == "" && s.from == "":
@@ -120,6 +130,21 @@ func Load(root string, engine *app.App, logger *log.Logger) (*Service, error) {
 	return s, nil
 }
 
+// alertSettings reads the operator-alert keys. An empty alert_recipient turns
+// alerting off. Zero throttle settings take the alert package's defaults,
+// whose daily cap is sized for the Mailgun Free plan.
+func alertSettings(cfg fileConfig) (string, alert.Config, error) {
+	to := strings.TrimSpace(cfg.AlertRecipient)
+	if to != "" && !strings.Contains(to, "@") {
+		return "", alert.Config{}, fmt.Errorf("alert_recipient %q is not an email address", to)
+	}
+	return to, alert.Config{
+		Cooldown:   time.Duration(cfg.AlertCooldownMinutes) * time.Minute,
+		MaxPerHour: cfg.AlertMaxPerHour,
+		MaxPerDay:  cfg.AlertMaxPerDay,
+	}, nil
+}
+
 // parseDelivery normalizes the address -> delivery mode map and rejects any
 // value that is not a known mode. An unrecognized mode is a typo, and a typo
 // that silently left the route on attachment delivery would be invisible.
diff --git a/internal/mailgun/delivery_test.go b/internal/mailgun/delivery_test.go
index 537687d..af98a54 100644
--- a/internal/mailgun/delivery_test.go
+++ b/internal/mailgun/delivery_test.go
@@ -48,8 +48,8 @@ func (p *fakePublisher) Delete(_ context.Context, fileID string) error {
 
 // sentMessage is one reply captured from the fake Mailgun endpoint.
 type sentMessage struct {
-	to, text    string
-	attachments []string
+	to, subject, text string
+	attachments       []string
 }
 
 // fakeMailgun is a stand-in Send API that records replies. Addresses listed in
@@ -81,6 +81,8 @@ func (m *fakeMailgun) start(t *testing.T) *httptest.Server {
 				msg.attachments = append(msg.attachments, part.FileName())
 			case part.FormName() == "to":
 				msg.to = string(buf[:n])
+			case part.FormName() == "subject":
+				msg.subject = string(buf[:n])
 			case part.FormName() == "text":
 				msg.text = string(buf[:n])
 			}
diff --git a/internal/mailgun/fake_test.go b/internal/mailgun/fake_test.go
index 56e4f31..e8be314 100644
--- a/internal/mailgun/fake_test.go
+++ b/internal/mailgun/fake_test.go
@@ -58,6 +58,7 @@ type fakeEngine struct {
 
 	deliveries      map[deliveryKey]store.Delivery
 	sweptDeliveries map[deliveryKey]bool
+	putDeliveryErr  error // when set, PutDelivery fails with it
 
 	// labels maps an operation to the report name a reply calls it by. An
 	// operation absent here falls back to its own name, as the real engine does.
@@ -174,6 +175,9 @@ type deliveryKey struct {
 
 func (f *fakeEngine) PutDelivery(submissionID int64, outputIndex int, fileID, link string) error {
 	f.calls = append(f.calls, fmt.Sprintf("PutDelivery(%d,%d)", submissionID, outputIndex))
+	if f.putDeliveryErr != nil {
+		return f.putDeliveryErr
+	}
 	key := deliveryKey{submissionID, outputIndex}
 	if _, exists := f.deliveries[key]; exists {
 		return nil // INSERT OR IGNORE: never overwrite a link already mailed out
diff --git a/internal/mailgun/outbound.go b/internal/mailgun/outbound.go
index cb345c9..ae82485 100644
--- a/internal/mailgun/outbound.go
+++ b/internal/mailgun/outbound.go
@@ -3,6 +3,7 @@ package mailgun
 import (
 	"bytes"
 	"context"
+	"errors"
 	"fmt"
 	"io"
 	"mime/multipart"
@@ -12,6 +13,7 @@ import (
 	"strings"
 	"time"
 
+	"filemill/internal/alert"
 	"filemill/internal/app"
 	"filemill/internal/store"
 )
@@ -20,6 +22,11 @@ import (
 // groups whose jobs have all finished.
 const deliveryPollInterval = time.Second
 
+// deliveryAlertAfter is how long a submission must keep failing before it is
+// reported. Delivery retries every second, and a Mailgun or Drive blip clears
+// well inside this.
+const deliveryAlertAfter = 5 * time.Minute
+
 // Deliver runs the outbound loop until ctx is cancelled: once every job in a
 // submission group is terminal, it mails one threaded reply carrying every
 // successful output. The inbound handler returns as soon as jobs are queued;
@@ -50,8 +57,9 @@ func (s *Service) Deliver(ctx context.Context) {
 // failure to read the queue itself aborts the tick.
 //
 // Per-submission failure counting and a dead-letter status are a follow-up; for
-// now a stuck submission is retried on every tick, and its log line is the
-// signal that something needs attention.
+// now a stuck submission is retried on every tick. Its log line, and an alert
+// once it has failed for deliveryAlertAfter, are the signal that something
+// needs attention.
 func (s *Service) deliverPending(ctx context.Context) error {
 	subs, err := s.engine.PendingEmails()
 	if err != nil {
@@ -63,7 +71,10 @@ func (s *Service) deliverPending(ctx context.Context) error {
 		}
 		if err := s.deliver(ctx, sub); err != nil {
 			s.log.Printf("delivery: submission %d (from %s): %v", sub.ID, sub.Sender, err)
+			s.deliveryFailed(sub, err)
+			continue
 		}
+		delete(s.failing, sub.ID)
 	}
 	return nil
 }
@@ -136,7 +147,7 @@ func (s *Service) deliver(ctx context.Context, sub store.EmailSubmission) error
 		}
 	}
 
-	if err := s.send(ctx, sub.Sender, sub.Subject, threadingID(sub.MessageID), text, attachments); err != nil {
+	if err := s.send(ctx, sub.Sender, replySubject(sub.Subject), threadingID(sub.MessageID), text, attachments); err != nil {
 		return err
 	}
 	return s.markDelivered(sub.ID)
@@ -152,12 +163,94 @@ func (s *Service) markDelivered(id int64) error {
 			s.sentUnmarked = map[int64]bool{}
 		}
 		s.sentUnmarked[id] = true
-		return fmt.Errorf("reply sent, but marking it delivered failed (retrying the mark only): %w", err)
+		return failedAt("delivery-mark", fmt.Errorf("reply sent, but marking it delivered failed (retrying the mark only): %w", err))
 	}
 	delete(s.sentUnmarked, id)
 	return nil
 }
 
+// deliveryError labels a delivery failure with the alert category it belongs
+// to. An unlabelled error is a failing reply: "delivery".
+type deliveryError struct {
+	category string
+	err      error
+}
+
+func (e *deliveryError) Error() string { return e.err.Error() }
+func (e *deliveryError) Unwrap() error { return e.err }
+
+func failedAt(category string, err error) error { return &deliveryError{category, err} }
+
+func categoryOf(err error) string {
+	var labelled *deliveryError
+	if errors.As(err, &labelled) {
+		return labelled.category
+	}
+	return "delivery"
+}
+
+// deliveryOutage is one submission's unbroken run of failed deliveries.
+type deliveryOutage struct {
+	since    time.Time
+	reported bool
+}
+
+// deliveryFailed reports a failing submission once per outage: after
+// deliveryAlertAfter, or at once for a failed mark, where every restart before
+// it heals sends another duplicate. An orphaned Drive file is reported every
+// time, since each names a different file that only a person can delete.
+// deliverPending ends the outage when the submission is delivered.
+func (s *Service) deliveryFailed(sub store.EmailSubmission, err error) {
+	now := s.clock()
+	category := categoryOf(err)
+	if category == "publish-orphan" {
+		s.report(alert.Alert{
+			Category: category,
+			Summary:  "published Drive file orphaned",
+			Detail: submissionDetail(sub, err) +
+				"\nThe file was uploaded to Google Drive, but its record was not saved, so the retention sweep will never delete it. Delete it by hand.\n",
+		})
+		category = "publish" // the submission itself still gets the grace period
+	}
+
+	if s.failing == nil {
+		s.failing = map[int64]*deliveryOutage{}
+	}
+	outage := s.failing[sub.ID]
+	if outage == nil {
+		outage = &deliveryOutage{since: now}
+		s.failing[sub.ID] = outage
+	}
+	grace := deliveryAlertAfter
+	if category == "delivery-mark" {
+		grace = 0
+	}
+	if outage.reported || now.Sub(outage.since) < grace {
+		return
+	}
+	outage.reported = true
+	s.report(alert.Alert{
+		Category: category,
+		Summary:  outageSummary(category),
+		Detail:   fmt.Sprintf("%s\nFailing since %s, retried every second.\n", submissionDetail(sub, err), outage.since.Format(time.RFC3339)),
+	})
+}
+
+func outageSummary(category string) string {
+	minutes := int(deliveryAlertAfter.Minutes())
+	switch category {
+	case "delivery-mark":
+		return "reply sent but not recorded as delivered"
+	case "publish":
+		return fmt.Sprintf("sheets-link publish failing for %d minutes", minutes)
+	}
+	return fmt.Sprintf("reply delivery failing for %d minutes", minutes)
+}
+
+func submissionDetail(sub store.EmailSubmission, err error) string {
+	return fmt.Sprintf("Submission: %d\nFrom: %s\nTo: %s\nSubject: %s\n\nError: %v\n", sub.ID, sub.Sender, sub.Recipient, sub.Subject, err)
+}
+
 // deliveryMode returns how replies to a recipient address are delivered.
 func (s *Service) deliveryMode(recipient string) string {
 	if mode := s.delivery[strings.ToLower(recipient)]; mode != "" {
@@ -182,13 +275,13 @@ func (s *Service) deliveryMode(recipient string) string {
 // distributed transaction against Google.
 func (s *Service) publish(ctx context.Context, submissionID int64, outputs []app.OutputFile) ([]string, error) {
 	if s.publisher == nil {
-		return nil, fmt.Errorf("sheets-link delivery is configured for this route but no publisher is available")
+		return nil, failedAt("publish", fmt.Errorf("sheets-link delivery is configured for this route but no publisher is available"))
 	}
 	links := make([]string, 0, len(outputs))
 	for i, out := range outputs {
 		record, published, err := s.engine.Delivery(submissionID, i)
 		if err != nil {
-			return nil, fmt.Errorf("look up published output %d: %w", i, err)
+			return nil, failedAt("publish", fmt.Errorf("look up published output %d: %w", i, err))
 		}
 		if published {
 			links = append(links, record.Link)
@@ -199,12 +292,12 @@ func (s *Service) publish(ctx context.Context, submissionID int64, outputs []app
 		title := strings.TrimSuffix(out.Name, filepath.Ext(out.Name))
 		fileID, link, err := s.publisher.Publish(ctx, out.Path, title)
 		if err != nil {
-			return nil, fmt.Errorf("publish %s: %w", out.Name, err)
+			return nil, failedAt("publish", fmt.Errorf("publish %s: %w", out.Name, err))
 		}
 		if err := s.engine.PutDelivery(submissionID, i, fileID, link); err != nil {
 			// The upload succeeded but is now unrecorded, so a retry would
 			// re-upload it. Name the orphan so it can be found by hand.
-			return nil, fmt.Errorf("record published file %s (now orphaned in Drive): %w", fileID, err)
+			return nil, failedAt("publish-orphan", fmt.Errorf("record published file %s (now orphaned in Drive): %w", fileID, err))
 		}
 		links = append(links, link)
 	}
@@ -263,8 +356,17 @@ func withLinks(text string, links, labels []string) string {
 	return b.String()
 }
 
-// send posts one reply to Mailgun's Send API, threaded into the original
-// conversation and carrying the given output files as attachments.
+// SendAlert sends one plain-text operator alert from REPLY_FROM, with its
+// subject as given: no "Re:", no threading, no attachments. It satisfies
+// alert.Mailer. It never reports its own failure: the alert channel is the
+// broken thing, and reporting it would only loop.
+func (s *Service) SendAlert(ctx context.Context, to, subject, text string) error {
+	return s.send(ctx, to, subject, "", text, nil)
+}
+
+// send posts one message to Mailgun's Send API with its subject as given.
+// With a messageID it is threaded into that conversation, and it carries the
+// given output files as attachments.
 func (s *Service) send(ctx context.Context, to, subject, messageID, text string, outputs []string) error {
 	var body bytes.Buffer
 	writer := multipart.NewWriter(&body)
@@ -272,7 +374,7 @@ func (s *Service) send(ctx context.Context, to, subject, messageID, text string,
 	fields := map[string]string{
 		"from":    s.from,
 		"to":      to,
-		"subject": replySubject(subject),
+		"subject": subject,
 		"text":    text,
 	}
 	if messageID != "" {
diff --git a/internal/mailgun/retention.go b/internal/mailgun/retention.go
index f8b9c08..1733937 100644
--- a/internal/mailgun/retention.go
+++ b/internal/mailgun/retention.go
@@ -2,7 +2,11 @@ package mailgun
 
 import (
 	"context"
+	"fmt"
+	"strings"
 	"time"
+
+	"filemill/internal/alert"
 )
 
 const (
@@ -43,8 +47,10 @@ func (s *Service) SweepExpired(ctx context.Context) {
 //
 // A file that fails to delete is logged and left unmarked, so the next sweep
 // tries again; one unreachable file must not strand the rest of the batch. The
-// delete itself is idempotent — a file already gone counts as deleted — so a
-// record marked after a crash-interrupted sweep is never a problem.
+// failures are reported together, once per sweep: each is a world-editable
+// copy of a sender's data outliving the retention promise. The delete itself
+// is idempotent — a file already gone counts as deleted — so a record marked
+// after a crash-interrupted sweep is never a problem.
 //
 // A sweep that deleted something says so. Retention runs unattended, on a daily
 // timer, and removes other people's data: if only failures were logged, a sweep
@@ -59,9 +65,11 @@ func (s *Service) sweepExpired(ctx context.Context) error {
 		return err
 	}
 	deleted := 0
+	var failed []string
 	for _, record := range expired {
 		if err := s.publisher.Delete(ctx, record.FileID); err != nil {
 			s.log.Printf("retention sweep: delete %s (submission %d): %v", record.FileID, record.SubmissionID, err)
+			failed = append(failed, fmt.Sprintf("%s (submission %d): %v", record.FileID, record.SubmissionID, err))
 			continue
 		}
 		deleted++
@@ -69,8 +77,17 @@ func (s *Service) sweepExpired(ctx context.Context) error {
 			s.log.Printf("retention sweep: record deletion of %s: %v", record.FileID, err)
 		}
 	}
+	days := int(retentionPeriod.Hours() / 24)
 	if deleted > 0 {
-		s.log.Printf("retention sweep: deleted %d published file(s) older than %d days", deleted, int(retentionPeriod.Hours()/24))
+		s.log.Printf("retention sweep: deleted %d published file(s) older than %d days", deleted, days)
+	}
+	if len(failed) > 0 {
+		s.report(alert.Alert{
+			Category: "sweep-drive",
+			Summary:  fmt.Sprintf("retention sweep could not delete %d Drive file(s)", len(failed)),
+			Detail: fmt.Sprintf("These published files are past the %d-day retention period and are still shared with anyone who has the link. The next sweep retries them.\n\n%s\n",
+				days, strings.Join(failed, "\n")),
+		})
 	}
 	return nil
 }
diff --git a/internal/mailgun/service.go b/internal/mailgun/service.go
index 711fcbd..8e26a84 100644
--- a/internal/mailgun/service.go
+++ b/internal/mailgun/service.go
@@ -20,6 +20,7 @@ import (
 	"net/http"
 	"time"
 
+	"filemill/internal/alert"
 	"filemill/internal/app"
 	"filemill/internal/store"
 )
@@ -100,8 +101,45 @@ type Service struct {
 	// delivered mark failed, so the delivery loop retries only the mark (see
 	// markDelivered). Only the delivery goroutine touches it; nil until needed.
 	sentUnmarked map[int64]bool
+
+	// Operator alerts (internal/alert). reporter is nil until SetReporter, and
+	// report treats nil as disabled, so a Service built in a test needs none.
+	reporter alert.Reporter
+	alertTo  string       // alert_recipient from email.yaml; empty disables alerting
+	alertCfg alert.Config // alert throttle settings from email.yaml
+
+	// failing holds each submission whose delivery keeps failing, for the grace
+	// period before it is reported (see deliveryFailed). Only the delivery
+	// goroutine touches it; nil until needed.
+	failing map[int64]*deliveryOutage
+	now     func() time.Time // nil means time.Now; tests set a fake clock
 }
 
 // Handler returns the webhook HTTP handler. It is mounted by cmd/filemill run
 // only in continuous worker mode.
 func (s *Service) Handler() http.Handler { return http.HandlerFunc(s.handle) }
+
+// SetReporter routes the adapter's systemic failures to r. Call it before the
+// webhook server and delivery loop start.
+func (s *Service) SetReporter(r alert.Reporter) { s.reporter = r }
+
+// AlertRecipient is the operator address from email.yaml. Empty means
+// alerting is off.
+func (s *Service) AlertRecipient() string { return s.alertTo }
+
+// AlertConfig is the alert throttle's settings from email.yaml. Zero fields
+// take the alert package's defaults.
+func (s *Service) AlertConfig() alert.Config { return s.alertCfg }
+
+func (s *Service) report(a alert.Alert) {
+	if s.reporter != nil {
+		s.reporter.Report(a)
+	}
+}
+
+func (s *Service) clock() time.Time {
+	if s.now != nil {
+		return s.now()
+	}
+	return time.Now()
+}
diff --git a/internal/mailgun/webhook.go b/internal/mailgun/webhook.go
index ed24775..43931d7 100644
--- a/internal/mailgun/webhook.go
+++ b/internal/mailgun/webhook.go
@@ -10,6 +10,7 @@ import (
 	"strconv"
 	"strings"
 
+	"filemill/internal/alert"
 	"filemill/internal/app"
 )
 
@@ -75,7 +76,13 @@ func (s *Service) receive(r *http.Request) (status int, reason string) {
 		// leaves the bytes behind a storage URL. Every real submission would
 		// vanish here, looking exactly like ordinary empty mail, so name it.
 		if declared := declaredAttachments(r); declared > 0 {
-			return http.StatusOK, fmt.Sprintf("WARNING: message declares %d attachment(s) but none arrived inline — is the Mailgun route using store(notify=) instead of forward(url)? (to %s from %s)", declared, recipient, sender)
+			reason := fmt.Sprintf("WARNING: message declares %d attachment(s) but none arrived inline — is the Mailgun route using store(notify=) instead of forward(url)? (to %s from %s)", declared, recipient, sender)
+			s.report(alert.Alert{
+				Category: "route-config",
+				Summary:  "Mailgun route is not forwarding attachments",
+				Detail:   reason + "\n\nEvery submission to this address is lost until its Mailgun route uses forward(url).\n",
+			})
+			return http.StatusOK, reason
 		}
 		// Routine, but still recorded: with nothing logged at all, a message
 		// that arrived and one that never did looked identical.
@@ -112,6 +119,12 @@ func (s *Service) receive(r *http.Request) (status int, reason string) {
 			// Mailgun stops retrying, and record who sent it and why.
 			return http.StatusOK, fmt.Sprintf("attachment rejected from %s: %v", sender, err)
 		}
+		s.report(alert.Alert{
+			Category: "intake",
+			Summary:  "webhook intake failing",
+			Detail: fmt.Sprintf("Recipient: %s\nSender: %s\nOperation: %s\n\nError: %v\n\nThe webhook returned 500, so Mailgun will retry this message for several hours.\n",
+				recipient, sender, operation, err),
+		})
 		return http.StatusInternalServerError, fmt.Sprintf("intake failed from %s: %v", sender, err)
 	}
 	return http.StatusOK, fmt.Sprintf("accepted: sender=%q recipient=%q operation=%s", sender, r.FormValue("recipient"), operation)

From c2dc0920e55e72487ee2ab319a30a450f34ec4bc Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 09:26:02 -0600
Subject: [PATCH 14/25] Wire operator alerts into the worker

In continuous mode with Mailgun configured and alert_recipient set, run
builds an alert Emailer over the Mailgun service and the App's ledger and
sets it as the reporter on both. It runs on its own context, stopped after
the job loop and webhook server have finished, so an alert raised during
shutdown still goes out. Without alert_recipient, or in --once mode, nothing
changes. email.yaml.example documents the new keys.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 cmd/filemill/main.go      | 32 ++++++++++++++++++++++++++++++++
 config/email.yaml.example | 14 ++++++++++++++
 2 files changed, 46 insertions(+)

diff --git a/cmd/filemill/main.go b/cmd/filemill/main.go
index 763bc1e..9f75408 100644
--- a/cmd/filemill/main.go
+++ b/cmd/filemill/main.go
@@ -13,6 +13,7 @@ import (
 	"syscall"
 	"time"
 
+	"filemill/internal/alert"
 	"filemill/internal/app"
 	"filemill/internal/mailgun"
 )
@@ -80,6 +81,13 @@ func main() {
 		defer cancel()
 		var server *http.Server
 		serverErrs := make(chan error, 1)
+		// The alert Emailer runs on a context of its own, stopped only after the
+		// job loop and the webhook server have finished, so an alert raised
+		// while they shut down still goes out. alertsDone closes once it has
+		// drained.
+		alertCtx, stopAlerts := context.WithCancel(context.Background())
+		defer stopAlerts()
+		var alertsDone chan struct{}
 		if !once {
 			mailLog := log.New(io.MultiWriter(os.Stderr, application.LogWriter()), "mailgun ", log.LstdFlags|log.LUTC)
 			mail, err := mailgun.Load(root, application, mailLog)
@@ -87,6 +95,17 @@ func main() {
 				fatal(err)
 			}
 			if mail != nil {
+				// Operator alerts go out through the Mailgun adapter, from
+				// REPLY_FROM, so they exist only when it does. The reporters are
+				// set before any goroutine starts, so nothing reports into the
+				// no-op default by accident.
+				var emailer *alert.Emailer
+				if to := mail.AlertRecipient(); to != "" {
+					alertLog := log.New(io.MultiWriter(os.Stderr, application.LogWriter()), "alert ", log.LstdFlags|log.LUTC)
+					emailer = alert.NewEmailer(mail, application, to, mail.AlertConfig(), time.Now, alertLog)
+					application.SetReporter(emailer)
+					mail.SetReporter(emailer)
+				}
 				server = &http.Server{Addr: os.Getenv("LISTEN_ADDR"), Handler: mail.Handler()}
 				if server.Addr == "" {
 					server.Addr = ":8080"
@@ -107,6 +126,13 @@ func main() {
 					fatal(fmt.Errorf("webhook server: %w", err))
 				}
 				mailLog.Printf("FileMill %s — webhook listening on %s; delivery loop started", version, server.Addr)
+				if emailer != nil {
+					alertsDone = make(chan struct{})
+					go func() { emailer.Run(alertCtx); close(alertsDone) }()
+					mailLog.Printf("operator alerts: sent to %s", mail.AlertRecipient())
+				} else {
+					mailLog.Print("operator alerts: disabled (no alert_recipient in email.yaml)")
+				}
 				go func() {
 					if err := server.Serve(listener); err != nil && err != http.ErrServerClosed {
 						mailLog.Printf("server: %v", err)
@@ -134,6 +160,12 @@ func main() {
 			_ = server.Shutdown(shutdownCtx)
 			cancelShutdown()
 		}
+		if alertsDone != nil {
+			// Everything that reports has stopped. The Emailer sends what is
+			// still queued, for up to 5 seconds, then returns.
+			stopAlerts()
+			<-alertsDone
+		}
 		// Checked before runErr: when the listener is what failed, Run returns
 		// nil (it stopped because its context was cancelled), and reporting a
 		// clean stop would hide the actual cause.
diff --git a/config/email.yaml.example b/config/email.yaml.example
index 1756e2f..99bec5b 100644
--- a/config/email.yaml.example
+++ b/config/email.yaml.example
@@ -26,3 +26,17 @@ max_attachment_bytes: 20971520
 # Timeout for an outbound Mailgun send (reply). Bounds a hung connection so it
 # can't stall the delivery loop. Defaults to 30 if unset or <= 0.
 send_timeout_seconds: 30
+
+# Operator alerts. Systemic failures (a crashed or timed-out transformer, replies
+# failing for 5 minutes, intake errors, retention sweeps that can't delete) are
+# emailed to this address, from REPLY_FROM. Empty or absent turns them off.
+#
+# Alerts are throttled: one per category every alert_cooldown_minutes (the next
+# email counts the ones held back), and at most alert_max_per_hour and
+# alert_max_per_day across every category. The daily cap is sized for the
+# Mailgun Free plan, whose 100 sends a day are shared with replies; raise it
+# only on a paid plan. The values shown are the defaults.
+alert_recipient: ""   # e.g. support@example.com
+# alert_cooldown_minutes: 15
+# alert_max_per_hour: 10
+# alert_max_per_day: 20

From 4242a356b82e09c32a5eff2d5aec8c06f9499064 Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 09:26:02 -0600
Subject: [PATCH 15/25] Record phase 3 in the alerting plan

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 ERROR-ALERTING-PLAN.md | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/ERROR-ALERTING-PLAN.md b/ERROR-ALERTING-PLAN.md
index 483ca7a..7077ef5 100644
--- a/ERROR-ALERTING-PLAN.md
+++ b/ERROR-ALERTING-PLAN.md
@@ -1,7 +1,8 @@
 # FileMill Error Alerting — Implementation Plan (issue #7)
 
-**Status:** Phase 1 (`internal/alert` core and the store `Ledger`) and phase 2
-(job taxonomy split) implemented 2026-09-11; phases 3–5 not started. First drafted 2026-07-19; **revised
+**Status:** Phases 1–3 implemented 2026-09-11: the `internal/alert` core and
+store `Ledger`, the job taxonomy split, and the Mailgun alert sites with their
+wiring. Alerting is off until `alert_recipient` is set. Phases 4–5 not started. First drafted 2026-07-19; **revised
 2026-09-11** against the current code. Since the first draft, FileMill gained the
 supervisor loop (#4), boot start, the two retention sweeps, sheets-link delivery,
 and non-blocking delivery. Each adds alert sites, and the supervisor changes how a
@@ -171,8 +172,8 @@ could spend the whole Free-plan budget.
   verbatim. Then add `SendAlert(ctx, to, subject, text)` as a thin call to `send`
   with no attachments and no threading headers.
 - Alerts come from `REPLY_FROM`. Subject: `[FileMill] <summary>`.
-- `Service` gets a `reporter alert.Reporter` field that defaults to `alert.Nop{}`,
-  plus `SetReporter`.
+- `Service` gets a `reporter alert.Reporter` field, where nil means disabled (most
+  tests build a `Service` literal), plus `SetReporter`.
 - `fileConfig` gets `alert_recipient` (empty means disabled),
   `alert_cooldown_minutes`, `alert_max_per_hour` (default 10) and
   `alert_max_per_day` (default 20).
@@ -231,7 +232,8 @@ which is left to heartbeat #5.
 
 ```
 app.Open → mailgun.Load → if alert_recipient set:
-    emailer := alert.NewEmailer(mail, store, …); go emailer.Run(ctx)
+    emailer := alert.NewEmailer(mail, application, …); go emailer.Run(ctx)
+    (the App is the Ledger: it hands the ledger calls to its store, like its other store methods)
     application.SetReporter(emailer); mail.SetReporter(emailer)
     report restart / interrupted jobs (§3.6)
 ```

From 39cf7e24e5a3b9a6d6ad3192a32f2f9195332fed Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 13:04:37 -0600
Subject: [PATCH 16/25] Add filemill alert-test to prove the alert channel

Sends one test alert to alert_recipient through the Mailgun service, so the
channel and its spam placement are checked before anything relies on it. It
bypasses the throttle, since the operator asked for exactly this message, but
is recorded in the ledger because it spends one of the day's Mailgun sends,
which keeps the daily cap exact. A ledger failure doesn't stop the send.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 README.md                       | 13 +++++++
 cmd/filemill/main.go            | 26 +++++++++++++-
 config/email.yaml.example       |  1 +
 internal/alert/sendtest.go      | 34 ++++++++++++++++++
 internal/alert/sendtest_test.go | 64 +++++++++++++++++++++++++++++++++
 5 files changed, 137 insertions(+), 1 deletion(-)
 create mode 100644 internal/alert/sendtest.go
 create mode 100644 internal/alert/sendtest_test.go

diff --git a/README.md b/README.md
index 5662992..6856a49 100644
--- a/README.md
+++ b/README.md
@@ -115,6 +115,19 @@ records, and (for a laptop with no public IP) a tunnel — that infrastructure
 setup is documented separately in [EMAIL-PIPELINE.md](EMAIL-PIPELINE.md) and
 diagrammed in [email-pipeline-diagram.html](email-pipeline-diagram.html).
 
+To have FileMill email you when it fails systemically (a crashed transformer,
+replies failing for five minutes, intake errors), set `alert_recipient` in
+`config/email.yaml`. Alerts go out through Mailgun from `REPLY_FROM`, and are
+throttled so they can't use up the day's send budget. Check the channel once
+before relying on it:
+
+```powershell
+.\bin\filemill.exe alert-test
+```
+
+It sends one test alert to `alert_recipient`, which counts toward the daily
+alert cap. Run it from a shell that has the Mailgun variables set.
+
 ### Run continuously, in the background
 
 FileMill should run under your Windows account so it can use locally
diff --git a/cmd/filemill/main.go b/cmd/filemill/main.go
index 9f75408..398b0c5 100644
--- a/cmd/filemill/main.go
+++ b/cmd/filemill/main.go
@@ -67,6 +67,30 @@ func main() {
 			fatal(err)
 		}
 		fmt.Printf("id: %s\noperation: %s\nstatus: %s\nmessage: %s\n", job.ID, job.Operation, job.Status, job.Message)
+	case "alert-test":
+		// Sends one alert to alert_recipient, so the channel is proven (and
+		// its spam placement known) before anything relies on it.
+		if len(os.Args) != 2 {
+			usage()
+			os.Exit(2)
+		}
+		mail, err := mailgun.Load(root, application, log.New(os.Stderr, "mailgun ", log.LstdFlags|log.LUTC))
+		if err != nil {
+			fatal(err)
+		}
+		if mail == nil {
+			fatal(fmt.Errorf("alert-test sends through Mailgun: set MAILGUN_API_KEY, MAILGUN_WEBHOOK_SIGNING_KEY, MAILGUN_DOMAIN, and REPLY_FROM"))
+		}
+		to := mail.AlertRecipient()
+		if to == "" {
+			fatal(fmt.Errorf("no alert_recipient in config/email.yaml"))
+		}
+		ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
+		defer cancel()
+		if err := alert.SendTest(ctx, mail, application, to, time.Now()); err != nil {
+			fatal(err)
+		}
+		fmt.Printf("test alert sent to %s; check that it arrives and isn't marked as spam\n", to)
 	case "run":
 		once := len(os.Args) == 3 && os.Args[2] == "--once"
 		if len(os.Args) > 2 && !once {
@@ -185,7 +209,7 @@ func main() {
 
 func usage() {
 	name := filepath.Base(os.Args[0])
-	fmt.Fprintf(os.Stderr, "Usage:\n  %s run [--once]\n  %s submit <operation> <file>\n  %s jobs get <job-id>\n  %s --version\n", name, name, name, name)
+	fmt.Fprintf(os.Stderr, "Usage:\n  %s run [--once]\n  %s submit <operation> <file>\n  %s jobs get <job-id>\n  %s alert-test\n  %s --version\n", name, name, name, name, name)
 }
 
 func fatal(err error) { fmt.Fprintln(os.Stderr, "filemill:", err); os.Exit(1) }
diff --git a/config/email.yaml.example b/config/email.yaml.example
index 99bec5b..55599df 100644
--- a/config/email.yaml.example
+++ b/config/email.yaml.example
@@ -30,6 +30,7 @@ send_timeout_seconds: 30
 # Operator alerts. Systemic failures (a crashed or timed-out transformer, replies
 # failing for 5 minutes, intake errors, retention sweeps that can't delete) are
 # emailed to this address, from REPLY_FROM. Empty or absent turns them off.
+# Check the channel once with `filemill alert-test`, which sends one test alert.
 #
 # Alerts are throttled: one per category every alert_cooldown_minutes (the next
 # email counts the ones held back), and at most alert_max_per_hour and
diff --git a/internal/alert/sendtest.go b/internal/alert/sendtest.go
new file mode 100644
index 0000000..633fa67
--- /dev/null
+++ b/internal/alert/sendtest.go
@@ -0,0 +1,34 @@
+package alert
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"time"
+)
+
+// testCategory records a test send in the Ledger apart from real alerts, so it
+// never holds a real category in its cooldown.
+const testCategory = "alert-test"
+
+// SendTest sends one test alert to to, straight through the Mailer. The
+// throttle is bypassed: the operator asked for exactly this message and needs
+// to see whether it arrives. The send is still recorded in the Ledger, since
+// it spends one of the day's Mailgun sends like any alert. A Ledger failure
+// doesn't stop the send; every error is returned.
+func SendTest(ctx context.Context, m Mailer, l Ledger, to string, now time.Time) error {
+	recordErr := l.RecordAlertSent(testCategory, now)
+	if recordErr != nil {
+		recordErr = fmt.Errorf("record the test send: %w", recordErr)
+	}
+	text := fmt.Sprintf("This is a test alert from `filemill alert-test`, sent %s.\n\n"+
+		"It reached you, so operator alerts work: FileMill emails its systemic failures to %s, from its reply address. "+
+		"If this landed in spam, mark it as not spam so real alerts don't.\n\n"+
+		"It counts toward the daily alert cap like any alert.\n",
+		now.Format("2006-01-02 15:04 MST"), to)
+	sendErr := m.SendAlert(ctx, to, subjectPrefix+"test alert", text)
+	if sendErr != nil {
+		sendErr = fmt.Errorf("send the test alert: %w", sendErr)
+	}
+	return errors.Join(sendErr, recordErr)
+}
diff --git a/internal/alert/sendtest_test.go b/internal/alert/sendtest_test.go
new file mode 100644
index 0000000..1364892
--- /dev/null
+++ b/internal/alert/sendtest_test.go
@@ -0,0 +1,64 @@
+package alert
+
+import (
+	"context"
+	"errors"
+	"strings"
+	"testing"
+	"time"
+)
+
+var testSendTime = time.Date(2026, 9, 11, 9, 0, 0, 0, time.UTC)
+
+// alert-test proves the channel works before anything relies on it: one
+// message to the configured recipient, through the same Mailer and subject
+// prefix real alerts use.
+func TestSendTestSendsOneMessageAndCountsIt(t *testing.T) {
+	m, l := &fakeMailer{}, newFakeLedger()
+
+	if err := SendTest(context.Background(), m, l, testRecipient, testSendTime); err != nil {
+		t.Fatalf("SendTest: %v", err)
+	}
+
+	sent := m.sends()
+	if len(sent) != 1 {
+		t.Fatalf("sent %d messages, want 1", len(sent))
+	}
+	if sent[0].to != testRecipient || sent[0].subject != "[FileMill] test alert" {
+		t.Errorf("sent to %q with subject %q", sent[0].to, sent[0].subject)
+	}
+	if !strings.Contains(sent[0].text, testRecipient) {
+		t.Errorf("the body should say where alerts go:\n%s", sent[0].text)
+	}
+	// It spends one of the day's Mailgun sends like any alert, so it counts
+	// against the daily cap, which then stays exact.
+	if len(l.sent) != 1 || !l.sent[0].Equal(testSendTime) {
+		t.Errorf("recorded sends = %v, want the test send at %v", l.sent, testSendTime)
+	}
+}
+
+func TestSendTestReturnsAFailedSend(t *testing.T) {
+	m := &fakeMailer{err: errors.New("mailgun send returned 401 Unauthorized: Forbidden")}
+
+	err := SendTest(context.Background(), m, newFakeLedger(), testRecipient, testSendTime)
+
+	if err == nil || !strings.Contains(err.Error(), "401 Unauthorized") {
+		t.Fatalf("SendTest = %v, want the send's error", err)
+	}
+}
+
+// A ledger that can't record the send must not stop the test: the operator
+// asked for it, and the ledger's failure is worth hearing about too.
+func TestSendTestStillSendsWhenTheLedgerFails(t *testing.T) {
+	m, l := &fakeMailer{}, newFakeLedger()
+	l.writeErr = errors.New("database is locked")
+
+	err := SendTest(context.Background(), m, l, testRecipient, testSendTime)
+
+	if n := len(m.sends()); n != 1 {
+		t.Fatalf("sent %d messages, want 1", n)
+	}
+	if err == nil || !strings.Contains(err.Error(), "database is locked") {
+		t.Errorf("SendTest = %v, want the ledger's error", err)
+	}
+}

From ff8cce36242fd31d0047fe004acd0f4493b1af7a Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 13:04:37 -0600
Subject: [PATCH 17/25] Use alert-test for phase 5's test send in the alerting
 plan

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 ERROR-ALERTING-PLAN.md | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/ERROR-ALERTING-PLAN.md b/ERROR-ALERTING-PLAN.md
index 7077ef5..c3685ba 100644
--- a/ERROR-ALERTING-PLAN.md
+++ b/ERROR-ALERTING-PLAN.md
@@ -296,6 +296,8 @@ during shutdown still goes out.
 
 **Phase 5 — live verification.**
 - Set the real `alert_recipient` in the gitignored `config/email.yaml`.
+- Send one test alert with `filemill alert-test` (it goes straight to the Mailer,
+  past the throttle, but counts toward the daily cap), and check its spam placement.
 - Trigger one alert per sink: a transformer that exits 1 with no result, a bad
   Mailgun domain for delivery, and a restart.
 - Confirm the throttle by forcing a repeated failure for 20 minutes and expecting

From 45eb72078189767e3ee159183aa20c29436b285e Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 13:17:09 -0600
Subject: [PATCH 18/25] Recover from panics in the delivery and sweep loops

Phase 4 of the error-alerting plan (#7). Each pass of the delivery loop, the
Drive retention sweep and the job sweep now recovers a panic, logs it and
reports panic with the stack, and the loop carries on. Before, a panic in any
of them ended the process, or with the supervisor restarting it, a crash-loop
if the panic repeats.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 internal/app/retention.go      | 24 ++++++++++++++---
 internal/mailgun/fake_test.go  | 11 +++++++-
 internal/mailgun/loops_test.go | 48 ++++++++++++++++++++++++++++++++++
 internal/mailgun/outbound.go   | 14 +++++++---
 internal/mailgun/retention.go  | 13 ++++++---
 internal/mailgun/service.go    | 17 ++++++++++++
 6 files changed, 117 insertions(+), 10 deletions(-)
 create mode 100644 internal/mailgun/loops_test.go

diff --git a/internal/app/retention.go b/internal/app/retention.go
index e6bfacb..2c92e92 100644
--- a/internal/app/retention.go
+++ b/internal/app/retention.go
@@ -5,6 +5,7 @@ import (
 	"fmt"
 	"os"
 	"path/filepath"
+	"runtime/debug"
 	"strings"
 	"time"
 
@@ -50,13 +51,30 @@ func (a *App) SweepExpiredJobs(ctx context.Context) {
 			return
 		case <-timer.C:
 		}
-		if err := a.sweepExpiredJobs(); err != nil {
-			a.log.Printf("job sweep: %v", err)
-		}
+		a.sweepJobsTick()
 		timer.Reset(jobSweepInterval)
 	}
 }
 
+// sweepJobsTick runs one sweep. A panic is logged and reported instead of
+// ending the loop, which would leave workspaces piling up past retention with
+// nothing to say so.
+func (a *App) sweepJobsTick() {
+	defer func() {
+		if r := recover(); r != nil {
+			a.log.Printf("job sweep: panic: %v", r)
+			a.reporter.Report(alert.Alert{
+				Category: "panic",
+				Summary:  fmt.Sprintf("job sweep panicked: %v", r),
+				Detail:   fmt.Sprintf("The job sweep recovered and runs again at its next interval.\n\nPanic: %v\n\n%s", r, debug.Stack()),
+			})
+		}
+	}()
+	if err := a.sweepExpiredJobs(); err != nil {
+		a.log.Printf("job sweep: %v", err)
+	}
+}
+
 // sweepExpiredJobs deletes the on-disk workspace of every completed job past
 // the retention horizon. Only completed_at is checked (see
 // store.ExpiredJobs), so a job still queued or running is never touched.
diff --git a/internal/mailgun/fake_test.go b/internal/mailgun/fake_test.go
index e8be314..bc9c90d 100644
--- a/internal/mailgun/fake_test.go
+++ b/internal/mailgun/fake_test.go
@@ -53,6 +53,7 @@ type fakeEngine struct {
 	calls []string // ordered log of mutating calls
 
 	pending    []store.EmailSubmission
+	panicWith  any // when set, PendingEmails and ExpiredDeliveries panic with it
 	outputs    map[string][]app.OutputFile
 	outputsErr map[string]error // job id -> error Outputs should return
 
@@ -149,7 +150,12 @@ func (f *fakeEngine) AddEmailJob(id int64, index int, jobID string) error {
 	return nil
 }
 
-func (f *fakeEngine) PendingEmails() ([]store.EmailSubmission, error) { return f.pending, nil }
+func (f *fakeEngine) PendingEmails() ([]store.EmailSubmission, error) {
+	if f.panicWith != nil {
+		panic(f.panicWith)
+	}
+	return f.pending, nil
+}
 
 func (f *fakeEngine) MarkEmailDelivered(id int64) error {
 	f.markCalls++
@@ -195,6 +201,9 @@ func (f *fakeEngine) Delivery(submissionID int64, outputIndex int) (store.Delive
 }
 
 func (f *fakeEngine) ExpiredDeliveries(cutoff time.Time) ([]store.Delivery, error) {
+	if f.panicWith != nil {
+		panic(f.panicWith)
+	}
 	var out []store.Delivery
 	for key, d := range f.deliveries {
 		if !f.sweptDeliveries[key] && d.CreatedAt.Before(cutoff) {
diff --git a/internal/mailgun/loops_test.go b/internal/mailgun/loops_test.go
new file mode 100644
index 0000000..3eea964
--- /dev/null
+++ b/internal/mailgun/loops_test.go
@@ -0,0 +1,48 @@
+package mailgun
+
+import (
+	"context"
+	"strings"
+	"testing"
+)
+
+// A panic in one delivery tick must not end the loop: the worker would keep
+// running with no replies going out. The tick reports it and the next one
+// runs normally.
+func TestDeliveryTickSurvivesPanic(t *testing.T) {
+	f, rep, _ := newAlertingFixture(t)
+	f.addSubmission(t, 1, "excel@mill.test", "schedule.xlsx")
+	f.engine.panicWith = "boom"
+
+	f.service.deliverTick(context.Background())
+
+	got := rep.only("panic")
+	if len(got) != 1 {
+		t.Fatalf("alerts = %+v, want one panic", rep.alerts)
+	}
+	for _, want := range []string{"boom", "goroutine"} {
+		if !strings.Contains(got[0].Detail, want) {
+			t.Errorf("panic detail lacks %q:\n%s", want, got[0].Detail)
+		}
+	}
+	if !strings.Contains(f.logs.String(), "boom") {
+		t.Errorf("the panic must be logged; got %q", f.logs.String())
+	}
+
+	f.engine.panicWith = nil
+	f.service.deliverTick(context.Background())
+	if !f.engine.delivered[1] {
+		t.Error("the tick after a panic must deliver normally")
+	}
+}
+
+func TestSweepTickSurvivesPanic(t *testing.T) {
+	f, rep, _ := newAlertingFixture(t)
+	f.engine.panicWith = "boom"
+
+	f.service.sweepTick(context.Background())
+
+	if got := rep.only("panic"); len(got) != 1 || !strings.Contains(got[0].Detail, "boom") {
+		t.Fatalf("alerts = %+v, want one panic naming it", rep.alerts)
+	}
+}
diff --git a/internal/mailgun/outbound.go b/internal/mailgun/outbound.go
index ae82485..f0322db 100644
--- a/internal/mailgun/outbound.go
+++ b/internal/mailgun/outbound.go
@@ -35,9 +35,7 @@ func (s *Service) Deliver(ctx context.Context) {
 	ticker := time.NewTicker(deliveryPollInterval)
 	defer ticker.Stop()
 	for {
-		if err := s.deliverPending(ctx); err != nil {
-			s.log.Printf("delivery: %v", err)
-		}
+		s.deliverTick(ctx)
 		select {
 		case <-ctx.Done():
 			return
@@ -46,6 +44,16 @@ func (s *Service) Deliver(ctx context.Context) {
 	}
 }
 
+// deliverTick runs one pass of the delivery loop. A panic is logged and
+// reported rather than ending the loop, which would leave the worker running
+// with no replies going out.
+func (s *Service) deliverTick(ctx context.Context) {
+	defer s.recoverLoop("delivery loop")
+	if err := s.deliverPending(ctx); err != nil {
+		s.log.Printf("delivery: %v", err)
+	}
+}
+
 // deliverPending sends a reply for every pending submission whose jobs have all
 // finished.
 //
diff --git a/internal/mailgun/retention.go b/internal/mailgun/retention.go
index 1733937..eb2626b 100644
--- a/internal/mailgun/retention.go
+++ b/internal/mailgun/retention.go
@@ -36,13 +36,20 @@ func (s *Service) SweepExpired(ctx context.Context) {
 			return
 		case <-timer.C:
 		}
-		if err := s.sweepExpired(ctx); err != nil {
-			s.log.Printf("retention sweep: %v", err)
-		}
+		s.sweepTick(ctx)
 		timer.Reset(sweepInterval)
 	}
 }
 
+// sweepTick runs one sweep. A panic is logged and reported rather than ending
+// the loop, which would leave expired files shared in Drive indefinitely.
+func (s *Service) sweepTick(ctx context.Context) {
+	defer s.recoverLoop("retention sweep")
+	if err := s.sweepExpired(ctx); err != nil {
+		s.log.Printf("retention sweep: %v", err)
+	}
+}
+
 // sweepExpired deletes every published file past the retention horizon.
 //
 // A file that fails to delete is logged and left unmarked, so the next sweep
diff --git a/internal/mailgun/service.go b/internal/mailgun/service.go
index 8e26a84..48bbae5 100644
--- a/internal/mailgun/service.go
+++ b/internal/mailgun/service.go
@@ -16,8 +16,10 @@ package mailgun
 
 import (
 	"context"
+	"fmt"
 	"log"
 	"net/http"
+	"runtime/debug"
 	"time"
 
 	"filemill/internal/alert"
@@ -137,6 +139,21 @@ func (s *Service) report(a alert.Alert) {
 	}
 }
 
+// recoverLoop, deferred by one pass of a background loop, turns a panic into a
+// log line and a panic alert, so the loop carries on with its next pass.
+func (s *Service) recoverLoop(loop string) {
+	r := recover()
+	if r == nil {
+		return
+	}
+	s.log.Printf("%s: panic: %v", loop, r)
+	s.report(alert.Alert{
+		Category: "panic",
+		Summary:  fmt.Sprintf("%s panicked: %v", loop, r),
+		Detail:   fmt.Sprintf("The %s recovered and carries on with its next pass.\n\nPanic: %v\n\n%s", loop, r, debug.Stack()),
+	})
+}
+
 func (s *Service) clock() time.Time {
 	if s.now != nil {
 		return s.now()

From 526f82e4a61b48d151260bbccb6edaf60cec7b6a Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 13:17:09 -0600
Subject: [PATCH 19/25] Report a worker restart and stop CLI commands
 interrupting live jobs

Phase 4 of the error-alerting plan (#7). A crashed process can't report its
own death, so the supervisor now passes FILEMILL_PREVIOUS_EXIT and
FILEMILL_RAPID_RESTARTS to each relaunch, and run reports one restart alert
from them, marked as a crash-loop at 4 rapid restarts. It folds in the jobs
the last worker left running, which also reveal an unclean stop with no
supervisor at all.

store.Open no longer marks running jobs interrupted. Every CLI command opens
the database, and submit, jobs get or alert-test run while the worker was
mid-job marked its live job interrupted, which the delivery loop treats as
finished. The starting worker does it instead, once it holds the webhook
port and just before the job loop. An interrupted job's message now asks the
sender to send the file again; its reply line used to be empty.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 README.md                        |  6 +-
 cmd/filemill/main.go             | 15 +++++
 cmd/filemill/restart.go          | 49 ++++++++++++++++
 cmd/filemill/restart_test.go     | 55 ++++++++++++++++++
 internal/app/app.go              | 12 ++++
 internal/app/loops_test.go       | 41 ++++++++++++++
 internal/store/interrupt_test.go | 95 ++++++++++++++++++++++++++++++++
 internal/store/store.go          | 25 +++++++--
 scripts/Supervise-FileMill.ps1   | 17 +++++-
 9 files changed, 306 insertions(+), 9 deletions(-)
 create mode 100644 cmd/filemill/restart.go
 create mode 100644 cmd/filemill/restart_test.go
 create mode 100644 internal/app/loops_test.go
 create mode 100644 internal/store/interrupt_test.go

diff --git a/README.md b/README.md
index 6856a49..08664ca 100644
--- a/README.md
+++ b/README.md
@@ -153,8 +153,10 @@ in**, and again at logon as a backstop. It launches a **supervisor**
 (`Supervise-FileMill.ps1`) that runs `filemill run` and **restarts it
 automatically if it crashes** — an immediate first retry, then escalating
 backoff (5s, 15s, 30s, 60s, 120s) for repeated rapid failures; a persistent
-crash-loop is logged (alerting is tracked in issue #7). A clean exit (Ctrl+C
-/ shutdown) stops the supervisor.
+crash-loop is logged. When operator alerts are on (`alert_recipient`), the
+restarted worker emails one `restart` alert saying how its predecessor ended,
+throttled so a crash-loop sends one per 15 minutes. A clean exit (Ctrl+C /
+shutdown) stops the supervisor.
 
 Two consequences of running before logon are worth knowing:
 
diff --git a/cmd/filemill/main.go b/cmd/filemill/main.go
index 398b0c5..500ffc6 100644
--- a/cmd/filemill/main.go
+++ b/cmd/filemill/main.go
@@ -112,6 +112,9 @@ func main() {
 		alertCtx, stopAlerts := context.WithCancel(context.Background())
 		defer stopAlerts()
 		var alertsDone chan struct{}
+		// reporter takes run's own restart alert: the Emailer once it is wired,
+		// otherwise nothing.
+		var reporter alert.Reporter = alert.Nop{}
 		if !once {
 			mailLog := log.New(io.MultiWriter(os.Stderr, application.LogWriter()), "mailgun ", log.LstdFlags|log.LUTC)
 			mail, err := mailgun.Load(root, application, mailLog)
@@ -129,6 +132,7 @@ func main() {
 					emailer = alert.NewEmailer(mail, application, to, mail.AlertConfig(), time.Now, alertLog)
 					application.SetReporter(emailer)
 					mail.SetReporter(emailer)
+					reporter = emailer
 				}
 				server = &http.Server{Addr: os.Getenv("LISTEN_ADDR"), Handler: mail.Handler()}
 				if server.Addr == "" {
@@ -176,6 +180,17 @@ func main() {
 			// nested under the mailgun branch above.
 			go application.SweepExpiredJobs(ctx)
 		}
+		// Only a starting worker may conclude that a job left running is
+		// orphaned, and only here: past the bind, so a second worker that lost
+		// the port has already exited without touching the first one's jobs.
+		// Other commands open the same database and never do this.
+		interrupted, err := application.InterruptLeftoverJobs()
+		if err != nil {
+			fatal(err)
+		}
+		if restart, ok := restartAlert(os.Getenv(previousExitEnv), os.Getenv(rapidRestartsEnv), interrupted); ok {
+			reporter.Report(restart)
+		}
 		runErr := application.Run(ctx, once)
 		if server != nil {
 			// Drain in-flight webhook requests on shutdown (Ctrl+C). Intake is
diff --git a/cmd/filemill/restart.go b/cmd/filemill/restart.go
new file mode 100644
index 0000000..039a4d1
--- /dev/null
+++ b/cmd/filemill/restart.go
@@ -0,0 +1,49 @@
+package main
+
+import (
+	"fmt"
+	"strconv"
+	"strings"
+
+	"filemill/internal/alert"
+)
+
+// The supervisor (scripts/Supervise-FileMill.ps1) sets these for each relaunch,
+// so the new worker can report how the last one ended: a process can't report
+// its own death.
+const (
+	previousExitEnv  = "FILEMILL_PREVIOUS_EXIT"  // the last exit code; unset on a first launch
+	rapidRestartsEnv = "FILEMILL_RAPID_RESTARTS" // restarts in quick succession so far
+)
+
+// crashLoopAt matches the supervisor's $crashLoopAt: this many rapid restarts
+// is a crash-loop.
+const crashLoopAt = 4
+
+// restartAlert builds the restart alert, if there is anything to report: a
+// supervisor relaunch, or jobs the last worker left running, which means it
+// didn't stop cleanly even when no supervisor was there to say so.
+func restartAlert(previousExit, rapidRestarts string, interrupted int) (alert.Alert, bool) {
+	if previousExit == "" && interrupted == 0 {
+		return alert.Alert{}, false
+	}
+	var summary, detail strings.Builder
+	if previousExit != "" {
+		fmt.Fprintf(&summary, "worker restarted after exit code %s", previousExit)
+		fmt.Fprintf(&detail, "The supervisor restarted the FileMill worker after exit code %s.\n", previousExit)
+		if rapid, err := strconv.Atoi(rapidRestarts); err == nil && rapid >= crashLoopAt {
+			fmt.Fprintf(&summary, " (crash-loop: %d rapid restarts)", rapid)
+			fmt.Fprintf(&detail, "It has restarted %d times in quick succession, a crash-loop. The supervisor backs off up to 2 minutes between attempts; data\\logs\\supervisor.log and filemill.log have the details.\n", rapid)
+		}
+		if interrupted > 0 {
+			fmt.Fprintf(&summary, ", %d job(s) interrupted", interrupted)
+		}
+	} else {
+		fmt.Fprintf(&summary, "worker started after an unclean stop: %d job(s) interrupted", interrupted)
+		detail.WriteString("The FileMill worker found jobs its predecessor left running, so the last worker did not stop cleanly: a crash, a kill, or a power loss.\n")
+	}
+	if interrupted > 0 {
+		fmt.Fprintf(&detail, "\n%d job(s) were running when the previous worker stopped. They are marked interrupted and not retried; each sender's reply asks them to send the file again.\n", interrupted)
+	}
+	return alert.Alert{Category: "restart", Summary: summary.String(), Detail: detail.String()}, true
+}
diff --git a/cmd/filemill/restart_test.go b/cmd/filemill/restart_test.go
new file mode 100644
index 0000000..93065d3
--- /dev/null
+++ b/cmd/filemill/restart_test.go
@@ -0,0 +1,55 @@
+package main
+
+import (
+	"strings"
+	"testing"
+)
+
+// A process can't report its own death, so the restarted worker reports it,
+// from what the supervisor passes it and the jobs its predecessor left
+// running.
+func TestRestartAlert(t *testing.T) {
+	for _, tc := range []struct {
+		name         string
+		previousExit string // FILEMILL_PREVIOUS_EXIT; empty on a first launch
+		rapid        string // FILEMILL_RAPID_RESTARTS
+		interrupted  int
+		wantReport   bool
+		wantSummary  string
+		wantDetail   []string
+	}{
+		{name: "first launch after a clean stop"},
+		{name: "restart after a crash", previousExit: "2", rapid: "1", wantReport: true,
+			wantSummary: "worker restarted after exit code 2", wantDetail: []string{"exit code 2"}},
+		{name: "below the crash-loop threshold", previousExit: "2", rapid: "3", wantReport: true,
+			wantSummary: "worker restarted after exit code 2"},
+		{name: "crash-loop", previousExit: "2", rapid: "4", wantReport: true,
+			wantSummary: "worker restarted after exit code 2 (crash-loop: 4 rapid restarts)", wantDetail: []string{"crash-loop"}},
+		{name: "unreadable restart count", previousExit: "1", rapid: "lots", wantReport: true,
+			wantSummary: "worker restarted after exit code 1"},
+		// Jobs left running mean the last worker didn't stop cleanly, even
+		// with no supervisor to say so.
+		{name: "interrupted jobs without a supervisor", interrupted: 3, wantReport: true,
+			wantSummary: "worker started after an unclean stop: 3 job(s) interrupted", wantDetail: []string{"3 job(s)"}},
+		{name: "crash with interrupted jobs", previousExit: "-1073741819", rapid: "0", interrupted: 1, wantReport: true,
+			wantSummary: "worker restarted after exit code -1073741819, 1 job(s) interrupted", wantDetail: []string{"1 job(s)"}},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			a, ok := restartAlert(tc.previousExit, tc.rapid, tc.interrupted)
+			if ok != tc.wantReport {
+				t.Fatalf("report = %t, want %t (alert %+v)", ok, tc.wantReport, a)
+			}
+			if !ok {
+				return
+			}
+			if a.Category != "restart" || a.Summary != tc.wantSummary {
+				t.Errorf("alert = %s %q, want restart %q", a.Category, a.Summary, tc.wantSummary)
+			}
+			for _, want := range tc.wantDetail {
+				if !strings.Contains(a.Detail, want) {
+					t.Errorf("detail lacks %q:\n%s", want, a.Detail)
+				}
+			}
+		})
+	}
+}
diff --git a/internal/app/app.go b/internal/app/app.go
index ea00eec..496b4d9 100644
--- a/internal/app/app.go
+++ b/internal/app/app.go
@@ -217,6 +217,18 @@ func (a *App) RecordAlertSuppressed(category string) error {
 	return a.store.RecordAlertSuppressed(category)
 }
 func (a *App) AlertSendsSince(t time.Time) ([]time.Time, error) { return a.store.AlertSendsSince(t) }
+
+// InterruptLeftoverJobs marks the jobs a dead predecessor left running as
+// interrupted and returns how many, for the restart alert. run calls it once,
+// just before the job loop starts; nothing else may (see
+// store.InterruptRunning).
+func (a *App) InterruptLeftoverJobs() (int, error) {
+	n, err := a.store.InterruptRunning()
+	if n > 0 {
+		a.log.Printf("worker startup: marked %d job(s) left running by the previous worker as interrupted", n)
+	}
+	return n, err
+}
 func (a *App) Run(ctx context.Context, once bool) error {
 	a.log.Printf("worker started once=%t", once)
 	var claims claimFailures
diff --git a/internal/app/loops_test.go b/internal/app/loops_test.go
new file mode 100644
index 0000000..520cd8e
--- /dev/null
+++ b/internal/app/loops_test.go
@@ -0,0 +1,41 @@
+package app
+
+import (
+	"strings"
+	"testing"
+)
+
+// A panic in one job sweep must not end the sweep loop, which would leave
+// workspaces piling up past retention with nothing to say so.
+func TestJobSweepTickSurvivesPanic(t *testing.T) {
+	a, rep := newTestApp(t, fakeTransformer(t, "succeed")...)
+	completedJob(t, a)
+	expireEverything(t)
+	saved := removeWorkspace
+	removeWorkspace = func(string) error { panic("boom") }
+	t.Cleanup(func() { removeWorkspace = saved })
+
+	a.sweepJobsTick()
+
+	alerts := rep.reports()
+	if len(alerts) != 1 || alerts[0].Category != "panic" {
+		t.Fatalf("alerts = %+v, want one panic", alerts)
+	}
+	for _, want := range []string{"boom", "goroutine"} {
+		if !strings.Contains(alerts[0].Detail, want) {
+			t.Errorf("panic detail lacks %q:\n%s", want, alerts[0].Detail)
+		}
+	}
+}
+
+// InterruptLeftoverJobs is what run calls at startup; it hands the store's
+// count back for the restart alert.
+func TestInterruptLeftoverJobsCountsRunningJobs(t *testing.T) {
+	a, _ := newTestApp(t, fakeTransformer(t, "succeed")...)
+	claimJob(t, a) // left running, as a crashed worker would leave it
+
+	n, err := a.InterruptLeftoverJobs()
+	if err != nil || n != 1 {
+		t.Fatalf("InterruptLeftoverJobs = %d, %v; want 1", n, err)
+	}
+}
diff --git a/internal/store/interrupt_test.go b/internal/store/interrupt_test.go
new file mode 100644
index 0000000..ac5c2cd
--- /dev/null
+++ b/internal/store/interrupt_test.go
@@ -0,0 +1,95 @@
+package store
+
+import (
+	"strings"
+	"testing"
+	"time"
+)
+
+// Only the worker marks jobs a dead predecessor left running. Open must not:
+// every CLI command opens the same database, and one run while the worker is
+// mid-job would mark that live job interrupted, which the delivery loop takes
+// as finished.
+func TestOpenLeavesRunningJobsAlone(t *testing.T) {
+	s, path := openTestStore(t)
+	if err := s.Create(Job{ID: "live", Operation: "op", InputName: "a.pdf", CreatedAt: time.Now()}); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := s.Next(); err != nil {
+		t.Fatal(err)
+	}
+
+	other, err := Open(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer other.Close()
+
+	got, err := other.Get("live")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got.Status != StatusRunning {
+		t.Errorf("a second Open changed a running job to %q", got.Status)
+	}
+}
+
+func TestInterruptRunningMarksAndCountsLeftovers(t *testing.T) {
+	s, _ := openTestStore(t)
+	t0 := time.Now().UTC()
+	for i, id := range []string{"left", "done", "queued"} {
+		if err := s.Create(Job{ID: id, Operation: "op", InputName: id + ".pdf", CreatedAt: t0.Add(time.Duration(i) * time.Second)}); err != nil {
+			t.Fatal(err)
+		}
+	}
+	// Claims go oldest first: "left" stays running, "done" finishes, "queued"
+	// is never claimed.
+	for range 2 {
+		if _, err := s.Next(); err != nil {
+			t.Fatal(err)
+		}
+	}
+	if err := s.Complete("done", StatusSucceeded, "ok"); err != nil {
+		t.Fatal(err)
+	}
+
+	n, err := s.InterruptRunning()
+	if err != nil || n != 1 {
+		t.Fatalf("InterruptRunning = %d, %v; want 1", n, err)
+	}
+	for id, want := range map[string]string{"left": StatusInterrupted, "done": StatusSucceeded, "queued": StatusQueued} {
+		got, err := s.Get(id)
+		if err != nil {
+			t.Fatal(err)
+		}
+		if got.Status != want {
+			t.Errorf("%s: status %q, want %q", id, got.Status, want)
+		}
+	}
+	// The reply is built from the message, so it has to tell the sender what
+	// to do: the job is not retried.
+	if got, _ := s.Get("left"); !strings.Contains(got.Message, "send the file again") {
+		t.Errorf("interrupted job message = %q, want it to ask for the file again", got.Message)
+	}
+	// It is completed, so the job sweep will eventually clear its workspace.
+	expired, err := s.ExpiredJobs(time.Now().UTC().Add(time.Hour))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !contains(expired, "left") {
+		t.Errorf("an interrupted job must be sweepable; expired = %v", expired)
+	}
+
+	if n, err := s.InterruptRunning(); err != nil || n != 0 {
+		t.Errorf("a second InterruptRunning = %d, %v; want 0", n, err)
+	}
+}
+
+func contains(ids []string, id string) bool {
+	for _, got := range ids {
+		if got == id {
+			return true
+		}
+	}
+	return false
+}
diff --git a/internal/store/store.go b/internal/store/store.go
index 94f9640..a85702c 100644
--- a/internal/store/store.go
+++ b/internal/store/store.go
@@ -130,10 +130,6 @@ func Open(path string) (*Store, error) {
 		db.Close()
 		return nil, fmt.Errorf("migrate email_submissions: %w", err)
 	}
-	if _, err := db.Exec("UPDATE jobs SET status=?, completed_at=? WHERE status=?", StatusInterrupted, time.Now().UTC().Format(time.RFC3339Nano), StatusRunning); err != nil {
-		db.Close()
-		return nil, err
-	}
 	return s, nil
 }
 
@@ -451,6 +447,27 @@ func (s *Store) Complete(id, status, message string) error {
 	_, err := s.db.Exec("UPDATE jobs SET status=?, message=?, completed_at=? WHERE id=?", status, message, time.Now().UTC().Format(time.RFC3339Nano), id)
 	return err
 }
+
+// interruptedMessage is an interrupted job's message. The sender's reply is
+// built from it and the job is not retried, so it says what to do.
+const interruptedMessage = "interrupted: FileMill stopped while this job was running; please send the file again"
+
+// InterruptRunning marks every running job interrupted and returns how many.
+// Only a worker starting up may call it: none of its own jobs is running yet,
+// so any job still running was orphaned by a predecessor that died. Open
+// doesn't do this, because every CLI command opens the same database, and one
+// run while the worker is mid-job would mark that live job interrupted, which
+// the delivery loop takes as finished.
+func (s *Store) InterruptRunning() (int, error) {
+	res, err := s.db.Exec("UPDATE jobs SET status=?, message=?, completed_at=? WHERE status=?",
+		StatusInterrupted, interruptedMessage, time.Now().UTC().Format(time.RFC3339Nano), StatusRunning)
+	if err != nil {
+		return 0, err
+	}
+	n, err := res.RowsAffected()
+	return int(n), err
+}
+
 // ExpiredJobs returns ids of jobs that finished before cutoff and whose
 // workspace the job sweep has not yet deleted from data/jobs. Only
 // completed_at is checked, not created_at, so a job still queued or running
diff --git a/scripts/Supervise-FileMill.ps1 b/scripts/Supervise-FileMill.ps1
index ed1d2fc..6d5b494 100644
--- a/scripts/Supervise-FileMill.ps1
+++ b/scripts/Supervise-FileMill.ps1
@@ -10,7 +10,10 @@
     lasts long enough to be "healthy" resets the backoff. A clean exit (code 0,
     i.e. an intentional Ctrl+C / OS shutdown signal) stops the supervisor.
 
-    A crash-loop is logged only, for now; alerting is tracked in issue #7.
+    A crashed worker can't report its own death, so each relaunch is told how
+    the last one ended: FILEMILL_PREVIOUS_EXIT (unset on the first launch) and
+    FILEMILL_RAPID_RESTARTS. The restarted worker emails one `restart` alert
+    from them when operator alerts are on (alert_recipient in email.yaml).
     Supervisor events are written to data\logs\supervisor.log; the worker's own
     output continues to go to data\logs\filemill.log.
 #>
@@ -46,12 +49,19 @@ function Write-SupervisorLog {
 # Restart policy.
 $backoffSeconds    = @(0, 5, 15, 30, 60, 120) # immediate first retry, then escalate (capped)
 $healthyRunSeconds = 60                        # a run at least this long resets the backoff
-$crashLoopAt       = 4                         # rapid restarts before logging a crash-loop warning
+$crashLoopAt       = 4                         # rapid restarts before logging a crash-loop warning (cmd/filemill/restart.go matches it)
 
 $rapidFailures = 0
+$previousExit = $null
+# The first launch is not a restart, whatever the environment says.
+Remove-Item Env:FILEMILL_PREVIOUS_EXIT, Env:FILEMILL_RAPID_RESTARTS -ErrorAction SilentlyContinue
 Write-SupervisorLog "starting; watching $Executable"
 
 while ($true) {
+    if ($null -ne $previousExit) {
+        $env:FILEMILL_PREVIOUS_EXIT = "$previousExit"
+        $env:FILEMILL_RAPID_RESTARTS = "$rapidFailures"
+    }
     $startedAt = Get-Date
     Write-SupervisorLog 'launching: filemill run'
     & $Executable run
@@ -65,6 +75,7 @@ while ($true) {
         Write-SupervisorLog 'clean exit (intentional stop); supervisor stopping'
         break
     }
+    $previousExit = $exitCode
 
     if ($ranSeconds -ge $healthyRunSeconds) {
         $rapidFailures = 0
@@ -73,7 +84,7 @@ while ($true) {
         $delay = $backoffSeconds[[Math]::Min($rapidFailures, $backoffSeconds.Count - 1)]
         $rapidFailures++
         if ($rapidFailures -ge $crashLoopAt) {
-            Write-SupervisorLog "WARNING crash-loop: $rapidFailures rapid restarts (log-only; alerting tracked in issue #7)"
+            Write-SupervisorLog "WARNING crash-loop: $rapidFailures rapid restarts"
         }
     }
 

From 0aa8f962749314949b93416e66bcb4dd27b08fbc Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 13:17:09 -0600
Subject: [PATCH 20/25] Record phase 4 in the alerting plan

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 ERROR-ALERTING-PLAN.md | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/ERROR-ALERTING-PLAN.md b/ERROR-ALERTING-PLAN.md
index c3685ba..5b86e76 100644
--- a/ERROR-ALERTING-PLAN.md
+++ b/ERROR-ALERTING-PLAN.md
@@ -1,8 +1,9 @@
 # FileMill Error Alerting — Implementation Plan (issue #7)
 
-**Status:** Phases 1–3 implemented 2026-09-11: the `internal/alert` core and
-store `Ledger`, the job taxonomy split, and the Mailgun alert sites with their
-wiring. Alerting is off until `alert_recipient` is set. Phases 4–5 not started. First drafted 2026-07-19; **revised
+**Status:** Phases 1–4 implemented 2026-09-11: the `internal/alert` core and
+store `Ledger`, the job taxonomy split, the Mailgun alert sites with their
+wiring, and crash reporting across restarts. Phase 4's manual kill test and
+phase 5's live verification are still to do. First drafted 2026-07-19; **revised
 2026-09-11** against the current code. Since the first draft, FileMill gained the
 supervisor loop (#4), boot start, the two retention sweeps, sheets-link delivery,
 and non-blocking delivery. Each adds alert sites, and the supervisor changes how a
@@ -220,8 +221,12 @@ The supervisor can't send email, and a crashed process can't report itself. So:
    `FILEMILL_PREVIOUS_EXIT` is set. The summary is "restarted after exit code N",
    plus "(crash-loop: K rapid restarts)" when K ≥ 4, the supervisor's existing
    threshold.
-3. `store.Open` already marks any `running` job `interrupted`. Return the count
-   from `Open` (or expose it) and fold it into the same `restart` alert.
+3. Fold the count of jobs left `running` into the same `restart` alert. As built,
+   `store.Open` no longer marks them `interrupted`: every CLI command opens the
+   database, and one run mid-job marked the worker's live job `interrupted`,
+   which delivery treats as finished. The worker does it instead, through
+   `Store.InterruptRunning`, once it holds the webhook port and just before the
+   job loop. The job's message asks the sender to send the file again.
 
 Because the throttle is persisted (§3.2), a crash-loop sends **one** `restart`
 alert per cooldown with a suppressed count, not one per restart. The case this

From f7d2118135aa16fd287bce600c2ff394d83c59a1 Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 15:39:09 -0600
Subject: [PATCH 21/25] Record the production verification of phases 4 and 5 in
 the alerting plan

Verified against a branch build (v0.2.1-20-g0aa8f96) running in production:
one restart email from a killed worker with four further restarts suppressed,
and 21 alert_probe jobs producing 2 emails and 19 suppressions, the second
email carrying the count held back. The bad-Mailgun-domain check is skipped
deliberately: it delays real replies and the send path is covered elsewhere.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 ERROR-ALERTING-PLAN.md | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/ERROR-ALERTING-PLAN.md b/ERROR-ALERTING-PLAN.md
index 5b86e76..c9c5b31 100644
--- a/ERROR-ALERTING-PLAN.md
+++ b/ERROR-ALERTING-PLAN.md
@@ -1,9 +1,14 @@
 # FileMill Error Alerting — Implementation Plan (issue #7)
 
-**Status:** Phases 1–4 implemented 2026-09-11: the `internal/alert` core and
-store `Ledger`, the job taxonomy split, the Mailgun alert sites with their
-wiring, and crash reporting across restarts. Phase 4's manual kill test and
-phase 5's live verification are still to do. First drafted 2026-07-19; **revised
+**Status:** Implemented and verified in production 2026-09-11. Phases 1–4 are
+built (the `internal/alert` core and store `Ledger`, the job taxonomy split, the
+Mailgun alert sites and wiring, crash reporting across restarts), and phase 5's
+live checks passed against a branch build (`v0.2.1-20-g0aa8f96`): the test send,
+one `restart` email from a killed worker with four further restarts suppressed,
+one `job-systemic` email from the `alert_probe` transformer, and a second one
+15 minutes later carrying "14 more". Only the bad-Mailgun-domain check is
+outstanding, and it is deliberately skipped: it delays real senders' replies,
+and the send path is covered by tests and by the four alerts above. First drafted 2026-07-19; **revised
 2026-09-11** against the current code. Since the first draft, FileMill gained the
 supervisor loop (#4), boot start, the two retention sweeps, sheets-link delivery,
 and non-blocking delivery. Each adds alert sites, and the supervisor changes how a
@@ -308,6 +313,13 @@ during shutdown still goes out.
 - Confirm the throttle by forcing a repeated failure for 20 minutes and expecting
   two emails, the second carrying a suppressed count.
 
+**Verified 2026-09-11** with an `alert_probe` transformer (`cmd /c exit 1`, no
+`result.json`, in the gitignored `transformers.yaml`, reachable by no route):
+21 submissions a minute apart produced 21 failed jobs, 2 emails and 19
+suppressions, the second email carrying "14 more since the last alert in this
+category". Killing the worker produced exactly one `restart` email; four more
+kills inside the cooldown were logged and suppressed.
+
 **Estimate:** about 3 days. Phase 1 (the throttle and persistence) is the bulk.
 Phases 2 and 3 are about 1 day together. Phase 4 is half a day.
 

From ba223676e1aac5741200667f34cac71067a1da64 Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 15:53:38 -0600
Subject: [PATCH 22/25] Interrupt leftover jobs only in continuous mode
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The call sat outside the `if !once` block, so `filemill run --once` — the
README quickstart command — marked a live worker`s running job interrupted.
The delivery loop treats interrupted as terminal, so the sender would get a
premature reply with outputs missing. That is the hazard this branch moved
out of store.Open, one layer up. Only the continuous worker does it, which by
then holds the webhook port; --once binds nothing and cannot tell whether the
real worker is mid-job.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 cmd/filemill/main.go | 26 +++++++++++++++-----------
 1 file changed, 15 insertions(+), 11 deletions(-)

diff --git a/cmd/filemill/main.go b/cmd/filemill/main.go
index 500ffc6..3bc2ad8 100644
--- a/cmd/filemill/main.go
+++ b/cmd/filemill/main.go
@@ -179,17 +179,21 @@ func main() {
 			// this sweep runs unconditionally in continuous mode rather than
 			// nested under the mailgun branch above.
 			go application.SweepExpiredJobs(ctx)
-		}
-		// Only a starting worker may conclude that a job left running is
-		// orphaned, and only here: past the bind, so a second worker that lost
-		// the port has already exited without touching the first one's jobs.
-		// Other commands open the same database and never do this.
-		interrupted, err := application.InterruptLeftoverJobs()
-		if err != nil {
-			fatal(err)
-		}
-		if restart, ok := restartAlert(os.Getenv(previousExitEnv), os.Getenv(rapidRestartsEnv), interrupted); ok {
-			reporter.Report(restart)
+
+			// Only a starting worker may conclude that a job left running is
+			// orphaned, and only the continuous one: it holds the webhook port
+			// by now, so a second worker that lost the bind has already exited
+			// without touching the first one's jobs. --once must never do this
+			// — it binds nothing, and `run --once` alongside the real worker
+			// would mark that worker's live job interrupted, which the delivery
+			// loop takes as finished. Other commands never do it either.
+			interrupted, err := application.InterruptLeftoverJobs()
+			if err != nil {
+				fatal(err)
+			}
+			if restart, ok := restartAlert(os.Getenv(previousExitEnv), os.Getenv(rapidRestartsEnv), interrupted); ok {
+				reporter.Report(restart)
+			}
 		}
 		runErr := application.Run(ctx, once)
 		if server != nil {

From 163a6bfee671a35a231699cbfffda7ed15dd868b Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 15:53:38 -0600
Subject: [PATCH 23/25] Keep the suppressed alert count when a send fails

RecordAlertSent zeroed the count before the send was attempted, so a failed
send threw away the backlog it was meant to report and no later email ever
mentioned it. Recording the send still happens first, to keep the daily cap
exact, but clearing the count is now a separate Ledger call made only after
an email actually goes out. A failed send instead adds itself to the backlog,
since nobody saw it either.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 internal/alert/alert.go        |  8 ++++++--
 internal/alert/emailer.go      | 18 ++++++++++++++++--
 internal/alert/emailer_test.go | 29 +++++++++++++++++++++++++++++
 internal/alert/fake_test.go    | 11 ++++++++++-
 internal/app/app.go            |  1 +
 internal/store/alert.go        | 14 +++++++++++---
 internal/store/alert_test.go   | 16 +++++++++++++---
 7 files changed, 86 insertions(+), 11 deletions(-)

diff --git a/internal/alert/alert.go b/internal/alert/alert.go
index 5c854a1..20993b9 100644
--- a/internal/alert/alert.go
+++ b/internal/alert/alert.go
@@ -39,11 +39,15 @@ type Ledger interface {
 	// have been suppressed since. A zero sentAt means it has never sent; it
 	// can still have a suppressed count, held back by a global cap.
 	LastAlert(category string) (sentAt time.Time, suppressed int, err error)
-	// RecordAlertSent records a send at at and resets category's suppressed
-	// count.
+	// RecordAlertSent records a send at at. It leaves the suppressed count
+	// alone: a send that then fails reported nothing, so the count must
+	// survive for the next email that does go out.
 	RecordAlertSent(category string, at time.Time) error
 	// RecordAlertSuppressed increments category's suppressed count.
 	RecordAlertSuppressed(category string) error
+	// ClearSuppressed zeroes category's suppressed count, once an email
+	// carrying it has been sent.
+	ClearSuppressed(category string) error
 	// AlertSendsSince returns the time of every send after t, in any
 	// category, oldest first. It must cover at least the last 24h.
 	AlertSendsSince(t time.Time) ([]time.Time, error)
diff --git a/internal/alert/emailer.go b/internal/alert/emailer.go
index 5baf94a..f2601b7 100644
--- a/internal/alert/emailer.go
+++ b/internal/alert/emailer.go
@@ -131,7 +131,9 @@ func (e *Emailer) drain(ctx context.Context) {
 //
 // The send is recorded before it is attempted, and a failed send is logged
 // and dropped, never retried or re-reported. It still counts against the
-// caps: a send that timed out may have reached Mailgun and been charged.
+// caps: a send that timed out may have reached Mailgun and been charged. The
+// suppressed count is the exception: only a delivered email clears it, since
+// it is the one place held-back alerts are ever reported.
 func (e *Emailer) handle(ctx context.Context, a Alert) {
 	defer func() {
 		if r := recover(); r != nil {
@@ -170,8 +172,19 @@ func (e *Emailer) handle(ctx context.Context, a Alert) {
 	subject, text := e.compose(a, suppressed, capReopens)
 	if err := e.mailer.SendAlert(ctx, e.to, subject, text); err != nil {
 		e.log.Printf("alert %s: send failed: %v; dropped: %s", a.Category, err, a.Summary)
+		// Nobody saw this one either, so it joins the backlog the next email
+		// reports, and the ones already counted stay counted.
+		e.mem.recordSuppressed(a.Category)
+		if err := e.ledger.RecordAlertSuppressed(a.Category); err != nil {
+			e.ledgerFailed(err)
+		}
 		return
 	}
+	// Only an email that went out clears the backlog it just reported.
+	e.mem.clearSuppressed(a.Category)
+	if err := e.ledger.ClearSuppressed(a.Category); err != nil {
+		e.ledgerFailed(err)
+	}
 	e.log.Printf("alert %s: sent: %s", a.Category, a.Summary)
 }
 
@@ -287,8 +300,9 @@ func (m *memLedger) read(category string, since time.Time) (time.Time, int, []ti
 
 func (m *memLedger) recordSent(category string, at time.Time) {
 	m.lastSent[category] = at
-	m.suppressed[category] = 0
 	m.sends = append(slices.DeleteFunc(m.sends, func(t time.Time) bool { return !t.After(at.Add(-day)) }), at)
 }
 
+func (m *memLedger) clearSuppressed(category string) { m.suppressed[category] = 0 }
+
 func (m *memLedger) recordSuppressed(category string) { m.suppressed[category]++ }
diff --git a/internal/alert/emailer_test.go b/internal/alert/emailer_test.go
index 82b7e17..7e2bdca 100644
--- a/internal/alert/emailer_test.go
+++ b/internal/alert/emailer_test.go
@@ -387,6 +387,35 @@ func TestSubjectIsOneLine(t *testing.T) {
 	}
 }
 
+// The next email that goes out is the only place held-back alerts are ever
+// reported, so a send that fails must not take the count with it. The alert
+// whose send failed joins the backlog: nobody saw it either.
+func TestFailedSendKeepsTheSuppressedCount(t *testing.T) {
+	h := newHarness(t)
+	h.report("delivery", "reply send failing")
+	for range 3 {
+		h.clock.Advance(time.Minute)
+		h.report("delivery", "reply send failing")
+	}
+
+	h.mailer.err = errors.New("mailgun: 500 Internal Server Error")
+	h.clock.Advance(15 * time.Minute)
+	h.report("delivery", "reply send failing")
+
+	h.mailer.err = nil
+	h.clock.Advance(15 * time.Minute)
+	h.report("delivery", "reply send failing")
+
+	sent := h.mailer.sends()
+	if len(sent) != 3 {
+		t.Fatalf("send attempts = %d, want 3 (first ok, second failed, third ok)", len(sent))
+	}
+	// 3 held back by the cooldown, plus the one whose send failed.
+	if !strings.Contains(sent[2].text, "4 more since the last alert") {
+		t.Errorf("the recovered email should carry every alert nobody saw:\n%s", sent[2].text)
+	}
+}
+
 func TestNopReportDoesNothing(t *testing.T) {
 	Nop{}.Report(Alert{Category: "panic", Summary: "ignored"})
 }
diff --git a/internal/alert/fake_test.go b/internal/alert/fake_test.go
index a2e3323..9ec08f2 100644
--- a/internal/alert/fake_test.go
+++ b/internal/alert/fake_test.go
@@ -74,11 +74,20 @@ func (l *fakeLedger) RecordAlertSent(category string, at time.Time) error {
 		return l.writeErr
 	}
 	l.lastSent[category] = at
-	l.suppressed[category] = 0
 	l.sent = append(l.sent, at)
 	return nil
 }
 
+func (l *fakeLedger) ClearSuppressed(category string) error {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	if l.writeErr != nil {
+		return l.writeErr
+	}
+	l.suppressed[category] = 0
+	return nil
+}
+
 func (l *fakeLedger) RecordAlertSuppressed(category string) error {
 	l.mu.Lock()
 	defer l.mu.Unlock()
diff --git a/internal/app/app.go b/internal/app/app.go
index 496b4d9..8dbebb3 100644
--- a/internal/app/app.go
+++ b/internal/app/app.go
@@ -216,6 +216,7 @@ func (a *App) RecordAlertSent(category string, at time.Time) error {
 func (a *App) RecordAlertSuppressed(category string) error {
 	return a.store.RecordAlertSuppressed(category)
 }
+func (a *App) ClearSuppressed(category string) error            { return a.store.ClearSuppressed(category) }
 func (a *App) AlertSendsSince(t time.Time) ([]time.Time, error) { return a.store.AlertSendsSince(t) }
 
 // InterruptLeftoverJobs marks the jobs a dead predecessor left running as
diff --git a/internal/store/alert.go b/internal/store/alert.go
index f1ff626..fa7b77a 100644
--- a/internal/store/alert.go
+++ b/internal/store/alert.go
@@ -40,8 +40,9 @@ func (s *Store) LastAlert(category string) (time.Time, int, error) {
 	return t, suppressed, nil
 }
 
-// RecordAlertSent records a send in category at at, resets its suppressed
-// count, and prunes sends too old for any cap window.
+// RecordAlertSent records a send in category at at and prunes sends too old
+// for any cap window. The suppressed count is left alone: a send that then
+// fails reported nothing, so ClearSuppressed is what zeroes it.
 func (s *Store) RecordAlertSent(category string, at time.Time) error {
 	tx, err := s.db.Begin()
 	if err != nil {
@@ -49,7 +50,7 @@ func (s *Store) RecordAlertSent(category string, at time.Time) error {
 	}
 	defer tx.Rollback()
 	if _, err := tx.Exec(`INSERT INTO alert_state(category,last_sent_at,suppressed) VALUES(?,?,0)
- ON CONFLICT(category) DO UPDATE SET last_sent_at=excluded.last_sent_at, suppressed=0`, category, formatAlertTime(at)); err != nil {
+ ON CONFLICT(category) DO UPDATE SET last_sent_at=excluded.last_sent_at`, category, formatAlertTime(at)); err != nil {
 		return err
 	}
 	if _, err := tx.Exec("INSERT INTO alert_sends(sent_at) VALUES(?)", formatAlertTime(at)); err != nil {
@@ -61,6 +62,13 @@ func (s *Store) RecordAlertSent(category string, at time.Time) error {
 	return tx.Commit()
 }
 
+// ClearSuppressed zeroes category's suppressed count, once an email carrying
+// it has gone out. A category with no row has nothing to clear.
+func (s *Store) ClearSuppressed(category string) error {
+	_, err := s.db.Exec("UPDATE alert_state SET suppressed=0 WHERE category=?", category)
+	return err
+}
+
 // RecordAlertSuppressed counts one held-back alert against category.
 func (s *Store) RecordAlertSuppressed(category string) error {
 	_, err := s.db.Exec(`INSERT INTO alert_state(category,suppressed) VALUES(?,1)
diff --git a/internal/store/alert_test.go b/internal/store/alert_test.go
index 351adbf..82a9948 100644
--- a/internal/store/alert_test.go
+++ b/internal/store/alert_test.go
@@ -54,7 +54,10 @@ func TestAlertSuppressedBeforeAnySend(t *testing.T) {
 	}
 }
 
-func TestRecordAlertSentStoresTimeAndResetsSuppressed(t *testing.T) {
+// Recording a send and clearing the backlog are separate steps: the send is
+// recorded before it is attempted, to keep the caps exact, but the count of
+// held-back alerts survives until one actually goes out.
+func TestRecordAlertSentKeepsSuppressedUntilCleared(t *testing.T) {
 	s, _ := openTestStore(t)
 	at := time.Date(2026, 9, 11, 9, 30, 15, 123456789, time.UTC)
 
@@ -68,8 +71,15 @@ func TestRecordAlertSentStoresTimeAndResetsSuppressed(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-	if !sentAt.Equal(at) || suppressed != 0 {
-		t.Fatalf("LastAlert = %v, %d; want %v, 0", sentAt, suppressed, at)
+	if !sentAt.Equal(at) || suppressed != 1 {
+		t.Fatalf("LastAlert = %v, %d; want %v, 1", sentAt, suppressed, at)
+	}
+
+	if err := s.ClearSuppressed("delivery"); err != nil {
+		t.Fatal(err)
+	}
+	if _, suppressed, _ := s.LastAlert("delivery"); suppressed != 0 {
+		t.Fatalf("suppressed after ClearSuppressed = %d, want 0", suppressed)
 	}
 	// Other categories are untouched.
 	if other, _, _ := s.LastAlert("intake"); !other.IsZero() {

From f47aec55747f5502b75a07faf2ebe2fe7348ed7d Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 15:53:38 -0600
Subject: [PATCH 24/25] Report a mark failure that follows another alert, and
 name orphaned files

Two fixes in the delivery loop`s reporting:
- The per-outage reported flag was a bool, so once a submission had reported
  publish or delivery, a later mark failure was silently swallowed. That is
  the one category with no grace period, because every restart before it
  heals sends the sender another duplicate. The flag now holds the category
  that was reported, and a different one reports on its own.
- Orphan alerts name their Drive file id in the summary. Only the first
  orphan per cooldown is emailed, and the rest are logged by summary alone,
  so the id has to be there: it is all anyone has to delete the file by hand.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 internal/mailgun/alerts_test.go | 32 +++++++++++++++++++++++++++
 internal/mailgun/outbound.go    | 38 +++++++++++++++++++++++++++------
 2 files changed, 63 insertions(+), 7 deletions(-)

diff --git a/internal/mailgun/alerts_test.go b/internal/mailgun/alerts_test.go
index 7f9f92d..d43cba4 100644
--- a/internal/mailgun/alerts_test.go
+++ b/internal/mailgun/alerts_test.go
@@ -153,6 +153,32 @@ func TestMarkFailureReportsAtOnce(t *testing.T) {
 	}
 }
 
+// The grace period is per submission, but delivery-mark exists to fire at
+// once. A submission that already reported a publish outage must still report
+// a reply that went out and could not be marked: until that is fixed, every
+// restart sends the sender another duplicate.
+func TestMarkFailureAfterAPublishOutageStillReports(t *testing.T) {
+	f, rep, clock := newAlertingFixture(t)
+	f.addSubmission(t, 1, "iwk@mill.test", "schedule.xlsx")
+	f.publisher.err = fmt.Errorf("google token expired")
+
+	tick(t, f)
+	clock.advance(5 * time.Minute)
+	tick(t, f)
+	if len(rep.only("publish")) != 1 {
+		t.Fatalf("alerts = %+v, want the publish outage reported first", rep.alerts)
+	}
+
+	// Drive recovers, the reply goes out, and now the mark fails.
+	f.publisher.err = nil
+	f.engine.markErr = fmt.Errorf("disk I/O error")
+	tick(t, f)
+
+	if got := rep.only("delivery-mark"); len(got) != 1 {
+		t.Fatalf("a mark failure after a publish outage went unreported; alerts = %+v", rep.alerts)
+	}
+}
+
 // An upload whose record failed leaves a world-editable file in Drive that the
 // retention sweep will never delete. Each one is reported at once, naming the
 // file, since only a person can clean it up.
@@ -172,6 +198,12 @@ func TestOrphanedDriveFileReportsAtOnce(t *testing.T) {
 		if !strings.Contains(got[i].Detail, id) {
 			t.Errorf("orphan alert %d does not name %s:\n%s", i+1, id, got[i].Detail)
 		}
+		// The throttle emails only the first orphan in a cooldown; the rest are
+		// logged by summary alone, so the summary has to carry the file id or
+		// the only record of what to delete by hand is lost.
+		if !strings.Contains(got[i].Summary, id) {
+			t.Errorf("orphan summary %d does not name %s: %q", i+1, id, got[i].Summary)
+		}
 	}
 	if len(rep.only("publish")) != 0 {
 		t.Error("the publish grace period still applies to the submission itself")
diff --git a/internal/mailgun/outbound.go b/internal/mailgun/outbound.go
index f0322db..637fbed 100644
--- a/internal/mailgun/outbound.go
+++ b/internal/mailgun/outbound.go
@@ -181,13 +181,22 @@ func (s *Service) markDelivered(id int64) error {
 // to. An unlabelled error is a failing reply: "delivery".
 type deliveryError struct {
 	category string
+	file     string // the Drive file id, for publish-orphan
 	err      error
 }
 
 func (e *deliveryError) Error() string { return e.err.Error() }
 func (e *deliveryError) Unwrap() error { return e.err }
 
-func failedAt(category string, err error) error { return &deliveryError{category, err} }
+func failedAt(category string, err error) error {
+	return &deliveryError{category: category, err: err}
+}
+
+// orphanedFile labels an upload whose record failed, naming the Drive file
+// left behind.
+func orphanedFile(fileID string, err error) error {
+	return &deliveryError{category: "publish-orphan", file: fileID, err: err}
+}
 
 func categoryOf(err error) string {
 	var labelled *deliveryError
@@ -197,10 +206,22 @@ func categoryOf(err error) string {
 	return "delivery"
 }
 
+func fileOf(err error) string {
+	var labelled *deliveryError
+	if errors.As(err, &labelled) {
+		return labelled.file
+	}
+	return ""
+}
+
 // deliveryOutage is one submission's unbroken run of failed deliveries.
 type deliveryOutage struct {
-	since    time.Time
-	reported bool
+	since time.Time
+	// reported is the category already reported for this outage, so the same
+	// failure isn't reported every tick. A different category is a different
+	// problem and reports on its own — delivery-mark above all, which must
+	// never wait behind an earlier publish or delivery alert.
+	reported string
 }
 
 // deliveryFailed reports a failing submission once per outage: after
@@ -212,9 +233,12 @@ func (s *Service) deliveryFailed(sub store.EmailSubmission, err error) {
 	now := s.clock()
 	category := categoryOf(err)
 	if category == "publish-orphan" {
+		// The summary names the file because the throttle emails only the
+		// first orphan per cooldown; the rest are logged by summary alone, and
+		// the id is all anyone has to delete it by hand.
 		s.report(alert.Alert{
 			Category: category,
-			Summary:  "published Drive file orphaned",
+			Summary:  fmt.Sprintf("published Drive file orphaned: %s", fileOf(err)),
 			Detail: submissionDetail(sub, err) +
 				"\nThe file was uploaded to Google Drive, but its record was not saved, so the retention sweep will never delete it. Delete it by hand.\n",
 		})
@@ -233,10 +257,10 @@ func (s *Service) deliveryFailed(sub store.EmailSubmission, err error) {
 	if category == "delivery-mark" {
 		grace = 0
 	}
-	if outage.reported || now.Sub(outage.since) < grace {
+	if outage.reported == category || now.Sub(outage.since) < grace {
 		return
 	}
-	outage.reported = true
+	outage.reported = category
 	s.report(alert.Alert{
 		Category: category,
 		Summary:  outageSummary(category),
@@ -305,7 +329,7 @@ func (s *Service) publish(ctx context.Context, submissionID int64, outputs []app
 		if err := s.engine.PutDelivery(submissionID, i, fileID, link); err != nil {
 			// The upload succeeded but is now unrecorded, so a retry would
 			// re-upload it. Name the orphan so it can be found by hand.
-			return nil, failedAt("publish-orphan", fmt.Errorf("record published file %s (now orphaned in Drive): %w", fileID, err))
+			return nil, orphanedFile(fileID, fmt.Errorf("record published file %s (now orphaned in Drive): %w", fileID, err))
 		}
 		links = append(links, link)
 	}

From 09e30b9010fdd9c41bc9137407b0f93546532577 Mon Sep 17 00:00:00 2001
From: Kevin Brown <git@whidit.com>
Date: Fri, 11 Sep 2026 15:53:38 -0600
Subject: [PATCH 25/25] Record the review fixes in the alerting plan

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---
 ERROR-ALERTING-PLAN.md | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/ERROR-ALERTING-PLAN.md b/ERROR-ALERTING-PLAN.md
index c9c5b31..17df0ff 100644
--- a/ERROR-ALERTING-PLAN.md
+++ b/ERROR-ALERTING-PLAN.md
@@ -110,8 +110,9 @@ type Mailer interface {
 // Ledger is the persisted throttle state. *store.Store satisfies it.
 type Ledger interface {
     LastAlert(category string) (sentAt time.Time, suppressed int, err error) // zero sentAt: never sent
-    RecordAlertSent(category string, at time.Time) error // resets suppressed
+    RecordAlertSent(category string, at time.Time) error // for the caps; leaves suppressed alone
     RecordAlertSuppressed(category string) error         // suppressed++
+    ClearSuppressed(category string) error               // after an email carrying the count went out
     AlertSendsSince(t time.Time) ([]time.Time, error)    // sends after t, oldest first
 }
 
@@ -248,7 +249,10 @@ app.Open → mailgun.Load → if alert_recipient set:
     report restart / interrupted jobs (§3.6)
 ```
 
-`--once` mode and a Mailgun-less configuration keep `Nop`. `emailer.Run` should
+`--once` mode and a Mailgun-less configuration keep `Nop`. Marking interrupted
+jobs and reporting `restart` are continuous-mode only, for a related reason:
+`--once` binds no port, so it cannot tell whether the real worker is mid-job,
+and `run --once` alongside it would mark that worker's live job `interrupted`. `emailer.Run` should
 drain the queue on shutdown within the existing 10s window, so an alert raised
 during shutdown still goes out.