Operator error alerting (#7)

[brocla]

Implements issue #7: FileMill emails the operator when it fails systemically, so an unattended worker no longer fails silently. Built in the five phases of ERROR-ALERTING-PLAN.md, which is updated here to match what was actually built.

What it does

• New leaf package internal/alert. Reporter (no-op by default), an Emailer that queues alerts and drains them on its own goroutine, and a throttle: 15 minutes per category, 10 an hour, 20 a day over a rolling 24h. The daily cap is sized for the Mailgun Free plan, whose 100 daily sends are shared with replies. The send that fills the cap says so.
• Throttle state lives in SQLite (alert_state, alert_sends), so a crash-looping worker can`t spend the whole budget. If the database fails, the throttle falls back to an in-memory copy rather than going silent, since a sick database is exactly what is worth alerting about.
• Alerts on systemic failures only. A transformer that rejects its input through the contract is the senders problem and stays silent; a crash, timeout, missing or invalid result.json, or a result contradicting its exit code is ours. Also: intake 500s, a Mailgun route that stores instead of forwarding, replies or Drive publishes failing for 5 minutes, a reply that cant be marked delivered, orphaned Drive files, failing job claims, both retention sweeps, and a restart after a crash.
• Crashes are reported by the next process. The supervisor passes the previous exit code and rapid-restart count to each relaunch, folded together with the count of jobs left running.
• Panic recovery in execute and in the delivery and sweep loops; each reports and carries on.
• filemill alert-test sends one test alert, to prove the channel before relying on it.

Fixes found along the way

• Reply resend storm (refs Reliability: at-least-once delivery can send a duplicate reply (#6) #6): when a reply was sent but couldnt be marked delivered, it was re-sent every second, which would flood the sender and spend the days Mailgun budget in under two minutes. Only the mark is retried now. Reliability: at-least-once delivery can send a duplicate reply (#6) #6`s remaining crash window is documented and accepted.
• CLI commands could interrupt a live job. store.Open marked every running job interrupted, so submit or jobs get run while the worker was mid-job made the delivery loop send a premature reply. Only the starting continuous worker does that now.
• Two misleading sender-facing messages: a result claiming success with a nonzero exit no longer reports the transformer`s success text, and an interrupted job now asks the sender to send the file again instead of leaving the line blank.

Verification

go build, go vet and go test ./... pass. Tests are isolation tests with fakes: the throttle tests use a fake clock, mailer and ledger, so they neither sleep nor touch SQLite or the network; execute's tests run the test binary itself as the transformer.

Verified in production against a build of this branch: a test send, one restart email from a killed worker with four further restarts suppressed, and 21 probe jobs producing 2 emails and 19 suppressions, the second email carrying the count held back. The bad-Mailgun-domain check was skipped deliberately: it delays real senders` replies, and that path is covered by tests and by the alerts above.

Notes

• Alerting is off until alert_recipient is set in the gitignored config/email.yaml; config/email.yaml.example documents it and the throttle settings.
• The last four commits fix correctness issues found in a review after that production run, including run --once reintroducing the interrupt hazard. A machine running an earlier build of this branch should be rebuilt.

🤖 Generated with Claude Code