fix(vahter): rebuild trgm index with fastupdate off, mark forensic indexes

[Szer]

Why

Every vahter 504 in the Envoy access logs (response_flags=UT, the 15 s upstream timeout on the Telegram webhook POST /bot) traces to one INSERT INTO event (the MessageReceived append) stalling for 8–30 s server-side. Nothing else runs on the DB at those moments, SELECTs on event keep working, and the second pod's identical INSERT (Telegram's 15 s redelivery) blocks until the first finishes.

Counting MessageReceived rows between consecutive stalls (2026-09-14 → 16, from Tempo + the event table):

stall intervals	rows between stalls	rawMessage bytes
12 consecutive	474 – 526	605 – 920 kB

A stall every ~500 inserts of ~1.4 kB text is the GIN pending list of idx_event_rawmessage_trgm (V47 forensic index, 653 MB, gin_pending_list_limit 4 MB) hitting its limit. The inserting backend flushes it synchronously and blocks other GIN inserters; on B1ms + P4 (120 IOPS) that takes tens of seconds. Envoy 504 UT on the vahter route: 1–5/day since 2026-08-28 (V47 landed), sporadic before.

What

V51 — DROP INDEX CONCURRENTLY IF EXISTS (old and new name, so a retry after a failed build is clean) + CREATE INDEX CONCURRENTLY idx_forensic_event_rawmessage_trgm ... WITH (fastupdate = off). Trigram entries then go straight into the index on each insert instead of accumulating into a periodic flush. The idx_forensic_ prefix records that this is a debug-session index and that GIN ones are built with fastupdate = off.

V52 — ALTER INDEX IF EXISTS ... RENAME TO idx_forensic_... (metadata only) for the other indexes that no bot query reads:

index	created by	evidence it is debug-only
idx_event_username_lower	V30 "debug indexes"	for v_users / v_recent_bans views; prod looks up usernames by stream_id (GetVahterActionStats)
idx_event_user_ban_by_stream	V30 "debug indexes"	no SELECT in DB.fs touches UserBanned/UserUnbanned rows; 418 scans since July come from debug views
idx_event_mlscored_score_created_at	V48 (#460)	LLM-band analysis query run by hand; no prod query reads score; 1 scan ever
idx_snapshot_user_username	V38	snapshot_user is only written by prod code (upsert + rebuild); 0 scans
idx_snapshot_user_banned	V38	same; 0 scans
idx_snapshot_user_spam_protection	V43	migration comment: "debug/ops queries"; 0 scans

Kept as-is despite low scan counts: idx_event_msg_marked_ham_text (0 scans, but GetUserStats' is_ham subquery filters on data->>'text' for MessageMarkedHam), the snapshot_message indexes (read by the report queries), and ix_bot_setting_feature_group (16 kB, not worth a migration).

Deploy notes

• The CONCURRENTLY build of a ~650 MB GIN index over 2.5 M rows on B1ms will take a while (tens of minutes is plausible). It does not block writes; the forensic index is simply absent until the build finishes. If the deploy job dies mid-build, the unconditional repair before the next migrate clears the failed row and V51 retries (the two IF EXISTS drops handle any leftover invalid index).
• Migrations run as the admin user, which owns every index here.
• Renames are metadata-only; no bot code or test references an index name.

Verify

• Tempo: { resource.service.name="vahter-bot" && name="postgresql" && duration > 2s } should stop showing the INSERT INTO event spans.
• Loki: {namespace="envoy-gateway-system"} |= "response_timeout" |= "vahter-bot-route" should go to zero per day.
• SELECT indexrelname, pg_get_indexdef(indexrelid) FROM pg_stat_user_indexes WHERE indexrelname LIKE 'idx_forensic_%' → 7 rows, the trgm one showing WITH (fastupdate='off').

🤖 Generated with Claude Code