feat(mining): incremental passive mining via per-source UID watermark

[malek10xdev]

Summary

Incremental passive mining with a clean design: enums instead of magic strings, getImapBoxes returns each folder's watermark + status, the backend derives the resume cursor from persisted config, and the mining-sources edge function is the single writer for source config.

What this changes

• Enums are the only vocabulary: MiningRunMode, TaskStatus, SourceHealthState, FolderStatus, SourceBadge are mirrored in the backend, edge functions and frontend. No 'full' / 'incremental' / "active" literals in code paths that write or compare domain values (legacy input decoding maps to the enum).
• getImapBoxes returns per-folder status: { name, key, total, attribs?, children?, watermark?, status, has_new_messages, latest_uid }, computed server-side by joining the live IMAP cursor with the persisted watermark. The frontend cursor/watermark state machine is gone.
• Resume is resolved server-side: the mine request sends intent only (boxes + miningMode); the backend builds resumeFrom = { folders: { <folder>: { uidvalidity, last_uid } } } from the source config and the emails-fetcher owns all IMAP logic (uidvalidity checks, [last_uid+1 .. uidNext-1], fallback, watermark emission).
• Run mode is transient: when selected pre-mined folders have new messages, a dialog asks Continue from last position vs Re-scan everything; the answer is sent as a request param. Nothing is persisted (no localStorage).
• mining-sources is the single config writer: callers send params (mining_flags, folders, passive_mining; internal health/mining); the edge function merges in JS (unknown keys preserved, null clears, per-folder cursors monotonic) and persists with supabase-js + a config_revision compare-and-swap. update_mining_source_config / jsonb_deep_merge are removed.
• One config migration: the intermediate RPC/merge migrations are squashed into 20260912010857_passive_mining_incremental_config.sql; the separate security migration is kept.

Verification

• Unit: backend 621/621, emails-fetcher 59/59, frontend focused 22/22, Deno (mining-sources config 5/5, mining-completion 4/4, enums).
• API E2E (local Supabase + Greenmail + Enron): getImapBoxes returns new_messages + watermark; the backend-built resumeFrom fetches only the new UIDs; mining-completion → mining-sources advances mining.last with the revision CAS; idempotent re-trigger.
• UI E2E (agent-browser): the Continue/Re-scan dialog appears only when pre-mined folders have new messages; "Continue" resumed 32:33 (2/6 fetched); no localStorage key is written.
• CI: backend, frontend, CodeQL, CodeSee, coverage, Snyk and DeepSource (JavaScript/Docker/SQL/Secrets/Test coverage) all green.

Notes

• Migration 20260912010857_passive_mining_incremental_config.sql: adds config + config_revision, has get_mining_source_credentials_for_user return config, and drops the superseded SQL merge helpers.
• Deploy order: migration → emails-fetcher → backend → edge functions → frontend.

[github-actions]

Coverage Report

✅ Passed

Commit: 0c7baf0

Summary

	Name	Stmts	Branch	Funcs	Lines
🟡	Total	46.4%	44.3%	52.5%	46.8%

All files

	Name	Stmts	Branch	Funcs	Lines
🔴	backend/src/controllers/contacts.controller.ts	0%	0%	0%	0%
🟢	backend/src/controllers/imap.controller.ts	81.3%	77.3%	100%	80.9%
🔴	backend/src/controllers/mining.controller.ts	0%	0%	0%	0%
🟡	backend/src/db/completion.ts	44.4%	0%	0%	44.4%
🟢	backend/src/db/types.ts	100%	100%	100%	100%
🟡	backend/src/db/pg/PgSmtpSenders.ts	45.8%	28.9%	55.6%	45.8%
🔴	backend/src/db/supabase/MiningSourceService.ts	11.4%	0%	12.5%	11.6%
🟡	backend/src/services/email-fetching/index.ts	51.6%	40%	60%	51.6%
🔴	backend/src/services/imap/ImapBoxesFetcher.ts	0%	0%	0%	0%
🟢	backend/src/services/tasks-manager-v2/Pipeline.ts	82.5%	80%	83.3%	85.1%
🔴	backend/src/services/tasks-manager-v2/factories.ts	0%	0%	0%	0%
🟢	backend/src/services/tasks-manager-v2/tasks/FetchTask.ts	100%	89.5%	100%	100%
🟢	backend/src/utils/helpers/imapTreeHelpers.ts	91.7%	85.4%	83.3%	91.5%
🟢	backend/src/validators/mining.schema.ts	100%	83.3%	100%	100%

No coverage changes

	Name	Stmts	Branch	Funcs	Lines
🔴	backend/src/controllers/contacts.controller.ts	0%	0%	0%	0%
🟢	backend/src/controllers/imap.controller.ts	81.3%	77.3%	100%	80.9%
🔴	backend/src/controllers/mining.controller.ts	0%	0%	0%	0%
🟡	backend/src/db/completion.ts	44.4%	0%	0%	44.4%
🟢	backend/src/db/types.ts	100%	100%	100%	100%
🟡	backend/src/db/pg/PgSmtpSenders.ts	45.8%	28.9%	55.6%	45.8%
🔴	backend/src/db/supabase/MiningSourceService.ts	11.4%	0%	12.5%	11.6%
🟡	backend/src/services/email-fetching/index.ts	51.6%	40%	60%	51.6%
🔴	backend/src/services/imap/ImapBoxesFetcher.ts	0%	0%	0%	0%
🟢	backend/src/services/tasks-manager-v2/Pipeline.ts	82.5%	80%	83.3%	85.1%
🔴	backend/src/services/tasks-manager-v2/factories.ts	0%	0%	0%	0%
🟢	backend/src/services/tasks-manager-v2/tasks/FetchTask.ts	100%	89.5%	100%	100%
🟢	backend/src/utils/helpers/imapTreeHelpers.ts	91.7%	85.4%	83.3%	91.5%
🟢	backend/src/validators/mining.schema.ts	100%	83.3%	100%	100%

Generated by Test Coverage Reporter for commit 0c7baf0