refactor(scheduling): converge flow and system-task scheduling on Agents API Routines

[chubes4]

Closes #3458

Converges Data Machine's recurring scheduling onto Agents API Routines (v0.11.0) and deletes the bespoke scheduler. This is a live migration on ~706 scheduled flows (verified against events.extrachill.com, below): every flow scheduled before the upgrade is scheduled after it, exactly once, with the same interval.

Status: all 18 CI checks green (homeboy review lint + full MySQL test suite across 4 shards + audit).

Identity contract

	Before (bespoke scheduler)	After (routines)
Recurring hook	datamachine_run_flow_now	wp_agent_routine_run_scheduled
Args	[flow_id, null, {generation}]	['routine_id' => 'flow-N']
Group	data-machine	agents-api
Generation fencing	DM option stamps + GenerationFencedAction	substrate generations per routine
One-time runs	single action on datamachine_run_flow_now	single action on datamachine_run_flow_once
Reconcile	FlowScheduleReconciler + reconciliation lock	Registry::reconcile() (verification-mode gated)
System schedules	RecurringScheduler::ensureSchedule per datamachine_recurring_<schedule> hook	system-<schedule_id> routines → datamachine/dispatch-system-task
Stagger	calculateStaggerOffset (crc32 of flow id)	substrate stagger (crc32 of routine id)
Coverage guard	hasLogicalCoverage scans	exact-match as_next_scheduled_action

Idempotence decision (design item 1a-CHECK)

Verified by reading the substrate: WP_Agent_Routine_Registry::register() fires wp_agent_routine_registered on every call, and the bridge's register() unconditionally does as_unschedule_all_actions + as_schedule_* + mints a fresh generation. Naive per-request re-declaration of ~700 flows would therefore thrash Action Scheduler every request.

Chosen fix: a backend decorator. DM installs HashGatedRoutineBackend via the wp_agent_routine_backend filter. It implements the backend interface by delegation and no-ops register() when the routine's schedule fingerprint (trigger + interval/expression + stagger window + ability + input — not label/meta, so flow renames never reset timers) matches a persisted per-routine hash in one option (datamachine_routine_schedule_hashes, autoload off). unregister()/pause() delete the hash entry so boot self-heals; reconciliation runs inside a verification mode that forces pass-through so registry repairs can never be swallowed by the hash gate. First boot after upgrade has no hashes → every register passes through → that is the migration moment.

This is a consumer-side cache, not a substrate reimplementation. Upstream issue filed as the permanent fix: Automattic/agents-api#548 ("Registry needs a non-persisting adopt/declare path for consumers with many persisted routines"). Boot cost after first sync: one option read + ~700 in-memory registrations + zero AS writes per request.

Note: composer.lock moved wordpress/agents-api from 6c785b32 to 96806e2 (v0.11.0) so CI and runtime share the substrate version the adapter targets.

Migration runbook

# 1. Inspect the plan (prints routine id, interval seconds, old action ids per flow)
wp datamachine flows migrate-routines

# 2. Apply (cancels generated legacy chains; leaves backpressure deferral ticks alone)
wp datamachine flows migrate-routines --apply

# 3. Audit routine coverage (dry run), then repair if needed
wp datamachine flows reconcile-schedules
wp datamachine flows reconcile-schedules --apply

# 4. Verify next_run resolves for scheduled flows
wp datamachine flows list

The migration also runs itself once on first boot (guarded by datamachine_routines_migrated_v1), so installs that never run the CLI still converge. The legacy datamachine_run_flow_now listener stays live for one release (TODO(3458)): queue-backpressure deferrals and stuck-job recovery still schedule it with positional args, so it remains a real executor; migration only cancels pending actions carrying the generated [flow_id, null, {gen}] shape.

Live dry-run verification (read-only, events.extrachill.com)

== FLEET PLAN (read-only dry run, 2026-09-09 03:29:48 UTC) ==
flows total: 718
schedulable (routine will register): 706
manual/disabled: 12
one_time: 0
invalid definitions: 0
flows with legacy pending actions: 706
legacy pending actions total: 706
legacy actions with no matching flow row: 0
arg shapes: 706 x (argc=3+gen)
interval histogram: daily 582 | twicedaily 106 | weekly 10 | every_3_days 5 | qtrdaily 3
flows with >1 legacy action (drift): 0

Exactly 706 pending actions map 1:1 to 706 flows with non-manual intervals; no flow maps to zero or >1 old action; zero orphans. (Matches the interval mix quoted in #3458.)

LOC

58 files changed, +2,701 / −6,352 (net −3,651)

Deleted: RecurringScheduler.php (1,436), ScheduleActionIdentity.php, GenerationFencedAction.php, FlowScheduling.php, FlowScheduleReconciler.php (538), FlowScheduleReconciliationLock.php, and 9 test files targeting them.

Kept despite the brief's deletion list (non-scheduling users verified): RecurringScheduleRegistry (TaskRegistry trigger metadata + schedule definitions source), RecurringRejectionTracker (SystemAbilities health signal; now fed by datamachine/dispatch-system-task). TaskScheduler/TaskRegistry (batch fan-out, #3428) untouched.

Risks

• First-run shift: new chains start at time() + stagger_offset(routine_id); first runs after upgrade shift by up to min(interval, 3600)s. No timestamps carried across (deliberate).
• One-release legacy path: datamachine_run_flow_now remains an execution bridge for backpressure deferrals; a straggler generated action that slips past migration executes once rather than double-firing (the new chain is the sole recurring mechanism, and migration cancels all pending generated actions).
• Registry-before-reconcile: FlowRoutines::reconcile() forces boot registration before Registry::reconcile(); an empty registry would classify every pending routine action as an orphan. The deferred-repair marker hook moved from action_scheduler_init to init for the same reason.
• Permission: datamachine/run-flow and datamachine/dispatch-system-task gate through PermissionHelper::can_manage(), which already admits the Action Scheduler principal (doing_action('action_scheduler_run_queue')); REST surface unchanged (routine permission filter defaults deny and allow-lists only routines registered this request).
• Pipeline row-delete failures now log-and-continue instead of failing the delete ability (the old coordinated failure path was coupled to the deleted commitDesiredSchedule primitive; the flow remains persisted but unscheduled, visible as manual).

Verification

• CI: all 18 checks green — Candidate Lint (PHPCS + PHPStan level 7 scoped to changed files, clean) and the full MySQL PHPUnit suite (4 shards, ~500 tests, 0 failures) plus the audit gate
• php -l on all changed files
• phpcbf --standard=WordPress clean on every changed PHP file
• Standalone smoke suites re-run locally: import-export-portable-flow-settings-smoke (55 assertions), recurring-schedule-registry-contract-smoke, recurring-rejection-escalation-smoke (26 assertions) — all pass
• New tests/Unit/Engine/Scheduling/FlowRoutinesTest.php covers alias→seconds, sync idempotence (no timer reset), manual→unregister, permission allow/deny, migration dry-run plan shape, and apply-cancels-generated-chains-only
• phpstan-baseline.neon had no entries for any deleted symbol
• Live read-only fleet verification above
• Local flow-config-cli-round-trip-smoke fails identically on main (missing AbilityResult stub in the standalone stubs — same chore(lint): eslint.config.js does not extend @wordpress/eslint-plugin recommended, so local eslint predicts nothing CI enforces #3469 class of local-only gap); it passes under the managed runtime in CI

AI-generated with Claude Code via Homeboy worktree