Executor selection has no capability matching, so a phase can be placed on a host that cannot satisfy it

docs/operating/remote-executors.md

@@ -13,9 +13,17 @@ phase-dispatch proxy that sends work to a registered executor.
 ## Dispatching phases to an executor
 
 `ExecutorPhaseProxy` (`src/CodeyBox.Orchestrator/ExecutorPhaseProxy.cs`)
-implements `IExecutorPhaseRunner`: it selects a registered executor from the
-worker registry using `ExecutorEligibility` (zero-capacity, cordoned and
-unhealthy hosts register but are never selected), stages the phase's single
+implements `IExecutorPhaseRunner`: it places each phase on a registered
+executor through the pure `ExecutorPlacement` decider
+(`src/CodeyBox.Core/ExecutorPlacement.cs`), matching the phase's requirements
+against each host's declared attributes — the agent credential the route
+needs against `DeclaredCredentials` (exact equality), the sandbox target's
+network profile against `AllowedNetworkProfiles` (empty means all), and the
+work item's `RequiredCapabilities` against the host's `DeclaredCapabilities`
+in the same case-insensitive capability vocabulary the agent-class router
+uses. Cordoned, unhealthy, runtime-backed-off and at-capacity hosts are
+excluded; among the eligible hosts the least-loaded wins (ties break by host
+id). The proxy then stages the phase's single
 bare repo to the host through `IExecutorPhaseTransport`, runs the phase
 there, and stages the repo back as a tar archive that is validated (archive
 bytes, entry count, expansion ratio, path containment) before anything is
@@ -27,23 +35,39 @@ transferred — never the whole repos root — so an executor receives only the
 repo for the item it is running.
 
 Delivery is idempotent through `IIdempotencyStore`: the key is work item +
-phase + attempt and the body hash covers the request, so a redelivered
+phase + attempt and the body hash covers the request (including the placement
+requirements when set), so a redelivered
 dispatch replays the original result instead of provisioning a second
 sandbox, while the same key with a different body is refused as a conflict
 and never executes. With no executor registered, dispatch falls back to the
 in-process runner with unchanged behaviour.
 
 An agent failure on the executor is returned as a result (`AgentFailed`); a
 host, connection or transfer problem throws `ExecutorPhaseTransportException`
-and stores nothing, so an unreachable host is retried elsewhere rather than
-charged against the work item as an agent failure. The proxy never touches
+and stores nothing, so an unreachable host fails over to the next eligible
+host (and, when every eligible host fails, the last host-attributed failure
+propagates) rather than being charged against the work item as an agent
+failure. A host that declared a credential it does not actually hold surfaces
+the same way — as a host-attributed failure with failover — never as an agent
+failure. When hosts are registered but none is currently eligible, the
+dispatch is deferred under `PlacementRecheckIn` so the work item is requeued
+rather than failed; when no registered host provides a required capability,
+the item is reported unplaceable naming the unmet tag instead of being
+dispatched and failed, and neither path consumes a rework iteration. Every
+decision is logged with the chosen host and the per-candidate refusal reason.
+The proxy never touches
 the work item table — the transport carries dispatch only, and re-dispatch
 after failure stays with the pipeline state machine.
 
 Bounds live under `CodeyBox:ExecutorPhaseDispatch` (`StageOutMaxArchiveBytes`,
 `StageOutMaxEntries`, `StageOutMaxExpansionRatio`, `IdempotencyTtl`,
 `MaxRequestPayloadBytes`, `MaxResultFindings`, `MaxFindingLengthChars`,
-`MaxResultErrorLengthChars`, `MaxStreamChunkChars`), hot-reloadable like the other dispatch knobs.
+`MaxResultErrorLengthChars`, `MaxStreamChunkChars`, `PlacementRecheckIn`,
+`RuntimeUnhealthyBackoff`), hot-reloadable like the other dispatch knobs.
+`PlacementRecheckIn` (default 15 s, mirroring the remote sandbox provider)
+is the requeue delay used when every eligible host is full, cordoned or
+unhealthy; `RuntimeUnhealthyBackoff` (default 1 min) is how long a host that
+fails dispatch is skipped before the next dispatch probes it again.
 
 ## Live agent-output relay
 
@@ -81,6 +105,7 @@ All operational values live under `CodeyBox:Executor` and are hot-reloadable
 | `MaxConcurrentSandboxes` | `int?` | `null` (uncapped) | Host-local sandbox capacity. `0` registers but is never selected |
 | `AllowedNetworkProfiles` | `string[]` | `[]` (all) | Network profiles this host accepts; `"*"` also means all |
 | `DeclaredCredentials` | `string[]` | `[]` | Agent credential sets this host holds (e.g. `claude`, `codex`) |
+| `DeclaredCapabilities` | `string[]` | `[]` | Clearance tags this host may handle, in the work item `RequiredCapabilities` vocabulary |
 | `Cordoned` | `bool` | `false` | Draining: registers and heartbeats but is never selected |
 | `Healthy` | `bool` | `true` | Health gate: `false` routes placements away without unregistering |
 | `LocalSandboxProvider` | `string` | `process` | `process` (dev runner, UNSAFE) or `bubblewrap` |

docs/reference/api.md

@@ -1352,6 +1352,7 @@ Response: `200 OK` with a JSON array:
 | `maxConcurrentSandboxes` | Declared host-local sandbox capacity (`null` = uncapped / non-executor row) |
 | `executorNetworkProfiles` | Declared network profiles the executor accepts (empty = all; `null` for non-executor rows) |
 | `executorCredentials` | Names of the agent credential sets the executor holds (`null` for non-executor rows) |
+| `executorCapabilities` | Clearance tags the executor declares, in the work item `RequiredCapabilities` vocabulary (`null` for non-executor rows) |
 | `cordoned` | Draining flag: registers and heartbeats but is never selected for new placements |
 | `healthy` | Operator health gate: `false` routes new placements away without removing the registration |
 
@@ -1369,6 +1370,7 @@ Request (`application/json`):
   "maxConcurrentSandboxes": 2,
   "allowedNetworkProfiles": ["restricted"],
   "declaredCredentials": ["claude"],
+  "declaredCapabilities": ["sensitive"],
   "cordoned": false,
   "healthy": true,
   "processId": 12345
@@ -1381,6 +1383,7 @@ Request (`application/json`):
 | `maxConcurrentSandboxes` | Host-local sandbox capacity, 0–100000. `0` registers the executor but leaves it never selected; omit for uncapped |
 | `allowedNetworkProfiles` | At most 64 entries; empty (or `"*"`) accepts every profile |
 | `declaredCredentials` | At most 64 opaque credential-set names, matched by exact equality |
+| `declaredCapabilities` | At most 64 clearance tags in the work item `RequiredCapabilities` vocabulary, matched case-insensitively |
 | `cordoned` | Draining flag: registers and heartbeats but is never selected for new placements |
 | `healthy` | Health gate, default `true` |
 | `processId` | Executor OS process id, informational only |

src/CodeyBox.Api/ExecutorEndpoints.cs

@@ -52,10 +52,12 @@ private static async Task<IResult> RegisterAsync(
 
         string[] profiles;
         string[] credentials;
+        string[] capabilities;
         try
         {
             profiles = NormalizeEntries(req.AllowedNetworkProfiles, nameof(req.AllowedNetworkProfiles));
             credentials = NormalizeEntries(req.DeclaredCredentials, nameof(req.DeclaredCredentials));
+            capabilities = NormalizeEntries(req.DeclaredCapabilities, nameof(req.DeclaredCapabilities));
         }
         catch (ArgumentException ex)
         {
@@ -74,6 +76,7 @@ private static async Task<IResult> RegisterAsync(
             MaxConcurrentSandboxes = req.MaxConcurrentSandboxes,
             ExecutorNetworkProfiles = profiles,
             ExecutorCredentials = credentials,
+            ExecutorCapabilities = capabilities,
             Cordoned = req.Cordoned,
             Healthy = req.Healthy ?? true,
         };
@@ -191,6 +194,7 @@ public sealed class ExecutorRegistrationRequest
         public int? MaxConcurrentSandboxes { get; set; }
         public List<string>? AllowedNetworkProfiles { get; set; }
         public List<string>? DeclaredCredentials { get; set; }
+        public List<string>? DeclaredCapabilities { get; set; }
         public bool Cordoned { get; set; }
         public bool? Healthy { get; set; }
         public int? ProcessId { get; set; }

src/CodeyBox.Api/WorkerRegistryEndpoints.cs

@@ -28,6 +28,7 @@ private static async Task<IResult> ListWorkersAsync(IWorkerRegistry registry, Ca
             maxConcurrentSandboxes = w.MaxConcurrentSandboxes,
             executorNetworkProfiles = w.ExecutorNetworkProfiles,
             executorCredentials = w.ExecutorCredentials,
+            executorCapabilities = w.ExecutorCapabilities,
             cordoned = w.Cordoned,
             healthy = w.Healthy,
         }));

src/CodeyBox.Core/ExecutorEligibility.cs

@@ -75,4 +75,77 @@ public static bool HoldsCredential(ExecutorRegistration registration, string cre
         }
         return false;
     }
+
+    /// <summary>
+    /// True when the executor's declared capabilities cover every required
+    /// tag. Uses the same vocabulary and comparison as the agent-class
+    /// router's <c>RequiredCapabilities</c> gate: ordinal, case-insensitive,
+    /// exact equality per tag. An empty required set is covered by any host.
+    /// </summary>
+    public static bool CoversRequiredCapabilities(
+        ExecutorRegistration registration,
+        IReadOnlyList<string>? required)
+    {
+        ArgumentNullException.ThrowIfNull(registration);
+        if (required is null || required.Count == 0)
+            return true;
+        if (registration.DeclaredCapabilities.Count == 0)
+            return false;
+        foreach (var tag in required)
+        {
+            if (string.IsNullOrWhiteSpace(tag))
+                continue;
+            var wanted = tag.Trim();
+            var hit = false;
+            foreach (var have in registration.DeclaredCapabilities)
+            {
+                if (string.Equals(have?.Trim(), wanted, StringComparison.OrdinalIgnoreCase))
+                {
+                    hit = true;
+                    break;
+                }
+            }
+            if (!hit)
+                return false;
+        }
+        return true;
+    }
+
+    /// <summary>
+    /// First required capability no host in <paramref name="hosts"/> declares
+    /// (ordinal, case-insensitive), or null when every required tag is held
+    /// by at least one host. Blank required entries are ignored.
+    /// </summary>
+    public static string? FindCapabilityNoHostProvides(
+        IEnumerable<ExecutorRegistration> hosts,
+        IReadOnlyList<string>? required)
+    {
+        ArgumentNullException.ThrowIfNull(hosts);
+        if (required is null || required.Count == 0)
+            return null;
+        foreach (var tag in required)
+        {
+            if (string.IsNullOrWhiteSpace(tag))
+                continue;
+            var wanted = tag.Trim();
+            var provided = false;
+            foreach (var host in hosts)
+            {
+                ArgumentNullException.ThrowIfNull(host);
+                foreach (var have in host.DeclaredCapabilities)
+                {
+                    if (string.Equals(have?.Trim(), wanted, StringComparison.OrdinalIgnoreCase))
+                    {
+                        provided = true;
+                        break;
+                    }
+                }
+                if (provided)
+                    break;
+            }
+            if (!provided)
+                return wanted;
+        }
+        return null;
+    }
 }

src/CodeyBox.Core/ExecutorPhase.cs

@@ -61,6 +61,33 @@ public sealed record ExecutorPhaseRequest
     /// by dispatch options; covered by the idempotency body hash.
     /// </summary>
     public required string PayloadJson { get; init; }
+
+    /// <summary>
+    /// Agent credential set the phase's route requires (for example "claude").
+    /// Null or blank means the phase needs no specific credential and any
+    /// host may run it. Matched against
+    /// <see cref="ExecutorRegistration.DeclaredCredentials"/> by exact
+    /// ordinal equality. At most 128 chars.
+    /// </summary>
+    public string? RequiredCredential { get; init; }
+
+    /// <summary>
+    /// Sandbox network profile the phase's sandbox target requires. Null or
+    /// blank means "(default)". Matched against
+    /// <see cref="ExecutorRegistration.AllowedNetworkProfiles"/> with the
+    /// same empty-means-all / "*" semantics. At most 128 chars.
+    /// </summary>
+    public string? RequiredNetworkProfile { get; init; }
+
+    /// <summary>
+    /// Clearance tags the phase's work item demands, in the same vocabulary
+    /// as <see cref="WorkItem.RequiredCapabilities"/>. Empty means no
+    /// clearance required. Placement only selects hosts whose
+    /// <see cref="ExecutorRegistration.DeclaredCapabilities"/> covers every
+    /// tag here (ordinal, case-insensitive). At most 16 entries, each at most
+    /// 128 chars.
+    /// </summary>
+    public IReadOnlyList<string> RequiredCapabilities { get; init; } = [];
 }
 
 /// <summary>