Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 32 additions & 1 deletion docs/reference/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -302,8 +302,39 @@ The composite form is unambiguous and works with all endpoints that accept `{id}
Returns `400 Bad Request` when the colon form has an empty project or externalId part.
Returns `404 Not Found` when the project exists but has no item with that externalId.

### `POST /workitems/{id}/replay`
### `POST /workitems/{id}/delegate`

Delegate a work item to the unconstrained delegation phase: one repair turn
with latitude the normal work/audit/rework cycle does not grant, verified by
the same audit and merge gates afterwards. Works from any non-terminal state
and from the terminal failure states (`Failed`, `AuditFailed`,
`MergeConflictResolutionFailed`, `AbandonedAfterRecoveryAttempts`); `Done`,
`Cancelled`, and `NoActionRequired` return `409`.

**Request body** (all optional):

```json
{
"note": "focus on the auth race; the token refresh path is suspect"
}
```

- `note` — operator direction for the attempt, stored on the item and
rendered into the convergence brief (max 4000 chars; control characters
other than newline/tab are rejected).

A worker-held in-flight item is fenced through worker recovery first; when
fencing fails closed the command returns `409` rather than racing the
pipeline. The delegated turn competes for the same worker and sandbox
capacity as normal work (priority preserved, explicit end-of-queue position,
shared dispatcher) so it cannot starve normal dispatch.

Returns `202 Accepted` with the `Delegating` item, the `trigger`
(`operator`), and the `priorState`. Fires a `work_item.delegated` webhook.
Automatic escalation uses the same transition — see
`CodeyBox:DelegationEscalation` in [`configuration.md`](configuration.md).

### `POST /workitems/{id}/replay`
Clone a terminal work item and run it with a different agent or model. See
[`replay.md`](../concepts/work-items.md) for full semantics.

Expand Down
30 changes: 30 additions & 0 deletions docs/reference/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -819,6 +819,36 @@ incidents. A parsed stream-json `turn.failed` event whose `error.message` is
exactly `timeout` is the exception because that is provider transport metadata,
not free-form build output.

## `DelegationEscalation`

Delegation triggers: the operator delegate command (`POST
/workitems/{id}/delegate`, always available) plus automatic escalation when an
item provably stops converging. Automatic escalation fires at most once per
item — a second delegation always requires an operator — and an item whose
delegation turn failed is never escalated automatically again. Escalation
preserves the failure signal (audit history, `LastError`, attempt counters)
and counts every trigger by condition on the `codeybox.delegation.triggers`
metric (`trigger` = `operator` | `audit-max-iterations` |
`repeated-terminal-failure`).

```json
"DelegationEscalation": {
"Enabled": false,
"OnAuditMaxIterations": true,
"OnRepeatedTerminalFailure": true,
"RepeatedTerminalFailureThreshold": 2,
"MaxNoteChars": 4000
}
```

| Key | Default | Description |
|-----|---------|-------------|
| `Enabled` | `false` | Master switch for automatic escalation. Default off so the feature is operator-only unless deliberately opted into. The operator command works regardless. |
| `OnAuditMaxIterations` | `true` | Escalate when audit iterations reach the configured maximum without passing (otherwise parked for the operator). Individually disableable. |
| `OnRepeatedTerminalFailure` | `true` | Escalate when an item terminally fails repeatedly after retry. Individually disableable. |
| `RepeatedTerminalFailureThreshold` | `2` | Terminal-failure episodes required before the repeated-failure condition fires. The count survives retries, so manual retries count too. |
| `MaxNoteChars` | `4000` | Upper bound on the operator note stored per delegation request. Longer notes are rejected by the API (400). |

## `ConfigValidation`

Optional startup cross-check that every `AgentClass` member's `ModelId`
Expand Down
1 change: 1 addition & 0 deletions docs/reference/webhooks.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ One event is fired per state transition. Events follow the naming convention `wo
| `work_item.merge_conflict_resolution_failed` | Merge conflict resolution was rejected by host verification or the scope fence; the item is terminal |
| `work_item.waiting_for_quota_reset` | Every eligible class member hit quota in one pickup; the item is parked, not failed. Details use the agent-fallback shape |
| `work_item.resumed` | An operator-cancelled item re-entered the pipeline via `POST /workitems/{id}/resume`. Details: `id`, `externalId`, resumed-from phase |
| `work_item.delegated` | An item entered the delegation phase via the operator delegate command or automatic escalation. Details: `trigger` (`operator` \| `audit-max-iterations` \| `repeated-terminal-failure`), `priorState`, `reason`, `note`, `terminalFailureCount`, `autoEscalated` |
| `work_item.check_followup_enqueued` | A check-and-act verdict matched and its follow-up item was queued. Details: `originCheckWorkItemId`, `followupWorkItemId` |
| `work_item.post_act_recheck_completed` | A post-act re-check finished. Details: iteration, `answer`, `actionableAnswer`, originating check id |
| `agent.fallback` | Routing moved a phase to another class member mid-item. Details: phase, iteration, from/to agent and model, reason |
Expand Down
21 changes: 19 additions & 2 deletions src/CodeyBox.Api/Program.cs
Original file line number Diff line number Diff line change
Expand Up @@ -3637,7 +3637,8 @@ static Func<TestRunOptions> DotnetTestRunOptionsAccessor(IServiceProvider sp)
flakeEscalationOptions: sp.GetRequiredService<NonDeterministicTestEscalationSnapshot>(),
briefComposer: sp.GetRequiredService<ConvergenceBriefComposer>(),
delegationEvents: sp.GetRequiredService<IDelegationEventStore>(),
delegationOptionsAccessor: () => sp.GetRequiredService<IOptionsMonitor<CodeyBoxOptions>>().CurrentValue.Delegation));
delegationOptionsAccessor: () => sp.GetRequiredService<IOptionsMonitor<CodeyBoxOptions>>().CurrentValue.Delegation,
delegationEscalation: sp.GetService<DelegationEscalationService>()));
builder.Services.AddSingleton<IPipelineRunner>(sp => sp.GetRequiredService<PipelineRunner>());
// Isolated base-branch fix-item spawner for NotDiffAttributable audit test
// failures. Constructed lazily from the store/queue plus the hot-reloadable
Expand Down Expand Up @@ -3753,6 +3754,18 @@ static Func<TestRunOptions> DotnetTestRunOptionsAccessor(IServiceProvider sp)
// Operators wire alternate classifiers (e.g. LLM-precision layer) by replacing
// the singleton registration; the service treats the interface as authoritative.
builder.Services.AddSingleton<ITerminalFailureClassifier, DefaultTerminalFailureClassifier>();
// --- Delegation triggers ----------------------------------------------------
// Single home for every transition into the delegation phase: the operator
// delegate command and both automatic-escalation conditions. Constructed
// lazily from the store/queue plus the hot-reloadable options so threshold
// edits apply to the next trigger without restart.
builder.Services.AddSingleton<DelegationEscalationService>(sp =>
new DelegationEscalationService(
sp.GetRequiredService<IWorkItemStore>(),
sp.GetService<ITaskQueue>(),
() => sp.GetRequiredService<IOptionsMonitor<CodeyBoxOptions>>().CurrentValue.DelegationEscalation,
sp.GetService<IWebhookDispatcher>(),
sp.GetService<IProjectRepository>()));
builder.Services.AddSingleton<TerminalFailureRecoveryService>(sp => new TerminalFailureRecoveryService(
sp.GetRequiredService<IWorkItemStore>(),
sp.GetRequiredService<WorkItemRetrier>(),
Expand All @@ -3768,7 +3781,8 @@ static Func<TestRunOptions> DotnetTestRunOptionsAccessor(IServiceProvider sp)
live.JitterFraction,
live.MaxAutoRetriesPerWorkItem);
},
sp.GetRequiredService<ILogger<TerminalFailureRecoveryService>>()));
sp.GetRequiredService<ILogger<TerminalFailureRecoveryService>>(),
delegationEscalation: sp.GetService<DelegationEscalationService>()));
builder.Services.AddHostedService(sp => sp.GetRequiredService<TerminalFailureRecoveryService>());

builder.Services.AddSingleton<OrchestratorOptions>(sp =>
Expand Down Expand Up @@ -5854,6 +5868,9 @@ public sealed class CodeyBoxOptions
/// <summary>Knobs for the operator-triggered delegation phase (result-diff bounds).</summary>
public DelegationOptions Delegation { get; set; } = new();

/// <summary>Triggers for the delegation phase: operator command plus automatic escalation.</summary>
public DelegationEscalationOptions DelegationEscalation { get; set; } = new();

/// <summary>Config-gated live human supervision and injection channel.</summary>
public AgentSupervisionOptions AgentSupervision { get; set; } = new();

Expand Down
180 changes: 180 additions & 0 deletions src/CodeyBox.Api/WorkItemEndpoints.cs
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ public static void Map(WebApplication app)
group.MapPost("/{id}/abandon", AbandonAsync);
group.MapPost("/{id}/promote", PromoteAsync);
group.MapPost("/{id}/retry", RetryAsync);
group.MapPost("/{id}/delegate", DelegateAsync);
group.MapPost("/{id}/replay", ReplayAsync);
group.MapGet("/", ListAsync);
group.MapGet("/{id}", GetAsync);
Expand Down Expand Up @@ -556,6 +557,175 @@ internal static bool IsStaleWorkerRetryEligible(WorkItem item, DateTimeOffset no
return null;
}

/// <summary>
/// Delegate a work item to the unconstrained delegation phase: one repair
/// turn with latitude the normal work/audit/rework cycle does not grant,
/// verified by the same audit and merge gates afterwards.
///
/// Works from any non-terminal state and from the terminal failure states
/// (Failed, AuditFailed, MergeConflictResolutionFailed,
/// AbandonedAfterRecoveryAttempts) — exactly the items that need it.
/// Done (nothing to repair), Cancelled (operator-stopped), and
/// NoActionRequired (resolved) return 409.
///
/// The optional <c>note</c> is stored on the item and rendered into the
/// convergence brief so the operator can direct the attempt. A
/// worker-held in-flight item is fenced through worker recovery first
/// (operator intent substitutes for a staleness verdict); when fencing
/// fails closed the command returns 409 rather than racing the pipeline.
///
/// The delegated turn competes for the same worker and sandbox capacity
/// as normal work through the shared dispatcher: priority is preserved,
/// an explicit end-of-queue position is stamped, and no lane or boost is
/// granted — so delegation cannot starve normal dispatch.
///
/// Returns:
/// - 202 with the Delegating item when the trigger is armed.
/// - 400 when the note violates its length/control-character guard.
/// - 404 when the item does not exist.
/// - 409 when the state is not delegable, operator questions are still
/// open, the worker fence fails closed, or the row advanced
/// concurrently.
/// </summary>
private static async Task<IResult> DelegateAsync(
string id,
DelegateWorkItemRequest? body,
IWorkItemStore store,
DelegationEscalationService delegationEscalation,
IWorkerRegistry registry,
ItemStaleProgressWatchdog staleWatchdog,
IWorkItemQuestionStore? questions,
IOptionsMonitor<CodeyBoxOptions> options,
CancellationToken ct)
{
var (item, err) = await ResolveWorkItemAsync(id, store, ct);
if (err is not null) return err;

var maxNoteChars = Math.Max(1, options.CurrentValue.DelegationEscalation.MaxNoteChars);
var note = body?.Note;
if (string.IsNullOrWhiteSpace(note))
{
note = null;
}
else if (note.Length > maxNoteChars)
{
return Results.BadRequest(new { error = $"note must be <= {maxNoteChars} chars" });
}
else if (note.Any(c => char.IsControl(c) && c is not ('\r' or '\n' or '\t')))
{
return Results.BadRequest(new { error = "note must not contain control characters" });
}

if (!DelegationEscalationPolicy.IsDelegableState(item!.State))
return Results.Conflict(new { error = $"cannot delegate item in state {item.State}; only non-terminal states and terminal failure states can be delegated" });

if (item.State == WorkItemState.NeedsOperatorInput && questions is not null)
{
var openQuestions = (await questions.ListByWorkItemAsync(item.Id.ToString(), ct))
.Where(q => string.Equals(q.State, "open", StringComparison.Ordinal))
.Select(q => q.QuestionId)
.Take(5)
.ToArray();
if (openQuestions.Length > 0)
{
return Results.Conflict(new
{
error = "cannot delegate item while operator questions are open; answer or dismiss them first",
openQuestions,
});
}
}

// A worker-held in-flight item cannot simply be flipped to Delegating
// under a live pipeline. Fence it through worker recovery first — the
// explicit operator command authorizes interrupting the current turn,
// so no staleness verdict is required; recovery still fails closed on
// unfenceable dispatch claims and concurrent advances.
if (WorkItemRecoveryPolicy.IsItemStaleWatchedState(item.State))
{
var fenceError = await TryFenceLiveWorkerItemForDelegateAsync(
item, registry, staleWatchdog, ct);
if (fenceError is not null)
return fenceError;
var fenced = await store.GetAsync(item.Id, ct);
if (fenced is null)
return Results.Conflict(new { error = "work item no longer exists" });
if (!DelegationEscalationPolicy.IsDelegableState(fenced.State))
return Results.Conflict(new { error = $"cannot delegate item in state {fenced.State} after fencing the previous worker; only non-terminal states and terminal failure states can be delegated" });
item = fenced;
}

var result = await delegationEscalation.DelegateAsync(
item,
DelegationTriggers.Operator,
note,
markAutoEscalated: false,
failureContext: null,
ct);
if (!result.Delegated)
return Results.Conflict(new { error = result.Error });

return Results.Accepted(
$"/workitems/{item.Id}",
new
{
id = item.Id.ToString(),
trigger = DelegationTriggers.Operator,
priorState = item.State.ToString(),
state = WorkItemState.Delegating.ToString(),
});
}

/// <summary>
/// Fences a worker-bound item so an operator delegate cannot race the live
/// pipeline. Returns null when delegation may proceed (no worker binds the
/// item, or recovery fenced it); otherwise the 409 result to return.
/// Unlike the retry fence, staleness is not required: the explicit
/// operator command itself authorizes interrupting the current turn.
/// Recovery fails closed on unfenceable dispatch claims, concurrent
/// advances, and exhausted attempt budgets that cannot park.
/// </summary>
private static async Task<IResult?> TryFenceLiveWorkerItemForDelegateAsync(
WorkItem item,
IWorkerRegistry registry,
ItemStaleProgressWatchdog staleWatchdog,
CancellationToken ct)
{
var idStr = item.Id.ToString();
var bound = false;
try
{
var workers = await registry.ListAsync(ct);
foreach (var worker in workers)
{
if (string.Equals(worker.CurrentWorkItemId, idStr, StringComparison.OrdinalIgnoreCase))
{
bound = true;
break;
}
}
}
catch (OperationCanceledException) when (ct.IsCancellationRequested) { throw; }
catch (Exception ex)
{
return Results.Conflict(new { error = $"cannot delegate worker-held item {item.Id}: failed to inspect worker bindings: {ex.Message}" });
}

if (!bound)
return null;

var recovery = await staleWatchdog.RecoverItemAsync(
item,
$"operator delegate fenced worker-held item in {item.State}",
ct);
if (!recovery.Recovered)
{
return Results.Conflict(new { error = $"cannot delegate worker-held item {item.Id}: {recovery.Error ?? "recovery did not transition the work item"}" });
}

return null;
}

/// <summary>
/// Create a replay of a terminal work item, optionally swapping the agent via agentClassId.
/// The new item gets the same prompt, base branch, and dependsOn list; it runs
Expand Down Expand Up @@ -2351,6 +2521,10 @@ private static WorkItemDto ToDto(
DelegationAttempts: item.DelegationAttempts,
DelegationRequested: item.DelegationRequested,
DelegationReason: item.DelegationReason,
DelegationNote: item.DelegationNote,
AutoDelegationEscalated: item.AutoDelegationEscalated,
DelegationFailed: item.DelegationFailed,
TerminalFailureCount: item.TerminalFailureCount,
Knobs: item.Knobs.Count == 0
? null
: item.Knobs.ToDictionary(kv => kv.Key, kv => kv.Value, StringComparer.OrdinalIgnoreCase));
Expand Down Expand Up @@ -2647,6 +2821,8 @@ public sealed record AgentControlDto(

public sealed record RetryWorkItemRequest(string? From);

public sealed record DelegateWorkItemRequest(string? Note = null);

public sealed record ResumeWorkItemRequest(string? From = null, string? Reason = null);

public sealed record PatchWorkItemRequest(
Expand Down Expand Up @@ -2794,6 +2970,10 @@ public sealed record WorkItemDto(
int DelegationAttempts = 0,
bool DelegationRequested = false,
string? DelegationReason = null,
string? DelegationNote = null,
bool AutoDelegationEscalated = false,
bool DelegationFailed = false,
int TerminalFailureCount = 0,
[property: JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingNull)]
IReadOnlyDictionary<string, string>? Knobs = null);

Expand Down
10 changes: 10 additions & 0 deletions src/CodeyBox.Core/CodeyBoxMeters.cs
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,16 @@ public static class CodeyBoxMeters
public static readonly Counter<long> Dispatches =
PipelineMeter.CreateCounter<long>("codeybox.dispatch.count", unit: "{dispatch}");

/// <summary>
/// One increment per delegation turn authorized (operator command or
/// automatic escalation). Tag: <c>trigger</c> (<c>operator</c> |
/// <c>audit-max-iterations</c> | <c>repeated-terminal-failure</c>). Lets
/// dashboards read a rise in delegations as a worsening convergence
/// problem rather than as the system working.
/// </summary>
public static readonly Counter<long> DelegationCounts =
PipelineMeter.CreateCounter<long>("codeybox.delegation.triggers", unit: "{delegation}");

/// <summary>
/// One increment per agent invocation attempt (work / rework / audit / merge /
/// upstream). Tags: <c>agent.kind</c>, <c>model</c>, <c>agent_class</c>,
Expand Down
Loading
Loading