feat(eval): interactive TUI wizard for batch-evaluation evaluate

[jariy17]

What

Adds an interactive TUI wizard for agentcore eval batch-evaluation evaluate, so a batch evaluation can be started without hand-assembling flags. Built on the shared wizard shell (src/components/wizard/).

Demo

Recorded end-to-end against a real account via the TUI harness: the final submit returns a job id and the get command to poll it.

Flow — 5 steps (agent-only)

1. name — validated against BatchEvaluationNameSchema (letters/digits/underscores, start with a letter, max 48) live as you type
2. source — target agent (harness or runtime ID)
3. lookback — days; converted to the source time window (now − N days → now)
4. evaluators — multi-select, populated live from listEvaluators, requires ≥1
5. review — summary → submit → job id + get command to poll

Endpoint, session IDs, ground truth, online-eval config, and the raw DataSourceConfig stay flag-only escape hatches to keep the happy path short.

Notes

• The shell ships no multi-select, so EvaluatorMultiSelect is a compound field (own useInput), following the ModelField pattern from project create. Cursor is clamped on read so it never depends on a stale list length, and the list is windowed so a long roster doesn't push the footer off-screen.
• Name validation reuses a new BatchEvaluationNameSchema in src/projectSchemas/, matching how the sibling eval resources (evaluator, online-eval-config) define their name schemas — surfaces the service pattern error at input rather than after submit.
• Evaluators load first page only (100) — pagination is a follow-up.
• Retargeted the CliOnlyScreen help-group test to simulate, since evaluate now renders a screen rather than command-line-only help.

Test

• bun test — full suite green
• Recorded submit → PENDING against a real runtime via the TUI harness

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)

[agentcore-devx-automation]

AgentCore Harness Review

Verdict: Changes requested

Nice, focused wizard. One real gap:

Missing tests for the new wizard

src/handlers/eval/batch-evaluation/evaluate/screen.tsx is a brand-new interactive wizard for a WRITE operation, and it introduces a bespoke EvaluatorMultiSelect compound field with non-trivial custom input handling (windowed list, cursor clamping, space-to-toggle, "select at least one" gating, escape→back). It ships with zero tests.

The convention elsewhere in this repo is to cover screens with a *.screen.test.tsx — e.g. src/handlers/eval/batch-evaluation/batch-evaluation.screen.test.tsx, src/handlers/eval/recommendation/recommendation.screen.test.tsx, and src/handlers/project/add/runtime/runtime.screen.test.tsx all use renderScreen(...) to drive the wizard and assert against lastFrame(). Please add coverage that exercises at least:

• the happy path (fills each step, verifies core.eval.startBatchEvaluation is called with the expected source/window/evaluatorIds, and shows the success next steps line);
• the evaluators step: empty-list message, "select at least one evaluator" gating on enter with no selections, space toggling on/off, and the cursor windowing when withId.length > WINDOW (this is the most bug-prone piece and currently has no safety net);
• the loading and error states of the listEvaluators query.

Everything else looks reasonable — the source mapping to SessionSourceValue matches what Core.startBatchEvaluation expects, the supportedTuiCommands("evaluate") addition preserves the CLI-only path for non-empty invocations, and the CliOnlyScreen help-groups test update to point at simulate is correct.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 52.07373% with 104 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.73%. Comparing base (e7a7aae) to head (e7e7dd8).

Files with missing lines	Patch %	Lines
...handlers/eval/batch-evaluation/evaluate/screen.tsx	48.51%	104 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           refactor    #2301      +/-   ##
============================================
- Coverage     96.98%   96.73%   -0.25%     
============================================
  Files           579      581       +2     
  Lines         39552    39768     +216     
============================================
+ Hits          38358    38470     +112     
- Misses         1194     1298     +104     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[agentcore-devx-automation]

Claude Security Review: no high-confidence findings. (run)