🤖 feat: Auto model routing experiment backed by TypeSafe Jev by ibetitsmike · Pull Request #4307 · coder/xum

ibetitsmike · 2026-09-20T14:36:32Z

Summary

Adds an experimental Auto choice to the composer's model picker. With Auto selected, the backend asks TypeSafe's Jev classifier how hard the prompt is, using difficulty tiers the user defines in Settings (easy / medium / hard / extreme by default), and runs the turn on the model mapped to the chosen tier. The composer's model stays the fallback whenever routing cannot happen, so nothing changes for users who never pick Auto.

Background

Switching models per message by hand is tedious: cheap fast models suit most prompts, expensive reasoning models only some. This experiment lets the user describe the tiers once, map each to a model and thinking level, and let a classifier pick per prompt. It is gated behind the auto-model-routing experiment and needs a TypeSafe API key.

Implementation

Auto is a flag, not a model string. SendMessageOptions.autoModelRouting: boolean rides alongside a concrete model, so every existing model validator (format checks, workspace AI settings extraction, gateway normalization) keeps seeing a real model. The literal "auto" never enters the model pipeline.
One resolution point. AgentSession.sendMessage resolves routing once before modelForStream, strips the flag before any snapshot, and skips classification for synthetic or agent-initiated turns, when the experiment is off, or when no tier has a model mapped. The chosen model passes the existing budgeted-goal pricing gate and the PDF attachment check; an unpriced or incompatible model falls back to the composer model. A tier may leave the model blank and only set a thinking level, which then applies to the composer model. Explicit per-send models (one-shot /model commands, Implement the plan) drop the flag so the classifier never overrides them.
Classifier client (src/node/services/autoModelRouter.ts): one Jev System One choice question whose options are the tier descriptions, an 8 s timeout, no retries, and every failure returns Err so the send falls back. Provider policy applies: under enforcement typesafe must be in provider_access or the classifier is skipped. The request carries the prompt (capped) plus up to three prior user prompts for context. The key lives in providers.jsonc under a typesafe entry (config, then apiKeyFile, then TYPESAFE_API_KEY / JEV_API_KEY env); typesafe is deliberately not a ProviderName, so it never appears in model lists, and it is a reserved custom provider id.
Config. Tiers live in config.json under autoModelRouting.tiers, mirroring modelFallbacks, since experiment flags are boolean-only. New oRPC routes: config.updateAutoModelRouting, config.getAutoModelRoutingClassifierStatus (returns only where the key came from), config.previewAutoModelRouting.
Per-turn record. An AutoModelRoutingRecord (routed / unmapped-tier / fallback, tier, confidence, probabilities, reason) is stored on the user message and flows through stream start/end into the assistant message, where AutoModelRoutingBadge renders it. A resume while Auto is still selected continues on the routed model and keeps the badge; a resume after leaving Auto uses the explicit model.
Renderer. ModelSelector gains a pinned, keyboard-reachable Auto row; the Auto choice persists per workspace and is copied from the creation composer to the new workspace; choosing a concrete model turns Auto off. A Settings sub-panel manages the key, the tier editor (inline validation, debounced text commits), and a "Test routing" preview.

Validation

Live classification against api.typesafe.ai with a real key (local UAT round 2, tested SHA 804751eb68): a rename prompt routed to Easy on Haiku 4.5, an architecture prompt to Extreme on Opus 5, each with confidence and per-tier probabilities in the badge tooltip and the Settings preview; a bogus key produced Classifier returned HTTP 401 and a fallback badge; stop then continue kept the routed model and record; two back-to-back Auto sends classified independently; a 32k-character prompt routed and completed; the creation composer carried Auto into the new workspace; no key material appeared in the page, localStorage, or logs. Verdict PASS. Later commits (polish ec6087b4f6, Codex fix rounds a0b41b28de and b51e13b7f6, main merge 7791a46625) are covered by unit tests and static checks only; the live UAT was not rerun on them.
Dogfood UAT round 1 ran remotely on dogfood.cdr.dev and found six UI/flow defects (fixed in the two fix(routing) commits). Coder Agents workspaces do not carry the Jev key, so the key-dependent round ran against a local make dev-server-sandbox instance with agent-browser.
Keyboard paths (ArrowUp to the Auto row, Enter to select) are covered by Storybook play tests because happy-dom does not deliver React onKeyDown; the equivalent bun tests were dropped rather than left vacuous.

Known limits and follow-ups

The Auto choice is stored like other composer state (usePersistedState, per workspace) and is browser-local: opening the same workspace from another browser or device starts with the concrete model. Making it follow the workspace means adding a field to the persisted WorkspaceAISettings, deferred to keep this PR's surface down.
On a routed turn the tier's "Inherit" thinking level goes through the existing per-model floor (getDefaultMinimumThinkingLevel), so a composer low shows as medium on models with an explicit thinking policy. Same rule as picking that model by hand.
A tier can name a model whose provider was later removed, disabled, or stripped of its key. That turn fails with the same actionable provider error an explicit send gets (formatSendMessageError points at Settings → Providers); it does not fall back to the composer model. Doing so needs a check-only path in ProviderModelFactory (or a session-side re-implementation of its provider checks), deferred as a follow-up rather than adding a fifth pre-stream gate here (Codex round 5, P2).

Risks

Send path. The only shared-path change is in AgentSession.sendMessage, and it is inert unless the experiment is on and the flag is set. With Auto on, the user message is appended after classification, so it appears ~2 s later than usual and up to 8 s on a classifier timeout. Every classifier failure falls back to the composer model and shows a fallback badge.
Privacy. With Auto on, the prompt text and up to three prior user prompts are sent to TypeSafe. The Settings panel states this next to the key field.
Compatibility. New optional config and message-metadata fields only; older builds ignore them.

Delivery record

Design: deep-codebase-exploration (5 lanes) established that Auto must be a flag beside a concrete model, that AgentSession.sendMessage is the single resolution point, and that the key belongs in providers.jsonc outside ProviderName.
Static review: read-only reviewer pass over 3e1241488b (2 P1, 2 P2, 9 P3). P1/P2 fixed in 7f865f194c; declined: clearing Auto before a cancellable model switch (there is no cancel path) and the aggregator writing undefined (matches adjacent metadata lines).
Remote UAT round 1: dogfood.cdr.dev chat 00d5e851-fd0d-438d-8e44-390e90234ada on 3e1241488b, BLOCKED for the classifier (no key in Coder Agents workspaces), 6 defects found (creation composer ignored Auto, resume lost the routed model/badge, tier edits committed only on blur, no validation feedback, raw error bodies surfaced, missing accessible names). Fixed in 7f865f194c and 804751eb68.
Local UAT round 2 on 804751eb68 with the real key: PASS, all six round 1 defects re-verified as fixed; three P3 notes adjudicated above (browser-local Auto, thinking floor, confidence 99% next to a probability rounded to 100% is the API's own numbers).
Polish: simplify + deslop pass, behavior-neutral (ec6087b4f6).
Codex review round 1 on ec6087b4f6: 6 findings. Fixed in a0b41b28de, each with a test that failed on the previous head: explicit per-send models bypassed by Auto (P1), thinking-only tiers skipped (P1), attachments not revalidated against the tier model (P2), TypeSafe key counted as a chat provider (P2), disclosure missing prior prompts (P2). Declined with evidence: atomic multi-draft unmount flush (unreachable, both fields commit on blur and only one can hold focus).
Codex review round 2 on a0b41b28de: 5 findings, all confirmed and fixed in b51e13b7f6 with tests that failed on the previous head: classifier ignored provider policy (P1), typesafe could collide with a custom provider id (P1, now reserved plus shape guards), thinking-only one-shot commands kept Auto (P2), on-send compaction dropped the routing record (P2), refusal fallback left the record on the refused model (P2). Evidence replies posted on all five threads and the threads resolved after the fix.
Merge of origin/main (7791a46625): one conflict in ProposePlanToolCall.tsx, where main consolidated both Implement sends into runPlanAction; the autoModelRouting: false opt-out moved into that single shared send. Typecheck, 2583 targeted tests, and make static-check green on the merged tree.
Codex review round 3 on 7791a46625, triggered automatically by the push ("New commits"), not requested: 3 findings. Fixed in 64c8167c98 with tests that were red on the previous head: typesafe rejected by PolicyProviderIdSchema so an enforced provider_access could never authorize the classifier (P1), and a policy-denied tier model failed the turn instead of falling back (P2). Declined with evidence: history loss when Stop cancels classification during an edit (P1) is unreachable, because the only cancelSignal producer is the bash-monitor wake, which routing skips.
Codex review round 4 on 64c8167c98, triggered by marking the PR ready at 20:25Z (code and security passes 8 and 9 against the six-pass cap): 4 P2 findings, all confirmed and fixed in acac7524ca with tests that were red on the previous head: a user-built compaction request (/compact plus follow-up, compact-and-retry) bypassed Auto for the follow-up (now classified once and stored on parsed.followUpContent), a resume naming the direct twin of a Coder-routed tier model kept the record (now modelSelectionEqualityKey), unbounded tier labels/descriptions (32/400-char caps in schema and UI), and an unbounded badge label (truncated, tooltip keeps the full label). Evidence replies posted on all four threads and the threads resolved.
Local UAT round 3 on 64c8167c98 with the real key (fresh sandbox root, plus policy-file runs): PASS on every headline flow and on the reachable Codex-round fixes, including the enforced-policy paths (classifier authorized when typesafe is listed, denied tier model falls back with a legible reason, policy without typesafe skips the classifier), thinking-only tiers, attachment revalidation, plan Implement on the concrete model, 375 px layout, zero console errors. One new P2: a text-only Auto send routed to a tier model whose provider rejects a PDF sent earlier in the conversation failed the turn and kept retrying. Fixed in 16c5679970: the tier-model attachment check now covers every user attachment still in the context window (test red on the previous head). Not reachable in that round: on-send compaction record carry-over (threshold is backend-owned), refusal fallback. Design note confirmed against the code: Implement-from-plan sends on the concrete model but does not clear the composer's Auto choice; only the send drops the flag. Evidence: /home/coder/.xum-uat-evidence/auto-model-routing/round-3/.
Local UAT round 3 delta on 16c5679970 with the real key (fresh sandbox root): PASS. Verified the history-attachment fix (PDF in history, Easy tier on xai:grok-4.6: classified, fell back to the composer model with a legible reason, no provider error, no retry loop), /compact plus follow-up under Auto (summary unrouted, follow-up routed once with badge and record), bare /compact unchanged, the 32/400 caps in the UI and on disk, badge truncation at 375 px with the full label in the tooltip, and easy/extreme regression sends; zero console errors at the end of the run. Skipped: Coder-route resume (no direct twin model configured in the sandbox; covered by the unit test). One P3 found: doubled period in the attachment fallback tooltip, fixed in 80684e010a (unit test red on the previous head). One-off React dev warning during creation-composer setup that did not reproduce in four controlled repeats, not attributed to routing code. Evidence: /home/coder/.xum-uat-evidence/auto-model-routing/round-3-delta/. The final head 80684e010a differs from the UAT-tested 16c5679970 only by that tooltip punctuation change.
Codex review round 5 on 80684e010a, triggered by the owner marking the PR ready at 08:59Z on 2026-09-21 after deciding to spend the passes (owner instruction: "work until it's done and then merge"): 5 findings. Two confirmed and fixed with tests that were red on 80684e010a: the classifier only checked isProviderAllowed("typesafe"), so an enforced policy listing typesafe with a model_access that excludes jev-latest still sent the prompt (P1, now isModelAllowed, one check instead of two), and min(1) on tier label/description accepted whitespace-only text from a hand-edited config (P2, now trimmed before the length checks). Declined with evidence replies: stale tier provider should fall back rather than fail (P2, scope growth, recorded above under follow-ups), an unawaited tier write racing Test routing (P2, requests are issued in order on one connection, mirrors useModelFallbacks, no reproduction), and Continue after a refusal fallback restarting on the refused tier model (P2, the product-wide resume contract: retrySendOptions is written at send time for every model selection). Threads resolved after the push.
Codex review round 6 on be00f62d69 (automatic "New commits" pair on the round 5 fix push): 2 P2 code findings plus 1 P1 security finding. The security finding was confirmed and fixed with a test that was red on be00f62d69: an enforced policy base_url for typesafe was ignored, so the classifier always posted the prompt and bearer credential to api.typesafe.ai (now resolved from getForcedBaseUrl("typesafe") when policy is enforced, shared by the Auto send and the Settings preview). The code findings were both confirmed and fixed with tests that were red on be00f62d69: an attachment-only send (empty text plus files, a valid send) still paid for a classifier call with an empty prompt (now falls back to the composer model with a legible reason before the round-trip), and a START_WORKSPACE_CREATION prefill carrying a concrete model left the project-scoped Auto flag on, so the creation send treated the explicit model as a mere fallback (the prefill now clears Auto, the same explicit-pick rule setWorkspaceModelWithOrigin applies). Threads resolved after the push.
Codex review round 7 on a16db9163a (automatic "New commits" pair on the round 6 fix push; security review completed with no new findings): 1 P2, declined with evidence: an optimistic experiment toggle racing an immediate Auto send is the pre-existing ExperimentsContext fire-and-forget pattern shared by every experiment gate, the override request precedes any later send on the same connection, and carrying the experiment value on the send would be a new experiments-plumbing contract outside this diff. Thread resolved.
Codex review round 8 on c476f0ff8c (automatic "New commits" pair on the round 7 fix push): both passes completed with zero findings. The board still listed the fixed security advisory without Codex's Resolved marker (the pass started 2 s before the thread was resolved), so the Codex Comments gate stayed red; a manual @codex security review (one pass, explicit "No security issues were found" comment) did not add the marker either.
Codex review round 9 on the same head c476f0ff8c (manual @codex review, requested only to obtain the Resolved marker): the security pass added the marker but raised a new Medium advisory, and the code pass returned 4 new findings on a head it had passed clean 35 minutes earlier. Confirmed and fixed with tests that were red on c476f0ff8c: TYPESAFE_API_KEY/JEV_API_KEY missing from providerSecretEnvVarNames so repo-automation-off subprocesses could inherit the classifier credential (P1), classifier context read through getLastMessages across the latest durable boundary so a /clear or compaction did not hide earlier prompts from Jev (P1, now getHistoryFromLatestBoundary), a persisted routing record with a non-string model throwing on Continue (P2, now read as absent), and unbounded classifier probabilities persisted per row (security Medium, now filtered to the configured tiers). Declined with evidence: byte-capping the classifier response body (no backend fetch bounds bodies, endpoint trusted by configuration) and per-tier thinking choices derived from the mapped model (documented per-model floor, shared selector behavior). Threads resolved before the push.
Codex review round 10 on 045a0754f9 (automatic "New commits" pair on the round 9 fix push; security pass added no advisory but also did not mark the persisted-probabilities advisory resolved despite its thread being resolved before the pass began): 2 P2 code findings, both confirmed and fixed with tests that were red on 045a0754f9: the routing-record lookup picked the last user row by role, so a completed subagent report card appended after an interrupted routed turn shadowed the record (now reuses findLastRetryUserMessage, the resume path's own predicate), and the tier-model attachment gate covered PDFs only, so an image in the send or the active context could be routed to a non-vision tier model (now falls back with a legible reason; the pre-existing send-time PDF check is unchanged). Threads resolved before the push.
Review budget (cap: six passes per PR, code and security counted separately): 26 of 6 consumed, namely the in-house reviewer pass, Codex code review x12, Codex security review x13. The round 5 pair, the automatic "New commits" pairs on the round 5, 6, 7, 9, and 10 fix pushes, the manual security-only pass, and the manual round 9 pair were authorized by the owner on 2026-09-21 ("work until it's done and then merge"). Convergence note: round 8 passed c476f0ff8c clean and round 9 found 5 issues on the identical head. Before that: 9 of 6, namely the in-house reviewer pass, Codex code review x4, Codex security review x4. Rounds 1 and 2 were requested; round 3 ran automatically on the merge push; round 4 ran because the PR was marked ready during the merge loop, which was a cap violation and has been corrected by a supervisor stop: no further Codex, security, or advisory review may be initiated for this PR. The PR was converted back to draft before pushing acac7524ca, 16c5679970, and 80684e010a so the ready-PR auto-review could not fire (verified after each push: no new board row, no reaction). That block was lifted by the owner on 2026-09-21 (marking the PR ready fired round 5; each fix push fired a "New commits" pair).

Generated with xum • Model: anthropic:claude-fable-5-1 • Thinking: xhigh • Cost: $110.64

Adds the auto-model-routing experiment: ordered difficulty tiers in config.json, an AutoModelRouter that asks Jev System One for a single tier choice, send-path resolution in AgentSession that swaps the tier's model in (falling back to the composer model on any classifier failure), provenance on the assistant message, a composer Auto entry, and a Settings sub-panel for the TypeSafe key, tier editor, and routing preview.

…ng stories Adds happy-dom tests for the ModelSelector Auto row and the auto-model-routing Settings panel, stories for the Experiments panel, the composer with Auto active, and assistant routing badges, and the story mock routes they need. Also clears the lint findings static-check raised on the first commit.

…mapped classification Review follow-ups on the auto-model-routing experiment: - The Auto row now lives at index -1 in the picker's arrow-key order, so ArrowUp from the first model highlights it and Enter selects it. Opening the picker while Auto is active starts on the Auto row. - A routed tier model passes the budgeted-goal pricing gate; an unpriced tier model falls back to the composer model instead of bypassing it. - No classifier call when no tier has a model mapped (nothing could change). - Explicit user/agent model picks (including accepted plans) turn Auto off centrally in setWorkspaceModelWithOrigin; sync-driven defaults keep it. - Failed tier writes surface inline instead of silently reverting; upstream error bodies stay in the debug log rather than persisted metadata. - Tier-count constants move to src/constants; shared percent formatter; story uses updatePersistedState and mirrors its pinned phone viewport.

… text edits Remote UAT round 1 findings on the auto-model-routing experiment: - Storage-based send options (creation send, resume, retry) now carry the Auto flag under the same experiment gate as the composer hook, and workspace creation copies the project-scoped Auto choice into the new workspace, so a first message sent with Auto is classified and the new composer keeps Auto. - The user row persists the routing record; resumeStream re-attaches it and, while Auto is still selected, continues on the routed model and thinking level instead of the composer fallback. Leaving Auto before resuming keeps the explicit model and drops the badge. - Tier label and description edits commit after a pause, on blur, or on Enter instead of only on blur; empty values and duplicate labels show an inline message and are never written. The per-tier model picker has an accessible name.

chatgpt-codex-connector · 2026-09-20T14:36:45Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-21T11:48:08.564144Z	`3673cd2`	New commits
🔒 Security Review	✅ Completed 2026-09-21T11:44:13.355534Z	`3673cd2`	New commits

Security findings

Advisory findings (2)

🟡 Honor the policy-forced TypeSafe base URL · High · Resolved
🟡 Bound classifier responses before buffering them · Medium

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

ibetitsmike · 2026-09-20T14:37:29Z

@codex review

chatgpt-codex-connector · 2026-09-20T14:44:12Z

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: ec6087b4f6

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ec6087b4f6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…iers Codex review round 1 on PR #4307: - explicit per-send models (one-shot /model commands, Implement the plan) drop the Auto flag so the classifier cannot override them - a routed tier model is re-checked against PDF attachments and falls back when it cannot accept them - a tier that keeps the composer model but sets a thinking level is now a real routing target instead of being skipped - the typesafe classifier key no longer counts as a configured chat provider - the Settings disclosure names the prior prompts sent with the request

ibetitsmike · 2026-09-20T15:16:24Z

@codex review

chatgpt-codex-connector · 2026-09-20T15:22:44Z

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: a0b41b28de

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a0b41b28de

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

… keep records through fallbacks Codex review round 2 on PR #4307: - the classifier honors provider policy (typesafe must be allowed under enforcement) - "typesafe" is a reserved custom provider id; a legacy custom provider under that id is never used as the classifier key and still counts as a chat provider - any one-shot /model or thinking override disables Auto for that turn - on-send compaction follow-ups carry the routing record so the redispatched turn keeps its badge and routed-model resume path - a refusal fallback swap updates the record's model to the model that answered

Resolve ProposePlanToolCall.tsx: main consolidated both Implement sends into runPlanAction, so the autoModelRouting: false opt-out moves into that single shared send.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7791a46625

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…y-denied tier models Reserving the typesafe custom-provider id (round 2) made PolicyProviderIdSchema reject it, so an enforced provider_access could never authorize the classifier. Admit the classifier id in the policy schema while keeping it reserved for custom-provider creation. A tier saved before a policy refresh could name a model the stream refuses with policy_denied; resolveAutoModelRouting now runs the same policy check and returns the fallback record instead.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 64c8167c98

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…tier text Codex round 4 on 64c8167, four P2 findings: - A user-built compaction request (/compact plus a follow-up, or compact-and-retry after a context overflow) deleted the Auto flag and never classified the follow-up, so the user's real prompt redispatched on the composer model without a badge. The follow-up is now classified once at request time and the routed model, thinking level and record are stored on parsed.followUpContent, which the dispatch path already reads. - A resume that names the direct twin of a Coder-routed tier model kept the Auto record because normalizeToCanonical collapses coder: routes. applyAutoRoutedResume now compares with modelSelectionEqualityKey. - Tier labels and descriptions were unbounded although descriptions are copied verbatim into the classifier criteria. Both are capped in the shared schema (32 / 400 chars) and as maxLength on the Settings inputs. - The badge label is bounded and truncated; the tooltip keeps the full label. The story uses a label at the cap so the snapshot covers it. Each backend fix has a test that failed on the previous head.

Local UAT round 3 on 64c8167 (P2): a text-only Auto send routed to a tier model whose provider rejects a PDF sent earlier in the conversation failed the turn with an opaque provider error and kept retrying. The tier-model attachment check now covers every user attachment still in the context window, not only this turn's file parts, so such a turn falls back to the composer model with the usual reason. Test red on the previous head.

UAT round 3 delta (P3): attachment fallback reasons are full sentences, so the tooltip read "does not support PDF input.. Used ...".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 80684e010a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…r text Codex review round 5 on 80684e0: - An enforced policy that lists typesafe but whose model_access excludes jev-latest still let the classifier send the prompt; the router now asks isModelAllowed(typesafe, jev-latest), which also covers the provider-not-listed case, replacing the provider-only check. - Tier label/description accepted whitespace-only text through min(1); the schema now trims before the length checks, so hand-edited configs are normalized and the oRPC boundary rejects blank tiers. Both tests were red on the previous head.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: be00f62d69

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

🛡️ Codex Security Review · Automatically triggered

Here are some automated security review suggestions for this pull request.

Reviewed commit: be00f62d69

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

…ave Auto Codex review round 6 on be00f62: - An attachment-only send (empty text plus files) still paid for a classifier call with an empty prompt; routing now falls back to the composer model before the round-trip and records why. - A START_WORKSPACE_CREATION prefill with a concrete model left the project-scoped Auto flag on, so the creation send treated the explicit model as the routing fallback; the prefill now clears Auto, matching setWorkspaceModelWithOrigin's explicit-pick rule. Both tests were red on the previous head.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a16db9163a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Codex security review on be00f62 (P1): an enforced policy that lists typesafe with a base_url still had the classifier post the prompt and bearer credential to api.typesafe.ai. The router now resolves the System One endpoint from getForcedBaseUrl("typesafe") when policy is enforced, composing the forced base with the endpoint path the same way the model factory applies forced baseURLs. Test was red on the previous head.

ibetitsmike · 2026-09-21T10:39:26Z

@codex security review

chatgpt-codex-connector · 2026-09-21T10:45:00Z

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: c476f0ff8c

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

ibetitsmike · 2026-09-21T10:47:30Z

@codex review

chatgpt-codex-connector

🛡️ Codex Security Review · Automatically triggered

Here are some automated security review suggestions for this pull request.

Reviewed commit: c476f0ff8c

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c476f0ff8c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…text to the boundary, harden the resume record Codex review round 9 on c476f0f: - TYPESAFE_API_KEY / JEV_API_KEY were missing from providerSecretEnvVarNames, so repo-automation-off subprocesses could inherit the classifier credential. - Classifier context came from getLastMessages, which continues into the sealed archive; it now reads from the latest durable boundary like every provider request, so /clear and compaction hide earlier prompts from Jev. - A persisted routing record whose model is not a string now reads as absent on resume instead of throwing in modelSelectionEqualityKey. - Only the configured tiers' probabilities are kept from the classifier response before the record is persisted. Every test was red on the previous head.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 045a0754f9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…ainst the tier model Codex review round 10 on 045a075: - The routing-record lookup selected the last user row by role, so a completed subagent report card appended after an interrupted routed turn shadowed the record; it now reuses findLastRetryUserMessage, the predicate the resume path already uses to pick the request. - The tier-model attachment gate covered PDFs only; images in the send or the active context now fall back to the composer model when the tier model has no vision support, with the same legible reason shape. Both tests were red on the previous head.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3673cd234d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-09-21T11:48:04Z

+    // History is untyped on disk; a hand-edited or damaged record must not brick Continue.
+    if (record == null || typeof record.model !== "string") return resumeOptions;


Validate the complete routing record before resuming

Fresh evidence after the corrupted-record fix: this guard checks only model, although AutoModelRoutingRecordSchema also requires requestedFallbackModel and a valid status. A damaged row such as { model: "openai:gpt", status: "fallback" } passes here and is attached to stream lifecycle events, which then fail the oRPC schema—or crash fallback badge formatting when the missing fallback model is used—so Continue remains broken until the history is repaired manually. Parse the complete record and treat any schema failure as absent.

AGENTS.md reference: AGENTS.md:L125-L129

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-09-21T11:48:04Z

+    const followUp = metadata.parsed.followUpContent;
+    const prompt = followUp?.text?.trim();
+    if (followUp == null || !prompt || followUp.autoModelRouting != null) return metadata;


Preserve Auto provenance for attachment-only follow-ups

Fresh evidence after the earlier compaction and attachment-only fixes: when /compact carries only files or reviews, followUp exists but its trimmed text is empty, so this return bypasses resolveAutoModelRouting() and its attachment-only fallback record. The later synthetic dispatch cannot classify, causing the response to lose its Auto badge and persisted resume provenance even though an equivalent direct attachment-only Auto send records the fallback; pass the empty prompt through the resolver instead.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-09-21T11:48:04Z

  },
 };
+
+export const AutoModelRoutingEnabled: Story = {


Pin the routing-settings story to a phone viewport

The new settings panel contains multiple sm: layout transitions, but this story specifies neither a Pixel viewport matrix nor matching viewport globals, so both snapshots and its play function exercise only desktop width and cannot catch narrow-layout overflow or stretching. Add an explicit phone variant and a narrow-width play/static contract as required for breakpoint-dependent UI.

AGENTS.md reference: AGENTS.md:L243-L243

Useful? React with 👍 / 👎.

ibetitsmike · 2026-09-21T11:49:54Z

Pausing the automated Codex loop on this PR for owner direction (repo rule: a review loop that does not converge stops and reports instead of adding mechanism).

Why: since the owner re-authorized reviews this morning, rounds 5 through 11 produced 5, 3, 1, 0, 5, 2, 3 findings respectively (26 review passes on the PR in total). Every confirmed finding was fixed with a test that was red on the previous head (9 fixes in rounds 5 to 10); 6 were declined with cited evidence. Round 8 passed c476f0ff8c with zero findings and round 9 then found 5 on the identical head, so the finding rate is not trending to zero.

Merge gate: Codex Comments (and therefore Required) also fails while the review board lists the advisory "Bound classifier responses before buffering them" without Codex's Resolved marker. Its thread is resolved (the persisted-probabilities half was fixed, the response byte cap was declined as hardening no other backend fetch has), but the security passes in rounds 10 and 11, both started after that resolution, did not mark it resolved. The marker appeared for the earlier advisory only after its fix landed, so this gate most likely stays red until the byte cap is implemented or the check is bypassed.

Open now (round 11, untriaged, left unresolved on purpose): full-record schema validation on resume (P2), Auto provenance for attachment-only /compact follow-ups (P2), phone-viewport pin for the routing settings story (P1, repo Storybook rule). All CI jobs other than the Codex gate are green on 3673cd234d.

Awaiting the owner's decision on how to proceed; no further pushes, review requests, or thread resolutions until then.

ThomasK33

Please don't hardcode a dependency on TypeSafe AI's jev.
Instead follow the docs here:

A user shall retain the choice of what evaluationModel they get to use.

ibetitsmike added 5 commits September 20, 2026 12:31

chore(routing): polish auto model routing (simplify + deslop)

ec6087b

chatgpt-codex-connector Bot reviewed Sep 20, 2026

View reviewed changes

Comment thread src/node/services/workspaceService.ts Outdated

Comment thread src/browser/features/ChatInput/prepareMessagePayload.ts Outdated

Comment thread src/node/services/agentSession.ts

Comment thread src/constants/autoModelRouting.ts

Comment thread src/node/services/agentSession.ts

ibetitsmike added 2 commits September 20, 2026 15:44

Merge origin/main into mike/auto-model-routing

7791a46

Resolve ProposePlanToolCall.tsx: main consolidated both Implement sends into runPlanAction, so the autoModelRouting: false opt-out moves into that single shared send.

chatgpt-codex-connector Bot reviewed Sep 20, 2026

View reviewed changes

Comment thread src/common/utils/providers/customProviders.ts

Comment thread src/node/services/agentSession.ts

Comment thread src/node/services/agentSession.ts

ibetitsmike marked this pull request as draft September 20, 2026 16:39

ibetitsmike marked this pull request as ready for review September 20, 2026 20:25

chatgpt-codex-connector Bot reviewed Sep 20, 2026

View reviewed changes

Comment thread src/node/services/agentSession.ts

Comment thread src/node/services/agentSession.ts Outdated

Comment thread src/common/types/autoModelRouting.ts Outdated

Comment thread src/browser/features/Messages/AutoModelRoutingBadge.tsx Outdated

ibetitsmike marked this pull request as draft September 20, 2026 20:50

ibetitsmike added 3 commits September 20, 2026 20:52

fix(routing): avoid a doubled period in the fallback tooltip

80684e0

UAT round 3 delta (P3): attachment fallback reasons are full sentences, so the tooltip read "does not support PDF input.. Used ...".

ibetitsmike marked this pull request as ready for review September 21, 2026 08:59

chatgpt-codex-connector Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread src/node/services/autoModelRouter.ts

Comment thread src/node/services/agentSession.ts

Comment thread src/common/types/autoModelRouting.ts Outdated

Comment thread src/browser/hooks/useAutoModelRouting.ts

Comment thread src/node/services/agentSession.ts

chatgpt-codex-connector Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread src/node/services/agentSession.ts

Comment thread src/browser/features/ChatInput/useCreationWorkspace.ts

chatgpt-codex-connector Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread src/node/services/autoModelRouter.ts Outdated

chatgpt-codex-connector Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread src/node/services/agentSession.ts

chatgpt-codex-connector Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread src/node/services/autoModelRouter.ts

chatgpt-codex-connector Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread src/constants/autoModelRouting.ts

Comment thread src/node/services/agentSession.ts Outdated

Comment thread src/node/services/agentSession.ts Outdated

Comment thread src/browser/features/Settings/Sections/AutoModelRoutingExperimentConfig.tsx

chatgpt-codex-connector Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread src/node/services/agentSession.ts Outdated

Comment thread src/node/services/agentSession.ts

chatgpt-codex-connector Bot reviewed Sep 21, 2026

View reviewed changes

ThomasK33 reviewed Sep 21, 2026

View reviewed changes

		// History is untyped on disk; a hand-edited or damaged record must not brick Continue.
		if (record == null \|\| typeof record.model !== "string") return resumeOptions;

Conversation

ibetitsmike commented Sep 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Background

Implementation

Validation

Known limits and follow-ups

Risks

Uh oh!

chatgpt-codex-connector Bot commented Sep 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Security findings

Advisory findings (2)

Uh oh!

ibetitsmike commented Sep 20, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 20, 2026

🛡️ Codex Security Review · Automatically triggered

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ibetitsmike commented Sep 20, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 20, 2026

🛡️ Codex Security Review · Automatically triggered

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

🛡️ Codex Security Review · Automatically triggered

Uh oh!

Uh oh!

ibetitsmike commented Sep 20, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 20, 2026 •

edited

Loading