Skip to content

Echo pure-Starlark runtime errors: index, key, and attribute failures return a bare invalid program #62

Description

@jmgilman

Problem

Issue #24 gave parse and resolve failures a model-derived suffix. Runtime failures raised by the Starlark interpreter itself — not by a handler, resolver, or policy — still return the bare sentinel, so the model gets no position and no message for a mistake entirely contained in its own program.

Measured against github.com/meigma/codemode v0.2.1 through the MCP execute tool of a downstream server:

Submitted source Tool error text
def main():\n return [1][5] invalid program
def main():\n d = {"a": 1}\n return d["b"] invalid program
def main():\n return undefined_name invalid program: <codemode>:2:9: undefined: undefined_name
def main()\n return 1 invalid program: <codemode>:2:1: got newline, want ':'

The first two rows are the gap. The underlying *starlark.EvalError already carries index 5 out of range / key "b" not in dict plus a CallStack whose frames point into the submitted source. Both the message and the position are model-derived: no host text is involved, because no host code ran.

Why it costs turns

This is not hypothetical. Building a program against a capability whose output shape I had misremembered, I read a field the capability does not return. The response was invalid program with no field name and no line, indistinguishable from a syntax error I could not find. It took two extra execute round trips plus a describe_api re-read to localize a one-token mistake, and the eventual fix was obvious once the field name was visible. A model has strictly less context than I did and will bisect by resubmission — exactly the blind-retry loop #24 set out to end.

Attribute access on a capability result is the common shape:

def main():
    guest = instance.get(sandbox="demo", name="gui")
    return guest.addresses     # capability returns `nics`, not `addresses`

Result: invalid program. Desired: the attribute name and the line.

Proposal

Extend the #24 selectivity table with one row, keeping its principle — echo only what the model already possesses:

Error class Echo to the model
Runtime error raised by the interpreter with no handler, resolver, or policy in the cause chain innermost CallStack position in the submitted source plus the interpreter message, e.g. invalid program: <codemode>:2:12: struct has no .addresses attribute

classifyRuntimeError already unwraps with errors.Is against each sentinel class, so the branch is the existing one: when a *starlark.EvalError unwraps to no host-authored cause, the error was produced by the interpreter and its Msg contains no host text. When it does unwrap to a handler, resolver, policy, or internal failure, the current coarse sentinel must stay — the danger note in #24 still applies unchanged.

Worth guarding explicitly in tests, since it is the one way this could leak: EvalError.Msg from a failed native call embeds the wrapped cause, so the discriminator must be the cause chain, not the error type.

Acceptance sketch

  • return [1][5], return {"a": 1}["b"], and a missing attribute on a capability result each return invalid program with a <codemode>:line:col: position and the interpreter's own message.
  • A handler returning errors.New("db password rejected") still yields exactly capability failed, with no host text in any content item.
  • Policy diagnostics, panic values, and Go stack detail remain absent.
  • The Errors table and the "Authoring and recovery" list in docs/docs/reference/mcp-tools.md describe runtime-error detail, since they are the model-facing contract.

Environment

github.com/meigma/codemode v0.2.1, stdio MCP adapter, Go 1.26.6. All four rows above are reproducible with any registered capability set; the first two need none.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions