Skip to content

th-d1754a: a panicking Go tool fails its call, not the process - #192

Open
brentrager wants to merge 1 commit into
mainfrom
th-d1754a-go-tool-panic
Open

brentrager wants to merge 1 commit into
mainfrom
th-d1754a-go-tool-panic

Conversation

@brentrager

Copy link
Copy Markdown
Contributor

Problem

Go escalates an unrecovered panic to a process-wide crash, and go/core had no recover() anywhere outside tests (grep -rn 'recover()' go/ | grep -v _test returned nothing). So a buggy host tool — a nil-map write, an index out of range — did not fail its tool call, it took down the pod and every other live turn on it.

The server-side recovery added in SmooAI/smooth-operator#504 cannot help, for two reasons:

  • core.SmoothAgent.RunStream (go/core/agent.go:739) spawns the whole turn on its own goroutine.
  • Under ParallelToolCalls, each dispatch runs on its own goroutine (agent.go:663 for Run, agent.go:898 for RunStream) — so even the non-streaming Run path was exposed.

recover() only catches panics on the goroutine that deferred it, so no caller-side guard reaches either.

Fix

Two guards, both in go/core/agent.go:

  • safeExecute wraps tool.Execute in the frame that calls it, turning a panic into an ordinary IsError tool result the model can react to. The stack goes to the process log, never into model-visible content. This is the root-cause placement: one guard covers Run, RunStream, serial and parallel dispatch. It also matches TypeScript (typescript/core/src/agent.ts:1023-1029), Python (agent.py:891-894) and .NET (SmoothAgent.cs:623-634), whose catch around execute() already converted a throwing tool into an error result.
  • The RunStream goroutine now recovers as a backstop, so a panic elsewhere in the turn (a hook, a checkpoint store, the streaming client) is reported through the documented Stream error contract — channel closed without a StreamDone, reason in Err() — instead of crashing. That is what Rust already gets for free by confining a turn to its JoinHandle. It is registered last so it runs first, ahead of the existing cancel and close(events) defers from th-e9cc8b: Go engine reports truncated streams as finished turns #190: Err() is set before a draining consumer sees the channel close.

All five engines checked

Engine Status
Rust Already correct — JoinHandle confines a panicking turn (agent.rs:626)
TypeScript Already correct — catch at tool.execute
Python Already correct — except Exception at tool.execute
.NET Already correct — catch (Exception) at function.InvokeAsync
Go The gap. Fixed here.

Also checked for the fire-and-forget variant the pearl warned about: no _ = Task./_ = *Async( in dotnet/core/src, no floating promises in typescript/core/src/agent.ts.

Proof the tests fail without the fix

Both guards reverted in place on this branch's base, then each test run alone:

Test Without the fix
TestPanickingToolFailsOnlyItsCall --- FAIL + panic: assignment to entry in nil map [recovered, repanicked]
TestPanickingToolInParallelDispatch panic: assignment to entry in nil mapbinary dies, no FAIL line
TestPanickingToolDuringRunStream --- FAIL + panic: assignment to entry in nil map
TestRunStreamRecoversPanicOutsideToolExecute panic: post-call hook explodedbinary dies, no FAIL line

The two that kill the whole go test binary without printing a FAIL line at all are the defect: a panic escaping onto a goroutine nobody owns. Guards restored, gofmt, go build ./..., go vet ./... and go test ./... all green.

Pearl: th-d1754a

Go turns an unrecovered panic into a process-wide crash, and go/core had no
recover() outside tests. A buggy host tool (nil-map write, index out of range)
therefore dropped every live turn on the pod instead of failing its own call.
The server-side recovery from #504 cannot reach it: recover() only catches
panics on the goroutine that deferred it, and under ParallelToolCalls each
dispatch runs on its own goroutine.

safeExecute wraps tool.Execute in the frame that calls it, so the panic becomes
an ordinary IsError result the model can react to — matching TypeScript, Python
and .NET, which already catch around execute(). The RunStream goroutine gets a
backstop recover so a panic elsewhere in the turn (hook, checkpoint store,
streaming client) surfaces through the documented Stream error contract rather
than crashing, which is what Rust gets from its JoinHandle.

Each test crashes the whole `go test` binary with the guards removed — two of
them without even printing a FAIL line, which is the defect exactly.
@changeset-bot

changeset-bot Bot commented Aug 22, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 8a54c37

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages
Name Type
@smooai/smooth-operator-core Patch
@smooai/smooth-operator-temporal Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant