th-d1754a: a panicking Go tool fails its call, not the process - #192
Open
brentrager wants to merge 1 commit into
Open
brentrager wants to merge 1 commit into
brentrager wants to merge 1 commit into
Conversation
Go turns an unrecovered panic into a process-wide crash, and go/core had no recover() outside tests. A buggy host tool (nil-map write, index out of range) therefore dropped every live turn on the pod instead of failing its own call. The server-side recovery from #504 cannot reach it: recover() only catches panics on the goroutine that deferred it, and under ParallelToolCalls each dispatch runs on its own goroutine. safeExecute wraps tool.Execute in the frame that calls it, so the panic becomes an ordinary IsError result the model can react to — matching TypeScript, Python and .NET, which already catch around execute(). The RunStream goroutine gets a backstop recover so a panic elsewhere in the turn (hook, checkpoint store, streaming client) surfaces through the documented Stream error contract rather than crashing, which is what Rust gets from its JoinHandle. Each test crashes the whole `go test` binary with the guards removed — two of them without even printing a FAIL line, which is the defect exactly.
🦋 Changeset detectedLatest commit: 8a54c37 The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Go escalates an unrecovered panic to a process-wide crash, and
go/corehad norecover()anywhere outside tests (grep -rn 'recover()' go/ | grep -v _testreturned nothing). So a buggy host tool — a nil-map write, an index out of range — did not fail its tool call, it took down the pod and every other live turn on it.The server-side recovery added in SmooAI/smooth-operator#504 cannot help, for two reasons:
core.SmoothAgent.RunStream(go/core/agent.go:739) spawns the whole turn on its own goroutine.ParallelToolCalls, each dispatch runs on its own goroutine (agent.go:663forRun,agent.go:898forRunStream) — so even the non-streamingRunpath was exposed.recover()only catches panics on the goroutine that deferred it, so no caller-side guard reaches either.Fix
Two guards, both in
go/core/agent.go:safeExecutewrapstool.Executein the frame that calls it, turning a panic into an ordinaryIsErrortool result the model can react to. The stack goes to the process log, never into model-visible content. This is the root-cause placement: one guard coversRun,RunStream, serial and parallel dispatch. It also matches TypeScript (typescript/core/src/agent.ts:1023-1029), Python (agent.py:891-894) and .NET (SmoothAgent.cs:623-634), whosecatcharoundexecute()already converted a throwing tool into an error result.RunStreamgoroutine now recovers as a backstop, so a panic elsewhere in the turn (a hook, a checkpoint store, the streaming client) is reported through the documentedStreamerror contract — channel closed without aStreamDone, reason inErr()— instead of crashing. That is what Rust already gets for free by confining a turn to itsJoinHandle. It is registered last so it runs first, ahead of the existingcancelandclose(events)defers from th-e9cc8b: Go engine reports truncated streams as finished turns #190:Err()is set before a draining consumer sees the channel close.All five engines checked
JoinHandleconfines a panicking turn (agent.rs:626)catchattool.executeexcept Exceptionattool.executecatch (Exception)atfunction.InvokeAsyncAlso checked for the fire-and-forget variant the pearl warned about: no
_ = Task./_ = *Async(indotnet/core/src, no floating promises intypescript/core/src/agent.ts.Proof the tests fail without the fix
Both guards reverted in place on this branch's base, then each test run alone:
TestPanickingToolFailsOnlyItsCall--- FAIL+panic: assignment to entry in nil map [recovered, repanicked]TestPanickingToolInParallelDispatchpanic: assignment to entry in nil map— binary dies, no FAIL lineTestPanickingToolDuringRunStream--- FAIL+panic: assignment to entry in nil mapTestRunStreamRecoversPanicOutsideToolExecutepanic: post-call hook exploded— binary dies, no FAIL lineThe two that kill the whole
go testbinary without printing aFAILline at all are the defect: a panic escaping onto a goroutine nobody owns. Guards restored,gofmt,go build ./...,go vet ./...andgo test ./...all green.Pearl: th-d1754a