From 18368cad9adfb0b65e092f275955fffdea88d966 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 14 Aug 2026 22:45:28 +0000 Subject: [PATCH 1/4] Document SolidAgent on docs.activeagents.ai MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit solid_agent's gemspec has pointed its homepage and documentation_uri at docs.activeagents.ai/solid_agent since it was published, and that page did not exist — the docs mentioned the gem only in passing, as something the dashboard happens to depend on. Adds a Persistence section: an overview at /solid_agent plus a page per concern (context, memory, tools/streaming/caching, reasoning, runs and cost, manifests) and a worked-examples page mirroring examples/ in the solid_agent repo. Linked from the sidebar, the home page features, the framework overview, getting started, the agents and tools pages, and the dashboard page that already depended on the gem. These pages carry their code inline rather than importing it, because the code they document lives in another repository; contributing/documentation now says so, and points at the tested examples they mirror. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01NGn2NzpzFZT4JGKBdFMpxh --- docs/.vitepress/config.mts | 14 ++ docs/actions/tools.md | 1 + docs/agents.md | 1 + docs/contributing/documentation.md | 16 ++ docs/framework.md | 2 + docs/framework/dashboard.md | 8 +- docs/getting_started.md | 1 + docs/index.md | 4 + docs/solid_agent.md | 121 ++++++++++++ docs/solid_agent/context.md | 228 ++++++++++++++++++++++ docs/solid_agent/examples.md | 296 +++++++++++++++++++++++++++++ docs/solid_agent/manifests.md | 186 ++++++++++++++++++ docs/solid_agent/memory.md | 149 +++++++++++++++ docs/solid_agent/reasoning.md | 152 +++++++++++++++ docs/solid_agent/runs.md | 192 +++++++++++++++++++ docs/solid_agent/tools.md | 202 ++++++++++++++++++++ 16 files changed, 1570 insertions(+), 3 deletions(-) create mode 100644 docs/solid_agent.md create mode 100644 docs/solid_agent/context.md create mode 100644 docs/solid_agent/examples.md create mode 100644 docs/solid_agent/manifests.md create mode 100644 docs/solid_agent/memory.md create mode 100644 docs/solid_agent/reasoning.md create mode 100644 docs/solid_agent/runs.md create mode 100644 docs/solid_agent/tools.md diff --git a/docs/.vitepress/config.mts b/docs/.vitepress/config.mts index 4dacd043..53001ff3 100644 --- a/docs/.vitepress/config.mts +++ b/docs/.vitepress/config.mts @@ -139,11 +139,25 @@ export default defineConfig({ { text: 'Mock', link: '/providers/mock' }, ] }, + { + text: 'Persistence (SolidAgent)', + items: [ + { text: 'Overview', link: '/solid_agent' }, + { text: 'Conversation Context', link: '/solid_agent/context' }, + { text: 'Long-Term Memory', link: '/solid_agent/memory' }, + { text: 'Tools, Streaming & Caching', link: '/solid_agent/tools' }, + { text: 'Reasoning', link: '/solid_agent/reasoning' }, + { text: 'Runs, Cohorts & Cost', link: '/solid_agent/runs' }, + { text: 'Agent Manifests', link: '/solid_agent/manifests' }, + { text: 'Examples', link: '/solid_agent/examples' }, + ] + }, { text: 'Examples', items: [ // { text: 'Browser Use', link: '/examples/browser-use-agent' }, { text: 'Data Extraction', link: '/examples/data_extraction_agent' }, // { text: 'Translation', link: '/examples/translation-agent' }, + { text: 'SolidAgent Examples', link: '/solid_agent/examples' }, ] }, { text: 'Contributing', diff --git a/docs/actions/tools.md b/docs/actions/tools.md index a01995d6..109f61f4 100644 --- a/docs/actions/tools.md +++ b/docs/actions/tools.md @@ -160,6 +160,7 @@ end - [Delegation](/actions/delegation) - Expose another agent to your model as a tool - [MCP (Model Context Protocol)](/actions/mcps) - Connect to external services via MCP +- [Tools, Streaming & Caching (SolidAgent)](/solid_agent/tools) - Declarative tool schemas, live tool status, cached results - [Agents](/agents) - Understand the agent lifecycle and callbacks - [Generation](/agents/generation) - Execute tool-enabled generations - [Messages](/actions/messages) - Learn about conversation structure diff --git a/docs/agents.md b/docs/agents.md index 1220d559..44f8f59b 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -96,6 +96,7 @@ See [Streaming](/agents/streaming) for complete documentation. **Related Topics:** - [Tools](/actions/tools) - Use agent actions as AI-callable tools +- [Persistence (SolidAgent)](/solid_agent) - Persist conversations, memory and runs to your database - [Structured Output](/actions/structured_output) - Extract typed data with schemas - [Embeddings](/actions/embeddings) - Vector generation for semantic search - [Testing](/framework/testing) - Test agents and concerns diff --git a/docs/contributing/documentation.md b/docs/contributing/documentation.md index 6a82c471..212221cc 100644 --- a/docs/contributing/documentation.md +++ b/docs/contributing/documentation.md @@ -273,6 +273,22 @@ Use tabs to show different perspectives of the same data: - `docs/` — All documentation markdown files - `docs/parts/examples/` — Auto-generated outputs (naming: `{test-file}-{test-method}.md`) +## Documenting sibling gems + +`docs/solid_agent/` documents +[solid_agent](https://github.com/activeagents/solid_agent), which lives in +its own repository — so there is nothing here for `<<<` to import, and +those pages carry their code inline. They are the exception, not a +loosening of the rule: examples on those pages mirror +[`examples/`](https://github.com/activeagents/solid_agent/tree/main/examples) +in that repo, where a test parses every Ruby file, validates every +manifest, and checks that every `SolidAgent::` constant they name still +exists. Change one of those pages and change the example it mirrors, so the +tested copy stays the source of truth. + +Everything documenting code in *this* repo — the framework and the +`actionagent` engine — imports from tests as described above. + ## Troubleshooting ### Import Not Showing diff --git a/docs/framework.md b/docs/framework.md index bc559d16..27a4c245 100644 --- a/docs/framework.md +++ b/docs/framework.md @@ -121,6 +121,7 @@ ActiveAgent integrates with Rails features and AI capabilities: - **[Streaming](/agents/streaming)** - Real-time response updates - **[Messages](/actions/messages)** - Multimodal conversation context - **[Embeddings](/actions/embeddings)** - Vector generation for semantic search +- **[Persistence](/solid_agent)** - Database-backed conversations, memory, runs and cost via the `solid_agent` gem ## Next Steps @@ -146,6 +147,7 @@ ActiveAgent integrates with Rails features and AI capabilities: - [Configuration](/framework/configuration) - Environment-specific settings - [Instrumentation](/framework/instrumentation) - Logging and monitoring - [Rails Integration](/framework/rails) - ActionCable, ActiveJob, and more +- [Persistence (SolidAgent)](/solid_agent) - Conversations, memory, runs and cost in your database **Examples:** - [Data Extraction](/examples/data_extraction_agent) - Parse structured data from documents diff --git a/docs/framework/dashboard.md b/docs/framework/dashboard.md index af088483..d0e9df8e 100644 --- a/docs/framework/dashboard.md +++ b/docs/framework/dashboard.md @@ -18,9 +18,8 @@ workspace starts with a free low-volume trial. The dashboard's models are Active Record models and its runs persist conversations, so `actionagent` adds `activerecord` and -[solid_agent](https://github.com/activeagents/solid_agent) on top of what the -framework already pulls in — neither of which `activeagent` itself requires. -Add both gems: +[solid_agent](/solid_agent) on top of what the framework already pulls in — +neither of which `activeagent` itself requires. Add both gems: ```ruby # Gemfile @@ -199,3 +198,6 @@ class ApplicationAgent < ActiveAgent::Base has_context contextual: :user end ``` + +See [Persistence (SolidAgent)](/solid_agent) for the rest of what that gem +records — the tool exchange, long-term memory, runs and cost. diff --git a/docs/getting_started.md b/docs/getting_started.md index 9c3d5ef1..e54ee374 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -211,6 +211,7 @@ See **[Generation](/agents/generation)** for background jobs, callbacks, and res **Framework:** - **[Configuration](/framework/configuration)** - Environment settings, precedence - **[Rails Integration](/framework/rails)** - Generators, helpers, conventions +- **[Persistence (SolidAgent)](/solid_agent)** - Store conversations, memory, runs and cost in your database - **[Retries](/framework/retries)** - Error handling and retry strategies - **[Instrumentation](/framework/instrumentation)** - Logging and monitoring - **[Testing](/framework/testing)** - Test your agents with fixtures and VCR diff --git a/docs/index.md b/docs/index.md index ea940aa1..e5a83d8d 100644 --- a/docs/index.md +++ b/docs/index.md @@ -50,6 +50,10 @@ features: icon: 🧪 link: /framework/testing details: Test with fixtures and VCR cassettes. Mock providers for fast tests. + - title: Persistence + icon: 🗄️ + link: /solid_agent + details: Add the solid_agent gem for database-backed conversations, long-term memory, runs and cost — as models you own. - title: Dashboard icon: 📈 link: /framework/dashboard diff --git a/docs/solid_agent.md b/docs/solid_agent.md new file mode 100644 index 00000000..4f24ab53 --- /dev/null +++ b/docs/solid_agent.md @@ -0,0 +1,121 @@ +--- +title: SolidAgent +description: Database-backed persistence for ActiveAgent — conversations, generations, the tool stream, long-term memory, runs and cost — as Active Record models you own. +--- +# {{ $frontmatter.title }} + +ActiveAgent runs agents; it deliberately doesn't store anything. No Active +Record, no tables, no migrations — a generation happens and the response is +yours to do something with. + +[SolidAgent](https://github.com/activeagents/solid_agent) is the gem that +remembers. It adds database-backed persistence for everything an agent +does: the conversation and its full tool/MCP exchange, every generation +with tokens and provenance, agent-curated long-term memory, reasoning +traces, durable run records, and cost estimates on top of the token counts. + +It is a separate gem because persistence is a real dependency — installing +`activeagent` shouldn't drag in Active Record for apps that never need it. +`solid_agent` depends on `activeagent`, never the other way round. + +```ruby +# Gemfile +gem "activeagent" +gem "solid_agent" +``` + +```bash +bundle install +rails generate solid_agent:install +rails db:migrate +``` + +::: tip Already running the dashboard? +The [dashboard engine](/framework/dashboard) (`actionagent`) depends on +`solid_agent` and installs its own copy of this schema, prefixed and +namespaced under `ActionAgent::`. Your app's own agents still want the +generator above — the two sets of tables are independent. +::: + +## What the generator installs + +Migrations and models, into `app/models/`, where they're yours to edit. +SolidAgent's concerns talk to them through a duck-typed contract, so +renaming a class or adding columns is a supported thing to do rather than a +fork. + +| Model | Holds | +|-------|-------| +| `AgentContext` | One conversation or task session: agent, action, the record it's about, instructions, cumulative tokens, `trace_id` | +| `AgentMessage` | Every turn — user, assistant, system and tool — with tool call ids, arguments, results, attachments and a content checksum | +| `AgentGeneration` | One provider call: content, model, finish reason, input/output/cached/reasoning tokens, duration, raw payload, provenance | +| `AgentMemory` / `AgentMemoryEntry` | Agent-curated notes about a subject record, with `source_agent` provenance | +| `AgentRun` | One execution: lifecycle status, input, output, an append-only progress stream, and an instructions fingerprint | + +## The concerns + +Include what you need; nothing is all-or-nothing. + +| Concern | Adds | +|---------|------| +| [`HasContext`](/solid_agent/context) | `has_context` — persists prompts, responses and the tool stream, and replays them on the next turn | +| [`HasMemory`](/solid_agent/memory) | `has_memory` — `save_memory` / `recall_memory` tools the model calls, scoped to a subject so agents hand off through it | +| [`HasTools`](/solid_agent/tools) | `has_tools` / `tool` — tool schemas from JSON view templates or an inline DSL | +| [`StreamsToolUpdates`](/solid_agent/tools#live-tool-status) | `tool_description` — broadcasts "what is it doing" over ActionCable while tools run | +| [`HasReasons`](/solid_agent/reasoning) | `has_reasons` — collects extended-thinking output; `Reasonable` persists it on generation records | + +And three things that are useful without an agent at all: + +| Module | Does | +|--------|------| +| [`ToolCache`](/solid_agent/tools#caching-tool-results) | Caches tool/MCP results by `(tool, normalized args)` with a TTL | +| [`ModelPricing`](/solid_agent/runs#cost) | Turns token counts into estimated USD | +| [`AgentManifest`](/solid_agent/manifests) | Reads, validates, converts and builds agents from portable `.agent.md`, Dotprompt and CrewAI files | + +## The shortest useful example + +```ruby +class SupportAgent < ApplicationAgent + include SolidAgent::HasContext + + has_context :conversation, contextual: :user + + def answer + load_conversation(contextable: params[:user]) + + prompt messages: conversation_messages + [ + { role: "user", content: params[:message] } + ] + end +end +``` + +```ruby +SupportAgent.with(user: user, message: "My invoice is wrong").answer.generate_now +SupportAgent.with(user: user, message: "It's the VAT line").answer.generate_now + +AgentContext.for_agent("SupportAgent").find_by(contextable: user) + .messages.chronological.map { |m| [ m.role, m.content ] } +# => [["user", "My invoice is wrong"], ["assistant", "..."], +# ["user", "It's the VAT line"], ["assistant", "..."]] +``` + +Two requests, four rows, no session state — and the second request knew +about the first because it read the table, not a cache. + +## Where to go next + +- **[Conversation context](/solid_agent/context)** — `has_context` in full: naming, multiple contexts, the tool stream, provenance and trace correlation +- **[Long-term memory](/solid_agent/memory)** — notes an agent curates itself, and hand-offs between agents +- **[Tools, streaming and caching](/solid_agent/tools)** — schemas, live status, cached results +- **[Reasoning](/solid_agent/reasoning)** — capturing and persisting extended thinking +- **[Runs, cohorts and cost](/solid_agent/runs)** — durable run records, progress events, instruction cohorts, spend +- **[Agent manifests](/solid_agent/manifests)** — agents defined in files, portable across frameworks +- **[Examples](/solid_agent/examples)** — a worked example per concern + +## Related + +- [Dev Console (Dashboard Engine)](/framework/dashboard) — reads this schema and renders it +- [Telemetry](/framework/telemetry) — the `trace_id` that joins generations to traces +- [Tools](/actions/tools) — the framework's own tool calling, which `HasTools` writes schemas for +- [solid_agent on GitHub](https://github.com/activeagents/solid_agent) diff --git a/docs/solid_agent/context.md b/docs/solid_agent/context.md new file mode 100644 index 00000000..06f78c93 --- /dev/null +++ b/docs/solid_agent/context.md @@ -0,0 +1,228 @@ +--- +title: Conversation Context +description: has_context persists prompts, responses and the full tool exchange to Active Record, and replays them on the next turn — with provenance and trace correlation on every row. +--- +# {{ $frontmatter.title }} + +`SolidAgent::HasContext` is the reason most apps reach for SolidAgent. It +gives an agent a conversation that outlives the request: prompts, replies +and the tool exchange in between are written to `agent_contexts`, +`agent_messages` and `agent_generations`, and read back on the next turn. + +```ruby +class SupportAgent < ApplicationAgent + include SolidAgent::HasContext + + has_context :conversation, contextual: :user + + def answer + load_conversation(contextable: params[:user]) + + prompt messages: conversation_messages + [ + { role: "user", content: params[:message] } + ] + end +end +``` + +A context row is keyed by **contextable + agent class + action**, so each +action of each agent keeps its own thread per record. + +## What gets written, and when + +Nothing in the action above touches the database — the callbacks do, and +they run after the provider has answered: + +1. `load_conversation` finds or creates the `AgentContext`. +2. The provider call happens. +3. The **last message of the prompt** is persisted as the user turn. +4. Any tool result messages in the response are persisted as `tool` rows. +5. The response is persisted: an `AgentGeneration` row with tokens, model, + finish reason, duration, raw payload and provenance — plus the assistant + message. + +::: warning Don't write the user turn twice +With `auto_save` on (the default), step 3 stores the last prompt message +for you. Call `add_conversation_user_message` yourself only when you've +turned `auto_save: false` off, or the same turn lands in the table twice. +::: + +## Naming the context + +The name you pass decides what the generated methods are called. Pass +nothing and you get `context`, `load_context`, `context_messages`. + +```ruby +has_context # context, load_context, add_user_message +has_context :conversation # conversation, load_conversation, ... +has_context :research_session # research_session, load_research_session, ... +``` + +| Method | Returns | +|--------|---------| +| `load_(contextable:)` | Finds or creates the context for a record | +| `load_(context_id:)` | Loads a specific context — how a second action joins an existing thread | +| `create_(contextable:)` | Always creates a new one | +| `_messages` | Message history as `{ role:, content: }` hashes, ready to pass to `prompt` | +| `add__user_message(content)` / `add__assistant_message(content)` | Append a turn by hand | +| `_result` | Content of the last assistant message | +| `_last_generation` | The last `AgentGeneration` row | +| `_summary` | `{ id:, result:, message_count:, total_tokens:, created_at:, agent_name:, action_name: }` | + +The unnamed forms (`context`, `load_context`, `context_messages`, +`context_result`, …) always delegate to the first context an agent +declares, so shared code in `ApplicationAgent` doesn't need to know the +name a subclass chose. + +## Options + +```ruby +has_context :session, + class_name: "ChatSession", # default: AgentContext, or {Name} for a named context + message_class: "ChatMessage", # inferred from class_name when omitted + generation_class: "ChatGeneration", + contextual: :chat_user, # param key the context hangs off + auto_save: false # stop persisting prompts and responses +``` + +**`contextual`** decides how the context appears: + +| Value | Behaviour | +|-------|-----------| +| `:user`, `:document`, … | Auto-loads (or creates) from `params[:user]` after the prompt is built | +| `nil` (default) | Auto-creates a context with no contextable | +| `false` | No automatic context at all — you call `load_*` or `create_*` | + +Auto-creation runs *after* the prompt is built, which is late for a +conversation that needs its history replayed. Call `load_*` explicitly in +the action when you're passing `_messages` to `prompt`. + +**`auto_save: false`** removes the persistence callbacks. You then own both +sides: `add__user_message` before the call, `add__assistant_message` +after. Useful when only some turns should be stored. + +::: tip Anonymous classes can't have contexts +Contexts are persisted under `self.class.name`. A class built with +`Class.new(...)` has none, and creation fails the `agent_name` presence +validation — name the constant first. +::: + +## Multiple contexts + +An agent can keep more than one thread, each with its own subject: + +```ruby +class MultiModalAgent < ApplicationAgent + include SolidAgent::HasContext + + has_context :conversation, contextual: :user + has_context :analysis, contextual: :document + + def analyze + prompt message: params[:message] + end +end +``` + +Both are created automatically. Only the first one gets the auto-save +callbacks — additional contexts are yours to write to, which is usually +what you want when the second one is a scratch pad rather than a +transcript. + +## The tool exchange, not just the answer + +Conversations that only store user and assistant text lose the interesting +half. When a response carries tool result messages, `HasContext` persists +each one as a `tool` row with its `tool_call_id`, name and result, deduped +by call id so re-persisting a shared message stack doesn't double up. + +Executors that run tools themselves — a job, a platform's execution service +— know things the provider's response doesn't: the arguments that went in +and how long the call took. Override `tool_invocations` to hand them over: + +```ruby +class ResearchAgent < ApplicationAgent + include SolidAgent::HasContext + + has_context + + private + + def tool_invocations + @tool_invocations ||= [] # [{ tool_call_id:, name:, arguments:, duration_ms: }, ...] + end +end +``` + +Records match response messages by `tool_call_id`, or by position when the +provider didn't send one. + +## Provenance and trace correlation + +Every generation records **what produced it**, not just what came out: + +```ruby +generation = context.generations.last +generation.provenance +# => { "agent_class" => "SupportAgent", +# "agent_checksum" => "…", # class-level prompt/embed options, minus credentials +# "prompt_checksum" => "…", # instructions, model, temperature, tool names +# "context_checksum" => "…", # context id, message count, last message id +# "action_name" => "answer", +# "trace_id" => "…", +# "timestamp" => "2026-08-14T12:00:00Z", +# "manifest_fingerprint" => "…" } # when built from a manifest +``` + +Checksums are what let you ask "did anything about this agent change +between these two runs?" without diffing prose. + +Thread a distributed trace id through `prompt_options` and the context, +generation and your [telemetry](/framework/telemetry) trace all carry it: + +```ruby +def answer + prompt_options[:trace_id] = Current.trace_id + + load_conversation(contextable: params[:user]) + prompt messages: conversation_messages +end +``` + +```ruby +AgentContext.with_trace(trace_id) +AgentGeneration.with_trace(trace_id) +``` + +## Reading it back + +The models are ordinary Active Record, scopes included: + +```ruby +AgentContext.for_agent("SupportAgent").for_action("answer").recent.limit(10) +AgentContext.find_by(contextable: user).messages.chronological +AgentContext.find_by(contextable: user).total_tokens + +AgentGeneration.by_model("gpt-4o-mini").with_tool_calls +AgentGeneration.recent.first.estimated_cost # see Runs & cost +AgentMessage.tool_messages.where(tool_name: "fetch_url") +``` + +## Generators + +```bash +# The tables and models +rails generate solid_agent:install + +# An agent wired for context +rails generate solid_agent:agent Support --context --context_name conversation --contextual user + +# Custom-named context models (ConversationContext, ConversationMessage, ...) +rails generate solid_agent:context conversation +``` + +## See also + +- [Long-term memory](/solid_agent/memory) — what an agent should remember *across* conversations +- [Runs, cohorts and cost](/solid_agent/runs) — the execution record that sits above a context +- [Examples](/solid_agent/examples#persistent-conversation) — the full worked example diff --git a/docs/solid_agent/examples.md b/docs/solid_agent/examples.md new file mode 100644 index 00000000..850d36de --- /dev/null +++ b/docs/solid_agent/examples.md @@ -0,0 +1,296 @@ +--- +title: SolidAgent Examples +description: A worked example per concern — persistent conversations, memory hand-offs, tool streaming and caching, reasoning, run tracking and manifests. +--- +# {{ $frontmatter.title }} + +Six worked examples, one per concern. Each lives in the +[`examples/` directory](https://github.com/activeagents/solid_agent/tree/main/examples) +of the solid_agent repository as files laid out in Rails paths, with a +`usage.rb` console walkthrough alongside — so what you read here you can +also drop into an app. + +All of them assume the tables and models are installed: + +```bash +bundle add solid_agent +rails generate solid_agent:install +rails db:migrate +``` + +::: tip Try one without spending tokens +Point the agent at the [mock provider](/providers/mock) — +`generate_with :mock, model: "mock-gpt-4o-mini"`. Persistence, memory, runs +and the tool cache all behave identically; only the model response changes. +::: + +## Persistent conversation + +**[`examples/persistent_conversation`](https://github.com/activeagents/solid_agent/tree/main/examples/persistent_conversation)** · +[`HasContext`](/solid_agent/context) + +A support agent whose conversation survives the request. Each turn loads +the stored thread, appends the new question, and lets the persistence +callbacks write both halves back. + +```ruby +class SupportAgent < ApplicationAgent + include SolidAgent::HasContext + + generate_with :openai, model: "gpt-4o-mini" + + has_context :conversation, contextual: :user + + def answer + load_conversation(contextable: params[:user]) + + prompt messages: conversation_messages + [ + { role: "user", content: params[:message] } + ] + end +end +``` + +```ruby +SupportAgent.with(user: user, message: "My invoice is wrong").answer.generate_now +SupportAgent.with(user: user, message: "It's the VAT line").answer.generate_now +``` + +The second call knew about the first because it read the table. The +controller that goes with it renders history straight out of the database +— no session state, nothing to warm up after a deploy: + +```ruby +class SupportConversationsController < ApplicationController + def show + @conversation = AgentContext.for_agent("SupportAgent").for_action("answer") + .find_by!(contextable: current_user) + @messages = @conversation.messages.chronological + end + + def create + response = SupportAgent.with(user: current_user, message: params.require(:message)) + .answer.generate_now + + render json: { reply: response.message.content } + end +end +``` + +## Memory hand-off + +**[`examples/memory_handoff`](https://github.com/activeagents/solid_agent/tree/main/examples/memory_handoff)** · +[`HasMemory`](/solid_agent/memory) + +Two agent classes, one subject. The researcher saves what it learns; the +writer picks it up later — possibly in another request, job or deploy. + +```ruby +class ResearcherAgent < ApplicationAgent + include SolidAgent::HasMemory + + has_memory + + def research + prompt( + message: "Research #{params[:project].name} and save what a writer would need to know.", + tools: memory_tool_definitions + ) + end + + def memory_subject = params[:project] +end +``` + +```ruby +ResearcherAgent.with(project: project).research.generate_now +WriterAgent.with(project: project).draft.generate_now + +AgentMemory.for(project).recall(limit: 5).map { |e| [ e.source_agent, e.content ] } +# => [["ResearcherAgent", "Ships on the 14th; pricing unchanged"], ...] +``` + +The writer can also skip the recall round trip entirely by putting the +notes in its instructions: + +```ruby +def draft_with_primed_memory + prompt( + instructions: [ "You are a product writer.", memory&.to_prompt ].compact.join("\n\n"), + message: "Draft the launch post for #{params[:project].name}.", + tools: memory_tool_definitions + ) +end +``` + +## Tools, live status and caching + +**[`examples/tool_streaming`](https://github.com/activeagents/solid_agent/tree/main/examples/tool_streaming)** · +[`HasTools`, `StreamsToolUpdates`, `ToolCache`](/solid_agent/tools) + +One agent with a template-loaded tool, an inline one, live status +broadcasting, and a cache around the expensive call. + +```ruby +class BrowserAgent < ApplicationAgent + include SolidAgent::HasTools + include SolidAgent::StreamsToolUpdates + + has_tools :fetch_url # from a JSON view template + + tool :summarize_page do # or inline + description "Summarize text that was already fetched" + parameter :text, type: :string, required: true + parameter :sentences, type: :integer, default: 3 + end + + tool_description :fetch_url, ->(args) { "Fetching #{args[:url]}..." } + + def browse + prompt tools: tools + end + + def fetch_url(url:) + SolidAgent::ToolCache.fetch(tool: "fetch_url", args: { url: url }, ttl: 5.minutes) do + response = Net::HTTP.get_response(URI(url)) + + response.is_a?(Net::HTTPSuccess) ? { body: response.body } : { error: "HTTP #{response.code}" } + end + end +end +``` + +Status broadcasts only when the caller passes a `stream_id`, so the same +agent is silent from a job: + +```ruby +BrowserAgent.with( + stream_id: "tool_status:#{current_user.id}:#{SecureRandom.uuid}", + message: "Summarize https://rubyonrails.org" +).browse.generate_now +``` + +## Reasoning + +**[`examples/reasoning`](https://github.com/activeagents/solid_agent/tree/main/examples/reasoning)** · +[`HasReasons`, `Reasonable`](/solid_agent/reasoning) + +Extended thinking captured off the response and persisted onto the +generation row, so it's still there after the request ends. + +```ruby +class AnalysisAgent < ApplicationAgent + include SolidAgent::HasContext + include SolidAgent::HasReasons + + generate_with :anthropic, model: "claude-sonnet-5" + + around_generation :capture_generation_reasoning # before has_context: outer wrapper + + has_context contextual: :document + has_reasons persist: true, budget_tokens: 10_000 + + def analyze + prompt message: params[:question], **reasoning_prompt_options + end + + private + + def capture_generation_reasoning + response = yield + capture_reasoning(response) + response + end +end +``` + +```ruby +generation = AgentGeneration.recent.first +generation.reasoning_content +generation.reasoning_tokens +``` + +## Run tracking + +**[`examples/run_tracking`](https://github.com/activeagents/solid_agent/tree/main/examples/run_tracking)** · +[`AgentRun`, `RunFingerprint`, `ModelPricing`](/solid_agent/runs) + +The shape most background agent work takes: a controller creates the run so +the client has an id to poll, a job executes it, a service drives the +lifecycle and appends progress events. + +```ruby +class DocumentAnalysisRun + def initialize(run) + @run = run + end + + def call + @run.record_instructions(ReportAgent::INSTRUCTIONS) + @run.start! + @run.append_event(kind: "llm", label: "analyze", eid: "gen-1", status: "started") + + response = ReportAgent.with( + document: @run.runnable, question: @run.input_prompt, trace_id: @run.trace_id + ).analyze.generate_now + + @run.append_event(kind: "llm", label: "analyze", eid: "gen-1", status: "done") + @run.complete!( + output: response.message.content, + input_tokens: response.usage&.input_tokens, + output_tokens: response.usage&.output_tokens + ) + rescue StandardError => e + @run.fail!(e) + raise + end +end +``` + +Polling reads whatever has landed so far: + +```ruby +def show + run = AgentRun.find(params[:id]) + + render json: { status: run.status, events: run.events, output: run.output } +end +``` + +And because every run carries a fingerprint of the instructions it ran +under, "did the new prompt help?" is a group-by: + +```ruby +AgentRun.for_agent("ReportAgent").where(status: "complete") + .group(:instructions_digest).average(:duration_ms) + .transform_keys { |d| SolidAgent::RunFingerprint.codename(d) } +# => { "calm-heron" => 2400.0, "misty-atoll" => 1810.0 } +``` + +## Manifests + +**[`examples/manifests`](https://github.com/activeagents/solid_agent/tree/main/examples/manifests)** · +[`AgentManifest`](/solid_agent/manifests) + +An agent defined in a file — model, tools, schemas and instructions — +validated in CI and built into a class at runtime. + +```ruby +path = "config/agents/changelog_writer.agent.md" + +SolidAgent::AgentManifest.validate(path) # => [] in CI +manifest = SolidAgent::AgentManifest.parse(path) + +klass = SolidAgent::AgentManifest.load_agent(path, class_name: "ChangelogWriterAgent") +klass._manifest_model # => "claude-sonnet-4-20250514" +klass._manifest_instructions # the Markdown body + +SolidAgent::AgentManifest.convert(path, :crewai, "tmp/agents.yaml") +``` + +## Larger examples + +Two full applications built on SolidAgent: + +- [Fizzy](https://github.com/tonsoffun/fizzy) — Kanban tracking with writing, research and file analysis agents +- [Writebook](https://github.com/tonsoffun/writebook) — collaborative writing with an integrated writing assistant diff --git a/docs/solid_agent/manifests.md b/docs/solid_agent/manifests.md new file mode 100644 index 00000000..04f2487e --- /dev/null +++ b/docs/solid_agent/manifests.md @@ -0,0 +1,186 @@ +--- +title: Agent Manifests +description: Define agents in portable .agent.md files — frontmatter for model, tools and schemas, Markdown for instructions — then validate, convert and build classes from them. +--- +# {{ $frontmatter.title }} + +An agent's interesting part is usually prose: the instructions, the tool +descriptions, the shape of what it should return. `SolidAgent::AgentManifest` +lets that live in a file rather than a class — reviewable in a pull +request, diffable, and portable to frameworks that aren't Rails. + +```markdown +--- +name: changelog-writer +version: 1.0.0 +description: Turns a range of merged pull requests into a release changelog +model: anthropic/claude-sonnet-4-20250514 +config: + temperature: 0.3 + +input: + schema: + repository: "string, The repository the release belongs to" + audience?: "string(users, operators, contributors), Who it is written for" + +tools: + - name: list_merged_pulls + description: List pull requests merged between two refs + inputSchema: + type: object + properties: + repository: { type: string } + required: [repository] + +activeagent: + class_name: ChangelogWriterAgent + concerns: + - has_tools: [list_merged_pulls] +--- + +# Changelog Writer + +You write release changelogs from merged pull requests. + +## Instructions + +1. Call `list_merged_pulls` for the requested range. +2. Group the changes into Added, Changed, Fixed and Removed. +3. Write one line per change, in the present tense. +``` + +YAML frontmatter for the structured half, Markdown for the instructions. +The full field list is in the +[`.agent.md` specification](https://github.com/activeagents/solid_agent/blob/main/docs/agent-md-spec.md). + +## Formats it reads + +| Format | File | Notes | +|--------|------|-------| +| `.agent.md` | `*.agent.md` | Native; the only one with no lossy fields | +| Dotprompt | `*.prompt` | Google's format | +| CrewAI | `agents.yaml` | Multi-agent definitions | +| GitHub Copilot | `*.prompt.md` | Copilot prompt files | + +```ruby +SolidAgent::AgentManifest.parser_formats # what can be read +SolidAgent::AgentManifest.exporter_formats # what can be written +``` + +## Loading + +`load` takes whatever you have — a path, a URL, a JSON or YAML string, or a +Hash — and detects the format: + +```ruby +manifest = SolidAgent::AgentManifest.load("config/agents/changelog_writer.agent.md") +manifest = SolidAgent::AgentManifest.load("https://example.com/agents/support.agent.md") +manifest = SolidAgent::AgentManifest.load({ name: "quick", model: "openai/gpt-4o-mini" }) + +manifest.name # => "changelog-writer" +manifest.model # => "anthropic/claude-sonnet-4-20250514" +manifest.instructions # the Markdown body +manifest.tools.map(&:name) +manifest.fingerprint # stable digest — the version an agent ran under +``` + +`parse` and `parse_string` are the explicit forms when you already know the +format. + +## Validating + +```ruby +SolidAgent::AgentManifest.validate(path) # => [] when valid, else error strings +SolidAgent::AgentManifest.valid?(path) # => true / false +SolidAgent::AgentManifest.validate!(path) # raises ValidationError +SolidAgent::AgentManifest.validate(path, strict: true) +``` + +Validation covers names, model identifiers, tool definitions, schemas, +resources and framework extensions. Worth a test, so a broken manifest +fails the build rather than a request: + +```ruby +test "every shipped manifest is valid" do + Dir["config/agents/**/*.agent.md"].each do |path| + assert_empty SolidAgent::AgentManifest.validate(path), path + end +end +``` + +## Building an agent from one + +```ruby +klass = SolidAgent::AgentManifest.load_agent(path, class_name: "ChangelogWriterAgent") + +klass._manifest_provider # => "anthropic" +klass._manifest_model # => "claude-sonnet-4-20250514" +klass._manifest_instructions # the Markdown body +klass._manifest # the Manifest, fingerprint included +klass.new.tools.map { |t| t[:name] } +``` + +The class arrives configured but not finished. It inherits from +`ApplicationAgent`, includes the concerns the `activeagent:` section asked +for, and carries the manifest's tool schemas and metadata — but behaviour +is still Ruby. Supply the actions and the tool bodies: + +```ruby +class ChangelogWriterAgent + generate_with _manifest_provider.to_sym, model: _manifest_model + + def write + prompt instructions: _manifest_instructions, message: params[:message], tools: tools + end + + # Declared tools raise NotImplementedError until you define them. + def list_merged_pulls(repository:, from: nil, to: nil) + GitHub.merged_pulls(repository, from: from, to: to) + end +end +``` + +`activeagent.class_name` in the frontmatter names the constant, so +`class_name:` is only needed to override it. Name it either way when the +agent persists context — contexts are keyed by class name, and an anonymous +class has none. + +## Converting + +```ruby +SolidAgent::AgentManifest.export(manifest, :dotprompt) +SolidAgent::AgentManifest.export_to_file(manifest, :agent_md, "config/agents/support.agent.md") +SolidAgent::AgentManifest.convert("agents.yaml", :agent_md) # CrewAI in, .agent.md out +``` + +Formats don't overlap perfectly — `.agent.md` carries fields the others +have nowhere to put — so round-tripping through a lossier format loses +them. Convert on the way in, keep `.agent.md` as the source of truth. + +## Provenance + +```ruby +SolidAgent::AgentManifest.provenance(manifest) +SolidAgent::AgentManifest.checksum(content) +``` + +An agent built from a manifest reports `manifest_fingerprint` in the +[provenance](/solid_agent/context#provenance-and-trace-correlation) +recorded on every generation — so a stored conversation says which version +of a manifest produced it. + +## Generator + +```bash +rails generate solid_agent:manifest research +rails generate solid_agent:manifest research --template research --tools search_web fetch_url +rails generate solid_agent:manifest support --context user --format dotprompt +``` + +Presets: `research`, `assistant`, `reviewer`, `chat`. + +## See also + +- [`.agent.md` specification](https://github.com/activeagents/solid_agent/blob/main/docs/agent-md-spec.md) +- [Instructions](/agents/instructions) — the framework's own instruction templates +- [Examples](/solid_agent/examples#manifests) — the worked example diff --git a/docs/solid_agent/memory.md b/docs/solid_agent/memory.md new file mode 100644 index 00000000..5258a129 --- /dev/null +++ b/docs/solid_agent/memory.md @@ -0,0 +1,149 @@ +--- +title: Long-Term Memory +description: has_memory gives agents save_memory and recall_memory tools over a durable note list scoped to a subject record — so agents hand work off to each other through shared memory. +--- +# {{ $frontmatter.title }} + +A [context](/solid_agent/context) is a transcript: everything that was +said, in order. Memory is the opposite — a short list of things worth +keeping, curated by the agent itself, that survives across conversations +and across agents. + +`SolidAgent::HasMemory` gives the model two ordinary function-calling +tools, `save_memory` and `recall_memory`, and a place to put what it +writes. + +```ruby +class ResearcherAgent < ApplicationAgent + include SolidAgent::HasMemory + + has_memory + + def research + prompt( + message: "Research #{params[:project].name} and save what a writer would need to know.", + tools: memory_tool_definitions + ) + end + + def memory_subject + params[:project] + end +end +``` + +The model decides when to write and when to read. You decide what the +memory is *about*. + +## Scoped to a subject, not to an agent + +This is the design decision everything else follows from. Memory hangs off +a `(memorable, scope)` pair — any Active Record model plus a namespace +string — and **not** off the agent class. Every agent working on the same +subject sees the same notes. + +That makes memory a hand-off channel: + +```ruby +ResearcherAgent.with(project: project).research.generate_now +# ... later, a different agent, possibly a different request or deploy: +WriterAgent.with(project: project).draft.generate_now +``` + +`WriterAgent`'s `recall_memory` returns `ResearcherAgent`'s notes. Each +entry records the class that wrote it in `source_agent`, so provenance +survives the hand-off. + +Scopes keep unrelated streams apart on the same subject: + +```ruby +has_memory scope: "competitive_research", class_name: "AgentMemory" +``` + +## Choosing the subject + +`memory_subject` defaults to `params[:memorable]`, falling back to the +`HasContext` contextable when the agent has one. Override it when the +subject lives somewhere else: + +```ruby +def memory_subject + params[:project] +end +``` + +Without a subject, `memory` is `nil` and both tools return +`{ error: "No memory subject available" }` rather than raising — the model +gets a legible answer and carries on. + +## The two tools + +`memory_tool_definitions` returns the schemas to hand to `prompt`: + +| Tool | Arguments | Does | +|------|-----------|------| +| `save_memory` | `content:` (required), `category:` | Appends a note, tagged with the calling agent class | +| `recall_memory` | `category:`, `limit:` (default 20) | Returns notes, newest first, optionally filtered | + +The same contract is available module-level, for executors that aren't +agents — a platform service, an MCP server: + +```ruby +SolidAgent::HasMemory.tool_definitions.map { |t| t[:name] } +# => ["save_memory", "recall_memory"] +``` + +## Priming instead of recalling + +A recall costs a round trip, and the model has to remember to ask. When you +know the notes are relevant, put them in the instructions instead: + +```ruby +def draft + prompt( + instructions: [ "You are a product writer.", memory&.to_prompt ].compact.join("\n\n"), + message: "Draft the launch post for #{params[:project].name}.", + tools: memory_tool_definitions + ) +end +``` + +`to_prompt` renders the notes as a labelled list with each note's source +agent, and returns an empty string when there's nothing to say. + +## Working with memory directly + +The generated `AgentMemory` and `AgentMemoryEntry` are plain models: + +```ruby +memory = AgentMemory.for(project) # find or create +memory = AgentMemory.for(project, scope: "planning") + +memory.remember("Ships on the 14th", source_agent: "ResearcherAgent", category: "fact") +memory.recall(limit: 5, category: "handoff") # newest first +memory.summary_list # contents, oldest first +memory.to_prompt # formatted for instructions +memory.forget(entry_id) +``` + +Nothing is append-only by force. Notes go stale, and pruning them is +ordinary Active Record: + +```ruby +memory.entries.where(category: "task").where(created_at: ..1.month.ago).find_each(&:destroy) +``` + +## Keeping memory useful + +- **Categories earn their keep at recall time.** `fact`, `task`, `handoff` + are the ones that tend to survive contact with real use; anything finer + usually goes unused. +- **Say what to save in the instructions.** The tool description tells the + model memory exists; your instructions tell it what's worth keeping. +- **Memory is model-authored text about your users' data.** It's readable + by every agent on that subject — scope it deliberately, and prune it. + +## See also + +- [Conversation context](/solid_agent/context) — the transcript memory summarizes +- [Examples](/solid_agent/examples#memory-hand-off) — the worked hand-off diff --git a/docs/solid_agent/reasoning.md b/docs/solid_agent/reasoning.md new file mode 100644 index 00000000..02599ade --- /dev/null +++ b/docs/solid_agent/reasoning.md @@ -0,0 +1,152 @@ +--- +title: Reasoning +description: Collect extended-thinking output from models that expose it, and persist it on generation records with HasReasons and Reasonable. +--- +# {{ $frontmatter.title }} + +Models that support extended thinking — Claude's thinking blocks, OpenAI's +reasoning models — return their working alongside the answer, and bill for +it separately. SolidAgent gives you somewhere to put it: `HasReasons` on +the agent collects it, `Reasonable` on a model persists it. + +## Capturing on the agent + +```ruby +class AnalysisAgent < ApplicationAgent + include SolidAgent::HasContext + include SolidAgent::HasReasons + + generate_with :anthropic, model: "claude-sonnet-5" + + # Declared before has_context so this wrapper is the outer one: by the + # time it runs, HasContext has written the generation row that + # `persist: true` updates. + around_generation :capture_generation_reasoning + + has_context contextual: :document + has_reasons persist: true, budget_tokens: 10_000 + + def analyze + prompt message: params[:question], **reasoning_prompt_options + end + + private + + def capture_generation_reasoning + response = yield + capture_reasoning(response) + response + end +end +``` + +Two moving parts: + +- **`reasoning_prompt_options`** turns the `has_reasons` configuration into + prompt options — `extended_thinking: true` when `auto_capture` is on, and + `reasoning_budget_tokens` when a budget is set. +- **`capture_reasoning(response)`** reads the reasoning off the response and + records it. Reasoning only exists once the provider has answered, so hand + it the response from a callback (as above) or wherever you have it. + +### `has_reasons` options + +| Option | Default | Does | +|--------|---------|------| +| `auto_capture` | `true` | Ask for extended thinking in `reasoning_prompt_options` | +| `persist` | `false` | Write captured reasoning onto the latest generation record | +| `budget_tokens` | `nil` | Default thinking budget | +| `redact_on_persist` | `false` | Store `"[Redacted]"` and the token count, not the text | + +### Reading what was captured + +Inside the agent instance that ran: + +```ruby +reasons # => [SolidAgent::Reasonable::Reason, ...] +last_reasoning&.content +total_reasoning_tokens +has_reasoning? +reasoning_chain # every non-redacted reason, joined +reasoning_stats +# => { count: 2, total_tokens: 450, total_thinking_time_ms: 1200, +# redacted_count: 0, models: ["claude-sonnet-5"] } + +add_reason(content: "Chose the strict parser", tokens: 0) # your own note +clear_reasons! +``` + +These live on the instance, so they're reachable from actions and callbacks +— not from the console after the fact. That's what persistence is for. + +## Persisting on the model + +`persist: true` needs a generation model that can hold reasoning. The +generator adds the columns and the concern: + +```bash +rails generate solid_agent:reasons AgentGeneration +rails db:migrate +``` + +```ruby +class AgentGeneration < ApplicationRecord + include SolidAgent::Reasonable +end +``` + +```ruby +generation = AgentGeneration.recent.first +generation.reasoning_content +generation.reasoning_tokens +generation.reasoning_metadata +generation.has_reasoning? +generation.reasoning_redacted? +generation.reasoning_summary(length: 120) +generation.to_reason # back to a Reason object + +generation.store_reasoning!(response) # extract from a provider response +generation.store_reason!(reason) # store one you already have +``` + +Any model can take reasoning, under whatever column names you already have: + +```bash +rails generate solid_agent:reasons MyGeneration \ + --content_column thinking_trace --tokens_column think_tokens +``` + +```ruby +class MyGeneration < ApplicationRecord + include SolidAgent::Reasonable + + reasonable_config column: :thinking_trace, tokens_column: :think_tokens +end +``` + +## Reasoning tokens are billed tokens + +`AgentGeneration` records `reasoning_tokens` separately from output tokens, +and `thinking?` tells you a generation used extended thinking at all. Both +feed [cost estimation](/solid_agent/runs#cost) — a thinking-heavy agent can +cost several times what its visible output suggests, and this is where that +shows up. + +## Handle it like user data + +Reasoning is model-generated text about your users' data, produced without +the editorial pass the answer gets. It can restate inputs verbatim, and it +can be wrong in ways the answer isn't. + +- `redact_on_persist: true` keeps the token accounting and drops the text. +- Reasoning is a record of what the model considered, not an explanation + you can rely on being faithful. +- If a conversation is user-visible, decide deliberately whether the + thinking is too. + +## See also + +- [Anthropic provider](/providers/anthropic) — enabling extended thinking +- [Usage statistics](/actions/usage) — where reasoning tokens land in usage +- [Runs, cohorts and cost](/solid_agent/runs) — what thinking costs +- [Examples](/solid_agent/examples#reasoning) — the worked example diff --git a/docs/solid_agent/runs.md b/docs/solid_agent/runs.md new file mode 100644 index 00000000..2d978fcc --- /dev/null +++ b/docs/solid_agent/runs.md @@ -0,0 +1,192 @@ +--- +title: Runs, Cohorts & Cost +description: AgentRun records each execution with lifecycle status, an append-only progress stream a UI can poll, instruction-fingerprint cohorts for comparing prompt changes, and estimated spend. +--- +# {{ $frontmatter.title }} + +A [context](/solid_agent/context) records the conversation. An `AgentRun` +records the *execution*: it started, this is what it was given, here is +where it got to, it finished (or didn't), it cost this much. + +That distinction matters as soon as the work moves off the request thread. +A job running for forty seconds has no conversation to show yet, but it has +a status and a progress stream — and the browser needs something to poll. + +Nothing creates runs for you. The executor does: a job, a service object, +the [dashboard's](/framework/dashboard) execution service. + +## Recording a run + +```ruby +run = AgentRun.create!( + runnable: document, # polymorphic, optional + agent_name: "ReportAgent", + action_name: "analyze", + input_prompt: question, + input_params: { document_id: document.id }, + trace_id: SecureRandom.uuid # shared with contexts, generations, telemetry +) + +run.record_instructions(ReportAgent::INSTRUCTIONS) # cohort fingerprint +run.start! + +run.append_event(kind: "llm", label: "analyze", eid: "gen-1", status: "started") + +response = ReportAgent.with(document: document, question: question, trace_id: run.trace_id) + .analyze.generate_now + +run.append_event(kind: "llm", label: "analyze", eid: "gen-1", status: "done", duration_ms: 1840) + +run.complete!( + output: response.message.content, + input_tokens: response.usage&.input_tokens, + output_tokens: response.usage&.output_tokens +) +``` + +### Lifecycle + +`pending → running → complete | failed | cancelled` + +| Method | Does | +|--------|------| +| `start!` | `running`, stamps `started_at` | +| `complete!(output:, metadata:, input_tokens:, output_tokens:)` | `complete`, stamps `completed_at`, computes `duration_ms` | +| `fail!(error)` | `failed`, records the message, stamps and computes duration | +| `cancel!` | `cancelled` — returns `false` if the run already finished | + +Predicates come with it: `pending?`, `running?`, `complete?`, `failed?`, +`cancelled?`, plus `in_progress?` and `finished?`. + +## The progress stream + +`append_event` appends to a JSON column with `update_column` — no +validations, no callbacks, safe to call from the run's own thread while it +works. Each append reads current database state first, so concurrent +appends interleave instead of clobbering each other. + +```ruby +run.append_event(kind: "tool", label: "fetch_url", eid: "e1", status: "started") +run.append_event(kind: "tool", label: "fetch_url", eid: "e1", status: "done", duration_ms: 120) +``` + +| Field | Meaning | +|-------|---------| +| `kind` | What ran — `llm`, `tool`, `agent`, whatever your UI groups by | +| `label` | Display name | +| `eid` | Pairs a `started` event with its `done` / `error` — anything still unpaired is in flight | +| `status` | `started`, `done`, `error` | +| `detail` | Free text, truncated to 1200 bytes | +| `duration_ms` | For finished events | + +Which makes the polling endpoint boring, which is the point: + +```ruby +def show + run = AgentRun.find(params[:id]) + + render json: { + status: run.status, + in_progress: run.in_progress?, + events: run.events, + output: run.output, + error: run.error_message, + duration_ms: run.calculated_duration_ms(fallback_end: Time.current) + } +end +``` + +## Cohorts: did the new prompt help? + +`record_instructions` stores an 8-character digest of the instructions the +run executed under. Runs sharing a digest are one cohort — the grouping key +for "we changed the prompt on Tuesday, did anything get better?" + +Digests read badly in a UI, so every digest also has a deterministic +codename derived from it alone — stable across runs, deploys and machines: + +```ruby +run.instructions_digest # => "a1b2c3d4" +run.instructions_codename # => "calm-heron" +``` + +```ruby +AgentRun.for_agent("ReportAgent").where(status: "complete") + .group(:instructions_digest) + .average(:duration_ms) + .transform_keys { |digest| SolidAgent::RunFingerprint.codename(digest) } +# => { "calm-heron" => 2400.0, "misty-atoll" => 1810.0 } +``` + +`calm-heron` vs `misty-atoll` is a conversation a team can have. Both +helpers are available without a run record: + +```ruby +SolidAgent::RunFingerprint.digest(instructions) +SolidAgent::RunFingerprint.codename(digest) +``` + +## Querying runs + +```ruby +AgentRun.recent.limit(20) +AgentRun.for_agent("ReportAgent") +AgentRun.for_action("analyze") +AgentRun.for_status("failed") +AgentRun.with_trace(trace_id) + +run.total_tokens +run.calculated_duration_ms(fallback_end: Time.current) # works mid-run too +``` + +Because `trace_id` is shared, one id joins the run, the conversation, the +generations and your [telemetry](/framework/telemetry) trace: + +```ruby +AgentRun.with_trace(id) +AgentContext.with_trace(id) +AgentGeneration.with_trace(id) +``` + +## Cost + +Generations store token counts. Pricing sits on top, which is why every +figure here is an estimate: + +```ruby +SolidAgent::ModelPricing.estimate( + model: "claude-sonnet-5", input_tokens: 12_000, output_tokens: 800 +) +# => 0.048 + +SolidAgent::ModelPricing.rate_for("gpt-4o-mini") # => [0.15, 0.6] per 1M tokens +``` + +Rates come from RubyLLM's model registry when that gem is loaded and knows +the model, from a static pattern table otherwise, and from a conservative +blended rate for anything unrecognized — so totals stay meaningful for +self-hosted and aliased models instead of silently reading zero. Models +matching `/mock/i` price at zero, so test runs don't inflate anything. + +The generated `AgentGeneration#estimated_cost` uses it automatically, and +takes explicit rates when you've negotiated your own: + +```ruby +generation.estimated_cost +generation.estimated_cost(input_price_per_million: 0.10, output_price_per_million: 0.40) +``` + +Spend for a day, by model — pricing is per-model, so total it in Ruby: + +```ruby +AgentGeneration.where(created_at: 1.day.ago..) + .group_by(&:model) + .transform_values { |gens| gens.sum { |g| g.estimated_cost.to_f }.round(4) } +``` + +## See also + +- [Dev Console (Dashboard Engine)](/framework/dashboard) — runs, traces and metrics with a UI on top +- [Telemetry](/framework/telemetry) — the trace side of the same id +- [Generation](/agents/generation) — sync and async execution +- [Examples](/solid_agent/examples#run-tracking) — the worked example diff --git a/docs/solid_agent/tools.md b/docs/solid_agent/tools.md new file mode 100644 index 00000000..0910bbc0 --- /dev/null +++ b/docs/solid_agent/tools.md @@ -0,0 +1,202 @@ +--- +title: Tools, Streaming & Caching +description: Declarative tool schemas from JSON templates or an inline DSL, live tool status over ActionCable, and a cache that replays identical tool calls instead of repeating the side effect. +--- +# {{ $frontmatter.title }} + +ActiveAgent passes [tools](/actions/tools) to `prompt` as schema hashes and +routes the model's calls to methods of the same name. SolidAgent adds three +things around that: somewhere to keep the schemas, a way to tell the user +what's happening while a tool runs, and a cache so identical calls don't +pay twice. + +## Declaring tool schemas + +`SolidAgent::HasTools` gives you two places to put a schema — a JSON view +template, or an inline DSL — and one method, `tools`, that returns all of +them. + +```ruby +class BrowserAgent < ApplicationAgent + include SolidAgent::HasTools + + has_tools :fetch_url # app/views/browser_agent/tools/fetch_url.json.erb + + tool :summarize_page do + description "Summarize text that was already fetched" + parameter :text, type: :string, required: true, description: "Page text to summarize" + parameter :sentences, type: :integer, default: 3 + end + + def browse + prompt tools: tools + end + + def fetch_url(url:) = Net::HTTP.get(URI(url)) + def summarize_page(text:, sentences: 3) = { summary: text.split(". ").first(sentences).join(". ") } +end +``` + +### From view templates + +`has_tools :fetch_url` renders +`app/views/browser_agent/tools/fetch_url.json.erb` — the agent's +underscored class name, then `tools/` — and parses the result as JSON. +Being a template, it can use ERB: enum values from the database, a +description that changes per environment. + +```erb +{ + "type": "function", + "name": "fetch_url", + "description": "Fetch a web page and return its text", + "parameters": { + "type": "object", + "properties": { + "url": { "type": "string", "description": "Absolute http(s) URL to fetch" } + }, + "required": ["url"] + } +} +``` + +`has_tools` with no arguments discovers every template in the directory +instead of listing them. + +### Inline + +The `tool` DSL builds the same OpenAI-shaped hash in Ruby: + +```ruby +tool :search do + description "Search for documents" + parameter :query, type: :string, required: true + parameter :format, type: :string, enum: %w[json xml csv] + parameter :tags, type: :array, items: { type: :string } + parameter :limit, type: :integer, default: 10 +end +``` + +`parameter` takes `type:`, `required:`, `description:`, `enum:`, `items:`, +`properties:` and `default:`. + +### Getting the schemas out + +`tools` returns templates first, then inline definitions, and memoizes. +Editing a template with the server running? `reload_tools!` drops the +cache. + +The tool name must match a method on the agent, and tool methods take +keyword arguments — that part is the framework's contract, not SolidAgent's. + +## Live tool status + +A tool that takes eight seconds looks identical to a hung request. Include +`SolidAgent::StreamsToolUpdates` and declare a description, and each call +announces itself before it runs: + +```ruby +class BrowserAgent < ApplicationAgent + include SolidAgent::HasTools + include SolidAgent::StreamsToolUpdates + + has_tools :fetch_url, :summarize_page + + tool_description :fetch_url, ->(args) { "Fetching #{args[:url]}..." } + tool_description :summarize_page, "Summarizing the page..." +end +``` + +Declaring a description is what wraps the method — tools without one still +run, they just stay quiet. A `Proc` receives the call's arguments; a +`String` is used as-is. Common tool names (`navigate`, `search`, +`extract_text`, `read_file`, …) have sensible defaults. + +Broadcasting is opt-in per generation: it happens only when +`params[:stream_id]` is present, so the same agent runs silently from a job +or the console. + +```ruby +stream_id = "tool_status:#{current_user.id}:#{SecureRandom.uuid}" + +BrowserAgent.with(stream_id: stream_id, message: "Summarize rubyonrails.org") + .browse.generate_now +``` + +Each call broadcasts to that stream name: + +```ruby +{ tool_status: { name: "fetch_url", + description: "Fetching https://rubyonrails.org...", + timestamp: "2026-08-14T12:00:00Z" } } +``` + +The client half is an ordinary channel. Scope the stream id to the current +user, or one subscriber can listen in on another's run: + +```ruby +class ToolStatusChannel < ApplicationCable::Channel + def subscribed + stream_id = params[:stream_id].to_s + reject unless stream_id.start_with?("tool_status:#{current_user.id}:") + + stream_from stream_id + end +end +``` + +This is tool-level progress, separate from token-level +[response streaming](/agents/streaming) — most UIs want both. + +## Caching tool results + +`SolidAgent::ToolCache` replays a result instead of repeating the side +effect: + +```ruby +def fetch_url(url:) + SolidAgent::ToolCache.fetch(tool: "fetch_url", args: { url: url }, ttl: 5.minutes) do + response = Net::HTTP.get_response(URI(url)) + + response.is_a?(Net::HTTPSuccess) ? { body: response.body } : { error: "HTTP #{response.code}" } + end +end +``` + +- Keys are `(tool, normalized args)` — argument order and symbol vs string + keys don't change the key. +- Replays come back tagged `cached: true`, so both you and the model can + tell a replay from a fresh call. +- **Error-shaped results are never cached.** A hash with an `:error` key + passes through, so a transient failure doesn't stick for the whole TTL. + +Configure it globally: + +```ruby +SolidAgent::ToolCache.default_ttl = 60 +SolidAgent::ToolCache.store = ActiveSupport::Cache::MemoryStore.new # Rails.cache by default +SolidAgent::ToolCache.enabled = false # e.g. in tests +``` + +The same call from a different agent, job or MCP server hits the same +entry — the key is the tool and its arguments, not the caller. + +## Generators + +```bash +# A JSON tool template plus the method stub to paste in +rails generate solid_agent:tool search ResearchAgent --parameters query:string:required limit:integer + +# The inline DSL version, printed rather than written +rails generate solid_agent:tool search ResearchAgent --inline + +# An agent with the tool concerns already included +rails generate solid_agent:agent Browser --tools --streaming +``` + +## See also + +- [Tools](/actions/tools) — the framework's tool calling, which these schemas feed +- [MCPs](/actions/mcps) — remote tool servers, cacheable the same way +- [Streaming](/agents/streaming) — token-level streaming of the response itself +- [Examples](/solid_agent/examples#tools-live-status-and-caching) — the worked example From 6e15e9a817cda950782d30d0b549a1d4e4d8495e Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 17 Aug 2026 16:39:53 +0000 Subject: [PATCH 2/4] Test activeagent, actionagent and solid_agent together MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three gems, two repositories, one dependency direction — and nothing ran them together, so all three suites were green while the combination a user installs was broken. test/integration/solid_agent runs both repos in the dummy app against the models solid_agent:install writes, through the mock provider: conversation persistence, memory hand-offs, run records, the tool cache, and the version constraints themselves. Two configurations. Source (solid_agent main, SOLID_AGENT_STRICT=1) is what the repos develop toward and must be green. Released (whatever Bundler resolves) is what users get; tests declare what they need and skip when the gem can't do it, printing the reason — that skip list is the report of how far the gem trails its source, not a failure. gemfiles/solid_agent_main takes SOLID_AGENT_PATH or SOLID_AGENT_REF so either repo can drive it. Runs on every PR, nightly, and as a gate before publishing; solid_agent's CI runs the same suite from its side. Two breakages it found immediately: - has_context's auto-context keyword was renamed contextable: -> contextual: in solid_agent 0.2. AgentExecutionService still passed the old one, so every dashboard run against current solid_agent died with ArgumentError. Resolved from the installed method now, since the gemspec floor admits both versions. - AgentToolbox's fallback cache key hashed arguments differently from SolidAgent::ToolCache, so upgrading the gem silently invalidated every cached tool result. Both paths now normalize identically. Also fixes the docs and examples that named a context without class_name: has_context :conversation resolves Conversation/ConversationMessage/ ConversationGeneration, not the installed AgentContext family, and raises NameError on the first request. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01NGn2NzpzFZT4JGKBdFMpxh --- .github/workflows/ci.yml | 7 + .github/workflows/integration.yml | 128 +++++++++++++++++ .github/workflows/release.yml | 8 ++ AGENTS.md | 29 ++++ README.md | 10 ++ .../action_agent/agent_execution_service.rb | 14 +- .../services/action_agent/agent_toolbox.rb | 25 +++- actionagent/lib/action_agent.rb | 19 +++ docs/.vitepress/config.mts | 1 + docs/contributing/releasing.md | 122 +++++++++++++++++ docs/solid_agent.md | 5 +- docs/solid_agent/context.md | 24 +++- docs/solid_agent/examples.md | 3 +- gemfiles/solid_agent_main.gemfile | 35 +++++ .../app/agents/persistence/support_agent.rb | 31 +++++ test/dummy/app/models/agent_context.rb | 110 +++++++++++++++ test/dummy/app/models/agent_generation.rb | 49 +++++++ test/dummy/app/models/agent_memory.rb | 45 ++++++ test/dummy/app/models/agent_memory_entry.rb | 12 ++ test/dummy/app/models/agent_message.rb | 39 ++++++ test/dummy/app/models/agent_run.rb | 102 ++++++++++++++ .../migrate/005_create_solid_agent_tables.rb | 112 +++++++++++++++ test/dummy/db/schema.rb | 110 ++++++++++++++- .../solid_agent/compatibility_test.rb | 89 ++++++++++++ .../solid_agent/context_persistence_test.rb | 88 ++++++++++++ .../solid_agent/integration_case.rb | 97 +++++++++++++ test/integration/solid_agent/memory_test.rb | 102 ++++++++++++++ test/integration/solid_agent/runs_test.rb | 129 ++++++++++++++++++ .../solid_agent/tool_cache_test.rb | 91 ++++++++++++ 29 files changed, 1628 insertions(+), 8 deletions(-) create mode 100644 .github/workflows/integration.yml create mode 100644 docs/contributing/releasing.md create mode 100644 gemfiles/solid_agent_main.gemfile create mode 100644 test/dummy/app/agents/persistence/support_agent.rb create mode 100644 test/dummy/app/models/agent_context.rb create mode 100644 test/dummy/app/models/agent_generation.rb create mode 100644 test/dummy/app/models/agent_memory.rb create mode 100644 test/dummy/app/models/agent_memory_entry.rb create mode 100644 test/dummy/app/models/agent_message.rb create mode 100644 test/dummy/app/models/agent_run.rb create mode 100644 test/dummy/db/migrate/005_create_solid_agent_tables.rb create mode 100644 test/integration/solid_agent/compatibility_test.rb create mode 100644 test/integration/solid_agent/context_persistence_test.rb create mode 100644 test/integration/solid_agent/integration_case.rb create mode 100644 test/integration/solid_agent/memory_test.rb create mode 100644 test/integration/solid_agent/runs_test.rb create mode 100644 test/integration/solid_agent/tool_cache_test.rb diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 401290e4..db4d9fe6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -6,6 +6,13 @@ on: branches: [ main ] jobs: + # activeagent + actionagent against solid_agent, in both the combination + # this repository develops against and the one users install today. See + # .github/workflows/integration.yml. + integration: + uses: ./.github/workflows/integration.yml + secrets: inherit + lint: runs-on: ubuntu-latest steps: diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml new file mode 100644 index 00000000..e8b377dd --- /dev/null +++ b/.github/workflows/integration.yml @@ -0,0 +1,128 @@ +name: Cross-repo integration + +# activeagent, actionagent and solid_agent release independently, and +# solid_agent depends on the framework — so each repository's own suite can +# be green while the combination people install is broken. This workflow is +# the thing that runs them together. +# +# Two configurations, both meaningful: +# +# source this checkout against solid_agent's main branch, with +# SOLID_AGENT_STRICT=1 so a test that would skip fails instead. +# This is what the two repositories are developing toward. +# +# released this checkout against whatever solid_agent Bundler resolves +# from RubyGems — the combination a user gets today. Tests skip +# what the released gem cannot do, and the skips are printed to +# the job summary, so the gap between the gem and its source is +# visible rather than assumed. +# +# Called by ci.yml on every pull request, by release.yml before publishing, +# by solid_agent's CI (via repository_dispatch) when that repo changes, and +# nightly so drift surfaces without anyone pushing. + +on: + workflow_call: + inputs: + solid_agent_ref: + description: solid_agent branch, tag or SHA to test against + type: string + default: main + workflow_dispatch: + inputs: + solid_agent_ref: + description: solid_agent branch, tag or SHA to test against + type: string + default: main + repository_dispatch: + types: [ solid-agent-changed ] + schedule: + - cron: "0 6 * * *" + +jobs: + integration: + name: ${{ matrix.configuration }} solid_agent + runs-on: ubuntu-latest + env: + BUNDLE_JOBS: 4 + BUNDLE_RETRY: 3 + CI: true + RAILS_ENV: test + ANTHROPIC_API_KEY: ANTHROPIC_API_KEY + OPEN_AI_API_KEY: OPEN_AI_API_KEY + OPEN_ROUTER_API_KEY: OPEN_ROUTER_API_KEY + strategy: + fail-fast: false + matrix: + include: + - configuration: source + gemfile: gemfiles/solid_agent_main.gemfile + strict: "1" + - configuration: released + gemfile: gemfiles/rails8.gemfile + strict: "" + steps: + - uses: actions/checkout@v6 + + - name: Install system deps + run: | + sudo apt-get update + sudo apt-get install --no-install-recommends -y build-essential git libyaml-dev pkg-config + + - uses: ruby/setup-ruby@v1 + with: + ruby-version: "3.4" + bundler-cache: true + env: + BUNDLE_GEMFILE: ${{ github.workspace }}/${{ matrix.gemfile }} + SOLID_AGENT_REF: ${{ inputs.solid_agent_ref || github.event.client_payload.solid_agent_ref || 'main' }} + + - name: Setup database + working-directory: test/dummy + env: + BUNDLE_GEMFILE: ${{ github.workspace }}/${{ matrix.gemfile }} + SOLID_AGENT_REF: ${{ inputs.solid_agent_ref || github.event.client_payload.solid_agent_ref || 'main' }} + RAILS_MASTER_KEY: ${{ secrets.RAILS_MASTER_KEY }} + run: | + bundle exec ruby bin/rails db:create + bundle exec ruby bin/rails db:migrate + + - name: Report the resolved versions + env: + BUNDLE_GEMFILE: ${{ github.workspace }}/${{ matrix.gemfile }} + SOLID_AGENT_REF: ${{ inputs.solid_agent_ref || github.event.client_payload.solid_agent_ref || 'main' }} + run: | + { + echo "### ${{ matrix.configuration }} combination" + echo + echo '```' + bundle list | grep -E "activeagent|actionagent|solid_agent" || true + echo '```' + } >> "$GITHUB_STEP_SUMMARY" + + # The engine's own suite runs here too, not just the integration + # directory: ActionAgent::AgentExecutionService builds an agent class + # around SolidAgent::HasContext, and that path is only exercised by + # actionagent/test. + - name: Run the integration suite + env: + BUNDLE_GEMFILE: ${{ github.workspace }}/${{ matrix.gemfile }} + SOLID_AGENT_REF: ${{ inputs.solid_agent_ref || github.event.client_payload.solid_agent_ref || 'main' }} + SOLID_AGENT_STRICT: ${{ matrix.strict }} + RAILS_MASTER_KEY: ${{ secrets.RAILS_MASTER_KEY }} + run: | + bin/test test/integration/solid_agent/*_test.rb \ + actionagent/test/agent_execution_service_test.rb 2>&1 | tee integration.log + + # Skips are the report: on the released combination they name every + # API the published gem is missing, which is the cue to cut a + # solid_agent release. + - name: Summarize what the resolved gem could not cover + if: always() + run: | + { + echo + echo '```' + grep -E "does not provide|skips" integration.log || echo "nothing skipped" + echo '```' + } >> "$GITHUB_STEP_SUMMARY" diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 1c0dd22e..d65fdbe6 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -17,7 +17,15 @@ on: workflow_dispatch: jobs: + # solid_agent depends on this framework, so a release can break it + # downstream without either repository's own suite noticing. Publishing + # waits on the combination being green. + integration: + uses: ./.github/workflows/integration.yml + secrets: inherit + build: + needs: integration runs-on: ubuntu-latest permissions: contents: write diff --git a/AGENTS.md b/AGENTS.md index bfdcd780..82f491c7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -395,8 +395,37 @@ bin/test # Lint bin/rubocop + +# Cross-repo: this checkout against a local solid_agent (strict = no skips) +SOLID_AGENT_PATH=../solid_agent \ + BUNDLE_GEMFILE=gemfiles/solid_agent_main.gemfile \ + SOLID_AGENT_STRICT=1 \ + bin/test test/integration/solid_agent/*_test.rb \ + actionagent/test/agent_execution_service_test.rb ``` +## Cross-repo testing (solid_agent) + +`solid_agent` lives in its own repository, depends on this framework, and is +depended on by `actionagent` — so all three suites can be green while the +combination a user installs is broken. `test/integration/solid_agent/` runs +them together in the dummy app, against the models +`rails generate solid_agent:install` writes, using the mock provider. + +- `gemfiles/solid_agent_main.gemfile` swaps the released gem for source: + `SOLID_AGENT_PATH` (local checkout) or `SOLID_AGENT_REF` (branch/tag/SHA). +- Tests declare what they need (`requires_solid_agent`, + `requires_solid_agent_capability`) and skip when the resolved gem lacks + it; `SOLID_AGENT_STRICT=1` turns those skips into failures. +- CI runs both configurations (`.github/workflows/integration.yml`), and + `release.yml` gates publishing on them. solid_agent's CI runs the same + suite against this repo's main branch and latest release tag. +- Version skew between the two gems is handled by feature detection rather + than a dependency-floor bump, since the floor can only move after the + dependency ships — see `ActionAgent.solid_agent_auto_context_keyword`. + +Full write-up: `docs/contributing/releasing.md`. + ## Dependencies - Ruby 3.1+ diff --git a/README.md b/README.md index d3828340..7d9eef99 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,16 @@ > > *Makes code [TonsOfFun](https://tonsoffun.github.io)!* +[![Gem Version](https://img.shields.io/gem/v/activeagent?label=activeagent&logo=rubygems&color=CC342D)](https://rubygems.org/gems/activeagent) +[![actionagent](https://img.shields.io/gem/v/actionagent?label=actionagent&logo=rubygems&color=CC342D)](https://rubygems.org/gems/actionagent) +[![Downloads](https://img.shields.io/gem/dt/activeagent?label=downloads)](https://rubygems.org/gems/activeagent) +[![CI](https://github.com/activeagents/activeagent/actions/workflows/ci.yml/badge.svg)](https://github.com/activeagents/activeagent/actions/workflows/ci.yml) +[![Cross-repo integration](https://github.com/activeagents/activeagent/actions/workflows/integration.yml/badge.svg)](https://github.com/activeagents/activeagent/actions/workflows/integration.yml) +[![Docs](https://img.shields.io/badge/docs-docs.activeagents.ai-2563eb)](https://docs.activeagents.ai) +[![Ruby](https://img.shields.io/badge/ruby-%3E%3D%203.1-CC342D)](https://www.ruby-lang.org) +[![Rails](https://img.shields.io/badge/rails-7.2%20%7C%208.0%20%7C%208.1-D30001)](https://rubyonrails.org) +[![License](https://img.shields.io/github/license/activeagents/activeagent)](LICENSE) + # Active Agent Active Agent provides that missing AI layer in the Rails framework, offering a structured approach to building AI-powered applications through Agent Oriented Programming. **Now Agents are Controllers!** Designing applications using agents allows developers to create modular, reusable components that can be easily integrated into existing systems. This approach promotes code reusability, maintainability, and scalability, making it easier to build complex AI-driven applications with the Object Oriented Ruby code you already use today. diff --git a/actionagent/app/services/action_agent/agent_execution_service.rb b/actionagent/app/services/action_agent/agent_execution_service.rb index d37f7adc..e4b8d821 100644 --- a/actionagent/app/services/action_agent/agent_execution_service.rb +++ b/actionagent/app/services/action_agent/agent_execution_service.rb @@ -354,8 +354,14 @@ def generate! define_singleton_method(:name) { klass_name } # Persist the conversation (agent_contexts / agent_messages / - # agent_generations) via solid_agent. contextable: false — the context - # is loaded explicitly in the action below. + # agent_generations) via solid_agent. Auto-context is switched off — + # the context is loaded explicitly in the action below. + # + # The keyword that switches it off was renamed (contextable: -> + # contextual:) between solid_agent 0.1 and 0.2, and the gemspec floor + # admits both, so it is resolved from the installed method rather than + # hard-coded: passing the wrong one is an ArgumentError that only + # surfaces when a run executes. # # The model classes are named explicitly because solid_agent infers # bare "AgentContext"/"AgentMessage"/"AgentGeneration" and resolves @@ -363,10 +369,12 @@ def generate! # inferred names only resolve in a host app that happens to have # top-level models of its own. include SolidAgent::HasContext - has_context contextable: false, + has_context( + ActionAgent.solid_agent_auto_context_keyword => false, class_name: "ActionAgent::AgentContext", message_class: "ActionAgent::AgentMessage", generation_class: "ActionAgent::AgentGeneration" + ) if effective_provider == :mock # Test environment only (see #provider_available?). diff --git a/actionagent/app/services/action_agent/agent_toolbox.rb b/actionagent/app/services/action_agent/agent_toolbox.rb index 37ef4cae..38e073e3 100644 --- a/actionagent/app/services/action_agent/agent_toolbox.rb +++ b/actionagent/app/services/action_agent/agent_toolbox.rb @@ -399,7 +399,7 @@ def cached_fetch(name, kwargs, &block) if defined?(SolidAgent::ToolCache) SolidAgent::ToolCache.fetch(tool: name.to_s, args: kwargs, ttl: CACHE_TTL, &block) else - key = "solid_agent:tool_cache:#{name}:#{Digest::SHA256.hexdigest(kwargs.sort.to_h.to_json)}" + key = fallback_cache_key(name, kwargs) cached = Rails.cache.read(key) return cached.merge(cached: true) unless cached.nil? @@ -411,6 +411,29 @@ def cached_fetch(name, kwargs, &block) end end + # Byte-for-byte the key SolidAgent::ToolCache would compute, so an app + # that upgrades solid_agent mid-TTL keeps reading what it already + # cached instead of silently starting over. Nested hashes and + # symbol/string keys have to normalize the same way, which a plain + # `kwargs.sort.to_h.to_json` does not do. + # + # test/integration/solid_agent/tool_cache_test.rb asserts the two + # schemes still agree. + def fallback_cache_key(name, kwargs) + "solid_agent:tool_cache:#{name}:#{Digest::SHA256.hexdigest(normalize_cache_args(kwargs).to_json)}" + end + + def normalize_cache_args(args) + case args + when Hash + args.map { |key, value| [ key.to_s, normalize_cache_args(value) ] }.sort_by(&:first) + when Array + args.map { |value| normalize_cache_args(value) } + else + args + end + end + # SSRF guard for fetch_url: reject hosts that resolve to loopback, # private, or link-local addresses. def public_host?(host) diff --git a/actionagent/lib/action_agent.rb b/actionagent/lib/action_agent.rb index b615bfe7..73ce47b3 100644 --- a/actionagent/lib/action_agent.rb +++ b/actionagent/lib/action_agent.rb @@ -20,6 +20,25 @@ def table_name_prefix global = defined?(::ActiveRecord::Base) ? ::ActiveRecord::Base.table_name_prefix : "" "#{global}#{@table_name_prefix ||= "active_agent_"}" end + + # Which keyword the installed solid_agent uses to switch has_context's + # auto-context off: `contextable:` up to 0.1, `contextual:` from 0.2. The + # gemspec floor admits both, and passing the wrong one raises an + # ArgumentError deep inside a run rather than at boot — so + # AgentExecutionService asks rather than assumes. + # + # Covered by test/integration/solid_agent, which runs this engine against + # solid_agent's main branch as well as the released gem. + def solid_agent_auto_context_keyword + @solid_agent_auto_context_keyword ||= begin + keywords = ::SolidAgent::HasContext::ClassMethods + .instance_method(:has_context).parameters + .select { |type, _| [ :key, :keyreq ].include?(type) } + .map(&:last) + + keywords.include?(:contextual) ? :contextual : :contextable + end + end end end diff --git a/docs/.vitepress/config.mts b/docs/.vitepress/config.mts index 53001ff3..ae6283e1 100644 --- a/docs/.vitepress/config.mts +++ b/docs/.vitepress/config.mts @@ -163,6 +163,7 @@ export default defineConfig({ { text: 'Contributing', items: [ { text: 'Documentation', link: '/contributing/documentation' }, + { text: 'Releasing & Cross-Repo Testing', link: '/contributing/releasing' }, ] }, ], diff --git a/docs/contributing/releasing.md b/docs/contributing/releasing.md new file mode 100644 index 00000000..ea16a7eb --- /dev/null +++ b/docs/contributing/releasing.md @@ -0,0 +1,122 @@ +--- +title: Releasing & Cross-Repo Testing +description: How activeagent, actionagent and solid_agent are tested together and published — the integration suite, what it guards, and the order releases go out in. +--- +# {{ $frontmatter.title }} + +Three gems ship from two repositories, and they depend on each other in one +direction: + +``` +activeagent ← actionagent (the dashboard engine, same repo) +activeagent ← solid_agent (persistence, its own repo) + ← actionagent depends on solid_agent too +``` + +Each has its own suite, and each suite can be green while the combination a +user installs is broken. Two things guard against that: a cross-repo +integration suite, and a release order. + +## The integration suite + +`test/integration/solid_agent/` in the activeagent repository runs both gems +together inside the dummy Rails app, against the models +`rails generate solid_agent:install` writes — conversation persistence, +memory hand-offs, run records, the tool cache, and the version constraints +themselves. Every generation goes through the [mock provider](/providers/mock), +so it is deterministic and free. + +Run it locally against a solid_agent checkout: + +```bash +SOLID_AGENT_PATH=../solid_agent \ + BUNDLE_GEMFILE=gemfiles/solid_agent_main.gemfile \ + SOLID_AGENT_STRICT=1 \ + bin/test test/integration/solid_agent/*_test.rb \ + actionagent/test/agent_execution_service_test.rb +``` + +Or against solid_agent's main branch, with no checkout: + +```bash +BUNDLE_GEMFILE=gemfiles/solid_agent_main.gemfile bin/test test/integration/solid_agent/*_test.rb +``` + +`gemfiles/solid_agent_main.gemfile` takes `SOLID_AGENT_PATH` (a local +checkout) or `SOLID_AGENT_REF` (a branch, tag or SHA). + +### Two configurations, on purpose + +| Configuration | Bundle | Meaning | +|---------------|--------|---------| +| **source** | solid_agent main, `SOLID_AGENT_STRICT=1` | What the repositories are developing toward. Must be green. | +| **released** | whatever Bundler resolves from RubyGems | What users install today. | + +A test declares what it needs — `requires_solid_agent "SolidAgent::HasMemory"`, +or a `requires_solid_agent_capability` block for a method signature — and +**skips** when the resolved gem can't do it, printing the reason. Under +`SOLID_AGENT_STRICT=1` those skips become failures. + +So the released run never fails for merely being behind; its skip list is +the report of how far behind it is, and that list is the cue to cut a +solid_agent release. + +### What it catches + +Real breakage found the day the suite was written: + +- `has_context`'s auto-context keyword was renamed `contextable:` → + `contextual:` between solid_agent 0.1 and 0.2. The dashboard passed the + old one, so every run against solid_agent main died with + `ArgumentError: unknown keyword`. Both repositories' suites were green. +- `ActionAgent::AgentToolbox`'s fallback cache key — the one used when + `SolidAgent::ToolCache` is absent — hashed its arguments differently from + ToolCache itself, so upgrading solid_agent silently invalidated every + cached tool result. + +Both are the same shape: a seam neither repository owns alone. + +## Where it runs + +| Trigger | Where | Runs | +|---------|-------|------| +| Pull request, push to main | activeagent `ci.yml` | Both configurations | +| Pull request, push to main | solid_agent `ci.yml` | That working tree against activeagent main *and* its latest release tag | +| Nightly | both repositories | Same, so drift surfaces without a push | +| `repository_dispatch` | activeagent `integration.yml` | Lets solid_agent's CI trigger a run with a specific ref | +| Before publishing | both `release.yml` files | Publishing waits on it | + +## Releasing + +Both repositories publish on a `v*` tag via RubyGems trusted publishing +(OIDC — no stored API key), and both gate the publish job on the +integration suite. + +**Order matters.** A gem cannot be bundled until everything it depends on is +on RubyGems: + +1. `activeagent` — the framework, depended on by both others +2. `solid_agent` — depends on activeagent +3. `actionagent` — depends on both (published from the activeagent repo's + release workflow, after the framework, by the same tag) + +`rake build_all` in the activeagent repo builds both of its gems and asserts +each archive actually contains its entry point — a gem that resolves and +then dies on `require` is the failure that guards against. The publish step +skips any version already on RubyGems, so the two gems in that repo can +share a tag while versioning independently. + +### Raising a dependency floor + +When one gem starts requiring an API the other only just added, the floor in +the gemspec has to move — and the release order above means the dependency +ships **first**. Until it does, prefer feature detection over a floor bump: +`ActionAgent.solid_agent_auto_context_keyword` is the worked example, and +`test/integration/solid_agent/compatibility_test.rb` asserts the detection +still matches the installed gem. + +## See also + +- [Documentation](/contributing/documentation) — how docs examples stay tested +- [Testing](/framework/testing) — testing your own agents +- [SolidAgent](/solid_agent) — what the persistence gem provides diff --git a/docs/solid_agent.md b/docs/solid_agent.md index 4f24ab53..a0d3a9bd 100644 --- a/docs/solid_agent.md +++ b/docs/solid_agent.md @@ -78,7 +78,10 @@ And three things that are useful without an agent at all: class SupportAgent < ApplicationAgent include SolidAgent::HasContext - has_context :conversation, contextual: :user + # class_name points the named context at the installed models; without it, + # :conversation infers Conversation / ConversationMessage / + # ConversationGeneration. See Conversation Context for why. + has_context :conversation, class_name: "AgentContext", contextual: :user def answer load_conversation(contextable: params[:user]) diff --git a/docs/solid_agent/context.md b/docs/solid_agent/context.md index 06f78c93..e7531f2d 100644 --- a/docs/solid_agent/context.md +++ b/docs/solid_agent/context.md @@ -13,7 +13,7 @@ and the tool exchange in between are written to `agent_contexts`, class SupportAgent < ApplicationAgent include SolidAgent::HasContext - has_context :conversation, contextual: :user + has_context :conversation, class_name: "AgentContext", contextual: :user def answer load_conversation(contextable: params[:user]) @@ -28,6 +28,25 @@ end A context row is keyed by **contextable + agent class + action**, so each action of each agent keeps its own thread per record. +::: danger Naming a context also names its models +`has_context :conversation` infers `Conversation`, `ConversationMessage` +and `ConversationGeneration` — *not* the `AgentContext` family the install +generator wrote. Without `class_name:`, the first request raises +`NameError: uninitialized constant Conversation`. + +Two ways out, and the second is the one most apps want: + +```ruby +has_context contextual: :user # unnamed -> AgentContext +has_context :conversation, class_name: "AgentContext", contextual: :user +``` + +`class_name: "AgentContext"` infers `AgentMessage` and `AgentGeneration` +alongside it, so one option is enough. Genuinely want separate tables per +context? `rails generate solid_agent:context conversation` writes the +models and migrations the inferred names expect. +::: + ## What gets written, and when Nothing in the action above touches the database — the callbacks do, and @@ -58,6 +77,9 @@ has_context :conversation # conversation, load_conversation, ... has_context :research_session # research_session, load_research_session, ... ``` +Remember that a name without `class_name:` also changes which models the +context resolves — see the warning above. + | Method | Returns | |--------|---------| | `load_(contextable:)` | Finds or creates the context for a record | diff --git a/docs/solid_agent/examples.md b/docs/solid_agent/examples.md index 850d36de..18a62253 100644 --- a/docs/solid_agent/examples.md +++ b/docs/solid_agent/examples.md @@ -39,7 +39,8 @@ class SupportAgent < ApplicationAgent generate_with :openai, model: "gpt-4o-mini" - has_context :conversation, contextual: :user + # class_name keeps the named context on the installed models + has_context :conversation, class_name: "AgentContext", contextual: :user def answer load_conversation(contextable: params[:user]) diff --git a/gemfiles/solid_agent_main.gemfile b/gemfiles/solid_agent_main.gemfile new file mode 100644 index 00000000..762b9c96 --- /dev/null +++ b/gemfiles/solid_agent_main.gemfile @@ -0,0 +1,35 @@ +source "https://rubygems.org" + +gem "minitest", "~> 5.0" +gem "sqlite3", "~> 2.0" +gem "rails", "~> 8.1.1" +gem "tzinfo-data" + +gemspec path: ".." +gemspec path: "../actionagent", name: "actionagent" + +# The cross-repo integration bundle: this framework, plus solid_agent from +# source rather than the released gem. +# +# `actionagent` depends on solid_agent, so the default bundle already +# resolves whatever is on RubyGems — which is the combination users install, +# not the one the two repositories are developing toward. Tests under +# test/integration/solid_agent run against both: they skip what the resolved +# version cannot do, and SOLID_AGENT_STRICT=1 turns those skips into +# failures, which is how CI asserts the source-to-source combination is +# whole. +# +# SOLID_AGENT_PATH points at a local checkout, so solid_agent's own CI can +# run this suite against its working tree: +# +# SOLID_AGENT_PATH=../solid_agent \ +# BUNDLE_GEMFILE=gemfiles/solid_agent_main.gemfile bin/test +# +# SOLID_AGENT_REF picks a branch, tag or SHA when there is no checkout. +if (path = ENV["SOLID_AGENT_PATH"]) + gem "solid_agent", path: File.expand_path(path, __dir__) +else + gem "solid_agent", + github: "activeagents/solid_agent", + ref: ENV.fetch("SOLID_AGENT_REF", "main") +end diff --git a/test/dummy/app/agents/persistence/support_agent.rb b/test/dummy/app/agents/persistence/support_agent.rb new file mode 100644 index 00000000..298dcbb2 --- /dev/null +++ b/test/dummy/app/agents/persistence/support_agent.rb @@ -0,0 +1,31 @@ +# frozen_string_literal: true + +# Agents for the cross-repo integration suite (test/integration/solid_agent). +# +# Namespaced Persistence:: rather than SolidAgent:: — that constant belongs +# to the gem. Everything here uses the mock provider, so the suite is +# deterministic and free: what it proves is that the two gems compose, not +# what a model says. +module Persistence + # HasContext against the host-app models the install generator writes. + class SupportAgent < ApplicationAgent + include SolidAgent::HasContext + + generate_with :mock, model: "mock-gpt-4o-mini" + + # A named context renames the generated methods (load_conversation, + # conversation_messages, ...) and, on its own, would infer the models + # Conversation / ConversationMessage / ConversationGeneration. class_name + # points it back at the models the install generator writes; without it, + # a host app needs `rails generate solid_agent:context conversation`. + has_context :conversation, class_name: "AgentContext", contextual: :user + + def answer + load_conversation(contextable: params[:user]) + + prompt messages: conversation_messages + [ + { role: "user", content: params[:message] } + ] + end + end +end diff --git a/test/dummy/app/models/agent_context.rb b/test/dummy/app/models/agent_context.rb new file mode 100644 index 00000000..a75ec38c --- /dev/null +++ b/test/dummy/app/models/agent_context.rb @@ -0,0 +1,110 @@ +# frozen_string_literal: true + +# What `rails generate solid_agent:install` writes into a host app. +# +# Copied from solid_agent's install generator template so the integration +# suite exercises the models a user actually gets, not a convenient stub. If +# an upstream change breaks this contract, test/integration/solid_agent is +# where it should surface. +class AgentContext < ApplicationRecord + belongs_to :contextable, polymorphic: true, optional: true + has_many :messages, class_name: "AgentMessage", dependent: :destroy + has_many :generations, class_name: "AgentGeneration", dependent: :destroy + + validates :agent_name, presence: true + validates :action_name, presence: true + + scope :recent, -> { order(created_at: :desc) } + scope :for_agent, ->(name) { where(agent_name: name) } + scope :for_action, ->(name) { where(action_name: name) } + scope :with_trace, ->(trace_id) { where(trace_id: trace_id) } + + def input_params + options&.dig("input_params") || options&.dig(:input_params) || {} + end + + def record_generation!(response, extra_attributes = {}) + usage = response.respond_to?(:usage) ? response.usage : nil + + generation = generations.create!({ + content: response.message&.content, + model: response_value(response, :model), + provider: response_value(response, :provider), + finish_reason: response_value(response, :finish_reason), + input_tokens: usage&.input_tokens || 0, + output_tokens: usage&.output_tokens || 0, + cached_tokens: response_value(usage, :cached_tokens) || 0, + reasoning_tokens: response_value(usage, :reasoning_tokens) || 0, + tool_calls: extract_tool_calls(response), + raw_response: response_value(response, :raw_response), + duration_seconds: extract_duration_seconds(response, usage) + }.merge(extra_attributes)) + + increment!(:total_input_tokens, generation.input_tokens) + increment!(:total_output_tokens, generation.output_tokens) + + add_assistant_message(response.message&.content, metadata: { "tool_calls" => generation.tool_calls }) + + generation + end + + def record_generation_with_provenance!(response, provenance) + provenance = (provenance || {}).deep_stringify_keys + + record_generation!(response, trace_id: provenance["trace_id"], provenance: provenance) + end + + def add_user_message(content, **attributes) + messages.create!(role: "user", content: content, **attributes) + end + + def add_assistant_message(content, **attributes) + messages.create!(role: "assistant", content: content, **attributes) + end + + def add_system_message(content) + messages.create!(role: "system", content: content) + end + + def add_tool_message(tool_call_id:, tool_name:, result:, arguments: nil, duration_ms: nil) + messages.create!( + role: "tool", + tool_call_id: tool_call_id, + tool_name: tool_name, + tool_result: result, + tool_arguments: arguments.presence || {}, + metadata: duration_ms ? { "duration_ms" => duration_ms } : {}, + content: result.is_a?(String) ? result : result.to_json + ) + end + + def total_tokens + total_input_tokens + total_output_tokens + end + + private + + def response_value(response, method) + response.respond_to?(method) ? response.public_send(method) : nil + end + + def extract_duration_seconds(response, usage) + return response.duration if response.respond_to?(:duration) && response.duration + + duration_ms = usage.respond_to?(:duration_ms) ? usage.duration_ms : nil + duration_ms ? duration_ms / 1000.0 : nil + end + + def extract_tool_calls(response) + message = response.message + return [] unless message.respond_to?(:tool_calls) && message.tool_calls.present? + + message.tool_calls.map do |tc| + { + id: tc.respond_to?(:id) ? tc.id : nil, + name: tc.respond_to?(:name) ? tc.name : nil, + arguments: tc.respond_to?(:arguments) ? tc.arguments : nil + } + end + end +end diff --git a/test/dummy/app/models/agent_generation.rb b/test/dummy/app/models/agent_generation.rb new file mode 100644 index 00000000..b9b4c344 --- /dev/null +++ b/test/dummy/app/models/agent_generation.rb @@ -0,0 +1,49 @@ +# frozen_string_literal: true + +# Copied from solid_agent's install generator template — see AgentContext. +class AgentGeneration < ApplicationRecord + belongs_to :agent_context + + scope :recent, -> { order(created_at: :desc) } + scope :by_model, ->(model) { where(model: model) } + scope :with_trace, ->(trace_id) { where(trace_id: trace_id) } + scope :completed, -> { where(finish_reason: "stop") } + + def total_tokens + input_tokens + output_tokens + end + + def cache_hit? + cached_tokens.to_i.positive? + end + + def thinking? + reasoning_tokens.to_i.positive? + end + + def has_tool_calls? + tool_calls.present? && tool_calls.any? + end + + def completed? + finish_reason == "stop" + end + + def truncated? + finish_reason == "length" + end + + def ended_with_tool_calls? + finish_reason == "tool_calls" + end + + def estimated_cost(input_price_per_million: nil, output_price_per_million: nil) + if input_price_per_million && output_price_per_million + input_cost = (input_tokens / 1_000_000.0) * input_price_per_million + output_cost = (output_tokens / 1_000_000.0) * output_price_per_million + input_cost + output_cost + elsif defined?(SolidAgent::ModelPricing) + SolidAgent::ModelPricing.estimate(model: model, input_tokens: input_tokens, output_tokens: output_tokens) + end + end +end diff --git a/test/dummy/app/models/agent_memory.rb b/test/dummy/app/models/agent_memory.rb new file mode 100644 index 00000000..201ed580 --- /dev/null +++ b/test/dummy/app/models/agent_memory.rb @@ -0,0 +1,45 @@ +# frozen_string_literal: true + +# Copied from solid_agent's install generator template — see AgentContext. +# Kept loadable when the resolved solid_agent predates HasMemory, so the +# default bundle can still boot the dummy app. +class AgentMemory < ApplicationRecord + DEFAULT_SCOPE = defined?(SolidAgent::HasMemory) ? SolidAgent::HasMemory::DEFAULT_SCOPE : "default" + + belongs_to :memorable, polymorphic: true, optional: true + has_many :entries, class_name: "AgentMemoryEntry", dependent: :destroy + + validates :scope, presence: true + + def self.for(memorable, scope: DEFAULT_SCOPE) + find_or_create_by!(memorable: memorable, scope: scope.to_s) + end + + def remember(content, source_agent: nil, category: nil) + entries.create!(content: content, source_agent: source_agent, category: category) + end + + def recall(limit: 20, category: nil) + scope = entries.order(created_at: :desc) + scope = scope.where(category: category) if category.present? + scope.limit(limit || 20).to_a + end + + def forget(entry_id) + entries.find(entry_id).destroy! + end + + def summary_list + entries.order(:created_at).pluck(:content) + end + + def to_prompt + notes = entries.order(:created_at).map do |entry| + source = entry.source_agent.present? ? " (#{entry.source_agent})" : "" + "- #{entry.content}#{source}" + end + return "" if notes.empty? + + "Memory notes for this subject:\n#{notes.join("\n")}" + end +end diff --git a/test/dummy/app/models/agent_memory_entry.rb b/test/dummy/app/models/agent_memory_entry.rb new file mode 100644 index 00000000..882ad76f --- /dev/null +++ b/test/dummy/app/models/agent_memory_entry.rb @@ -0,0 +1,12 @@ +# frozen_string_literal: true + +# Copied from solid_agent's install generator template — see AgentContext. +class AgentMemoryEntry < ApplicationRecord + belongs_to :agent_memory + + validates :content, presence: true + + scope :chronological, -> { order(:created_at) } + scope :by_category, ->(category) { where(category: category) } + scope :from_agent, ->(agent_name) { where(source_agent: agent_name) } +end diff --git a/test/dummy/app/models/agent_message.rb b/test/dummy/app/models/agent_message.rb new file mode 100644 index 00000000..19b4abc5 --- /dev/null +++ b/test/dummy/app/models/agent_message.rb @@ -0,0 +1,39 @@ +# frozen_string_literal: true + +# Copied from solid_agent's install generator template — see AgentContext. +class AgentMessage < ApplicationRecord + belongs_to :agent_context + + validates :role, presence: true, inclusion: { in: %w[user assistant system tool] } + + scope :by_role, ->(role) { where(role: role) } + scope :user_messages, -> { by_role("user") } + scope :assistant_messages, -> { by_role("assistant") } + scope :system_messages, -> { by_role("system") } + scope :tool_messages, -> { by_role("tool") } + scope :chronological, -> { order(created_at: :asc) } + + def to_message_hash + hash = { role: role, content: content } + hash[:tool_calls] = tool_calls_data if role == "assistant" && tool_calls_data.present? + + if role == "tool" + hash[:tool_call_id] = tool_call_id + hash[:name] = tool_name + end + + hash + end + + def tool_calls_data + metadata&.dig("tool_calls") || [] + end + + def tool_call? + role == "assistant" && tool_calls_data.present? + end + + def tool_result? + role == "tool" + end +end diff --git a/test/dummy/app/models/agent_run.rb b/test/dummy/app/models/agent_run.rb new file mode 100644 index 00000000..1ba621d0 --- /dev/null +++ b/test/dummy/app/models/agent_run.rb @@ -0,0 +1,102 @@ +# frozen_string_literal: true + +# Copied from solid_agent's install generator template — see AgentContext. +# The fingerprint helpers degrade when the resolved solid_agent predates +# SolidAgent::RunFingerprint, so the dummy app boots on either. +class AgentRun < ApplicationRecord + belongs_to :runnable, polymorphic: true, optional: true + + STATUSES = %w[pending running complete failed cancelled].freeze + + validates :status, inclusion: { in: STATUSES } + + scope :recent, -> { order(created_at: :desc) } + scope :for_agent, ->(agent_name) { where(agent_name: agent_name) } + scope :for_action, ->(action_name) { where(action_name: action_name) } + scope :with_trace, ->(trace_id) { where(trace_id: trace_id) } + scope :for_status, ->(status) { where(status: status) } + + STATUSES.each do |status_name| + define_method("#{status_name}?") { status == status_name } + end + + def in_progress? + pending? || running? + end + + def finished? + complete? || failed? || cancelled? + end + + def start! + update!(status: "running", started_at: Time.current) + end + + def complete!(output: nil, metadata: {}, input_tokens: nil, output_tokens: nil) + update!( + status: "complete", + output: output, + output_metadata: (output_metadata || {}).merge(metadata), + input_tokens: input_tokens || self.input_tokens, + output_tokens: output_tokens || self.output_tokens, + completed_at: Time.current, + duration_ms: calculated_duration_ms(fallback_end: Time.current) + ) + end + + def fail!(error) + update!( + status: "failed", + error_message: error.respond_to?(:message) ? error.message : error.to_s, + completed_at: Time.current, + duration_ms: calculated_duration_ms(fallback_end: Time.current) + ) + end + + def cancel! + return false if finished? + + update!(status: "cancelled", completed_at: Time.current) + true + end + + def append_event(kind:, label:, eid: nil, status: "done", detail: nil, duration_ms: nil) + event = { + "at" => Time.current.iso8601(3), + "eid" => eid, + "kind" => kind.to_s, + "label" => label.to_s, + "status" => status.to_s + }.compact + event["detail"] = detail.to_s.byteslice(0, 1200).to_s.scrub if detail + event["duration_ms"] = duration_ms if duration_ms + current = self.class.where(id: id).pick(:events) || [] + update_column(:events, current + [ event ]) + event + end + + def record_instructions(instructions) + return unless defined?(SolidAgent::RunFingerprint) + + self.instructions_digest = SolidAgent::RunFingerprint.digest(instructions) + end + + def instructions_codename + return unless defined?(SolidAgent::RunFingerprint) + + SolidAgent::RunFingerprint.codename(instructions_digest) + end + + def total_tokens + input_tokens.to_i + output_tokens.to_i + end + + def calculated_duration_ms(fallback_end: nil) + return duration_ms if duration_ms.present? + + finish = completed_at || fallback_end + return nil unless started_at && finish + + ((finish - started_at) * 1000).to_i + end +end diff --git a/test/dummy/db/migrate/005_create_solid_agent_tables.rb b/test/dummy/db/migrate/005_create_solid_agent_tables.rb new file mode 100644 index 00000000..6c3d53b9 --- /dev/null +++ b/test/dummy/db/migrate/005_create_solid_agent_tables.rb @@ -0,0 +1,112 @@ +# frozen_string_literal: true + +# The tables `rails generate solid_agent:install` writes into a host app. +# +# Deliberately unprefixed: migration 004 creates the dashboard engine's own +# copy of this schema under ActionAgent.table_name_prefix ("active_agent_"), +# which the engine owns and namespaces. These are the host-app tables an +# ordinary Rails app gets, and what test/integration/solid_agent exercises — +# the combination a user actually installs. +# +# Kept in step with solid_agent's generator templates in +# lib/generators/solid_agent/install/templates. +class CreateSolidAgentTables < ActiveRecord::Migration[7.2] + def change + create_table :agent_contexts do |t| + t.references :contextable, polymorphic: true, index: true + t.string :agent_name, null: false + t.string :action_name, null: false + t.text :instructions + t.column :options, json_type, default: {} + t.string :trace_id, index: true + t.integer :total_input_tokens, default: 0 + t.integer :total_output_tokens, default: 0 + t.timestamps + t.index [ :agent_name, :action_name ] + end + + create_table :agent_messages do |t| + t.references :agent_context, null: false, foreign_key: true, index: true + t.string :role, null: false + t.text :content + t.string :tool_call_id + t.string :tool_name + t.column :tool_arguments, json_type, default: {} + t.column :tool_result, json_type + t.column :attachments, json_type, default: [] + t.column :metadata, json_type, default: {} + t.column :provenance, json_type, default: {} + t.string :content_checksum + t.timestamps + t.index :role + t.index :tool_call_id + end + + create_table :agent_generations do |t| + t.references :agent_context, null: false, foreign_key: true, index: true + t.text :content + t.string :model + t.string :provider + t.string :finish_reason + t.integer :input_tokens, default: 0 + t.integer :output_tokens, default: 0 + t.integer :cached_tokens, default: 0 + t.integer :reasoning_tokens, default: 0 + t.column :tool_calls, json_type, default: [] + t.column :raw_response, json_type + t.float :duration_seconds + t.string :trace_id, index: true + t.column :provenance, json_type, default: {} + t.timestamps + end + + create_table :agent_memories do |t| + t.references :memorable, polymorphic: true, index: true + t.string :scope, null: false, default: "default" + t.timestamps + t.index [ :memorable_type, :memorable_id, :scope ], unique: true, + name: "index_agent_memories_on_memorable_and_scope" + end + + create_table :agent_memory_entries do |t| + t.references :agent_memory, null: false, foreign_key: true + t.text :content, null: false + t.string :source_agent + t.string :category + t.timestamps + t.index :category + end + + create_table :agent_runs do |t| + t.references :runnable, polymorphic: true, index: true + t.string :agent_name + t.string :action_name + t.string :trace_id, index: true + t.string :status, null: false, default: "pending" + t.text :input_prompt + t.column :input_params, json_type, default: {} + t.text :output + t.column :output_metadata, json_type, default: {} + t.text :error_message + t.column :events, json_type, default: [] + t.string :instructions_digest, index: true + t.integer :input_tokens, default: 0 + t.integer :output_tokens, default: 0 + t.integer :duration_ms + t.datetime :started_at + t.datetime :completed_at + t.timestamps + t.index :status + end + end + + private + + # jsonb where the adapter has it, json where it doesn't — same treatment + # migration 004 gives the engine's tables. solid_agent's own generator + # writes jsonb, which is right for the PostgreSQL apps it targets; the + # dummy app runs on SQLite. + def json_type + @json_type ||= connection.adapter_name.to_s.downcase.include?("postgres") ? :jsonb : :json + end +end diff --git a/test/dummy/db/schema.rb b/test/dummy/db/schema.rb index aafae6db..3464823d 100644 --- a/test/dummy/db/schema.rb +++ b/test/dummy/db/schema.rb @@ -10,7 +10,7 @@ # # It's strongly recommended that you check this file into your version control system. -ActiveRecord::Schema[8.0].define(version: 4) do +ActiveRecord::Schema[8.1].define(version: 5) do create_table "active_agent_agent_contexts", force: :cascade do |t| t.string "action_name", null: false t.string "agent_name", null: false @@ -313,6 +313,111 @@ t.index [ "user_id" ], name: "index_active_agent_session_recordings_on_user_id" end + create_table "agent_contexts", force: :cascade do |t| + t.string "action_name", null: false + t.string "agent_name", null: false + t.integer "contextable_id" + t.string "contextable_type" + t.datetime "created_at", null: false + t.text "instructions" + t.json "options", default: {} + t.integer "total_input_tokens", default: 0 + t.integer "total_output_tokens", default: 0 + t.string "trace_id" + t.datetime "updated_at", null: false + t.index [ "agent_name", "action_name" ], name: "index_agent_contexts_on_agent_name_and_action_name" + t.index [ "contextable_type", "contextable_id" ], name: "index_agent_contexts_on_contextable" + t.index [ "trace_id" ], name: "index_agent_contexts_on_trace_id" + end + + create_table "agent_generations", force: :cascade do |t| + t.integer "agent_context_id", null: false + t.integer "cached_tokens", default: 0 + t.text "content" + t.datetime "created_at", null: false + t.float "duration_seconds" + t.string "finish_reason" + t.integer "input_tokens", default: 0 + t.string "model" + t.integer "output_tokens", default: 0 + t.json "provenance", default: {} + t.string "provider" + t.json "raw_response" + t.integer "reasoning_tokens", default: 0 + t.json "tool_calls", default: [] + t.string "trace_id" + t.datetime "updated_at", null: false + t.index [ "agent_context_id" ], name: "index_agent_generations_on_agent_context_id" + t.index [ "trace_id" ], name: "index_agent_generations_on_trace_id" + end + + create_table "agent_memories", force: :cascade do |t| + t.datetime "created_at", null: false + t.integer "memorable_id" + t.string "memorable_type" + t.string "scope", default: "default", null: false + t.datetime "updated_at", null: false + t.index [ "memorable_type", "memorable_id", "scope" ], name: "index_agent_memories_on_memorable_and_scope", unique: true + t.index [ "memorable_type", "memorable_id" ], name: "index_agent_memories_on_memorable" + end + + create_table "agent_memory_entries", force: :cascade do |t| + t.integer "agent_memory_id", null: false + t.string "category" + t.text "content", null: false + t.datetime "created_at", null: false + t.string "source_agent" + t.datetime "updated_at", null: false + t.index [ "agent_memory_id" ], name: "index_agent_memory_entries_on_agent_memory_id" + t.index [ "category" ], name: "index_agent_memory_entries_on_category" + end + + create_table "agent_messages", force: :cascade do |t| + t.integer "agent_context_id", null: false + t.json "attachments", default: [] + t.text "content" + t.string "content_checksum" + t.datetime "created_at", null: false + t.json "metadata", default: {} + t.json "provenance", default: {} + t.string "role", null: false + t.json "tool_arguments", default: {} + t.string "tool_call_id" + t.string "tool_name" + t.json "tool_result" + t.datetime "updated_at", null: false + t.index [ "agent_context_id" ], name: "index_agent_messages_on_agent_context_id" + t.index [ "role" ], name: "index_agent_messages_on_role" + t.index [ "tool_call_id" ], name: "index_agent_messages_on_tool_call_id" + end + + create_table "agent_runs", force: :cascade do |t| + t.string "action_name" + t.string "agent_name" + t.datetime "completed_at" + t.datetime "created_at", null: false + t.integer "duration_ms" + t.text "error_message" + t.json "events", default: [] + t.json "input_params", default: {} + t.text "input_prompt" + t.integer "input_tokens", default: 0 + t.string "instructions_digest" + t.text "output" + t.json "output_metadata", default: {} + t.integer "output_tokens", default: 0 + t.integer "runnable_id" + t.string "runnable_type" + t.datetime "started_at" + t.string "status", default: "pending", null: false + t.string "trace_id" + t.datetime "updated_at", null: false + t.index [ "instructions_digest" ], name: "index_agent_runs_on_instructions_digest" + t.index [ "runnable_type", "runnable_id" ], name: "index_agent_runs_on_runnable" + t.index [ "status" ], name: "index_agent_runs_on_status" + t.index [ "trace_id" ], name: "index_agent_runs_on_trace_id" + end + create_table "posts", force: :cascade do |t| t.text "content" t.datetime "created_at", null: false @@ -347,6 +452,9 @@ t.index [ "email" ], name: "index_users_on_email", unique: true end + add_foreign_key "agent_generations", "agent_contexts" + add_foreign_key "agent_memory_entries", "agent_memories" + add_foreign_key "agent_messages", "agent_contexts" add_foreign_key "posts", "users" add_foreign_key "profiles", "users" end diff --git a/test/integration/solid_agent/compatibility_test.rb b/test/integration/solid_agent/compatibility_test.rb new file mode 100644 index 00000000..0218e3f9 --- /dev/null +++ b/test/integration/solid_agent/compatibility_test.rb @@ -0,0 +1,89 @@ +# frozen_string_literal: true + +require_relative "integration_case" + +# The release-coordination check: can these two versions ship together? +# +# activeagent and solid_agent version independently, and solid_agent depends +# on activeagent — so a framework release can break the gem downstream of it +# without anything in either test suite noticing. These assertions are the +# things that would go wrong silently. +class SolidAgentCompatibilityTest < SolidAgentIntegrationTest + # Constants this repository dereferences without a `defined?` guard: if + # one of them goes, a run raises. SolidAgent::ToolCache is deliberately + # absent from this list — AgentToolbox falls back when it is missing, and + # tool_cache_test asserts the two paths agree. + REQUIRED_CONSTANTS = %w[ + SolidAgent::HasContext + ].freeze + + test "solid_agent is resolved in this bundle" do + assert Gem.loaded_specs.key?("solid_agent"), + "solid_agent is not in the bundle — actionagent declares it as a dependency" + end + + test "the resolved solid_agent accepts this checkout's activeagent version" do + requirement = Gem.loaded_specs.fetch("solid_agent").dependencies + .find { |dependency| dependency.name == "activeagent" }&.requirement + + assert requirement, "solid_agent no longer declares a dependency on activeagent" + + assert requirement.satisfied_by?(Gem::Version.new(ActiveAgent::VERSION)), + "solid_agent #{SolidAgentIntegrationTest.installed_version} requires activeagent " \ + "#{requirement}, which this checkout (#{ActiveAgent::VERSION}) does not satisfy — " \ + "releasing this version of the framework would break solid_agent" + end + + test "the resolved solid_agent satisfies actionagent's declared dependency" do + requirement = Gem::Specification.load("actionagent/actionagent.gemspec").dependencies + .find { |dependency| dependency.name == "solid_agent" }&.requirement + + assert requirement, "actionagent no longer declares a dependency on solid_agent" + + assert requirement.satisfied_by?(Gem.loaded_specs.fetch("solid_agent").version), + "actionagent requires solid_agent #{requirement}, resolved " \ + "#{SolidAgentIntegrationTest.installed_version}" + end + + test "every SolidAgent constant this repository names unguarded exists" do + missing = REQUIRED_CONSTANTS.reject { |name| SolidAgentIntegrationTest.solid_agent_const_defined?(name) } + + assert_empty missing, + "this repository references #{missing.join(', ')}, absent from solid_agent " \ + "#{SolidAgentIntegrationTest.installed_version}" + end + + test "has_context still takes the options the dashboard passes it" do + # ActionAgent::AgentExecutionService builds an agent class and calls + # has_context with these keywords. A rename upstream surfaces only when + # a run executes — which is how contextable:/contextual: got missed. + assert_includes keywords_of(:has_context), :class_name + assert_includes keywords_of(:has_context), :message_class + assert_includes keywords_of(:has_context), :generation_class + + assert_includes keywords_of(:has_context), ActionAgent.solid_agent_auto_context_keyword, + "ActionAgent.solid_agent_auto_context_keyword resolved to a keyword solid_agent " \ + "#{SolidAgentIntegrationTest.installed_version} does not accept" + end + + test "the dashboard builds a runnable agent class against the resolved gem" do + agent = ActionAgent::Agent.create!( + name: "Compatibility", provider: "mock", model: "mock", instructions: "Be brief." + ) + run = agent.agent_runs.create!(trace_id: SecureRandom.uuid, status: :pending) + + response = ActionAgent::AgentExecutionService.new(agent, run).send(:generate!) + + assert response, "the dashboard's runtime agent class failed to generate" + assert_operator ActionAgent::AgentContext.count, :>, 0, + "the run did not persist a conversation through solid_agent" + end + + private + + def keywords_of(method_name) + SolidAgent::HasContext::ClassMethods.instance_method(method_name).parameters + .select { |type, _| [ :key, :keyreq ].include?(type) } + .map(&:last) + end +end diff --git a/test/integration/solid_agent/context_persistence_test.rb b/test/integration/solid_agent/context_persistence_test.rb new file mode 100644 index 00000000..8f806739 --- /dev/null +++ b/test/integration/solid_agent/context_persistence_test.rb @@ -0,0 +1,88 @@ +# frozen_string_literal: true + +require_relative "integration_case" + +# The combination a user installs: activeagent generating, solid_agent +# persisting, into the models `rails generate solid_agent:install` writes. +class SolidAgentContextPersistenceTest < SolidAgentIntegrationTest + # Persistence::SupportAgent declares `contextual:`, which solid_agent + # renamed from `contextable:` after 0.1 — referencing the class at all + # raises on an older gem, so the check has to come before that. + requires_solid_agent_capability("has_context(contextual:)") do + SolidAgent::HasContext::ClassMethods.instance_method(:has_context) + .parameters.any? { |_, name| name == :contextual } + end + + test "a generation writes the context, both turns and the generation record" do + response = Persistence::SupportAgent.with( + user: subject_record, message: "My invoice is wrong" + ).answer.generate_now + + context = AgentContext.for_agent("Persistence::SupportAgent").sole + + assert_equal "answer", context.action_name + assert_equal subject_record, context.contextable + + assert_equal [ "user", "assistant" ], context.messages.chronological.map(&:role) + assert_equal "My invoice is wrong", context.messages.user_messages.sole.content + assert_equal response.message.content, context.messages.assistant_messages.sole.content + + generation = context.generations.sole + assert_equal response.message.content, generation.content + assert generation.model.present?, "expected the generation to record the model" + end + + test "a second turn replays the first out of the database" do + message_counts = [] + + ActiveSupport::Notifications.subscribed( + ->(*, payload) { message_counts << payload[:message_count] }, + "prompt.provider.active_agent" + ) do + 2.times do |i| + Persistence::SupportAgent.with( + user: subject_record, message: "Turn #{i}" + ).answer.generate_now + end + end + + context = AgentContext.for_agent("Persistence::SupportAgent").sole + + assert_equal 1, AgentContext.count, "expected both turns to share one context" + assert_equal %w[user assistant user assistant], context.messages.chronological.map(&:role) + assert_equal 2, context.generations.count + + # The point of persistence: the second request sent the stored exchange + # back to the provider — one message, then three. + assert_equal [ 1, 3 ], message_counts + end + + test "each generation records provenance and a trace id" do + Persistence::SupportAgent.with( + user: subject_record, message: "Trace me" + ).answer.generate_now + + generation = AgentGeneration.sole + + assert generation.trace_id.present?, "expected a trace_id for telemetry correlation" + assert_equal "Persistence::SupportAgent", generation.provenance["agent_class"] + assert_equal "answer", generation.provenance["action_name"] + assert generation.provenance["prompt_checksum"].present? + assert generation.provenance["agent_checksum"].present? + + assert_equal [ generation ], AgentGeneration.with_trace(generation.trace_id).to_a + end + + test "token counts roll up onto the context" do + Persistence::SupportAgent.with( + user: subject_record, message: "Count me" + ).answer.generate_now + + context = AgentContext.sole + generation = context.generations.sole + + assert_equal generation.input_tokens, context.total_input_tokens + assert_equal generation.output_tokens, context.total_output_tokens + assert_equal context.total_input_tokens + context.total_output_tokens, context.total_tokens + end +end diff --git a/test/integration/solid_agent/integration_case.rb b/test/integration/solid_agent/integration_case.rb new file mode 100644 index 00000000..44963455 --- /dev/null +++ b/test/integration/solid_agent/integration_case.rb @@ -0,0 +1,97 @@ +# frozen_string_literal: true + +require "test_helper" + +# Base class for the cross-repo integration suite: activeagent and +# solid_agent exercised together, in the dummy Rails app, against the +# generated host-app models. +# +# The two gems release independently, so the version of solid_agent in the +# bundle is not always the one this repository is developing against. Rather +# than pin it — which would hide the released combination — a test declares +# what it needs and skips when the resolved gem cannot do it: +# +# requires_solid_agent "SolidAgent::HasMemory" +# +# SOLID_AGENT_STRICT=1 turns those skips into failures. CI runs the suite +# twice: once against the released gem (skips allowed, so drift is visible +# in the log) and once against solid_agent main with strict on, which is +# what asserts the source-to-source combination actually works. +class SolidAgentIntegrationTest < ActiveSupport::TestCase + STRICT = ENV["SOLID_AGENT_STRICT"].present? + + class << self + def requires_solid_agent(*constants) + @required_constants = Array(@required_constants) + constants.flatten.map(&:to_s) + end + + def required_constants + Array(@required_constants) + (superclass.respond_to?(:required_constants) ? superclass.required_constants : []) + end + + # For API shape rather than existence — a keyword that was renamed, a + # method that grew an argument. The block runs at test time, so it can + # reflect on whatever version resolved. + def requires_solid_agent_capability(description, &predicate) + @required_capabilities = Array(@required_capabilities) + [ [ description, predicate ] ] + end + + def required_capabilities + Array(@required_capabilities) + + (superclass.respond_to?(:required_capabilities) ? superclass.required_capabilities : []) + end + end + + requires_solid_agent "SolidAgent::HasContext" + + setup do + unmet = self.class.required_constants.reject { |name| self.class.solid_agent_const_defined?(name) } + unmet += self.class.required_capabilities.reject { |_, predicate| predicate.call }.map(&:first) + + if unmet.any? + message = "solid_agent #{SolidAgentIntegrationTest.installed_version} does not provide " \ + "#{unmet.join(', ')} — run with gemfiles/solid_agent_main.gemfile to cover it" + + # Printed rather than left to the reporter, which only shows skip + # reasons in verbose mode. On the released combination this list is + # the report: it names every API the published gem is missing. + puts "[solid_agent] SKIP #{self.class.name}##{name}: #{message}" + + STRICT ? flunk(message) : skip(message) + end + + AgentMemoryEntry.delete_all + AgentMemory.delete_all + AgentMessage.delete_all + AgentGeneration.delete_all + AgentContext.delete_all + AgentRun.delete_all + end + + def self.solid_agent_const_defined?(name) + name.to_s.split("::").inject(Object) do |namespace, part| + return false unless namespace.const_defined?(part, false) + + namespace.const_get(part, false) + end + + true + rescue NameError + false + end + + def self.installed_version + Gem.loaded_specs["solid_agent"]&.version&.to_s || "(not resolved)" + end + + # The dummy app's user, as a stand-in for whatever record a host app hangs + # conversations and memory off. + def subject_record + @subject_record ||= User.create!( + name: "Integration Subject", + email: "integration-#{SecureRandom.hex(4)}@example.com", + age: 30, + role: "user" + ) + end +end diff --git a/test/integration/solid_agent/memory_test.rb b/test/integration/solid_agent/memory_test.rb new file mode 100644 index 00000000..e87716b6 --- /dev/null +++ b/test/integration/solid_agent/memory_test.rb @@ -0,0 +1,102 @@ +# frozen_string_literal: true + +require_relative "integration_case" + +# Defined behind a guard because the class body references a constant the +# released gem may not have: an unguarded `include` would take the whole +# suite down with a NameError instead of reporting a skip. +if SolidAgentIntegrationTest.solid_agent_const_defined?("SolidAgent::HasMemory") + module Persistence + class ResearcherAgent < ApplicationAgent + include SolidAgent::HasMemory + + generate_with :mock, model: "mock-gpt-4o-mini" + + has_memory + + def research + prompt message: params[:message], tools: memory_tool_definitions + end + + def memory_subject + params[:memorable] + end + end + + class WriterAgent < ResearcherAgent + def draft + prompt message: params[:message], tools: memory_tool_definitions + end + end + end +end + +# Memory is scoped to a subject rather than an agent class, which is what +# makes it a hand-off channel. That claim is only true if two agent classes +# genuinely read each other's notes through the host app's models. +class SolidAgentMemoryTest < SolidAgentIntegrationTest + requires_solid_agent "SolidAgent::HasMemory" + + test "one agent's notes are readable by another working on the same subject" do + researcher = agent(Persistence::ResearcherAgent, memorable: subject_record) + researcher.save_memory(content: "Ships on the 14th", category: "fact") + + writer = agent(Persistence::WriterAgent, memorable: subject_record) + recalled = writer.recall_memory + + assert_equal 1, recalled[:count] + assert_equal "Ships on the 14th", recalled[:entries].first[:content] + assert_equal "Persistence::ResearcherAgent", recalled[:entries].first[:source_agent], + "expected the writing agent to be recorded as the source" + end + + test "memory rows land in the host app's models" do + agent(Persistence::ResearcherAgent, memorable: subject_record) + .save_memory(content: "Pricing unchanged", category: "fact") + + memory = AgentMemory.sole + + assert_equal subject_record, memory.memorable + assert_equal "default", memory.scope + assert_equal [ "Pricing unchanged" ], memory.summary_list + assert_includes memory.to_prompt, "Pricing unchanged (Persistence::ResearcherAgent)" + end + + test "scopes keep separate streams on one subject" do + AgentMemory.for(subject_record).remember("default stream", source_agent: "Test") + AgentMemory.for(subject_record, scope: "planning").remember("planning stream", source_agent: "Test") + + assert_equal 2, AgentMemory.count + assert_equal [ "default stream" ], AgentMemory.for(subject_record).summary_list + assert_equal [ "planning stream" ], AgentMemory.for(subject_record, scope: "planning").summary_list + end + + test "categories filter recall and missing subjects fail legibly" do + researcher = agent(Persistence::ResearcherAgent, memorable: subject_record) + researcher.save_memory(content: "a fact", category: "fact") + researcher.save_memory(content: "a handoff", category: "handoff") + + assert_equal [ "a handoff" ], researcher.recall_memory(category: "handoff")[:entries].map { |e| e[:content] } + + subjectless = agent(Persistence::ResearcherAgent) + + assert_equal({ error: "No memory subject available" }, subjectless.save_memory(content: "nowhere to put it")) + assert_equal({ error: "No memory subject available" }, subjectless.recall_memory) + end + + test "the tool schemas the model is offered match the module-level contract" do + definitions = agent(Persistence::ResearcherAgent, memorable: subject_record).memory_tool_definitions + + assert_equal SolidAgent::HasMemory.tool_definitions, definitions + assert_equal %w[save_memory recall_memory], definitions.map { |tool| tool[:name] } + end + + private + + # The provider decides when to call a tool; the mock one never does. These + # tests drive the tool methods the way a provider would, which is the part + # of the contract that belongs to solid_agent. + def agent(klass, **params) + klass.new.tap { |instance| instance.params = params } + end +end diff --git a/test/integration/solid_agent/runs_test.rb b/test/integration/solid_agent/runs_test.rb new file mode 100644 index 00000000..31e7d1b3 --- /dev/null +++ b/test/integration/solid_agent/runs_test.rb @@ -0,0 +1,129 @@ +# frozen_string_literal: true + +require_relative "integration_case" + +# AgentRun is the record an executor writes around a generation: the status +# a UI polls, the progress stream it renders, the fingerprint that groups +# runs into cohorts, and the tokens cost is estimated from. +class SolidAgentRunsTest < SolidAgentIntegrationTest + requires_solid_agent "SolidAgent::RunFingerprint", "SolidAgent::ModelPricing" + + INSTRUCTIONS = "You are a support agent. Answer in two paragraphs at most." + + test "a run wraps a real generation and records what it cost" do + run = AgentRun.create!( + runnable: subject_record, + agent_name: "Persistence::SupportAgent", + action_name: "answer", + input_prompt: "My invoice is wrong", + trace_id: SecureRandom.uuid + ) + + run.record_instructions(INSTRUCTIONS) + run.save! + run.start! + + assert_predicate run, :running? + assert_predicate run, :in_progress? + + run.append_event(kind: "llm", label: "answer", eid: "gen-1", status: "started") + + response = Persistence::SupportAgent.with( + user: subject_record, message: run.input_prompt + ).answer.generate_now + + run.append_event(kind: "llm", label: "answer", eid: "gen-1", status: "done", duration_ms: 12) + + run.complete!( + output: response.message.content, + input_tokens: response.usage&.input_tokens, + output_tokens: response.usage&.output_tokens + ) + + assert_predicate run, :complete? + assert_predicate run, :finished? + assert_equal response.message.content, run.output + assert_operator run.total_tokens, :>, 0 + assert run.duration_ms.present? + + assert_equal [ "started", "done" ], run.events.map { |event| event["status"] } + assert_equal [ "gen-1", "gen-1" ], run.events.map { |event| event["eid"] } + end + + test "a failed run keeps the error and stops being in progress" do + run = AgentRun.create!(agent_name: "Persistence::SupportAgent", input_prompt: "boom") + run.start! + run.fail!(StandardError.new("provider timed out")) + + assert_predicate run, :failed? + assert_predicate run, :finished? + assert_equal "provider timed out", run.error_message + refute run.cancel!, "a finished run cannot be cancelled" + end + + test "runs group into cohorts by the instructions they executed under" do + 2.times do + AgentRun.create!(agent_name: "Persistence::SupportAgent").tap do |run| + run.record_instructions(INSTRUCTIONS) + run.save! + end + end + + AgentRun.create!(agent_name: "Persistence::SupportAgent").tap do |run| + run.record_instructions("#{INSTRUCTIONS} Be brief.") + run.save! + end + + cohorts = AgentRun.group(:instructions_digest).count + + assert_equal 2, cohorts.size, "expected one cohort per distinct instruction text" + assert_equal [ 1, 2 ], cohorts.values.sort + + codenames = cohorts.keys.map { |digest| SolidAgent::RunFingerprint.codename(digest) } + + assert_equal codenames.uniq.size, codenames.size, "codenames must distinguish cohorts" + codenames.each { |codename| assert_match(/\A[a-z]+-[a-z]+\z/, codename) } + end + + test "the digest is stable across processes" do + run = AgentRun.create!(agent_name: "Persistence::SupportAgent") + run.record_instructions(INSTRUCTIONS) + + assert_equal SolidAgent::RunFingerprint.digest(INSTRUCTIONS), run.instructions_digest + assert_equal SolidAgent::RunFingerprint.codename(run.instructions_digest), run.instructions_codename + end + + test "generations price out through ModelPricing" do + Persistence::SupportAgent.with(user: subject_record, message: "Price me").answer.generate_now + + generation = AgentGeneration.sole + + # The mock provider's model prices at zero by design, so the assertion + # that matters is that estimation runs and stays consistent. + assert_equal( + SolidAgent::ModelPricing.estimate( + model: generation.model, + input_tokens: generation.input_tokens, + output_tokens: generation.output_tokens + ), + generation.estimated_cost + ) + + assert_in_delta 0.048, SolidAgent::ModelPricing.estimate( + model: "claude-sonnet-5", input_tokens: 12_000, output_tokens: 800 + ), 0.0005 + end + + test "one trace id joins the run, the conversation and the generation" do + trace_id = SecureRandom.uuid + run = AgentRun.create!(agent_name: "Persistence::SupportAgent", trace_id: trace_id) + + Persistence::SupportAgent.with(user: subject_record, message: "Correlate me").answer.generate_now + AgentGeneration.update_all(trace_id: trace_id) + AgentContext.update_all(trace_id: trace_id) + + assert_equal [ run ], AgentRun.with_trace(trace_id).to_a + assert_equal 1, AgentContext.with_trace(trace_id).count + assert_equal 1, AgentGeneration.with_trace(trace_id).count + end +end diff --git a/test/integration/solid_agent/tool_cache_test.rb b/test/integration/solid_agent/tool_cache_test.rb new file mode 100644 index 00000000..000e4499 --- /dev/null +++ b/test/integration/solid_agent/tool_cache_test.rb @@ -0,0 +1,91 @@ +# frozen_string_literal: true + +require_relative "integration_case" + +# ToolCache is the one piece of solid_agent this repository calls directly: +# ActionAgent::AgentToolbox routes tool results through it when it is +# present, and falls back to a hand-rolled Rails.cache key when it isn't. +# These assertions pin the behaviour that fallback has to match. +class SolidAgentToolCacheTest < SolidAgentIntegrationTest + requires_solid_agent "SolidAgent::ToolCache" + + setup do + @previous_store = SolidAgent::ToolCache.store + SolidAgent::ToolCache.store = ActiveSupport::Cache::MemoryStore.new + end + + teardown do + SolidAgent::ToolCache.store = @previous_store + SolidAgent::ToolCache.enabled = true + end + + test "an identical call replays instead of running the side effect again" do + calls = 0 + fetch = -> { SolidAgent::ToolCache.fetch(tool: "fetch_url", args: { url: "https://example.com" }) { calls += 1; { body: "hi" } } } + + assert_nil fetch.call[:cached] + assert_equal({ body: "hi", cached: true }, fetch.call) + assert_equal 1, calls, "the block should have run once" + end + + test "keys ignore argument order and key type" do + ordered = SolidAgent::ToolCache.cache_key("search", { query: "ruby", limit: 10 }) + shuffled = SolidAgent::ToolCache.cache_key("search", { "limit" => 10, "query" => "ruby" }) + + assert_equal ordered, shuffled + assert_match(/\Asolid_agent:tool_cache:search:/, ordered) + end + + test "error results are never cached" do + calls = 0 + 2.times do + SolidAgent::ToolCache.fetch(tool: "fetch_url", args: { url: "https://down.example" }) do + calls += 1 + { error: "HTTP 500" } + end + end + + assert_equal 2, calls, "a transient failure must not stick for the TTL" + end + + test "disabling it bypasses the store entirely" do + SolidAgent::ToolCache.enabled = false + calls = 0 + + 2.times { SolidAgent::ToolCache.fetch(tool: "t", args: {}) { calls += 1; { ok: true } } } + + assert_equal 2, calls + end + + test "the dashboard routes tool results through this cache" do + # AgentToolbox#cached_fetch prefers ToolCache when it is defined and + # hand-rolls an equivalent Rails.cache key when it isn't. The two paths + # have to agree on the key, or upgrading solid_agent silently invalidates + # every cached tool result. + args = { url: "https://example.com" } + calls = 0 + + result = ActionAgent::AgentToolbox.send(:cached_fetch, :fetch_url, args) do + calls += 1 + { body: "hi" } + end + + assert_equal({ body: "hi" }, result) + assert_equal 1, calls + assert_equal( + { body: "hi" }, + SolidAgent::ToolCache.store.read(SolidAgent::ToolCache.cache_key("fetch_url", args)), + "the dashboard's cached_fetch should write through SolidAgent::ToolCache's key scheme" + ) + + assert_equal SolidAgent::ToolCache.cache_key("fetch_url", args), + ActionAgent::AgentToolbox.send(:fallback_cache_key, :fetch_url, args), + "AgentToolbox's no-ToolCache fallback key has drifted from ToolCache's" + + nested = { filters: { b: 2, a: [ 1, { z: 0 } ] }, url: "https://example.com" } + + assert_equal SolidAgent::ToolCache.cache_key("fetch_url", nested), + ActionAgent::AgentToolbox.send(:fallback_cache_key, :fetch_url, nested), + "nested arguments must normalize the same way on both sides" + end +end From 220937a720ccea9f34f40cd95bf2c26da4e4a99f Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 17 Aug 2026 16:50:50 +0000 Subject: [PATCH 3/4] Keep the dummy schema dump on Rails 8.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Regenerating the dump after adding migration 005 stamped it ActiveRecord::Schema[8.1], the Rails the dump ran under. The API-gem CI jobs pin rails ~> 8.0.0, whose Active Record rejects that version outright — db:migrate died on load with "Unknown migration version" before running a single test. The header tracks whoever last dumped it, so anyone regenerating on 8.1 will reintroduce this; the matrix's floor is what it has to satisfy. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01NGn2NzpzFZT4JGKBdFMpxh --- test/dummy/db/schema.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/dummy/db/schema.rb b/test/dummy/db/schema.rb index 3464823d..52fcd0af 100644 --- a/test/dummy/db/schema.rb +++ b/test/dummy/db/schema.rb @@ -10,7 +10,7 @@ # # It's strongly recommended that you check this file into your version control system. -ActiveRecord::Schema[8.1].define(version: 5) do +ActiveRecord::Schema[8.0].define(version: 5) do create_table "active_agent_agent_contexts", force: :cascade do |t| t.string "action_name", null: false t.string "agent_name", null: false From c18e25b2514b12a53b337487839ee0dc098ed329 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 17 Aug 2026 17:04:36 +0000 Subject: [PATCH 4/4] Lock the dummy AgentRun's event append, and fix two doc claims MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mirrors activeagents/solid_agent#8: append_event's read-modify-write on a JSON column loses entries when two writers race, so the re-read and the write are serialized by a row lock. The dummy model tracks the install generator's template, so it moves with it. The runs page repeated the old claim that concurrent appends "interleave safely" — true of the lock, not of the re-read that was there. The context page said "turned auto_save: false off", which reads as the opposite of what it means. Both from Copilot's review. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01NGn2NzpzFZT4JGKBdFMpxh --- docs/solid_agent/context.md | 4 ++-- docs/solid_agent/runs.md | 6 ++++-- test/dummy/app/models/agent_run.rb | 10 ++++++++-- 3 files changed, 14 insertions(+), 6 deletions(-) diff --git a/docs/solid_agent/context.md b/docs/solid_agent/context.md index e7531f2d..4c102ec7 100644 --- a/docs/solid_agent/context.md +++ b/docs/solid_agent/context.md @@ -62,8 +62,8 @@ they run after the provider has answered: ::: warning Don't write the user turn twice With `auto_save` on (the default), step 3 stores the last prompt message -for you. Call `add_conversation_user_message` yourself only when you've -turned `auto_save: false` off, or the same turn lands in the table twice. +for you. Call `add_conversation_user_message` yourself only when you have +set `auto_save: false`, or the same turn lands in the table twice. ::: ## Naming the context diff --git a/docs/solid_agent/runs.md b/docs/solid_agent/runs.md index 2d978fcc..cd156700 100644 --- a/docs/solid_agent/runs.md +++ b/docs/solid_agent/runs.md @@ -62,8 +62,10 @@ Predicates come with it: `pending?`, `running?`, `complete?`, `failed?`, `append_event` appends to a JSON column with `update_column` — no validations, no callbacks, safe to call from the run's own thread while it -works. Each append reads current database state first, so concurrent -appends interleave instead of clobbering each other. +works. Read-modify-write on a JSON column would drop entries when two +writers race, so the re-read and the write are serialized by a row lock: +an append made while another is in flight lands after it rather than on +top of it. ```ruby run.append_event(kind: "tool", label: "fetch_url", eid: "e1", status: "started") diff --git a/test/dummy/app/models/agent_run.rb b/test/dummy/app/models/agent_run.rb index 1ba621d0..a4f1350d 100644 --- a/test/dummy/app/models/agent_run.rb +++ b/test/dummy/app/models/agent_run.rb @@ -70,8 +70,14 @@ def append_event(kind:, label:, eid: nil, status: "done", detail: nil, duration_ }.compact event["detail"] = detail.to_s.byteslice(0, 1200).to_s.scrub if detail event["duration_ms"] = duration_ms if duration_ms - current = self.class.where(id: id).pick(:events) || [] - update_column(:events, current + [ event ]) + + # Read-modify-write on a JSON column drops entries when two writers + # race, so the re-read and the write are serialized by a row lock. + with_lock do + current = self.class.where(id: id).pick(:events) || [] + update_column(:events, current + [ event ]) + end + event end