⚡ SpecTool

High-Performance Speculative Tool-Execution Runtime with Conservative Dependency & Effect Verification

Decouple LLM token generation from high-latency external tool I/O through provably safe, isolated speculative execution.

Key Features • Installation & PyPI • Architecture • Execution Timeline • Safety Model • Benchmarks • Quickstart • Documentation

---

🚀 The Core Problem

Modern autonomous AI agents spend 60%–85% of their total wall-clock lifetime stalled on synchronous external tool I/O (web searching, API queries, database queries, reading/writing files).

Traditional runtimes force a strictly serialized lockstep:

[ Turn 1: LLM Reasoning ] ──▶ [ Tool I/O (Wait) ] ──▶ [ Turn 2: LLM Reasoning ] ──▶ [ Tool I/O (Wait) ] ...

SpecTool brings speculative execution principles to LLM agent runtimes:

• Predicts likely next tool invocations and concrete arguments from execution history or token streams.
• Verifies data independence and side-effect disjointness with conservative fail-closed safety.
• Executes admitted candidates asynchronously in quarantined background tasks.
• Reconciles candidate results with authoritative model requests in 0ms exposed latency upon hit, or cleans up silently upon miss with zero state pollution.

---

📦 Installation & PyPI Packages

SpecTool is published as an asynchronous, zero-dependency core package with optional provider extras for OpenAI and Anthropic SDKs.

# Install core runtime
pip install spectool

# Install with OpenAI Async SDK support
pip install "spectool[openai]"

# Install with Anthropic Async SDK support
pip install "spectool[anthropic]"

# Install with all LLM provider adapters
pip install "spectool[all]"

For local development and running the benchmark suites:

git clone https://github.com/Akgithub2028/SpecTool.git
cd spectool
pip install -e ".[dev]"

---

✨ Key Features

Feature	Description
⚡ Latency Hiding	Overlaps background I/O with forward token generation, yielding 1.25x–2.0x wall-time speedups.
🛡️ Fail-Closed Safety (Invariant I1)	Any unmodeled effect, unknown resource, or unproven commutativity automatically rejects speculation.
🔬 Static AST Dependency Verification	Parses Python tool calls using Python AST analysis to extract data dependencies and block dangerous dynamic reflection (eval, exec, unverified imports).
🔒 Quarantined State Isolation	Speculative executions reside strictly in the isolated SpeculationStore and never leak into the authoritative agent history without explicit reconciliation.
🔄 Deterministic Argument Canonicalization	Schema-aware argument sorting and default value injection guarantees exact cache hits regardless of argument key ordering.
🔌 Provider Agnostic	Native adapters for OpenAI, Anthropic, and custom model streams normalize events into unified runtime interfaces.
📊 Complete Observability	Fine-grained trace recording with correlation IDs (session_id, turn_id, invocation_id, prediction_id) and automated metric aggregation.

---

🏛️ System Architecture

The SpecTool runtime decouples forward LLM generation from tool I/O through a modular pipeline spanning stream adaptation, transition prediction, AST/effect verification, admission scoring, isolated execution, and transactional reconciliation:

  [ LLM Stream ] ──▶ [ Predictor ] ──▶ [ Canonicalizer ] ──▶ [ Verifier ] ──▶ [ Admission ] ──▶ [ Scheduler ]
                                                                                                        │
                                                                                                        ▼
  [ Authoritative Commit ] ◀── [ Reconciliation Engine ] ◀── [ Isolated Store ] ◀── [ Speculative Executor ]

📖 Deep Dive: See the full interactive flowchart in the Architecture Specification Document.

---

⏱️ Execution Timeline & Latency Hiding

In dependency chains (such as search -> fetch), SpecTool overlaps background tool execution with the agent's turn reasoning. Upon the LLM requesting the tool, the cached result is promoted instantly with zero exposed wait time:

Sequential (B0):   [ LLM Turn 1 ] ──▶ [ Search (50ms) ] ──▶ [ LLM Turn 2 ] ──▶ [ Fetch (50ms) ] (Total: 250ms)
                                                                 ▲
                                                                 │ (Parallel Overlap)
Speculative (B4):  [ LLM Turn 1 ] ──▶ [ Search (50ms) ] ──▶ [ LLM Turn 2 ] ──▶ [ PROMOTE (0ms) ] (Total: 200ms — 1.25x Speedup)
                                                └──▶ [ Spec Fetch (50ms) ] ──┘

📖 Sequence Diagram: See the complete timeline visualization in Benchmarking Methodology & Timelines.

---

🛡️ Fail-Closed Safety & AST Verification

SpecTool enforces strict mathematical non-interference before admitting any candidate for background execution:

1. Policy Gating: Enforces READ_ONLY, SANDBOX_ONLY, DRY_RUN_ONLY, or NEVER.
2. Resource Footprints: Checks hierarchical namespaces (web:*, file:/tmp/*, db:users).
3. Static AST Analysis: Statically validates Python code blocks to identify data dependencies and disallow unsafe constructs (eval, exec, unverified imports).
4. Hazard Detection: Automatically blocks Read-After-Write (RAW), Write-After-Read (WAR), and Write-After-Write (WAW) hazards.

📖 Verification Flow: See the full safety funnel flowchart in Safety Model & Invariants.

---

📈 Empirical Benchmark Results

Measured across 5 canonical workloads and 5 baseline configurations (3 trials each, 50ms simulated I/O latency):

Workload	Baseline	p50 Latency	p90 Latency	Speedup	Hit Rate	Promoted	Wasted	Status
W1 (Independent Reads)	B0 (Sequential)	253.4ms	253.5ms	1.00x	0.0%	0	0	Baseline
	B4 (Full SpecTool)	254.3ms	254.5ms	1.00x	0.0%	0	0	Safe Read
W2 (Dependency Chain)	B0 (Sequential)	253.9ms	254.0ms	1.00x	0.0%	0	0	Baseline
	B1 (Naive Speculation)	253.8ms	254.0ms	1.00x	0.0%	0	0	Verified
	B2 (Predictor Only)	203.7ms	203.8ms	1.25x	100.0%	1	0	Hit
	B3 (Predictor + Verifier)	204.5ms	204.6ms	1.24x	100.0%	1	0	Verified
	B4 (Full SpecTool)	203.5ms	204.3ms	1.25x	100.0%	1	0	⚡ Optimal
W3 (Conflicting File Ops)	B0 (Sequential)	253.3ms	253.5ms	1.00x	0.0%	0	0	Baseline
	B4 (Full SpecTool)	253.9ms	253.9ms	1.00x	0.0%	0	0	🛡️ Protected
W4 (Disjoint Writes)	B0 (Sequential)	253.5ms	253.7ms	1.00x	0.0%	0	0	Baseline
	B4 (Full SpecTool)	253.5ms	254.2ms	1.00x	0.0%	0	0	Safe Write
W5 (Prediction Miss)	B0 (Sequential)	253.7ms	253.8ms	1.00x	0.0%	0	0	Baseline
	B4 (Full SpecTool)	254.6ms	254.6ms	1.00x	0.0%	0	1	🧹 Clean Discard

---

🛠️ Quickstart & Developer Ergonomics

1. Declare Tools with @tool

import asyncio
from spectool import EffectSet, SpeculationPolicy, tool


@tool(
    effects=EffectSet(reads=frozenset({"web:search"}), network=True),
    speculation_policy=SpeculationPolicy.READ_ONLY,
    description="Search technical documentation",
)
async def search_docs(query: str, limit: int = 10) -> list[str]:
    """Parameter types and defaults are automatically inferred into JSON schema."""
    await asyncio.sleep(0.05)
    return [f"https://spectool.ai/docs/{query}"]

2. Run a Speculative Agent Harness

import asyncio
from spectool import (
    AdmissionController,
    AgentMessage,
    IndependenceVerifier,
    MessageRole,
    PatternPredictor,
    SpeculativeAgentHarness,
    ToolRegistry,
)
from spectool.providers.openai import OpenAIProvider


async def main():
    registry = ToolRegistry()
    registry.register(search_docs)

    predictor = PatternPredictor(top_k=2, confidence_threshold=0.5)

    harness = SpeculativeAgentHarness(
        provider=OpenAIProvider(client=my_openai_client),
        registry=registry,
        predictor=predictor,
        admission_controller=AdmissionController(),
        verifier=IndependenceVerifier(),
        speculation_enabled=True,
    )

    result = await harness.run(
        [AgentMessage(role=MessageRole.USER, content="Look up SpecTool architecture")]
    )
    print(
        f"Completed in {result.turn_count} turns. Hit Rate: {result.trace_snapshot.speculative_hit_rate * 100:.1f}%"
    )


asyncio.run(main())

---

💻 CLI & Developer Commands

# Run benchmark suite via CLI
spectool benchmark run --workload all --trials 3 --baselines B0 B4

# Run full evaluation matrix script
PYTHONPATH=src:. python3 scripts/evaluate_runtime.py

# Run comprehensive benchmark execution
PYTHONPATH=src:. python3 scripts/run_benchmarks.py

# Run complete quality suite
pytest && ruff check . && ruff format --check . && mypy

---

📚 Documentation Index

Document	Description
Architecture Specification	In-depth breakdown of provider adapters, predictors, verifiers, schedulers, and store.
Safety Model & Invariants	Formal invariant specifications (I1–I5), resource namespaces, and hazard proofs.
Benchmarking Methodology	Standard workload designs (W1–W5), baseline configs (B0–B4), and latency metrics.
Runtime Evaluation Matrix	12-dimension verification matrix with empirical proof.
Research Positioning	Literature review and positioning relative to PASTE, SPORK, ToolSpec, and AsyncFC.

---

_{Built with strict typing, fail-closed safety, and performance-first engineering. Released under the MIT License.}