Skip to content

Fix a test that had been failing for two weeks, and run the suite in CI - #27

Merged
michalakmarcinth merged 1 commit into
mainfrom
claude/eval-skill-check-dbe48l
Aug 24, 2026
Merged

Fix a test that had been failing for two weeks, and run the suite in CI#27
michalakmarcinth merged 1 commit into
mainfrom
claude/eval-skill-check-dbe48l

Conversation

@michalakmarcinth

@michalakmarcinth michalakmarcinth commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #26. Adding that docs job was the first time anything in this repo ran a test — release.yml only publishes. Pointing it at the whole suite turned up four failures on a clean checkout, and they weren't the same kind of thing.

Split out: the follow-on work that un-skips the framework integration tests (and fixes three more stale ones it uncovered) is #28, stacked on this branch.

Three were missing optional extras

test_span_tree.py and test_integrations.py import agentx.integrations.google_adk at call time. That module raises ImportError by design when google-adk is absent, so the tests failed instead of skipping — while their crewai, litellm, llamaindex and autogen neighbours in the same files guard the identical situation with pytest.importorskip.

They now do too. With the extra installed all three run and pass, so this skips what genuinely isn't installed rather than hiding anything.

The fourth was real — and stale by two weeks

test_trace_tool_call_emits_real_child_span asserted the root trace carries no flat tool_calls list when a tool call became a real child span:

assert "tool_calls" not in root or root.get("tool_calls") in (None, [])

That was true when the test was written on 2026-08-07. Four days later, 40c6f6e deliberately added the dual-write — the code says why:

The child span above is only for the trace detail's span tree — the engine's built-in "Tool failure" check and the dashboard's Tool quality column read the ROOT trace's flat tool_calls list, so a summary lands there too... the reason a failed trace_tool_call() used to be invisible to both surfaces.

The behaviour is right; the assertion was stale. It had been failing since 2026-08-11 — unnoticed, because no CI ran it. "Fixing" the code to satisfy it would have re-broken the Tool failure check and the Tool quality column.

So the assertion is corrected, and corrected upward:

assert [tc["name"] for tc in root["tool_calls"]] == ["policy_lookup"]
assert root["tool_calls"][0]["output"] == "digital purchases are final"

It now pins the dual-write positively and pins it at exactly one entry. Verified against a modified tree: removing the dual-write fails it, and writing a third copy fails it too — neither of which the old assertion could catch, since it only checked for absence.

Widening the CI

docs.yml becomes tests.yml and runs the whole suite. The old name stopped being true once it covered more than the docs.

Install Result
base (.[langchain]) 73 passed, 28 skipped, 0 failed
with google-adk 76 passed, 25 skipped, 0 failed

Optional-extra tests skip themselves, so the base install is green — a skip is honest about not being covered, where a failure would have drowned out real ones. #28 takes the next step and actually runs them. test_integration.py talks to the hosted API and skips itself without a key; no secret is passed on purpose, so fork PRs behave the same.

Verification

  • Exact CI command run on both paths — with and without google-adk
  • The 3 adk tests confirmed to run and pass with the extra, and to skip with a clear reason without it (not silently vanish)
  • The corrected assertion confirmed to fail in both directions it's meant to catch
  • tests.yml validated as YAML
  • Eval-skill audit re-run against the freshly published 0.6.37 — clean, 29 breaks still caught, no regression from the release

Tests and CI only; no library code changed, so no version bump is warranted by the diff.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WYi6akpDa8qfiojRed2cAE

Adding the docs job yesterday was the first time anything in this repo ran a
test. Pointing it at the whole suite turned up four failures on a clean
checkout, and they were not the same kind of thing.

Three were tests/test_span_tree.py and tests/test_integrations.py importing
agentx.integrations.google_adk at call time. That module raises ImportError by
design when google-adk is absent, so the tests failed instead of skipping -
while their crewai, litellm, llamaindex and autogen neighbours in the same
files guard the same situation with pytest.importorskip. They now do too. With
the extra installed all three run and pass, so this skips what is genuinely not
installed rather than hiding anything.

The fourth was real. test_trace_tool_call_emits_real_child_span asserted the
root trace carries no flat tool_calls list when a tool call became a real child
span. That was true when the test was written on 2026-08-07; four days later
40c6f6e deliberately added the dual-write, because the engine's built-in "Tool
failure" check and the dashboard's Tool quality column read the root's flat
list and a failed trace_tool_call() was invisible to both. The behaviour is
right and documented in the code; the assertion was stale, and had been failing
ever since - unnoticed, because no CI ran it.

So the assertion is corrected rather than the code, and corrected upward: it
now pins the dual-write positively and pins it at exactly one entry. Removing
the dual-write fails it, and writing a third copy fails it too - neither of
which the old assertion could catch, since it only checked for absence.

docs.yml becomes tests.yml and runs the whole suite; the old name stopped being
true once it covered more than the docs. Optional-extra tests skip themselves,
so the base install is green: 73 passed, 28 skipped without extras, 76 passed,
25 skipped with google-adk. Installing the extras to actually run those is a
fair next step and a slower job, not a different one.

Verified on both paths with the exact CI command, and the corrected assertion
verified to fail against a tree with the dual-write removed and against one
that writes three times.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WYi6akpDa8qfiojRed2cAE
@michalakmarcinth
michalakmarcinth merged commit 7e167d0 into main Aug 24, 2026
2 checks passed
@michalakmarcinth michalakmarcinth changed the title Fix a test that had been failing for two weeks, and run the suite in CI Run the suite in CI, and fix the four stale tests that turned up Aug 24, 2026
@michalakmarcinth michalakmarcinth changed the title Run the suite in CI, and fix the four stale tests that turned up Fix a test that had been failing for two weeks, and run the suite in CI Aug 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants