Found by: examples/agents/90_guardrail_e2e_tests.py, the repo's own guardrail E2E harness
Symptom: The harness runs 27 cases (tool INPUT/OUTPUT × Regex/LLM/Custom × RETRY/RAISE/FIX) and asserts each outcome. 24 pass, 3 fail:
# 8 aout_custom_raise: expected FAILED, got COMPLETED
# 9 aout_custom_fix: missing 'REDACTED'
#11 tin_regex_raise: expected FAILED, got COMPLETED
Cause: Two distinct enforcement gaps.
- RAISE does not fail the execution. A guardrail configured to raise lets the workflow complete instead. Affects a custom guardrail on tool output (
# 8) and a regex guardrail on tool input (#11), so it is not specific to one guardrail type or one direction.
- FIX does not redact. The fixed output is missing the expected
REDACTED marker (# 9).
A guardrail that silently fails to enforce is worse than no guardrail: callers believe the check ran.
Reproduce:
CONDUCTOR_AGENT_LLM_MODEL=openai/gpt-4o python examples/agents/90_guardrail_e2e_tests.py
Takes about 5 minutes; it prints a per-case table and a FAILURES section. The harness prints nothing between cases, so any timeout under ~5 minutes kills it mid-run and looks like a hang — that is how this went unnoticed.
Verify: All 27 cases pass.
Note: Each failing case prints its workflow id, so the server-side execution can be inspected directly.
Found by:
examples/agents/90_guardrail_e2e_tests.py, the repo's own guardrail E2E harnessSymptom: The harness runs 27 cases (tool INPUT/OUTPUT × Regex/LLM/Custom × RETRY/RAISE/FIX) and asserts each outcome. 24 pass, 3 fail:
Cause: Two distinct enforcement gaps.
# 8) and a regex guardrail on tool input (#11), so it is not specific to one guardrail type or one direction.REDACTEDmarker (# 9).A guardrail that silently fails to enforce is worse than no guardrail: callers believe the check ran.
Reproduce:
Takes about 5 minutes; it prints a per-case table and a FAILURES section. The harness prints nothing between cases, so any timeout under ~5 minutes kills it mid-run and looks like a hang — that is how this went unnoticed.
Verify: All 27 cases pass.
Note: Each failing case prints its workflow id, so the server-side execution can be inspected directly.