[fix] Repair web acceptance tests and shorten preview CI

[mmabrouk]

Context

Preview web acceptance tests repeatedly waited for controls that changed during the UI migration. Retries consumed the 45-minute job limit, and cancellation left us without the browser traces needed to diagnose the failures.

Changes

Use accessible control names and checked states instead of old Ant Design classes. Configure the existing mock provider before the attachment test, and make app-creation waits target the actual confirmation dialog.

Do not reuse seed-only agents as completion prompts. Keep the configured-revision requirement and cover the cross-test fixture bug with a regression test. Check evaluation selection summaries after the dialog advances to the next tab.

Compare deleted scenario IDs without assuming database return order. This fixes the API acceptance failure while preserving the ID, count, and repeated-delete checks.

Keep all acceptance tests, with one retry instead of two. Stop after five failures or 20 minutes of browser tests, inside a 30-minute job. Preserve the configured HTML/JUnit reporters and upload raw results even for interrupted runs, excluding browser authentication state.

Tests

• Acceptance discovery: 80 tests across 21 files.
• Frontend lint: all 25 tasks passed. Pre-commit formatting, lint, and secret scan passed.
• Observability UI package build and lint passed.
• Fixture regression: failed before the fix, then all 5 helper unit tests passed.
• Targeted browser checks against the live preview passed for attachment upload, prompt creation, human/auto evaluation, testset editing and row/column changes, and both API snippet flows.
• Full web acceptance on final commit e2741843bb: 50 passed, 0 failed, 30 skipped; no retries needed. Browser tests took 12.4 minutes. No tests were removed or newly skipped by this PR.
• API acceptance: 757 passed, 30 skipped, zero failures.
• All CI checks are green on the final commit, including the complete preview build, deployment, and test matrix.

How to review

Start with the acceptance helpers and selectors, then review the Playwright limits and Railway reporting changes. This PR starts from release/v0.114.8 and does not change application behavior.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
agenta-documentation	Ready	Preview	Sep 5, 2026 8:35am UTC

[mmabrouk]

🤖 The AI agent says: @coderabbitai review. This fixes the diagnosed preview acceptance failures and bounds retries/runtime without removing test coverage. Local formatting, lint, package build, and test discovery passed; live preview validation is pending.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Team

Run ID: 798103dc-8fef-414e-930c-4354c45da8ba

📥 Commits

Reviewing files that changed from the base of the PR and between 93f703f and e274184.

📒 Files selected for processing (1)

• api/oss/tests/pytest/acceptance/evaluations/test_evaluation_scenarios_basics.py

Included review availability: Your plan provides up to 8 included reviews per hour; 6 remain after this review.

---

📝 Summary

Summary by CodeRabbit

• Bug Fixes

  • Improved end-to-end test reliability for attachment scenarios, evaluation selection, observability, and creation workflows.
  • Corrected completion-prompt matching for apps containing only seed revisions.

• Tests

  • Updated automated tests to use accessible labels and stable identifiers.
  • Added validation for enabled attachment controls and selected options.

• Chores

  • Optimized CI test execution with shorter timeouts, controlled retries, failure limits, and artifact uploads for interrupted runs.

Walkthrough

The pull request updates Playwright CI execution, test setup, accessible selectors, selection assertions, and trace-tree targeting. It also changes artifact collection to include successful and interrupted runs.

Changes

Playwright reliability updates

Layer / File(s)	Summary
CI execution and artifact handling .github/workflows/44-railway-tests.yml, web/tests/playwright.config.ts	CI retries, timeouts, failure limits, reporters, and result artifact handling were updated.
Test setup and creation flow web/oss/tests/playwright/acceptance/agent-chat/attach-send-render-reload.spec.ts, web/oss/tests/playwright/acceptance/app/test.ts, web/oss/tests/playwright/acceptance/prompts/test.ts, web/tests/tests/fixtures/base.fixture/apiHelpers/index.ts, web/oss/tests/playwright/unit/api-helpers.spec.ts, api/oss/tests/pytest/acceptance/evaluations/test_evaluation_scenarios_basics.py	Tests ensure a provider exists, verify attachment-button state, identify the create dialog by name, coordinate creation actions with response waits, validate seed-revision matching, and compare scenario IDs without relying on order.
Accessible selectors and state assertions web/oss/tests/playwright/acceptance/auto-evaluation/tests.ts, web/oss/tests/playwright/acceptance/human-annotation/tests.ts, web/oss/tests/playwright/acceptance/testsset/testset-management.ts, web/oss/tests/playwright/acceptance/use-api/index.ts, web/oss/tests/playwright/acceptance/playground/tests.ts	Acceptance tests replace Ant Design-specific selectors with roles, labels, stable row locators, and explicit selection-state assertions.
Trace-tree test hook web/packages/agenta-observability-ui/src/traceDrawer/TraceTree.tsx, web/oss/tests/playwright/acceptance/observability/index.ts	The trace tree now exposes data-testid="trace-tree", and observability tests verify the selected span, accessible controls, and radio state.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Merge Risk: 🟡 Moderate · up to e2741

This updates acceptance-test reliability and makes evaluation scenario deletion assertions order-independent, but two acceptance flows may still validate the wrong revision or pass without proving the intended row was selected. Resolve those test assertions before merge to retain reliable coverage.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 60.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 14 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the two main changes: repairing web acceptance tests and reducing preview CI duration.
Description check	✅ Passed	The description directly explains the test repairs, CI limits, reporting changes, regression coverage, and validation results.

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/web-acceptance-1148

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

@mmabrouk I will review the changes in #6566.

⚠️ Action not completed

Already reviewed the last commit. Use @coderabbitai full review to rerun a review of the entire changeset.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[github-actions]

Railway Preview Environment

Status	Destroyed (PR closed)

Updated at 2026-09-05T09:00:45.107Z

[coderabbitai]

Actionable comments posted: 2

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Team

Run ID: 372efaa9-5a11-408a-bab4-f98527fabf52

📥 Commits

Reviewing files that changed from the base of the PR and between 17b3b77 and 93f703f.

📒 Files selected for processing (8)

• web/oss/tests/playwright/acceptance/app/test.ts
• web/oss/tests/playwright/acceptance/auto-evaluation/tests.ts
• web/oss/tests/playwright/acceptance/human-annotation/tests.ts
• web/oss/tests/playwright/acceptance/prompts/test.ts
• web/oss/tests/playwright/acceptance/testsset/testset-management.ts
• web/oss/tests/playwright/acceptance/use-api/index.ts
• web/oss/tests/playwright/unit/api-helpers.spec.ts
• web/tests/tests/fixtures/base.fixture/apiHelpers/index.ts

Included review availability: Your plan provides up to 8 included reviews per hour; 6 remain after this review.