test: make the live Rollbar API specs conditional on ROLLBAR_TOKEN - #100
Open
resparza95 wants to merge 4 commits into
Open
test: make the live Rollbar API specs conditional on ROLLBAR_TOKEN#100resparza95 wants to merge 4 commits into
resparza95 wants to merge 4 commits into
Conversation
The rollbar-client suite runs its ping/createItem/reportDeploy specs against the real Rollbar API, so the whole CI turned red when the Rollbar account behind the ROLLBAR_TOKEN secret was deactivated. Those specs are contract checks (the API accepted our payload and the response parsed), not logic coverage, so they are now marked pending when ROLLBAR_TOKEN is unset or empty and run unchanged when it is set. The wai and yesod specs never call the API but read the token through rollbar.yaml anyway; they now build their Settings inline, so the whole test suite runs without any Rollbar credentials. The workflow stops injecting the deactivated secret; a comment documents how to re-enable the live specs once an active account exists. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A missing secret renders as an empty string, which the test suite already treats as "skip the live API specs", so the workflow never needs editing again: deleting the stale secret disables them, setting a valid one re-enables them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
GitHub retired the macos-13 image on 2025-12-04, so jobs targeting it queue forever. macos-latest runs on Apple Silicon, for which all the tested GHC versions ship bindists. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The spec asserted on an IORef written by the forkIO'd reporter thread after a 500 microsecond sleep, which lost the race on a busy runner (it just failed on the macos-latest CI job). The fake createItem now fills an MVar the test blocks on with a five second timeout, so the positive case is deterministic and finishes as soon as the item lands. tryPutMVar keeps the first reported item, so a second connection-level exception from warp can no longer overwrite the one under test. The negative case keeps a delay, raised to 100ms, before asserting the MVar is still empty. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
resparza95
marked this pull request as ready for review
August 20, 2026 15:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CI currently fails on every branch because the Rollbar account behind the
ROLLBAR_TOKENsecret has been deactivated ("This account has been deactivated. To reactivate it, log in to Rollbar and choose a plan."), and the rollbar-client suite runsping/createItem/reportDeployagainst the real API. This PR makes CI independent of that account, as discussed in #open-source:liveApiSpecsand marked pending whenROLLBAR_TOKENis unset or empty; with a token set they run exactly as before. They're contract checks (Rollbar accepted the payload, the response parsed) rather than logic coverage, so skipping them doesn't lose unit coverage.rollbar.yamlanyway, which made them crash without the env var; they now buildSettingsinline (same pattern ClientSpec already used), so the entire test suite runs with no Rollbar credentials.${{ secrets.ROLLBAR_TOKEN }}unconditionally — a missing secret renders as an empty string, which the suite treats as "skip". Enabling/disabling the live specs is therefore pure secret management; the workflow never needs editing again.One admin action required
The stale
ROLLBAR_TOKENsecret must be deleted (Settings → Secrets → Actions) for the skip to kick in — while it exists, the live specs still run against the deactivated account and fail. When a new Rollbar account is created, setting the secret to itspost_server_itemtoken re-enables the contract checks with no code changes.Verified
On GHC 9.8.4:
cabal test allwithROLLBAR_TOKENunset → all four suites pass, client reports1 pending (ROLLBAR_TOKEN is not set); with a token set, the live specs run (verified via thepingspec).Unblocks #99 once merged (it will need a rebase/merge from main). Getting a new free Rollbar account to re-enable the contract checks is tracked separately.
🤖 Generated with Claude Code