Conversation
…at [RED-943]
`checkly deploy` now asks Checkly what the deploy would change before it
changes anything, prints that plan, and — once it is accepted — deploys the
plan it printed rather than whatever the account looks like by then. The plan
comes back with a token that identifies the state it was computed against; the
deploy sends that token, and Checkly refuses the deploy, having written
nothing, if anything moved in between.
The order of work changes as a result. Parsing and bundling happen first, the
plan is fetched, and only then does the command ask for confirmation — so the
one prompt of the command describes the actual deploy, naming every resource
that would be deleted, where the old prompt could only warn that deletion was
possible. Nothing is uploaded until the plan is accepted: the payload describes
the code bundle and every visual snapshot by content hash, which the CLI can
compute from the files on disk, so finding out what a deploy would change costs
no uploads. The consequence, stated plainly: in agent or CI mode without
`--force`, the CLI parses and bundles the project before printing the
`confirmation_required` envelope and exiting 2.
Payload additions, all computable before an upload: `sha256` per snapshot
(hashed where snapshots are detected), and `codeBundleSha256` for Playwright
suites, computed when the archive is finalized on disk and also sent as the
upload's `x-bundle-checksum-sha256` header. The bundle hash rides the same kind
of mutable marker as the bundle path, because the archive does not exist yet
when a check's payload is built; before it is filled in, the key is simply
absent rather than present and wrong.
New flags: `--plan-token <token>` deploys only if the plan still matches, and
aborts otherwise; `--prune-relations` deletes the alert channel subscriptions
and private location assignments on this project's checks and groups that the
project does not manage; `--dry-run` prints the shared `dry_run` envelope and
exits 0, like the eight other write commands. `CommandPreview` and the agent
envelope gain a structured `preview: { planToken, diff }` next to the
human-readable `changes` lines, and the echoed `confirmCommand` carries
`--plan-token`, so an approved re-run applies the plan the user was shown. The
envelope's copy of the plan drops each resource's full state and any value too
large to inline; the terminal output is where a rendered diff belongs.
Compatibility. An API without the preview endpoint answers 404: the CLI says so
and falls back to the older dry-run diff and its delete guard, sending no plan
token. A preview that fails for any other reason falls back the same way — one
resource Checkly cannot report on must not make a project undeployable — and
`--plan-token` and `--prune-relations` are refused whenever there is no plan,
the first because it cannot be checked and the second because nothing could say
what it would delete.
The payload is sent in its pre-preview-endpoint form in exactly the two cases
where the full one cannot be sent: an API that lacks the endpoint rejects the
fields that arrived with it (`sourceFile`, each snapshot's `sha256`,
`codeBundleSha256`), and a run that skipped the uploads describes snapshots it
has no storage key for, which every write route requires. Deliberately not for
every missing plan: a transient failure against a current API leaves those
fields perfectly acceptable, and stripping them would blank the stored hashes
and make the next deploy report every Playwright suite as changed.
A busy project answers 409, and the preview is retried up to three times,
honouring `retry-after` up to a cap. A deploy refused as `PLAN_STALE` is never
retried with the same token, since the deploy's own commit would invalidate it
anyway: an unattended run plans again and deploys that, and any other run prints
the current plan and says to re-run. Waiting out another deployment is treated
the same way — reaching a final state is what invalidates a token, so the CLI
says so instead of spending the payload on a certain refusal, unless it
cancelled that deployment itself or gave up waiting on one still running.
`DETACH` joins `DETACHED` in the rendered output, and `UNCHANGED` resources are
counted rather than listed as updates, so `Update and Unchanged:` becomes
`Update:` and `Unchanged: <n>`. Relations the project does not manage print
under their own `Prune` heading.
The agent-facing reference docs are corrected: the passage telling agents that
deploy's confirmation happens before parsing and therefore cannot name
deletions — true until this change — is replaced, and `deploy` is added to the
commands that pin a resolved target into the `confirmCommand`.
Includes AI-522's two CLI commits as its base (`sourceFile` on each resource,
resolved against the repository root), because this command's payload has to
carry that key and the backend now expects it there; drop them from this branch
if that PR lands first.
Two things the content hash had to earn. The archive is reproducible from its
content alone: every entry's timestamp is pinned, because a tar entry otherwise
carries the file's mtime — or, for an entry the bundler generates and for every
symlink, the clock — and a CI pipeline that clones fresh per run would report
every Playwright suite as changed on every deploy. Entry order is by code point
rather than `localeCompare`, so the bytes do not depend on the machine's locale.
Snapshots are re-hashed as they upload, since the confirmation now sits between
detection and upload, and a file that moved in between is named in a warning.
`checkly test` sends neither hash: its payloads are synthesized the same way,
but only a deploy compares them, and only the deploy schemas accept them.
Five review rounds ran, ending with no critical or major findings. What they
caught is worth recording: an unattended deploy pinned to a plan nobody had
reviewed (it now re-plans once rather than failing a pipeline because someone
touched the account mid-upload); `--cancel-in-progress-deployment` cancelling a
predecessor and then refusing to deploy; a plan token re-sent after waiting out
a predecessor, which that predecessor's commit had necessarily invalidated; a
resource labelled "Update" forever because someone had attached an alert channel
to it in the web app; the payload stripping keyed on a 404 rather than on what
the deploy route can actually accept; and the two halves of the archive hash
above. The minor findings taken after the last round — the double-reported
pruned relation, stripping that would have blanked the stored hashes after a
transient failure, the locale-dependent entry order, and the snapshot-drift
warning — are therefore unreviewed, and the spec's phase row says so.
Tests: the confirm flow end to end (one preview then one prompt, the plan and
token in the envelope, `--force`, `--preview`, `--dry-run`, a pinned token that
matches and one that does not, `--prune-relations` against an API without the
endpoint and against one with it, a failed preview deploying without a token, an
unattended re-plan that carries the new token, and a refused plan), the REST
layer (query strings, both flags, the 404 and 409 paths, the superseded-plan
rules, `PLAN_STALE` carrying the fresh plan, and the stripped legacy payload),
the check-run payload shape, and the content hashes in synthesized payloads
before and after the upload — including that the same content hashes the same
whatever the timestamps are, which fails without the pinning.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…RED-943] Two primitives for showing what a deploy would change to a resource: render the construct as Checkly currently has it and as the local code would make it, then diff the two. This commit adds only the primitives. Nothing calls them yet and no command behaves differently; the renderer that does is separate, so that the parts with no observable behaviour can be reviewed and read on their own. `renderConstruct(codegen, logicalId, resource)` renders a single construct to a string, writing nothing to disk and leaving out the scaffolding of the file it would otherwise live in: no generated-file header, and no import list, whose paths would be invented for a render that never becomes a file. Behind it, `GeneratedFile.render()` takes a `scaffolding` switch that defaults to today's behaviour, and `Program` exposes the construct files it has handed out, which is the only way to tell a construct apart from the script, snippet and stylesheet files the same `gencode` call registers beside it. Three things about it a caller has to know, all of them in its doc comment. It watches `prepare` and `gencode` together, because several codegens create and register their construct file in `prepare` and generate into whatever the registration returns. Errors from the codegens themselves are left exactly as they are, the way the import planner takes them — an unsupported resource type throws a plain `Error`, an unparseable script an `UnsupportedScriptError` — so a caller rendering for a reader catches `Error`, not only the `ConstructRenderError` raised here. And a script body is not part of a construct: codegens write it to a spec or support file and leave an `entrypoint` path behind, so two resources differing only in a check's script render to the same string, and a caller must read equal renders as "nothing I can show", never as "nothing changed". `unifiedDiff(before, after)` is a line diff over a longest common subsequence, formatted as a unified diff. It is written here rather than taken from a package: the inputs are a few dozen lines of generated code, the output is read by a person in a terminal, and this is a CLI that ships to users. Identical head and tail lines — the bulk of any real comparison — are matched off before the quadratic part runs and put back as context, so the line numbering needs no adjustment afterwards. It returns an empty array for two sides that are the same and `undefined`, never a throw, for a comparison it declines, because the caller has a coarser listing to fall back to. Both limits it declines on are needed: one on the lines, for output nobody would read, and one on the cells, because the cost is the product of the two sides and grows far faster than either. Headers and hunk numbering follow `git diff` and GNU `diff -u` — the spelling a reader recognises and the one their CI prints — down to omitting a side's line count when it is exactly 1. Every expected header in the tests was taken from `git diff --no-index` on the same two inputs. Worth recording, because the two disagree: BSD `diff`, and so the binary on a macOS developer's path, numbers an empty range at the very start of a side `1,0` where git says `0,0`. Where two sides can be aligned in more than one minimal way, which script you get is a matter of heuristics and this one's is not git's; the result is always a valid minimal diff of the same texts, but it may group its hunks differently. Deviation from the implementation spec, which puts the render-one-construct entry point in `sourcegen`: it lives beside `Codegen` instead. It has to know `Codegen` and `Context`, and importing those into sourcegen's barrel would make that barrel import the modules that import it. Sourcegen gains the piece it was actually missing — rendering a file's sections to a string. Two review rounds. The first confirmed nothing: every major finding described a failure in a caller this commit deliberately does not have, which is the argument for splitting it out. Its minors were worth taking, and one was a real breach of the stated contract — a single-line side was spelled `-1,1` where every diff spells it `-1`. The second round found no majors either, and its two minors were about the claim rather than the code: the numbering rule had been derived from the macOS binary rather than from git, and the script-body limitation above was undocumented. The numbering now follows git, checked against it over 400 generated cases spanning replace, delete and insert at every position at four context widths, matching byte for byte in headers and bodies. Tests: the diff against `git diff`'s own output (insertion, deletion, replacement, an empty side either way, context widths, hunk coalescing and the boundary where two changes stay separate, a trailing newline as a terminator rather than an empty line, and both decline limits including that the cell budget returns rather than throws); and the renderer (no scaffolding in the output, a reference resolved to the variable a context names and falling back to `fromId` when it does not, the same resource rendered identically twice, a script change rendering identically, and both arms of the one error it raises). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A deploy preview's `before` is the resource in the import format, the same
payload `checkly import` receives, so the codegen renders it as it is. The
local side is the payload this CLI synthesized for the deploy, which differs
from that format in a few known ways; `services/deploy-diff/import-shape.ts`
bridges exactly those: `id` from the plan's physical ids (a stable synthetic
one for a resource being created), every `{ ref }` under the reference keys
replaced by the target's physical id, deploy-only keys dropped, the agentic
runtime under the name the codegen reads. No canonicalization of nulls,
defaults or ordering: the local payload has none.
Relations are folded onto a check or group by registering them on the
codegen context, so both sides need theirs: the deployed side's are the
subscription and assignment rows its `before` carries, the local side's are
the project's own relation constructs plus every deployed row the deploy
keeps — a row the code removed (named by a DELETE or DETACH relation entry
folded into the parent) or that --prune-relations deletes is left out, and
one a local construct already stands for is not repeated.
`blankRedacted` applies the redaction rule table the API reports with
`before` — the resource type's whole table, each rule naming what it blanks
to — onto the local payload, so a credential is blank on both sides, to the
same placeholder, and never a phantom change nor printed. `registerProject`
registers every member construct under its physical id, so a reference
renders as the construct's variable on both sides; a `fromId(...)` reference
stays unregistered and renders as the `fromId(...)` it is. A payload that
cannot be shaped throws `UnshapeableError`, which the renderer turns into a
listing of the reported changes.
`rest/projects.ts` documents the new contract: `before` is the import format,
`redactions` lists the rules applied to it, and a moved secret is reported by
presence (`secret: true`) with no values.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…D-943] `checkly deploy --preview` and `--output` print, under each updated resource, a unified diff of the construct as Checkly has it against the construct as the local code would make it. Both sides are rendered by the import codegen — the deployed side from the preview's `before`, the local side from the payload shaped like an import resource — so style noise cancels and only real differences remain. Both flags now ask the preview for `detail=full`, which is what carries the deployed state. Per resource, the first of these applies: one `changed: <causes>` line when every change is a bundle or cache; the construct diff, followed by the text diff of any change outside the construct and a line for any change that is only a cause; when the two renderings agree and every change is one of the known shape changes, `payload format changed (CLI upgrade)` — the paths alone cannot tell an upgrade from an edit of the same property, so a real edit shows as construct lines first; when the renderings agree otherwise, a text diff per content change (a script, a body) read from `before` and the local payload; and a listing of the reported changes, with the reason, when a side cannot be shaped or rendered or the diff is too large. A moved secret is named after the block, never shown. A render failure falls back to the listing and can never fail a deploy. The local side also takes from the deployed row every property the payload leaves out and the plan does not report as changed: the defaults the deploy's validation fills into an absent property (`activated`, a check type's response-time limits, an empty header list), which the row always carries and the codegen prints. The plan compared the validated payload against the row, so an absent, unreported property is one the deploy leaves as it is; copying it to both sides cannot hide a change and keeps the CLI free of any copy of a backend default. A property a codegen writes to a file of its own (a browser or multi-step check's script, an API check's setup and teardown scripts, a dashboard's CSS) is invisible in the construct diff, so a change to one is shown as a text diff of its own, beside the construct diff when there is one. The rendered construct's own variable is named after its logical id on both sides, since a codegen names it after content (an alert channel after its address) that may be the very change. A deployed setup or teardown snippet reference is left out of the rendering: it resolves through files an import registers and a preview has not, and a deploy clears it. After a deploy, `--output` renders from the plan the deploy was confirmed against, joined to the deploy's own result for the ids it created. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…diff [RED-943] A check or group adopted from the web app can carry a setup or teardown snippet reference, which the codegen resolves through snippet files an import registers and a preview has not. The renderer dropped it from the deployed side only, then filled the local side from the unmodified deployed state — and a snippet reference is a bookkeeping column the plan never reports, so the fill copied it across and the codegen threw, turning the resource into the coarse listing. The reference is now dropped once, before either side reads the deployed state; `UNRENDERED_KEYS` sits with the other key lists in `import-shape.ts`. The CLI-upgrade spec case gets the fixture a real upgrade produces: the deployed row holds `doubleCheck: true` beside the retry strategy derived from it, the local construct sends the same strategy and no `doubleCheck`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The API withholds the list that holds a changed secret whole, as one `secret: true` change with no values, plain siblings' edits included. The renderer dropped secret changes before deciding whether to render, so a resource whose only reported change was that list printed nothing but the secret line, and an edited plain variable beside a rotated locked one was invisible; a bundle change beside it took the cause-only shortcut and hid it the same way. Such an entry now renders its construct diff, both sides blanked, with the cause lines after it. Rendered only when a reported redaction rule reaches every secret's path: the local side's blanks come from that table, and a gap in it must not print what the API withheld — the entry lists its other changes instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…RED-943] The API now reports every sensitive position of a change as a marker that says whether that element's secret moved. The preview renders a secret inline like any other property, its value masked as `'********'` on both sides — the deployed side's blanks are masked the same way, so a blank never passes for a value — and the element the report marks reads `'******** (changed)'` on the local side, or `'******** (changed in Checkly)'` on the deployed side, so the diff names the secret beside its key without ever holding a value. A mark is matched to the payload's element by key when unique in the report and the payload, by position only when the lists are equal in length and the key at that index is the same, and never by index alone: the deployed row can order its list differently from the report, and a name must never be wrong. Each change writes its own unguessable sentinel, and only a change whose sentinel reached the returned lines counts as shown; a secret change whose mark did not (an unmatched element, a position the rules blank to null, a block the codegen does not print, a response without markers) is still named as `secret changed: <path>`. A `secret: true` variable, which an import prints as a generated `secret()` reference, prints its masked value beside the flag under the preview's codegen context; the codegen prints a secret only as a string the preview itself wrote, so a value that merely looks masked prints as the plain mask. `checkly import` output is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
sorccu
force-pushed
the
simo/red-943-deploy-diff
branch
from
September 20, 2026 09:23
09d7a18 to
4c0395a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
checkly deploynow asks Checkly what a deploy would change before it writes anything, shows the plan, and pins the deploy to it.--forceskips the prompt but still plans, so an unchanged project deploys nothing.PLAN_STALE); a forced run then plans again, a pinned run (--plan-token) stops.--dry-runprints the plan as JSON for scripts and agents.--previewand--outputprint, under each updated resource, a unified diff of the construct as Checkly has it against the construct the local code would produce. Both sides render through the import codegen, so only real differences remain. Changes outside the construct (a script, a request body) print as text diffs; a moved secret is named, never shown; a CLI upgrade that only respells the payload is reported as such.--prune-relationsdeletes alert channel subscriptions and private location assignments added outside the project; without it they are kept and listed.Requires the backend preview endpoint. Against a backend without it, the CLI falls back to the previous listing and warns.
Commits
Seven commits, each reviewed separately: the preview and confirm flow, the single-construct renderer and unified diff, the import-shape adapter, the renderer wiring, a fix for deployed snippet references, a fix that renders the construct diff for a resource whose only reported change is a secret list, and the inline masked secret: an encrypted value renders as
'********'on both sides and the element the API marks as moved reads'******** (changed)'(or'******** (changed in Checkly)'on the deployed side), matched by key so the name is never wrong; a mark that cannot be shown inline falls back tosecret changed: <path>.Verification
Full unit suite (182 files, 3112 tests), eslint,
tsc --noEmit,test:types,prepare:distandprepackpass on the rebased branch. The end-to-end deploy test needs a live account and was not run.Related
Ticket: RED-943
🤖 Generated with Claude Code