Skip to content

ci: releases/symlink deploy with health-check rollback and smoke-test gate - #441

Merged
saqibmanan merged 3 commits into
devfrom
ci/release-rollback-quality-gate
Aug 27, 2026
Merged

ci: releases/symlink deploy with health-check rollback and smoke-test gate#441
saqibmanan merged 3 commits into
devfrom
ci/release-rollback-quality-gate

Conversation

@saqibmanan

Copy link
Copy Markdown
Contributor

Summary

  • Replace the plain mv .next2 .next in-place swap with a releases/<timestamp-sha>/ + symlink layout, so a bad deploy can be rolled back instead of leaving the app half-updated.
  • npm ci (not npm install) plus a guard step that aborts the build if the installed Next.js version drifts from package.json — a prior incident let a different/canary Next.js version get silently resolved via plain npm install.
  • Each deploy extracts into its own releases/<release>/, installs deps there, re-verifies the Next version, then atomically flips the DataExFrontend symlink and runs a boot health check — auto-rolling back to the previous release if the health check fails.
  • New on dev: promote-dev/rollback-dev jobs make the existing smoke-tests job (CivicDataSpace-test's run-smoke.yml) an actual deploy gate — a release is only marked .last_good on a smoke-test pass, and reverted on failure. Production has no smoke-test job yet, so its only gate remains the boot health check.
  • Runtime secrets: shared/.env.local, symlinked into each release. Confirmed via pm2 env on the dev box that the app's real secrets file is .env.local inside DataExFrontend, not the .env one level up in ~/DataExchange/ (that one belongs to a separate docker-compose stack — DataExAuth/DataExBackend/DataExKeycloak).
  • Adds deploy/ec2-migrate-to-releases.sh, the one-time manual migration each EC2 box needs before its first deploy under this workflow.

Scope — dev only for now

This workflow file is shared between dev and main (environment-scoped only by GH Actions vars/secrets), but the two boxes are not equally ready:

Node 24.13.0 Migration run
dev ✅ installed ✅ done, verified shared/.env.local wired correctly
prod ❌ only v20.11.1 ❌ not run, still a plain directory

Do not merge devmain until prod has Node v24.13.0 installed via nvm and deploy/ec2-migrate-to-releases.sh has been run there. That's on the user to do, not something to automate/assume.

Test plan

  • actionlint on the workflow file — clean
  • bash -n on the migration script — clean
  • Verified live over SSH on the dev box: Node 24.13.0 present, migration already applied, DataExFrontend/.env.local correctly symlinked to shared/.env.local (1915-byte real file, no secrets missing)
  • First real deploy through this workflow on dev, to confirm the release/health-check/rollback flow end-to-end (not yet run)

… gate

- npm ci (not npm install) so the build always matches package-lock.json,
  plus a guard that aborts if the installed Next.js version drifts from
  package.json
- ship a versioned release.tar.gz, extract into releases/<release>/ on
  EC2, install deps there, verify the Next.js version again, then flip
  the DataExFrontend symlink atomically
- boot health check after activation; auto-rolls back to the previous
  release on failure
- dev: promote-dev/rollback-dev jobs make smoke-tests an actual deploy
  gate — release is only marked last-known-good on a smoke-test pass,
  and reverted on a smoke-test failure
- prod: no smoke-test job exists yet, so the boot health check is its
  only gate; release is marked good immediately after
- add deploy/ec2-migrate-to-releases.sh, the one-time manual migration
  from a plain DataExFrontend directory to the releases/ layout
The app's actual runtime secrets file on EC2 is .env.local (inside
DataExFrontend itself), not .env one level up in DataExchange/ - that
one belongs to a separate docker-compose stack (DataExAuth/DataExBackend/
DataExKeycloak). Confirmed via `pm2 env` showing no secrets in PM2's
own captured environment, and .env.local's size/presence matching
.env.local.example.

Without this fix the new releases/ workflow would have symlinked the
wrong file and every release would boot with no runtime config.
…ality-gate

# Conflicts:
#	.github/workflows/deploy-Dataspace.yml
@saqibmanan
saqibmanan deployed to development August 27, 2026 08:40 — with GitHub Actions Active
@saqibmanan
saqibmanan merged commit 5be47e4 into dev Aug 27, 2026
1 check passed
saqibmanan added a commit that referenced this pull request Aug 27, 2026
PR #441's deploy just failed: appleboy/ssh-action's non-interactive shell
never sources nvm, leaving PATH pointed at the ancient system node (v10).
npm and pm2 are both scripts with a '#!/usr/bin/env node' shebang, so
invoking them by absolute path wasn't enough - env still re-resolved
node via PATH and picked v10, which can't parse npm v24's node: imports
('Cannot find module node:path').

Failed before touching the DataExFrontend symlink, so the previous
release stayed live throughout - no downtime from this.

Fix: export NODE_BIN onto PATH before any npm/pm2 invocation, in both
the deploy workflow (build-and-deploy activation step, rollback-dev)
and the migration script. Verified live on dev-cds that node/npm/pm2 all
resolve to v24 with this PATH export.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant