v1.7: Postgres backend with verified SQLite migration

.env.example

@@ -6,6 +6,31 @@ JWT_SECRET=
 
 PORT=3000
 
+# --- Database ---
+# Set this to use Postgres (recommended). Leave blank to keep using the
+# SQLite file at DB_PATH.
+#
+# On first boot with this set, if the SQLite database still exists and the
+# Postgres database is empty, the server migrates your data across
+# automatically, verifies it, and only then starts serving. Your SQLite file
+# is never modified or deleted - to roll back, blank this out and restart.
+#
+# Either postgresql:// or postgres:// works - they are the same scheme to
+# the driver. The host must not be `localhost` from inside a container - use
+# the Postgres container/host's own address (e.g. its Docker Compose service
+# name, or the host's LAN IP on Unraid).
+DATABASE_URL=
+
+# Where the SQLite database lives. Still used as the migration source even
+# after DATABASE_URL is set, so do not remove the volume mapping.
+#
+# Left commented on purpose. The default is <repo>/data/rackstack.db, which
+# is what you want for local development; /app/data only exists inside the
+# container, where the Dockerfile sets DB_PATH itself. Setting the container
+# path here would make `npm run dev` fail at startup, because the SQLite
+# driver creates the parent directory and a normal user cannot mkdir /app.
+# DB_PATH=/app/data/rackstack.db
+
 # Comma-separated user ids (provider:providerId, e.g. github:37058311,
 # discord:123456789012345678) that are always treated as owning every role
 # (admin + event_coordinator), regardless of what's stored in the DB. This

.github/workflows/test.yml

@@ -0,0 +1,30 @@
+name: Tests
+on: [push, pull_request]
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        backend: [pg, sqlite]
+    services:
+      postgres:
+        image: postgres:16
+        env:
+          POSTGRES_PASSWORD: postgres
+          POSTGRES_USER: postgres
+          POSTGRES_DB: postgres
+        ports: ['5432:5432']
+        options: >-
+          --health-cmd pg_isready --health-interval 10s
+          --health-timeout 5s --health-retries 5
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+      - run: npm ci
+      - run: npm test
+        env:
+          TEST_BACKEND: ${{ matrix.backend }}
+          TEST_DATABASE_URL: postgresql://postgres:postgres@localhost:5432/postgres

CHANGELOG.md

@@ -1,5 +1,47 @@
 # Changelog
 
+## v1.7.0
+
+Postgres support, with automatic migration from SQLite.
+
+- **Postgres backend**: `DATABASE_URL` selects Postgres as the persistence
+  backend instead of the local SQLite file. `server/db.js` fronts one async
+  repository interface (`server/db/index.js`) implemented by two drivers -
+  `server/db/driver.pg.js` and `server/db/driver.sqlite.js` - so every
+  caller (routes, services, minigames) is backend-agnostic. SQLite remains
+  fully supported and is still the default when `DATABASE_URL` is unset.
+- **Automatic migration on boot**: with `DATABASE_URL` set, if a SQLite
+  database still exists and the target Postgres database is empty, the
+  server migrates every table across in a single transaction, verifies row
+  counts match, and only then starts serving. Your SQLite file is never
+  modified or deleted, so rolling back is just unsetting `DATABASE_URL` and
+  restarting. If verification fails, the container refuses to start on
+  purpose rather than serve an empty game over live save data - the log
+  names the table that failed. The same logic is available standalone via
+  `npm run migrate:pg`.
+- **Per-user request serialization**: save updates are a read-modify-write
+  (load, evaluate, persist). Under SQLite those three steps used to run in a
+  single uninterruptible turn, because every database call was synchronous;
+  making the interface async removed that guarantee. Requests for one account
+  are now queued behind each other explicitly, so two open tabs can't load the
+  same state and have one overwrite the other's progress. Different players
+  never block each other.
+- **`identities` table**: authentication identity records (provider +
+  provider id) are now split from `users` into their own table, in
+  preparation for the SuperTokens migration planned for v1.8.
+- **Two-backend test matrix**: `npm run test:all` runs the full suite against
+  both SQLite and Postgres; CI does the same. A Postgres test harness
+  (`tests/setup/pg-global.js`, `tests/helpers/backend.js`) provisions an
+  isolated database per test file via Testcontainers (or a Docker/Podman
+  service container in CI).
+- **Operator note**: whether you're on SQLite or have moved to Postgres,
+  keep the `/app/data` volume mapping in place. It is the migration source
+  on first cutover and your rollback path afterward - removing it is the one
+  irreversible mistake in the whole process. See the
+  [migration runbook](docs/postgres-migration-runbook.md) or the README's
+  [Migrating from SQLite to Postgres](README.md#migrating-from-sqlite-to-postgres)
+  section for the full walkthrough.
+
 ## v1.6.0
 
 Onboarding & quality of life: a guided tour for new and existing players,

Dockerfile

@@ -33,12 +33,18 @@ COPY --from=client-build /app/client/dist ./client/dist
 
 ENV NODE_ENV=production
 ENV PORT=3000
+# Still the default so a container without DATABASE_URL behaves exactly as
+# before, and so the migrator knows where to find the source database.
 ENV DB_PATH=/app/data/rackstack.db
 
 LABEL org.opencontainers.image.source="https://github.com/NeverEndingCode/rackstack-server"
 LABEL org.opencontainers.image.description="RackStack self-hosted server"
 LABEL org.opencontainers.image.licenses="MIT"
-LABEL org.opencontainers.image.version="1.6.0"
+# The GHCR publish workflow (.github/workflows/docker-publish.yml) triggers
+# only on a pushed vX.Y.Z tag, and docker/metadata-action derives the
+# published image's version label from that tag - so this literal only
+# affects locally-built images, not what GHCR publishes.
+LABEL org.opencontainers.image.version="1.7.0"
 
 VOLUME ["/app/data"]
 EXPOSE 3000

README.md

@@ -1,8 +1,14 @@
 # RACKSTACK server
 
-Self-hosted version of the game: Discord/GitHub OAuth login, SQLite for
-persistence, and a server-authoritative economy (hard-capped at 72 hours of
-offline progress regardless of upgrades).
+Self-hosted version of the game: Discord/GitHub OAuth login, Postgres or
+SQLite for persistence, and a server-authoritative economy (hard-capped at 72
+hours of offline progress regardless of upgrades).
+
+As of v1.7, Postgres is a supported (and recommended) persistence backend
+alongside SQLite, selected by whether `DATABASE_URL` is set. Existing SQLite
+installs migrate across automatically and losslessly on first boot with
+`DATABASE_URL` set - see
+[Migrating from SQLite to Postgres](#migrating-from-sqlite-to-postgres) below.
 
 As of v1.3, a Cold Storage tab unlocks at the Server Room tier: a 16-block
 passive reward track that refills on a timer, one offline-only archival job
@@ -32,10 +38,12 @@ as you play. A daily login streak sits in the header. See
   separate client copy to keep in sync.
 - `server/` - Express API. Passport handles the Discord/GitHub OAuth
   handshake; on success we issue our own JWT in an httpOnly cookie (no
-  server-side session store needed). SQLite (`better-sqlite3`) persists
-  users, saves, roles, a versioned tunables config, and minigame sessions.
-  The client no longer computes or stores the economy itself - it dispatches
-  actions to `POST /api/actions` and renders whatever `GET /api/state`
+  server-side session store needed). `server/db.js` fronts one async
+  repository interface backed by either Postgres (`pg`, recommended) or
+  SQLite (`better-sqlite3`) - selected by whether `DATABASE_URL` is set -
+  persisting users, saves, roles, a versioned tunables config, and minigame
+  sessions. The client no longer computes or stores the economy itself - it
+  dispatches actions to `POST /api/actions` and renders whatever `GET /api/state`
   returns, with offline gain computed lazily on load rather than by an
   always-on background worker. As of v1.2 the old client-computed save flow
   (`GET`/`POST`/`DELETE /api/save`) is gone - if you had anything external
@@ -93,21 +101,30 @@ Users.
 docker compose up -d --build
 ```
 
-This builds the client, starts the server on port 3000 (mapped in
-`docker-compose.yml` - change the host side if you want a different port),
-and persists the SQLite file to `./data/rackstack.db` via a bind-mounted
-volume, so it survives container rebuilds.
+This builds the client, starts a `postgres:16` container plus the server on
+port 3000 (mapped in `docker-compose.yml` - change the host side if you want
+a different port), and persists Postgres's data under `./pgdata`. The
+`./data:/app/data` mapping is also present for the server - on a fresh
+install nothing uses it, but keep it mapped anyway: if you later point an
+existing SQLite-backed install at this compose file, it is the migration
+source and your rollback path (see
+[Migrating from SQLite to Postgres](#migrating-from-sqlite-to-postgres)).
 
 Point your reverse proxy / Cloudflare tunnel at `http://<host>:3000`.
 
 ## Upgrading
 
-Back up `rackstack.db` before upgrading across a major/minor version (e.g.
-v1.1.x -> v1.2.x): stop the container, copy the file
-(`./data/rackstack.db` for Docker Compose, or
-`<data path>/rackstack.db` for Unraid), then start the upgraded container.
-The database uses WAL mode, so copying it while the server is still running
-can grab an inconsistent snapshot - stopping first avoids that.
+If you're running on Postgres, back it up with your usual `pg_dump` practice
+before upgrading; nothing below is Postgres-specific.
+
+If you're still on SQLite, back up `rackstack.db` before upgrading across a
+major/minor version (e.g. v1.1.x -> v1.2.x): stop the container, copy **all
+three** `rackstack.db*` files - `rackstack.db`, `rackstack.db-wal`,
+`rackstack.db-shm` (recent progress lives in the `-wal` file, so copying only
+`rackstack.db` can lose it) - from `./data/` for Docker Compose or
+`<data path>/` for Unraid, then start the upgraded container. The database
+uses WAL mode, so copying it while the server is still running can grab an
+inconsistent snapshot - stopping first avoids that.
 
 That said, upgrading in place should just work without a backup too: v1.1
 saves are migrated to the current shape automatically and losslessly the
@@ -136,19 +153,58 @@ as a reference. Two things matter for updates to be safe:
 
 - **Data path** must be a stable host path (e.g.
   `/mnt/user/appdata/rackstack-server/data`) mapped to the container's
-  `/app/data` - this holds the entire SQLite database (saves + users) and is
-  untouched by "Apply Update," since that only swaps the image and reuses the
-  existing volume/variable config.
+  `/app/data`, and left in place even after moving to Postgres - it is the
+  migration source on first cutover and your rollback path afterward. On
+  SQLite it also holds the entire database (saves + users). It is untouched
+  by "Apply Update," since that only swaps the image and reuses the existing
+  volume/variable config.
 - **`JWT_SECRET`** must be set once as a container Variable and never changed
   afterward - it signs the 90-day login cookie, so rotating it logs every
   user out (no data loss, just re-login required). The other OAuth variables
   mirror `.env.example`.
+- **`DATABASE_URL`** (optional, recommended) points at a Postgres database
+  instead of the local SQLite file - see
+  [Migrating from SQLite to Postgres](#migrating-from-sqlite-to-postgres)
+  below.
 
 Once installed this way, updates are just Unraid's Docker tab -> "Check for
 Updates" / "Apply Update" whenever a new `:latest` digest is published.
 
-**Cutting a release:** bump `version` in `package.json` and
-`client/package.json`, commit, then:
+### Migrating from SQLite to Postgres
+
+Full runbook (backup, cutover, verification, rollback):
+[`docs/postgres-migration-runbook.md`](./docs/postgres-migration-runbook.md).
+The two things operators most often get wrong:
+
+- **Back up all three `rackstack.db*` files**, not just `rackstack.db`.
+  Recent progress lives in the `-wal` file - copying only the `.db` is the
+  most likely way to lose data during this migration.
+- **Leave the `/app/data` volume mapping in place** after setting
+  `DATABASE_URL`. It is the migration source and your rollback path;
+  removing it is the one irreversible mistake in the whole process.
+
+Short version: add a `postgres:16` container with its own appdata path and a
+`rackstack` database, stop rackstack, back up as above, set `DATABASE_URL`
+(the host must not be `localhost` from inside a container), and start
+rackstack. Watch the log for `[migrate]` - you should see a verified row
+count for each table, then `committed`. If verification fails the container
+refuses to start on purpose, so it never serves an empty game over your save
+data; your SQLite data is untouched either way.
+
+To roll back, blank out `DATABASE_URL` and restart — but note where that
+variable actually lives for your deployment:
+
+| Deployment | Where to blank `DATABASE_URL` |
+|---|---|
+| Unraid / plain `docker run` | The container's Variable in the Unraid UI (or the `-e` flag) |
+| Docker Compose | `.env` — `docker-compose.yml` reads it via `${DATABASE_URL:-...}` |
+| Local `npm start` | `.env` |
+
+**Cutting a release:** bump `version` in `package.json` (the single release-
+version authority - `client/vite.config.js` reads it for `__APP_VERSION__`,
+and `client/package.json`'s own version is deliberately not kept in sync),
+update `CHANGELOG.md` and the Dockerfile's
+`org.opencontainers.image.version` label, commit, then:
 
 ```bash
 git tag vX.Y.Z
@@ -175,6 +231,46 @@ npm run dev
 
 Visit the client dev server's printed URL (usually `http://localhost:5173`).
 
+## Running the tests
+
+```bash
+npm test          # Postgres backend (default) - boots a throwaway container
+npm run test:sqlite   # SQLite backend, no container needed
+npm run test:all      # both, sqlite then pg
+```
+
+The Postgres backend needs a container runtime that speaks the Docker API.
+`tests/setup/pg-global.js` boots one shared Postgres 16 container via
+[Testcontainers](https://node.testcontainers.org) and each test file carves
+out its own database from it (`tests/helpers/backend.js`), so files can't
+see each other's rows.
+
+- **Docker**: works out of the box, nothing to configure.
+- **Podman** (what this repo's containers were validated against; no
+  `docker` binary required): start the user socket once per login session
+  and Testcontainers will find it automatically -
+  `tests/setup/pg-global.js` points `DOCKER_HOST` at the Podman socket
+  itself if `DOCKER_HOST` isn't already set, so no per-developer config is
+  needed:
+
+  ```bash
+  systemctl --user start podman.socket
+  ```
+
+  Rootless Podman can't grant Testcontainers' Ryuk reaper the privileges it
+  wants, so the harness also sets `TESTCONTAINERS_RYUK_DISABLED=true` by
+  default when using Podman. With Ryuk off, the container is stopped and
+  removed by `teardown()` in `pg-global.js` at the end of the run instead -
+  if a run is killed hard enough to skip that (e.g. `SIGKILL`), clean up any
+  leftovers with `podman ps -a` / `podman rm -f`.
+- **CI** sets `TEST_DATABASE_URL` directly against a Postgres service
+  container (see `.github/workflows/test.yml`) and never touches
+  Testcontainers at all.
+
+To point manually at a different runtime or disable Ryuk yourself, set
+`DOCKER_HOST` and/or `TESTCONTAINERS_RYUK_DISABLED` before running the
+tests - the harness only fills these in when they're unset.
+
 ## Live Events
 
 At most one event is active globally at a time. Each is a set of tunable
@@ -246,28 +342,20 @@ introduces a new currency.
 
 ## Notes / things worth knowing
 
-- **SQLite, not Postgres**: chosen for zero-config, single-file persistence
-  that's easy to back up - see [Upgrading](#upgrading) for the safe way (stop
-  the container, then copy the file; it runs in WAL mode, so a bare `cp`
-  against a live server can grab an inconsistent snapshot). If you need a
-  backup without stopping the server, use SQLite's own online-safe backup
-  command instead: `sqlite3 data/rackstack.db ".backup data/rackstack.db.bak"`.
-  Fine for a personal or small-group deployment. If you outgrow it (many
-  concurrent users, wanting replication, etc.), `server/db.js` is still the
-  only module that touches the database, but the porting surface is its full
-  export list now, not a handful of functions: saves (`getSave`/`putSave`,
-  consumed by `stateService.js` and, for `putSave`, also `routes/api.js`'s
-  minigame handler; `deleteSave` is exported but currently unused), users
-  (`upsertUser` in `auth.js`; `getUserById`, `getAllUsersWithSaves`,
-  `setUsername` in `routes/api.js`), roles (`getRoles` in both `auth.js` and
-  `routes/api.js`; `setRoles` in `routes/api.js`), the tunables config
-  (`getConfigRow`/`putConfigRow`/`getConfigHistory`, all consumed by
-  `configService.js`), minigame sessions (`createMinigameSession`/
-  `getMinigameSession`/`getOpenMinigameSession`/`finishMinigameSession`, all
-  consumed by `routes/api.js`), and `dedupeUsernames`, which db.js calls on
-  itself at boot rather than exposing to a caller. Swapping in a `pg`
-  version behind the same signatures still wouldn't require touching those
-  callers, but it's a bigger module to port than it used to be.
+- **Postgres or SQLite**: `server/db.js` fronts one async repository
+  interface (`server/db/index.js`) implemented by two drivers -
+  `server/db/driver.pg.js` and `server/db/driver.sqlite.js` - selected by
+  whether `DATABASE_URL` is set. Postgres is recommended; SQLite remains
+  fully supported for zero-config, single-file personal deployments. On
+  SQLite, back up the way [Upgrading](#upgrading) describes (stop the
+  container, then copy all three `rackstack.db*` files; it runs in WAL mode,
+  so a bare `cp` of just `rackstack.db` against a live server grabs an
+  inconsistent, incomplete snapshot). If you need a backup without stopping
+  the server, use SQLite's own online-safe backup command instead:
+  `sqlite3 data/rackstack.db ".backup data/rackstack.db.bak"`. On Postgres,
+  use your usual `pg_dump`/`pg_basebackup` practice. Every caller goes
+  through the same interface regardless of backend, so nothing outside
+  `server/db/` needs to know or care which one is active.
 - **JWT cookie, not sessions**: avoids needing a session store. The cookie
   is httpOnly and `secure` in production, valid for 90 days.
 - **Multi-user by default**: every Discord/GitHub login gets its own