feat(storage): add an optional FTWDB beta candidate

.changeset/ftwdb-live-history-candidate.md

@@ -0,0 +1,4 @@
+---
+"ftw": minor
+---
+Add an optional FTWDB beta sidecar that copies committed live site history through a bounded memory queue. Keep SQLite and Parquet authoritative and report candidate gaps, errors and durable acknowledgements in health.

.github/workflows/ftwdb-shadow-contract.yml

@@ -0,0 +1,46 @@
+name: FTWDB shadow contract
+
+on:
+  pull_request:
+    paths:
+      - ".github/workflows/ftwdb-shadow-contract.yml"
+      - "docker-compose.ftwdb-shadow.yml"
+      - "go/internal/ftwdbshadow/**"
+      - "go/internal/state/history_feed*"
+      - "go/internal/state/store_ts.go"
+      - "go/cmd/ftw/main.go"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  go-rust-contract:
+    name: Go to Rust contract
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+      - name: Check out pinned FTWDB
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          repository: srcfl/ftwdb
+          ref: 7bbae63532f695b10aca548bf4ee58c6d7ebb3a8
+          path: .ftwdb-contract
+      - uses: actions/setup-go@v7
+        with:
+          go-version-file: go/go.mod
+      - uses: dtolnay/rust-toolchain@2c7215f132e9ebf062739d9130488b56d53c060c # master
+        with:
+          toolchain: 1.97.1
+      - name: Check the frozen v1 contract
+        run: diff -ru .ftwdb-contract/testdata/shadow-protocol-v1 go/internal/ftwdbshadow/testdata/shadow-protocol-v1
+      - name: Build the pinned sidecar and reconcile tool
+        run: cargo build --locked --manifest-path .ftwdb-contract/Cargo.toml --bin ftwdb-shadow --bin ftwdb-shadow-reconcile
+      - name: Exercise live copy, lost ACK, limits, SIGKILL, restart and reconcile
+        working-directory: go
+        env:
+          FTWDB_SHADOW_BIN: ${{ github.workspace }}/.ftwdb-contract/target/debug/ftwdb-shadow
+          FTWDB_RECONCILE_BIN: ${{ github.workspace }}/.ftwdb-contract/target/debug/ftwdb-shadow-reconcile
+          FTWDB_SHADOW_FIXTURES: ${{ github.workspace }}/.ftwdb-contract/testdata/shadow-protocol-v1
+        run: go test -race ./internal/ftwdbshadow ./internal/state -run 'TestBeta|TestHistoryMapping|TestRustSidecarInterop|TestLiveHistory|TestHealthOps|TestV1Golden|TestVendored' -count=1 -v

docker-compose.ftwdb-shadow.yml

@@ -0,0 +1,31 @@
+# Optional beta overlay. Core and SQLite keep their normal lifecycle.
+# Pin matches .github/workflows/ftwdb-shadow-contract.yml.
+services:
+  ftw:
+    environment:
+      FTWDB_SHADOW_SOCKET: /run/ftwdb-shadow/shadow.sock
+    volumes:
+      - ftwdb-shadow-run:/run/ftwdb-shadow
+
+  ftwdb-shadow:
+    profiles: [ftwdb-shadow]
+    build:
+      context: https://github.com/srcfl/ftwdb.git#7bbae63532f695b10aca548bf4ee58c6d7ebb3a8
+    image: ftwdb-shadow:7bbae63532f695b10aca548bf4ee58c6d7ebb3a8
+    user: "100:101"
+    network_mode: none
+    read_only: true
+    cap_drop: [ALL]
+    security_opt: [no-new-privileges:true]
+    cpus: 0.25
+    mem_limit: 256m
+    pids_limit: 64
+    restart: "no"
+    stop_grace_period: 30s
+    volumes:
+      - ftwdb-shadow-data:/var/lib/ftwdb-shadow
+      - ftwdb-shadow-run:/run/ftwdb-shadow
+
+volumes:
+  ftwdb-shadow-data:
+  ftwdb-shadow-run:

docs/architecture.md

@@ -65,6 +65,11 @@ The control loop computes a site target, allocates it across capable assets,
 applies safety constraints, then sends commands through the driver registry.
 Planner output is an input to that loop, never a direct device command.
 
+The optional [FTWDB beta candidate](ftwdb-shadow.md) copies committed numeric
+site history through a bounded memory queue to a local sidecar. It reports
+session gaps and durable receipts. SQLite and Parquet keep serving all reads;
+the candidate has no role in control, config or forecasting.
+
 ## Drivers
 
 The public `srcfl/device-drivers` repo owns editable driver source, versions,

docs/ftwdb-shadow.md

@@ -0,0 +1,92 @@
+# FTWDB beta candidate
+
+The optional FTWDB sidecar copies five numeric fields from successful live
+SQLite history writes: grid power, PV power, battery power, house load and
+battery state of charge. Watts keep the site sign convention; SoC stays a
+0–1 fraction. SQLite and Parquet still serve history. Config, forecasts, learned
+models, schedules and control continue to use their current stores.
+
+This is a bounded session recording. It does not copy old data, driver samples,
+SQL imports, retention deletes, the energy ledger or forecast archives. It is
+not a complete replica or a backup. Each Core start has a new session ID. Pending
+memory work can be lost on restart; the SQLite source remains available.
+
+## Enable on a beta test box
+
+Use a Core beta that contains this integration. From the FTW checkout:
+
+```sh
+docker compose -f docker-compose.yml -f docker-compose.ftwdb-shadow.yml \
+  --profile ftwdb-shadow build ftwdb-shadow
+docker compose -f docker-compose.yml -f docker-compose.ftwdb-shadow.yml \
+  --profile ftwdb-shadow up -d ftw ftwdb-shadow
+```
+
+The overlay builds a pinned FTWDB commit. It gives the sidecar its own data
+volume, no network, a 256 MiB memory limit and a quarter CPU. Only the private
+Unix socket volume is shared with Core. Both processes use UID 100, GID 101.
+There is no startup or health dependency from Core to FTWDB.
+
+For a native Linux install, use the pinned
+[systemd service example](https://github.com/srcfl/ftwdb/blob/7bbae63532f695b10aca548bf4ee58c6d7ebb3a8/packaging/systemd/ftwdb-shadow.service)
+with the same user as Core. That service listens on
+`/run/ftwdb-shadow/ftwdb-shadow.sock`. Pass that path with
+`-ftwdb-shadow-socket` or `FTWDB_SHADOW_SOCKET` to Core.
+An empty value disables the candidate.
+This is an install option; household Settings do not expose an experimental
+storage switch.
+
+## Read the result
+
+Read `ftwdb_shadow` from `GET /api/health`. Its state is independent of Core
+health. Check these fields together:
+
+- `session`, `started_at` and `scope` identify the covered run.
+- `offered_ticks`, `queued_ticks`, `pending_ticks` and `dropped_ticks` show
+  collection and overload. `gaps` means at least one offered tick was lost.
+- `acknowledged_ticks`, `durable_through_sequence` and `last_ack_at` report
+  durable sidecar receipts. A sent batch is not yet an acknowledgement.
+  `last_ack_ms` and `max_ack_ms` measure the commit request and durable reply.
+- `errors` and `last_error` explain a pause. `sidecar` counters have their own
+  `sidecar_checked_at`; they can precede the latest batch acknowledgement.
+
+The queue holds at most 256 small numeric records, plus one pending batch of
+at most 128. Core tries a batch every 30 seconds. Connect, encode, socket I/O
+and retry happen on a separate goroutine with two-second I/O deadlines. A full
+queue drops candidate work and increments its counter. It never waits for the
+sidecar from a device or control loop.
+
+The sequence follows delivery of committed writes, not measurement time.
+Late and same-time live history writes therefore remain distinct. Retries keep
+one source ID, sequence, commit ID and the exact encoded bytes. The sidecar uses
+always-sync durability. Core also pauses new writes once its reported store
+size reaches 512 MiB. The sidecar enforces its own space limits. Store limits
+are test budgets, not a claim that shared-disk I/O has no effect on control.
+
+## Stop the experiment
+
+Stop the sidecar, then recreate Core with the normal Compose file:
+
+```sh
+docker compose -f docker-compose.yml -f docker-compose.ftwdb-shadow.yml \
+  --profile ftwdb-shadow stop ftwdb-shadow
+docker compose -f docker-compose.yml up -d --no-deps ftw
+```
+
+Keep the candidate data volume when collecting a report. This flow does not
+remove SQLite or Parquet. Do not use `down -v` to disable the experiment.
+
+## Validation
+
+The contract workflow pins the same FTWDB commit as the overlay. It compares
+shared byte fixtures, sends committed SQLite history to the real Rust process,
+drops an acknowledgement, retries exact bytes, kills the process, checks the
+reopened durable receipt and reconciles all copied points offline. Tests also
+cover absent, unhealthy, non-durable and full sidecars, a full client queue,
+failed SQLite commits, late writes, SI units and concurrent status reads.
+
+Before increasing the scope, measure control latency, CPU, RSS, disk growth,
+sync rate and gaps on a real box for at least 72 hours. Test disk pressure and
+physical power loss on that hardware. Host tests and SIGKILL do not prove SD-card
+power-loss behavior. Keep SQLite/Parquet as the source until those results and
+an explicit data migration justify a separate replacement change.

go/cmd/ftw/main.go

@@ -49,6 +49,7 @@ import (
 	"github.com/srcfl/ftw/go/internal/drivers"
 	"github.com/srcfl/ftw/go/internal/events"
 	"github.com/srcfl/ftw/go/internal/forecast"
+	"github.com/srcfl/ftw/go/internal/ftwdbshadow"
 	"github.com/srcfl/ftw/go/internal/gatewayidentity"
 	"github.com/srcfl/ftw/go/internal/ha"
 	"github.com/srcfl/ftw/go/internal/loadmodel"
@@ -318,6 +319,7 @@ func main() {
 	}
 
 	configPath := flag.String("config", "config.yaml", "Path to config.yaml")
+	shadowSocket := flag.String("ftwdb-shadow-socket", os.Getenv("FTWDB_SHADOW_SOCKET"), "Optional local FTWDB beta socket; empty disables the candidate")
 	webDir := flag.String("web", "web", "Path to static web UI directory")
 	driverDirFlag := flag.String("drivers", "", "Path to drivers directory (default: <config-dir>/drivers)")
 	userDriversDirFlag := flag.String("user-drivers", "", "Path to PERSISTENT user-drivers directory (overlay on top of -drivers). Searched first; falls back to -drivers when a file isn't found here. Designed for docker deploys.")
@@ -713,6 +715,8 @@ func main() {
 	var forecastConfigMu sync.RWMutex
 	var ocppSrv *ocpp.Server
 	forecastSettings := newForecastSiteConfig(st)
+	shadow := ftwdbshadow.Start(ctx, st, *shadowSocket, forecastSettings.Snapshot().SiteID, Version)
+	defer shadow.Close()
 	forecastSettings.identity = func(name string) (string, bool) {
 		if id, ok := runningDeviceID(reg, name); ok {
 			return id, true
@@ -2356,6 +2360,7 @@ func main() {
 		ColdDir:           coldDir,
 		DataDir:           dataDir,
 		StatePath:         statePath,
+		FTWDBShadow:       shadow,
 		BackupDir:         backupDir,
 		DataMaintenanceMu: dataMaintenanceMu,
 		// Snapshots live next to the rest of the persistent data so

go/internal/api/api.go

@@ -41,6 +41,7 @@ import (
 	"github.com/srcfl/ftw/go/internal/events"
 	"github.com/srcfl/ftw/go/internal/fleetping"
 	"github.com/srcfl/ftw/go/internal/forecast"
+	"github.com/srcfl/ftw/go/internal/ftwdbshadow"
 	"github.com/srcfl/ftw/go/internal/ha"
 	"github.com/srcfl/ftw/go/internal/loadmodel"
 	"github.com/srcfl/ftw/go/internal/loadpoint"
@@ -71,6 +72,8 @@ const (
 // One instance is shared across all handlers; mutations use the contained
 // mutexes from each package.
 type Deps struct {
+	FTWDBShadow *ftwdbshadow.Beta
+
 	// MutationPolicy protects every state-changing route at the shared
 	// Handler boundary. Production requires tokens for non-local hostnames;
 	// the zero value retains local/test embedding compatibility.
@@ -699,6 +702,9 @@ func (s *Server) handleHealth(w http.ResponseWriter, r *http.Request) {
 		}
 		resp["storage"] = storage
 	}
+	if s.deps.FTWDBShadow != nil {
+		resp["ftwdb_shadow"] = s.deps.FTWDBShadow.Status()
+	}
 	writeJSON(w, 200, resp)
 }