fix(storage): restore SQLite and Parquet with verified beta conversion - #1248
Conversation
Signed-off-by: Fredrik Ahlgren <fredrik@sourceful-labs.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
…recovery-20260913
Signed-off-by: Fredrik Ahlgren <fredrik@sourceful-labs.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 76bf6c4103
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| func (s *Store) ImportedHistoryFiles(ctx context.Context) (map[string]bool, error) { | ||
| return map[string]bool{}, nil |
There was a problem hiding this comment.
When an existing Parquet day is truncated or otherwise unreadable, returning an empty imported-file set causes the backup to include that file, but backup.Verify checks only its self-recorded hash and never calls walkParquetRows. The archive is therefore reported as a verified full backup even though restoring it leaves that portion of history unreadable; validate every retained Parquet file before publishing the backup.
AGENTS.md reference: AGENTS.md:L24-L25
Useful? React with 👍 / 👎.
| } else if active == "" && restore == "" { | ||
| return errors.New("unbound SQLite history; complete its migration before starting") |
There was a problem hiding this comment.
If power is lost after migrateSQLiteHistory commits the generation: marker but before SaveConfig writes history_sqlite_generation, the next boot enters this branch and permanently refuses to start. This can happen during either a fresh installation or the first legacy SQLite migration, and there is no converter or automatic retry path for it, so the binding protocol needs a recoverable pending marker or another way to distinguish its own completed migration from an unrelated unbound database.
AGENTS.md reference: AGENTS.md:L24-L25
Useful? React with 👍 / 👎.
Signed-off-by: Fredrik Ahlgren <fredrik@sourceful-labs.com>
1 participant
Core's DuckDB history path can fail during full backup and adds a native runtime/build dependency to every installation. This change restores SQLite history plus verified Parquet archives, with state.db retaining goals, identities and models separately from history.db.
Beta candidate. Backup can still delay durable goal saves on Raspberry Pi (#1246), and a new safety fault can still wait behind an already blocked control tick (#1247). These issues carry release-blocker and prevent stable promotion. No production history has been converted and this storage candidate is not installed on the live site.
Requires merged #1240 at 67439f4 and the paired driver pin from srcfl/device-drivers#116. All EV hardenings from those changes remain included.
The normal Core and backup builds use CGO_ENABLED=0 and have no DuckDB dependency. Fresh and older SQLite/Parquet installations start directly. Only DuckDB beta installations use the separate offline converter under go/tools/history-migrate. It fingerprints frozen sources, resumes bounded verified copies, reconciles catalog IDs by name, preserves older summaries and energy counters, and publishes only after readback, sync and a recovery receipt. It retains original history files. Schema 4 blocks image-only rollback across the history format.
History writes keep bounded queues and durable commit receipts. Archive writes use bounded staging, verified Parquet publication and value-matched pruning. Raw reads, chart buckets and JSON output have explicit limits. Hourly series summaries and older daily energy totals survive retention. Full backup uses verified read snapshots and paced writes. Known EV safety stops now precede session IO for all chargers in a tick; this does not yet cover a new fault arriving while an earlier tick is already blocked.
Full backups now read retained Parquet pages and compare row counts, so matching hashes cannot hide a damaged source. The first SQLite migration records its intended generation before copying and can finish a binding interrupted by power loss, while still refusing unrelated files.
Closes #1250. Closes #1251.
Validation:
New regressions reproduce both review findings before the fixes and pass after them: four first-binding restart cases, truncated Parquet, and damaged pages under an intact footer. Backup tests cover both creation and independent archive verification. The new binding, corruption and full backup/restore tests also pass on the ARM64 Pi using isolated synthetic data; local make verify passes on f69110e.
Full master-target CI found that Windows cannot sync a read-only file handle. The converter now opens only its new destination read/write before the final sync. The existing interruption/resume tests cover this path; conversion sources remain read-only.
Local make verify, state/backup/loadpoint/MPC race suites, migration interruption/source-change/receipt tests, corruption/pruning tests, process-kill recovery and full backup/restore tests pass. Final exact-commit logs are linked in the local delivery report.
The new blocked-session test fails before the safety-ordering fix and passes after it, for stale site-meter permission and unavailable charger power with two chargers.
A real DuckDB fixture with one million interleaved samples passes locally and on ARM64 Pi (204 s, 98.5 MiB measured peak process RSS). The same fixture passes locally inside an enforced 256 MiB container limit. The Pi kernel does not enable its memory controller, so its RSS result is not an enforced-memory test.
The 48 MiB backup fixture restores all rows/goals locally with concurrent durable saves and commits. On Pi the backup still produces multi-second goal-save delays; a normal-priority run reached 14.94 s. An idle-priority run reached 45.18 s, but also deprioritized goal saves. These failures remain visible and block stable promotion; the 2 s acceptance limit has not been relaxed.
The standalone converter needs a compatible glibc/C++ runtime; an isolated compatible container works on the older Pi host. The normal pure-Go Core does not inherit that requirement.
Before converting a live DuckDB site: estimate disk space, verify a full backup and separate restore on the box, then validate migration, restart, EV identity/SoC/goal, forecasts and recent/old history on the exact candidate. Stable promotion requires corrections and passing target evidence for #1246 and #1247; both findings remain open. No household records are included in this PR.
Related: #1234, #1246, #1247. Operator steps are in docs/history-conversion.md.