Conversation
OpenBackupSource no longer inherits the live 100ms yield that made a two-hour export deadline unreachable for large converted histories. Live Core backups still pause between copy batches. Verification extracts beside the archive, and create refuses to start without room for the raw export, compressed file and extract. Addresses #1259 Signed-off-by: Fredrik Ahlgren <fredrik@sourceful-labs.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cee71eaf82
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
| seen := make(map[string]bool, len(want)) | ||
| tmpDir, err := os.MkdirTemp("", ".ftw-backup-verify-") | ||
| tmpDir, err := os.MkdirTemp(filepath.Dir(archivePath), ".ftw-backup-verify-") |
There was a problem hiding this comment.
When an archive is stored on a read-only USB mount or in a directory the invoking user cannot modify, creating this temporary directory fails even though the archive is readable and the restore destination is writable. Because both Inspect and RestoreContents call Verify, this also prevents inspecting or restoring such backups; use a caller-writable workspace or restrict same-filesystem staging to the creation path.
AGENTS.md reference: AGENTS.md:L24-L25
Useful? React with 👍 / 👎.
| if err := os.MkdirAll(outputDir, 0o700); err != nil { | ||
| return Info{}, fmt.Errorf("backup: create output dir: %w", err) | ||
| } | ||
| if err := state.EnsureDiskSpace(outputDir, state.BackupArchiveScratch(opts.State.BackupSourceBytes(), 0)); err != nil { |
There was a problem hiding this comment.
When dataDir contains substantial cold Parquet history or other persistent files, this preflight counts only the SQLite state/history files because extraBytes is always zero, while collectSources later adds those files to the archive. A target with room for three copies of a small database but not the cold archive therefore passes the promised refusal check and fills during creation; inventory and include the non-database source sizes before checking free space.
AGENTS.md reference: AGENTS.md:L24-L25
Useful? React with 👍 / 👎.
1 participant
Problem and result
#1259 shows that a large converted history cannot finish a portable backup through
ftw-backup.exportHistoryToSQLitealways paused 100 ms after at most 1,024 rows, including when the store came from read-onlyOpenBackupSource. For ~175 million sample rows those pauses alone take about 4 h 44 m, so the live two-hour export deadline always fires.The offline helper now copies without that live yield and without the two-hour deadline. Live Core backups still pause between copy batches so charging-goal saves keep their latency protection. Create refuses to start when the destination cannot hold the raw export, compressed archive and verification extract. Verify extracts beside the archive instead of into process temp, which is too small for these files.
Scope and safety
go/internal/statebackup copy/export andgo/internal/backupcreate/verify. No UI.origin/fix/converter-throughput-20260914is conversion batching, not backup pacing.quick_checkand restore checks.Verification
Ran:
go test ./internal/state(pass, 16.7 s)go test ./internal/backup(pass)go test ./cmd/ftw-backup(pass)go test ./internal/api -run Backup(pass)New coverage: live vs offline yield, no offline deadline, scratch-space preflight, offline create/verify/restore with process temp blocked.
Not run:
FTW_STORAGE_ADMISSION=1on a Raspberry Pi; 175 million-row offline export/verify/restore.Checklist