Improve wallet sync throughput, binary transport, and checkpoint recovery

[BraydenLangley]

Results at a glance

Backups now transfer oversized records in authenticated pieces, verify integrity before import, and resume interrupted uploads.

Live E2E testing used a large wallet in the native desktop client, covering
complete local copies, restart recovery, and repeat sync. Transfer size was
measured separately with a synthetic fixture.

Test	Verified result
Full native desktop restores	Two complete copies; identical entity counts. Second run: 12m 50s vs 16m 30s (22.3% less time).
Retained local backup	Completed after cancellation, a connectivity pause, and restart recovery; all 12 entity-store counts preserved on another restart.
Local reads and repeat sync	Transaction/output reads passed; sampled transaction bytes matched; repeat sync made 0 inserts, 0 updates.
Transfer size	62.7% smaller encoded byte payload in a synthetic fixture.
Fresh large-wallet copy	134,240 records / 817 pages completed; cleanup passed. All six existing profiles exercised with saved local copies.
Oversized individual record	7 MiB restored into real browser IndexedDB with matching SHA-256. Interrupted upload resumed at 512 KiB; corrupt download rejected; repeat sync unchanged.
Automated integration	Authenticated HTTP backup/restore, interrupted pages, lost acknowledgements, binary byte round-trips, user isolation, and legacy compatibility covered.

Timing compares successive candidates, not a controlled comparison against upstream
main. Byte verification was sampled, not database-wide. See
test methods and limits for details.

Implementation, detailed test evidence, and release notes

Program and scope

• Tracker: Improve wallet sync throughput, binary transport, and checkpoint recovery #486, wallet storage replication throughput and recovery.

• Problem: large syncs spend time in small source queries, repeated transactions, oversized checkpoint downloads, and decimal JSON byte payloads.

• Implementation: adaptive bounded source reads and proof indexes; atomic page/checkpoint commits; IndexedDB identity and relation lookups that avoid wallet-wide scans; optional totals; writer-scoped checkpoint selection; negotiated compact checkpoints reused after each committed page; compact binary sync bytes in both directions; bounded retries for oversized read responses; no late authentication dispatch or session recovery after the caller's deadline.

• Compatibility: legacy peers retain full checkpoints and array payloads. Binary uploads remain opt-in and negotiated. Gateway failures do not cause write replay. Complete ID mappings remain durable on the writer.

• Out of scope: merging this PR, publishing npm packages, changing wallet ownership or primary-provider policy.

• Rebased onto upstream main at c42233af7cfd143b74154c89df65a7ea60384477 (this repository has no upstream master).

• Earlier throughput revision: 7c39d891dfbe654ff1ebc7f79d6cb4d7ca0d2a20; health, authenticated identity/capability, configuration preservation, and artifact verification passed. That deployed revision passed the complete merge gate (CI run 34292518689), CodeQL (34292518596), and conformance workflow (34292518668).

• Deployed production implementation: 1901c63108ec136ddce901eca6293f3bc08b0613. Adds negotiated bounded record transfers: raw binary framing, authenticated pieces, durable staged uploads, integrity verification and a matching-checkpoint commit guard. Existing ordinary-page and BRC-38/39 file formats are retained. This exact revision is deployed and has passed public/local health, authenticated storage-identity and transfer-capability checks, artifact verification, and configuration preservation. An offline migration/rollback rehearsal preserved all 23 existing application tables; a fresh off-host database snapshot was verified before cutover. Earlier timing results below describe preceding revisions.

• Review follow-up: afe5a64287d97f57345e20785ff77cd4729f0126 changes only tests; all 1,180 packed runtime files match the deployment. Thirteen focused cases passed, including explicit staged-read fallback, malformed acknowledgements, upload cleanup, and rejection of identity changes despite a valid frame hash. All 38 applicable follow-up checks passed, including the external Codecov patch check at 92.82% against the unchanged 90% target (repository CI run 34401057984); four scope-based skips were accepted by the merge gate.

• Current local evidence: 230-suite Toolbox regression passed; focused final transfer, migration/replay and authenticated user-isolation checks passed. Desktop: 208 tests, TypeScript, and production builds passed with both linked candidates and an isolated published-dependency fixture; lint and security checks passed. Real-browser synthetic HTTP/IndexedDB transfer, interruption, corruption rejection and unchanged resync passed in 10.4 seconds on the final runtime. Four fixed portable framing vectors passed; conformance now has 6,483 passing and 211 existing skipped cases.

• Health, lint, formatting, typecheck, packed Node consumers and Vite/esbuild/Metro/Hermes checks passed locally. The deployed runtime revision passed all 37 repository hosted checks, including the merge gate, combined coverage, mutation, conformance, CodeQL, zero-new-Sonar, and browser/mobile gates (CI run 34398746700). Four scope-based skips were accepted by the merge gate. No new open code-scanning alerts or unresolved review threads remain. Linux and macOS artifact measurements are documented for review. Proposed narrow bundle-budget increases require explicit review; measured composition and cost are documented in the transfer guide.

Fresh large-wallet validation: a default-provider-to-device copy completed 134,240 records / 817 pages with successful temporary cleanup and both configured backups preserved. All six existing profiles were exercised with saved local copies. The full copy took 45m 29s, including 43m 25s remote reads; this validates completion, not a speedup against the separately optimized provider.

Post-deployment native sample: 20 pages / 4,770 records copied from the deployed revision into temporary IndexedDB in 16.4s; cleanup passed. Foreground timer-delay p95 22 ms, maximum 44 ms across 115 samples. This bounded sample is not a full-wallet benchmark or a controlled speed comparison.

Impact

• Public package source or manifest changed: SDK and all three toolbox distributions.
• Public API/types and portable behavior changed additively.
• Security-sensitive authentication, remote serialization, and persistence boundaries changed.
• Documentation, changelog, release notes, and generated migration guidance updated.

Release candidates: @bsv/sdk 2.5.0 (upstream's existing unreleased minor candidate, with this PR's compatible timeout and large-payload hashing fixes); toolbox/server, client, and mobile 2.11.0. Publication remains a separate release workflow.

Verification

Earlier throughput-candidate results (current transfer validation is summarized above):

• pnpm health:check, pnpm lint, pnpm format:check, pnpm typecheck: passed.
• pnpm --filter @bsv/sdk test: 158 suites, 5,950 tests passed.
• pnpm --filter @bsv/wallet-toolbox test: 228 suites, 2,202 tests passed; one existing skip.
• Browser/client tests: 19 passed. Mobile tests: 42 passed.
• Authenticated HTTP integration: multipage backup, complete restore into a fresh IndexedDB store, no-change resync, compact checkpoint reuse, and binary upload byte round-trip.
• Recovery: interrupted pages and lost commit acknowledgements resume from durable progress. User isolation, malformed checkpoints, legacy support, binary marker collisions, and byte preservation are covered.
• Packed Node/ESM/CJS consumers and browser/mobile platform contracts were exercised. Local full toolbox coverage passed, and the combined current-toolbox/prior-identical-SDK report passes the unchanged patch gate at 97.36%. Hosted CI passed for that earlier deployed revision (run 34292518689), including the aggregate coverage and merge gates.
• pnpm conformance: all 6,690 vectors in 76 files parsed/validated structurally; this command is not a claim that every external implementation executed them.
• pnpm test:mutation --target sdk-auth-http: 85.47%; 72 killed, 28 timed out, 17 survived. No mutation threshold or exclusion was changed.
• pnpm audit:security: passed the existing governed audit policy; existing advisory dispositions remain. No new override or suppression.

Measured improvements on synthetic fixtures:

• Nine alternating parser samples on identical 1 MiB numeric-array data: initial run median 501.2 ms with the historical reviver versus 10.1 ms with the new parser. A repeat measured 569.2 ms versus 10.3 ms. These are local CPU measurements, not network speedup claims.
• A 50,000-entry synthetic ID map occupied 678,850 bytes; its compact checkpoint occupied 432 bytes.
• Negotiated binary sync encoding reduced a synthetic byte-field payload from 3,743,771 to 1,398,163 bytes, a 62.7% reduction.
• The authenticated source benchmark fills a 250-record page in three source queries (limits 10, 80, 160).
• In a 1,000-row synthetic local fixture, transaction matching previously visited 1,002 rows and now visits only matching rows; commission matching falls from 1,000 rows to one. Tests retain user filtering, pagination, duplicate txids, version-5-to-6 migration, and reopen coverage.
• Native WebView synthetic 1 MiB SHA-256 takes 274–309 ms synchronously versus 0–3 ms through asynchronous Web Crypto (coarse timer resolution). Large ProtoWallet signing/verification payloads now use the latter with a snapshot-preserving fallback; six tests cover deterministic signature parity, altered payload rejection, small/direct-digest behavior, legacy byte coercion, unavailable hosts, and failed/malformed native results. This is component evidence, not a measured full-sync speedup.
• Portable bundle growth and measured ceilings are recorded in the toolbox changelog. No Node storage backend enters portable bundles.

Live native integration results:

• A client-only sample against the older server did not improve overall throughput. After binary-capable server deployment, a comparable partial sample took 28.9 seconds versus 34.6 seconds initially. These are observational samples, not a full-wallet comparison against upstream main.

• The first completed disposable full restore took 990.3 seconds. Its proof-batch preflight scans and large synchronous signature-payload hashing motivated the indexed lookup and asynchronous hashing fixes in this PR.

• The second complete restore took 769.5 seconds versus 990.3 seconds (22.3% less elapsed time), with identical entity counts and successful temporary database cleanup. Reads took 619.2 seconds, writes 146.0 seconds, and measurement 4.1 seconds. Proof-page writes fell from 75.7 to 9.8 seconds. This sequential comparison between candidates does not isolate each optimization or prove a full-wallet speedup against upstream main.

• The second full restore's all-state timer-delay p95 was 908 ms. Focus was not recorded, so that metric cannot establish foreground latency. Four subsequent 90-second foreground probes across transactions, outputs, heavier outputs, and proof requests measured p95 delays of 5, 8, 10, and 5 ms; the largest sampled foreground delay was 298 ms. These selected-phase samples do not guarantee latency throughout a full restore.

• A retained local backup completed after safe cancellation, a connectivity pause, and checkpoint recovery. A subsequent full native app restart preserved all twelve entity-store counts exactly. Local transaction/output reads passed, a sampled transaction-byte SHA-256 matched its pre-pause fingerprint, and an explicit resync reported zero inserts and zero updates. The original remote primary remained selected. This is not independent byte-for-byte verification of the whole database, nor an offline spending test. This retained restore used the older primary provider, so its duration is not an optimized-server benchmark.

• A final 20-page diagnostic against deployed head 7c39d891dfbe654ff1ebc7f79d6cb4d7ca0d2a20 completed in 17.7 seconds. All 166 timer samples were visible/focused: p95 delay 24 ms, maximum 48 ms. Temporary database cleanup succeeded. This deliberately bounded sample is not a full restore; network conditions had changed, so it is not a controlled speed comparison.

• Native development testing exposed a linked-dependency browser-cache issue: rebuilding a candidate could retain an immutable Vite URL and load an older IndexedDB schema. The companion desktop PR fingerprints linked package JavaScript in Vite's optimizer cache identity; reopening the retained backup subsequently passed twice without clearing wallet data. Regression tests cover unchanged package versions and lockfiles. This fix is in metanet-client-desktop#42.

• The final deployed revision passed artifact verification, local/public health, authenticated storage identity/capability, and configuration-preservation checks. Health remained successful after the final sample. Private operational evidence stays outside application repositories and PR discussion.

• Self-reviewed correctness, security, compatibility, API, artifacts, dependencies, documentation, and operational impact.

• All applicable final-head checks reached terminal success, including the separately reported Codecov patch check; no unresolved review/security findings remain.

Security and dependencies

• Updated the existing transitive js-yaml 3.x resolution from 3.15.1 to 3.15.2 after the audit gate began reporting GHSA-2883-xcg3-v3hh. Reviewed the upstream patch: it charges empty merge sources and rejects abnormally long merge sequences. Only Jest coverage configuration and documentation frontmatter consume this workspace dependency; their tests and the complete docs build passed. Dependency ranges, peer graph, wallet runtime dependencies, and audit policy are unchanged. No new override, release-age exception, or advisory dismissal.
• Peer/runtime compatibility, package consumers, and existing audit dispositions reviewed.
• Negative tests cover authentication deadlines, unauthorized checkpoint access, malformed progress, binary decoding, and unchanged legacy behavior.
• Final review-head CodeQL verified; no open new alerts.
• Final review-head repository quality and merge gates verified (CI run 34401057984), plus external Codecov patch coverage at 92.82%.
• No new override, advisory dismissal, suppression, or skipped test. The 97.35% hosted patch report initially failed because the declaration-only settings interface has no LCOV entry. RPC-only capability typing now belongs to the remote client, leaving the persisted table declaration unchanged. No coverage rule or exclusion was added.
• Workflow permissions and lifecycle-script policy are unchanged.

Release and operations

• No npm publication or PR merge performed.
• Compatible release decisions and migration guidance recorded.
• Database migration rollback behavior is covered by retained tests; operator deployment uses a separately verified backup and staged candidate.
• Exact transfer revision deployed and validated; post-deployment native copying and normal backup sync passed. Prior throughput revisions completed the historical restore/recovery checks above.

Completion evidence and release recommendation

Ready for maintainer review: exact-head repository and external coverage gates, migration/rollback rehearsal, deployment, fresh native large-wallet completion, and final normal backup sync passed. The follow-up changes only tests; its 1,180 packed runtime files are identical to the deployed implementation. The proposed artifact-budget increases require explicit review; transport remains bounded to 64 MiB frames. The earlier whole-wallet throughput results remain valid for that dataset; they did not prove oversized-record support. The new bounded transfer specifically addresses that missing protocol path. No merge or npm publication has been performed.

The desktop application still declares published SDK 2.4.1 / toolbox-client 2.10.2; these integration tests used the linked SDK 2.5.0 / toolbox 2.11.0 candidates. Coordinate publication and reviewed desktop dependency updates before shipping the combined behavior. A candidate-upgraded IndexedDB database requires a schema-compatible client; reverting to the older published client is not a compatible database downgrade.

The measurements support the improvements described above, not a universal throughput or latency guarantee. Public fixtures are synthetic; personal infrastructure, identities, credentials, wallet contents, and raw operational output are excluded.

Record-size boundary and standards

Upgraded peers negotiate version 1 transfers with frames up to 64 MiB and pieces no larger than 256 KiB. Complete frames are still materialized in memory. Uploads resume durable pieces after restart; interrupted downloads resume at the last committed record and may re-download an incomplete record. Larger records fail explicitly and are never skipped. Legacy providers require an upgrade when a single record exceeds their HTTP limits.

BRC-40 is transport-agnostic; reassembled records keep its ordering, watermark, ID maps and commit semantics. BRC-38/39 export/encryption formats are unchanged and their existing regression tests pass. A live IndexedDB copy is not a portable export file. A pre-existing legacy BRC-38 restore migration changes a default-basket liquidity value, so these results are not a blanket claim of exact preservation of every exported value. No spec revision is proposed.

[codecov]

Codecov Report

❌ Patch coverage is 92.82051% with 42 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...-toolbox/src/storage/remoting/StorageClientBase.ts	90.69%	0 Missing and 8 partials ⚠️
...llet-toolbox/src/storage/remoting/StorageServer.ts	88.23%	2 Missing and 6 partials ⚠️
...lbox/src/storage/remoting/KnexSyncTransferStore.ts	94.73%	0 Missing and 5 partials ⚠️
...allet-toolbox/src/storage/remoting/SyncTransfer.ts	94.18%	0 Missing and 5 partials ⚠️
...ox/src/storage/remoting/entityValidationHelpers.ts	71.42%	2 Missing and 2 partials ⚠️
...s/wallet/wallet-toolbox/src/storage/StorageKnex.ts	66.66%	1 Missing and 2 partials ⚠️
packages/sdk/src/wallet/ProtoWallet.ts	80.00%	0 Missing and 2 partials ⚠️
...es/wallet/wallet-toolbox/src/storage/StorageIdb.ts	94.28%	0 Missing and 2 partials ⚠️
...llet/wallet-toolbox/src/storage/StorageProvider.ts	80.00%	0 Missing and 2 partials ⚠️
...wallet/wallet-toolbox/src/storage/StorageReader.ts	0.00%	1 Missing ⚠️
... and 2 more

📢 Thoughts on this report? Let us know!

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.1% Duplication on New Code

See analysis details on SonarQube Cloud