feat: implement virtio based host-guest communication

.github/workflows/Fuzzing.yml

@@ -22,7 +22,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        target: ['fuzz_host_print', 'fuzz_guest_call', 'fuzz_host_call', 'fuzz_guest_estimate_trace_event', 'fuzz_guest_trace', 'fuzz_virtq_packed_ring']
+        target: ['fuzz_host_print', 'fuzz_guest_call', 'fuzz_host_call', 'fuzz_guest_estimate_trace_event', 'fuzz_guest_trace', 'fuzz_virtq_malformed', 'fuzz_virtq_roundtrip']
     uses: ./.github/workflows/dep_fuzzing.yml
     with:
       target: ${{ matrix.target }}

.github/workflows/ValidatePullRequest.yml

@@ -213,7 +213,7 @@ jobs:
     if: ${{ !cancelled() && !failure() }}
     strategy:
       matrix:
-        target: ['fuzz_host_print', 'fuzz_guest_call', 'fuzz_host_call', 'fuzz_guest_estimate_trace_event', 'fuzz_guest_trace', 'fuzz_virtq_packed_ring']
+        target: ['fuzz_host_print', 'fuzz_guest_call', 'fuzz_host_call', 'fuzz_guest_estimate_trace_event', 'fuzz_guest_trace', 'fuzz_virtq_malformed', 'fuzz_virtq_roundtrip']
         arch:
           - X64
         # arm64 fuzzing runs on the daily schedule (DailyArm64.yml) instead of on

.github/workflows/dep_build_test.yml

@@ -141,6 +141,7 @@ jobs:
           HYPERLIGHT_INITIAL_SURROGATES: "0"
         run: |
           PROFILE=${{ inputs.config == 'debug' && 'dev' || inputs.config }}
+          # No-surrogate mode permits only one live VM per process.
           # Verify each expected test actually ran (guards against silent
           # renames where cargo test exits 0 with 0 matches).
           # NOTE: keep in sync with `just test-no-surrogate`.
@@ -156,12 +157,12 @@ jobs:
           OUT=$(cargo test -p hyperlight-host --profile=$PROFILE --lib -- no_surrogate_tests --test-threads=1 2>&1) || { echo "$OUT"; exit 1; }
           echo "$OUT"
           assert_ran "$OUT" single_vm_lifecycle
-          OUT=$(cargo test -p hyperlight-host --profile=$PROFILE --test integration_test -- guest_malloc guest_panic corrupt_output_size_prefix_rejected --test-threads=1 2>&1) || { echo "$OUT"; exit 1; }
+          OUT=$(cargo test -p hyperlight-host --profile=$PROFILE --test integration_test -- guest_malloc guest_panic --test-threads=1 2>&1) || { echo "$OUT"; exit 1; }
           echo "$OUT"
-          assert_ran "$OUT" guest_malloc guest_panic corrupt_output_size_prefix_rejected
-          OUT=$(cargo test -p hyperlight-host --profile=$PROFILE --test sandbox_host_tests -- --exact callback_test float_roundtrip --test-threads=1 2>&1) || { echo "$OUT"; exit 1; }
+          assert_ran "$OUT" guest_malloc guest_panic
+          OUT=$(cargo test -p hyperlight-host --profile=$PROFILE --test sandbox_host_tests -- --exact callback_test float_roundtrip guest_external_bytes_round_trip_and_retention oversized_host_response_returns_transport_error --test-threads=1 2>&1) || { echo "$OUT"; exit 1; }
           echo "$OUT"
-          assert_ran "$OUT" callback_test float_roundtrip
+          assert_ran "$OUT" callback_test float_roundtrip guest_external_bytes_round_trip_and_retention oversized_host_response_returns_transport_error
           OUT=$(cargo test -p hyperlight-host --profile=$PROFILE --lib -- snapshot_evolve_restore_handles_state_correctly restore_from_loaded_snapshot --test-threads=1 2>&1) || { echo "$OUT"; exit 1; }
           echo "$OUT"
           assert_ran "$OUT" snapshot_evolve_restore_handles_state_correctly restore_from_loaded_snapshot

CHANGELOG.md

@@ -5,11 +5,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 ## [Prerelease] - Unreleased
 
 ### Added
-* Add per-direction virtqueue configuration and account its allocations in
-  scratch sizing.
+* Per-direction virtqueue configuration through `SandboxConfiguration` and
+  `SandboxBuilder`, with allocations included in scratch sizing.
 * Shared virtqueue framing with a 12-byte `MsgHeader` and external byte values.
+* `ExternalValueSource` implementations for `RecvChain` and `Segments`.
 * Producer batch completion without notification and segmented payload
-  extraction without flattening.
+  assembly and extraction without flattening.
 
 ### Changed
 * Support overriding the guest log level when building or restoring initialized
@@ -29,25 +30,47 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
   the configured level is above `OFF` rather than whether the tracing state was
   allocated.
 * **Breaking:** Virtqueue rings and pools occupy host-owned scratch before page
-  tables. Snapshots use ABI 4 and config schema v2. Existing snapshots must be
+  tables. Snapshots use ABI 5 and config schema v3. Existing snapshots must be
   regenerated.
 * Host virtqueue access uses checked copies and atomics across mapped scratch.
   Snapshot admission checks geometry, canonical ring state, and H2G buffer shape.
   Consumers validate descriptors and payload accesses during use.
 * Virtqueue producers use concrete `SlotPool` allocation and `BufferLease`
   ownership. `BufferMap` supplies complete owners exposing initialized bytes.
-* `VirtqProducer::reset` is unsafe and requires a stopped peer with no live
-  consumer-side chain handles.
+* `VirtqProducer::reset` and `GuestContext::prepare_snapshot` are unsafe.
+  Peers must stay stopped with no live consumer-side chain handles until
+  their consumers are reset or replaced.
 * `ChainBuilder::build()` allocates readable and writable requests.
   `writable_avail()` reserves available upper-tier slots within the descriptor
   budget. It may add zero slots to a nonempty chain.
+* Require guest logs and all host and guest function calls to use virtqueues.
+* Keep registered Rust guest return values typed until transport encoding so
+  external byte results avoid intermediate FlatBuffer copies.
+* Store canonical virtqueue rings in versioned OCI transport layers. Config v3
+  rejects snapshots without transport state.
+* Running snapshots checkpoint dirty virtqueues before capture. Ordinary calls
+  keep their deferred result path.
+* Reject snapshot capture while guest-owned transport buffers are retained.
+* Use the reclaimed stack pages to raise the default G2H and H2G pools to 12
+  and 8 pages.
 
 ### Removed
 * `RunPool` and the run-specific `AllocError::InvalidAlign` variant.
+* Remove legacy stack I/O, its `GuestHandle` methods, and its sandbox
+  configuration and builder options.
+* Embedded byte payload tables and their value-union variants.
 
 ### Fixed
+* Allow reclaimed virtqueue completions to span multiple ring reuse cycles.
+* Virtqueue consumers return errors when payload copies or runtime bookkeeping
+  cannot be allocated.
 * Use a 16 KiB-aligned default scratch size for Apple Silicon compatibility.
 * Guest virtqueue copies reject overlapping buffers before accessing memory.
+* Keep sandboxes usable after an H2G request exceeds available virtqueue capacity.
+* Snapshot checkpoints ignore idle cancellation and clear partial abort state.
+* Keep sandboxes usable when G2H calls exhaust reply capacity.
+* Reject incompatible transport snapshots before changing sandbox state.
+* Allow guest host-return conversions to call or log to the host.
 
 ## [v0.17.0] - 2026-08-27
 

Justfile

@@ -114,10 +114,10 @@ test-like-ci config=default-target hypervisor="kvm":
     just test-compilation-no-default-features {{config}}
 
     @# test the crashdump feature
-    just test-rust-crashdump {{config}}
+    {{ set-env-command }}RUST_LOG='debug'; just test-rust-crashdump {{config}}
 
     @# test the tracing related features
-    {{ if os() == "linux" { "just test-rust-tracing " + config + " " + if hypervisor == "mshv3" { "mshv3" } else { "kvm" } } else { "" } }}
+    {{ if os() == "linux" { "RUST_LOG=debug just test-rust-tracing " + config + " " + if hypervisor == "mshv3" { "mshv3" } else { "kvm" } } else { "" } }}
 
 code-checks-like-ci config=default-target hypervisor="kvm":
     @# Ensure up-to-date Cargo.lock
@@ -170,13 +170,13 @@ build-test-like-ci config=default-target hypervisor="kvm":
     {{ if os() == "linux" { if hypervisor == "mshv3" { "just test " + config + " mshv3,hw-interrupts" } else { "just test " + config + " kvm,hw-interrupts" } } else { "just test " + config + " hw-interrupts" } }}
 
     @# Run Rust Gdb tests
-    just test-rust-gdb-debugging {{config}}
+    {{ set-env-command }}RUST_LOG='debug'; just test-rust-gdb-debugging {{config}}
 
     @# Run Rust Crashdump tests
-    just test-rust-crashdump {{config}}
+    {{ set-env-command }}RUST_LOG='debug'; just test-rust-crashdump {{config}}
 
     @# Run Rust Tracing tests
-    {{ if os() == "linux" { "just test-rust-tracing " + config } else { "" } }}
+    {{ if os() == "linux" { "RUST_LOG=debug just test-rust-tracing " + config } else { "" } }}
 
 run-examples-like-ci config=default-target hypervisor="kvm":
     @# Run Rust examples - Windows
@@ -218,8 +218,8 @@ like-ci config=default-target hypervisor="kvm":
     just fuzz-like-ci fuzz_host_call {{config}} {{hypervisor}}
     just fuzz-like-ci fuzz_guest_estimate_trace_event {{config}} {{hypervisor}}
     just fuzz-like-ci fuzz_guest_trace {{config}} {{hypervisor}}
-    just fuzz-like-ci fuzz_virtq_packed_ring {{config}} {{hypervisor}}
-    just fuzz-like-ci fuzz_push_pop_buffer {{config}} {{hypervisor}}
+    just fuzz-like-ci fuzz_virtq_malformed {{config}} {{hypervisor}}
+    just fuzz-like-ci fuzz_virtq_roundtrip {{config}} {{hypervisor}}
 
     @# spelling
     typos
@@ -241,7 +241,7 @@ test-loom:
 # runs tests that requires being run separately, for example due to global state
 test-isolated target=default-target features="" :
     {{ cargo-cmd }} test {{ if features =="" {''} else if features=="no-default-features" {"--no-default-features" } else {"--no-default-features -F " + features } }} --profile={{ if target == "debug" { "dev" } else { target } }} {{ target-triple-flag }} -p hyperlight-host --lib -- sandbox::uninitialized::tests::test_log_trace --exact --ignored
-    {{ cargo-cmd }} test {{ if features =="" {''} else if features=="no-default-features" {"--no-default-features" } else {"--no-default-features -F " + features } }} --profile={{ if target == "debug" { "dev" } else { target } }} {{ target-triple-flag }} -p hyperlight-host --lib -- sandbox::outb::tests::test_log_outb_log --exact --ignored
+    {{ cargo-cmd }} test {{ if features =="" {''} else if features=="no-default-features" {"--no-default-features" } else {"--no-default-features -F " + features } }} --profile={{ if target == "debug" { "dev" } else { target } }} {{ target-triple-flag }} -p hyperlight-host --lib -- sandbox::outb::tests::test_log_emit_guest_log --exact --ignored
     {{ cargo-cmd }} test {{ if features =="" {''} else if features=="no-default-features" {"--no-default-features" } else {"--no-default-features -F " + features } }} --profile={{ if target == "debug" { "dev" } else { target } }} {{ target-triple-flag }} -p hyperlight-host --lib -- sandbox::initialized_multi_use::tests::from_snapshot::max_guest_log_level_is_honored_from_snapshot --exact --ignored
     {{ cargo-cmd }} test {{ if features =="" {''} else if features=="no-default-features" {"--no-default-features" } else {"--no-default-features -F " + features } }} --profile={{ if target == "debug" { "dev" } else { target } }} {{ target-triple-flag }} -p hyperlight-host --lib -- sandbox::initialized_multi_use::tests::from_snapshot::max_guest_log_level_overrides_initialized_snapshot --exact --ignored
     {{ cargo-cmd }} test {{ if features =="" {''} else if features=="no-default-features" {"--no-default-features" } else {"--no-default-features -F " + features } }} --profile={{ if target == "debug" { "dev" } else { target } }} {{ target-triple-flag }} -p hyperlight-host --lib -- sandbox::initialized_multi_use::tests::from_snapshot::max_guest_log_level_setter_survives_restore --exact --ignored
@@ -282,14 +282,15 @@ test-compilation-no-default-features target=default-target:
     {{ if os() == "linux" { cargo-cmd + " check -p hyperlight-host --no-default-features --features mshv3" } else { "" } }}  {{ target-triple-flag }}
 
 # runs a subset of existing tests with HYPERLIGHT_MAX_SURROGATES=0 (Windows only).
-# Covers: guest calls, host callbacks, in-memory snapshot/restore, and
-# save/load snapshot from disk.
+# Covers guest calls, host callbacks, virtqueue buffers and errors,
+# in-memory snapshot/restore, and save/load from disk.
+# No-surrogate mode permits only one live VM per process, so tests run serially.
 # NOTE: if any of the test names below are renamed, update both this
 # recipe AND the matching CI step in .github/workflows/dep_build_test.yml.
 test-no-surrogate target=default-target:
     {{ set-env-command }}HYPERLIGHT_MAX_SURROGATES=0; {{ set-env-command }}HYPERLIGHT_INITIAL_SURROGATES=0; {{ cargo-cmd }} test -p hyperlight-host --profile={{ if target == "debug" { "dev" } else { target } }} --lib -- no_surrogate_tests --test-threads=1
-    {{ set-env-command }}HYPERLIGHT_MAX_SURROGATES=0; {{ set-env-command }}HYPERLIGHT_INITIAL_SURROGATES=0; {{ cargo-cmd }} test -p hyperlight-host --profile={{ if target == "debug" { "dev" } else { target } }} --test integration_test -- guest_malloc guest_panic corrupt_output_size_prefix_rejected --test-threads=1
-    {{ set-env-command }}HYPERLIGHT_MAX_SURROGATES=0; {{ set-env-command }}HYPERLIGHT_INITIAL_SURROGATES=0; {{ cargo-cmd }} test -p hyperlight-host --profile={{ if target == "debug" { "dev" } else { target } }} --test sandbox_host_tests -- --exact callback_test float_roundtrip --test-threads=1
+    {{ set-env-command }}HYPERLIGHT_MAX_SURROGATES=0; {{ set-env-command }}HYPERLIGHT_INITIAL_SURROGATES=0; {{ cargo-cmd }} test -p hyperlight-host --profile={{ if target == "debug" { "dev" } else { target } }} --test integration_test -- guest_malloc guest_panic --test-threads=1
+    {{ set-env-command }}HYPERLIGHT_MAX_SURROGATES=0; {{ set-env-command }}HYPERLIGHT_INITIAL_SURROGATES=0; {{ cargo-cmd }} test -p hyperlight-host --profile={{ if target == "debug" { "dev" } else { target } }} --test sandbox_host_tests -- --exact callback_test float_roundtrip guest_external_bytes_round_trip_and_retention oversized_host_response_returns_transport_error --test-threads=1
     {{ set-env-command }}HYPERLIGHT_MAX_SURROGATES=0; {{ set-env-command }}HYPERLIGHT_INITIAL_SURROGATES=0; {{ cargo-cmd }} test -p hyperlight-host --profile={{ if target == "debug" { "dev" } else { target } }} --lib -- snapshot_evolve_restore_handles_state_correctly restore_from_loaded_snapshot --test-threads=1
 
 # runs tests that exercise gdb debugging
@@ -530,7 +531,7 @@ coverage-run hypervisor="kvm": ensure-cargo-llvm-cov
 
     # isolated tests (require running separately due to global state)
     cargo +nightly test -p hyperlight-host --lib -- sandbox::uninitialized::tests::test_log_trace --exact --ignored
-    cargo +nightly test -p hyperlight-host --lib -- sandbox::outb::tests::test_log_outb_log --exact --ignored
+    cargo +nightly test -p hyperlight-host --lib -- sandbox::outb::tests::test_log_emit_guest_log --exact --ignored
     cargo +nightly test -p hyperlight-host --lib -- sandbox::initialized_multi_use::tests::from_snapshot::max_guest_log_level_is_honored_from_snapshot --exact --ignored
     cargo +nightly test -p hyperlight-host --lib -- sandbox::initialized_multi_use::tests::from_snapshot::max_guest_log_level_overrides_initialized_snapshot --exact --ignored
     cargo +nightly test -p hyperlight-host --lib -- sandbox::initialized_multi_use::tests::from_snapshot::max_guest_log_level_setter_survives_restore --exact --ignored

docs/README.md

@@ -35,6 +35,7 @@ This project is composed internally of several components, depicted in the below
 
 * [Security guidance for developers](./security-guidance-for-developers.md)
 * [Paging Development Notes](./paging-development-notes.md)
+* [Virtqueue host and guest communication](./virtio-host-guest-communication.md)
 * [How to debug a Hyperlight guest](./how-to-debug-a-hyperlight-guest.md)
 * [How to use Flatbuffers in Hyperlight](./how-to-use-flatbuffers.md)
 * [How to make a Hyperlight release](./how-to-make-releases.md)

docs/paging-development-notes.md

@@ -139,13 +139,10 @@ calls, i.e. there may be no calls in flight at the time of
 snapshotting. This is not enforced, but odd things may happen if it is
 violated.
 
-Buffer management between the host and guest is needed to pass call
-arguments and return values. Ideally, buffers would be dynamically
-allocated from the scratch region as needed.
-
-Currently, I/O buffers are statically allocated at the bottom of the
-scratch region. This is a stopgap pending improved
-physical allocation and buffer management.
+Host and guest calls use two virtqueues in a fixed transport arena at
+the bottom of scratch. The arena contains both rings and their
+fixed-slot buffer pools. Copied page tables follow the arena. Dynamic
+scratch allocations begin after the copied page tables.
 
 The minimum scratch size is calculated by `min_scratch_size()` in the
 architecture-specific layout modules under `hyperlight_common`; see
@@ -177,4 +174,3 @@ paging) and enables PAE.  The guest is always entered in long mode.
 
 Hyperlight unconditionally uses 48-bit virtual addresses. Hyperlight
 presently only uses addresses in the lower (ttbr0) half of the address range.
-