add simple virtio-based block device driver and basic support of a FAT file system - #2658
Draft
stlankes wants to merge 12 commits into
Draft
add simple virtio-based block device driver and basic support of a FAT file system#2658stlankes wants to merge 12 commits into
stlankes wants to merge 12 commits into
Conversation
Member
|
A student of ours has already done this. We need to compare before we merge either of the two. |
Contributor
Author
|
I know, I was interested. It isn't necessary to merge the PR. I was more interested in Hadris FAT. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Benchmark Results
Details
| Benchmark | Current: a31b73d | Previous: 2e23902 | Performance Ratio |
|---|---|---|---|
| startup_benchmark Build Time | 97.47 s |
80.34 s |
1.21 ❗ |
| startup_benchmark File Size | 0.79 MB |
0.80 MB |
0.99 ❗ |
| Startup Time - 1 core | 0.75 s (±0.02 s) |
0.75 s (±0.02 s) |
1.00 |
| Startup Time - 2 cores | 0.72 s (±0.02 s) |
0.74 s (±0.02 s) |
0.98 |
| Startup Time - 4 cores | 0.72 s (±0.02 s) |
0.74 s (±0.02 s) |
0.98 |
| multithreaded_benchmark Build Time | 95.55 s |
82.11 s |
1.16 ❗ |
| multithreaded_benchmark File Size | 0.84 MB |
0.86 MB |
0.98 ❗ |
| Multithreaded Pi Efficiency - 2 Threads | 65.88 % (±5.89 %) |
85.89 % (±6.61 %) |
0.77 ❗ |
| Multithreaded Pi Efficiency - 4 Threads | 40.61 % (±2.56 %) |
43.43 % (±2.56 %) |
0.93 |
| Multithreaded Pi Efficiency - 8 Threads | 19.99 % (±1.44 %) |
25.76 % (±1.53 %) |
0.78 ❗ |
| micro_benchmarks Build Time | 215.97 s |
80.40 s |
2.69 ❗ |
| micro_benchmarks File Size | 0.84 MB |
0.86 MB |
0.98 ❗ |
| Scheduling time - 1 thread | 157.94 ticks (±31.40 ticks) |
62.65 ticks (±4.06 ticks) |
2.52 ❗ |
| Scheduling time - 2 threads | 92.96 ticks (±23.62 ticks) |
34.08 ticks (±4.10 ticks) |
2.73 ❗ |
| Micro - Time for syscall (getpid) | 9.44 ticks (±4.87 ticks) |
3.45 ticks (±0.58 ticks) |
2.74 ❗ |
| Memcpy speed - (built_in) block size 4096 | 53669.36 MByte/s (±39121.38 MByte/s) |
82448.38 MByte/s (±56997.13 MByte/s) |
0.65 |
| Memcpy speed - (built_in) block size 1048576 | 13272.15 MByte/s (±10805.94 MByte/s) |
30585.98 MByte/s (±24707.84 MByte/s) |
0.43 |
| Memcpy speed - (built_in) block size 16777216 | 11598.54 MByte/s (±9579.86 MByte/s) |
26340.06 MByte/s (±21720.96 MByte/s) |
0.44 |
| Memset speed - (built_in) block size 4096 | 53888.38 MByte/s (±39291.23 MByte/s) |
82292.76 MByte/s (±56891.50 MByte/s) |
0.65 |
| Memset speed - (built_in) block size 1048576 | 13508.74 MByte/s (±10905.14 MByte/s) |
31323.85 MByte/s (±25145.86 MByte/s) |
0.43 |
| Memset speed - (built_in) block size 16777216 | 11844.53 MByte/s (±9701.56 MByte/s) |
27104.68 MByte/s (±22209.94 MByte/s) |
0.44 |
| Memcpy speed - (rust) block size 4096 | 49737.37 MByte/s (±36223.35 MByte/s) |
74097.96 MByte/s (±51811.44 MByte/s) |
0.67 |
| Memcpy speed - (rust) block size 1048576 | 13554.28 MByte/s (±11440.92 MByte/s) |
30361.60 MByte/s (±24602.37 MByte/s) |
0.45 |
| Memcpy speed - (rust) block size 16777216 | 11527.26 MByte/s (±9495.59 MByte/s) |
27625.34 MByte/s (±22806.88 MByte/s) |
0.42 |
| Memset speed - (rust) block size 4096 | 50006.74 MByte/s (±36412.97 MByte/s) |
74373.47 MByte/s (±51976.48 MByte/s) |
0.67 |
| Memset speed - (rust) block size 1048576 | 13876.24 MByte/s (±11603.12 MByte/s) |
31110.89 MByte/s (±25033.24 MByte/s) |
0.45 |
| Memset speed - (rust) block size 16777216 | 11794.45 MByte/s (±9640.22 MByte/s) |
28386.93 MByte/s (±23265.03 MByte/s) |
0.42 |
| alloc_benchmarks Build Time | 214.21 s |
74.76 s |
2.87 ❗ |
| alloc_benchmarks File Size | 0.86 MB |
0.87 MB |
0.99 ❗ |
| Allocations - Allocation success | 91.38 % |
91.31 % |
1.00 ❗ |
| Allocations - Deallocation success | 100.00 % |
100.00 % |
1 |
| Allocations - Pre-fail Allocations | 61.60 % |
61.44 % |
1.00 ❗ |
| Allocations - Average Allocation time | 24427.07 Ticks (±1476.94 Ticks) |
5860.58 Ticks (±98.43 Ticks) |
4.17 ❗ |
| Allocations - Average Allocation time (no fail) | 25477.66 Ticks (±1996.28 Ticks) |
6554.81 Ticks (±92.86 Ticks) |
3.89 ❗ |
| Allocations - Average Deallocation time | 7466.13 Ticks (±2119.70 Ticks) |
1805.01 Ticks (±250.35 Ticks) |
4.14 ❗ |
| mutex_benchmark Build Time | 220.60 s |
79.82 s |
2.76 ❗ |
| mutex_benchmark File Size | 0.84 MB |
0.86 MB |
0.98 ❗ |
| Mutex Stress Test Average Time per Iteration - 1 Threads | 35.22 ns (±7.95 ns) |
12.10 ns (±0.41 ns) |
2.91 ❗ |
| Mutex Stress Test Average Time per Iteration - 2 Threads | 34.12 ns (±10.86 ns) |
40.26 ns (±1.68 ns) |
0.85 |
This comment was automatically generated by workflow using github-action-benchmark.
The driver operates the single request virtqueue and completes requests by polling the used ring through Virtq::dispatch_blocking, so a completion never needs an interrupt.
FatStream turns the sector-granular block driver into the byte-granular async Read/Write/Seek that hadris-fat expects, backed by a 16-sector LRU cache. VfatDirectory resolves paths component by component, because hadris-fat addresses everything by directory entry. File handles keep only a path and an offset.
Before these changes VirtioBlkDriver::flush answered Errno::Nosys when the device had not negotiated F::FLUSH, and that would now fail every sync on such a device. Now, the driver response Ok, if the flush feature isn't available.
The FAT volume writes through a sector cache, so a returned write() or unlink() is not yet on disk, and nothing forced it out: applications had no way to ask, and an orderly exit left whatever was still dirty in memory. On FAT that surfaces as lost cluster chains, because a directory entry and the FAT sectors of its chain reference each other while the cache evicts them independently. sys_fsync is backed by a new ObjectInterface::fsync that defaults to a no-op for objects without backing storage. Both vfat handles route it to a volume-wide sync, including the directory handle. syscalls::shutdown runs fs::sync() before anything else.
Every request the driver issued moved exactly one sector: I counted 44134 read requests for 44134 sectors and 23806 writes for 23806. A request costs a round trip and barely cares about its size, so a sequential pass over a 1 MiB file paid 2048 of them where a few dozen would do. A cache miss now fetches RUN_SECTORS following sectors in one request. Sectors of the run that are already cached keep their cached copy, which may be dirty and would then be newer than what the device just handed back. flush_all sorts the dirty slots by sector and writes each run of neighbours as a single request. CACHE_SECTORS grows to 128 so that one read-ahead cannot displace the FAT and directory sectors the cache exists for, and the staging buffers are split in two because a read-ahead that evicts a dirty sector flushes from inside its own transfer. Together that is 96 KiB instead of 8 KiB. Requests drop to 1602 reads and 1119 writes. Sequential reads with a 64 KiB buffer go from 29.3 to 320.4 MiB/s, writes with a 16 KiB buffer from 1.8 to 21.9 MiB/s, and creating a file from 565 to 201 us.
If the device supports multiqueues, create for every core a separate queue. In this case, the lock around the driver is bottleneck. Moving the lock into the driver solve this issue and improves the scalabillity.
dispatch is async and the queues use an async lock, so a task waiting for the device yields instead of spinning with interrupts masked. slot, install and flush_all follow
hadris-fat locks its sector stream per access, not per operation, so a lookup and the write that follows it were separate critical sections. Two cores creating files could pick the same directory slot, and the loser's file was gone: a concurrent benchmark failed in two of four runs on two cores, and in none of eight now. The volume lives in an async_lock::Mutex, so it cannot be reached without locking, and resolve, find and readdir_at take the volume as an argument instead of a static. FatDir now borrows from the guard rather than being 'static, which is what keeps a directory handle from outliving it.
Requests were awaited one at a time, leaving the device idle between them. Batch submits without waiting and finish collects every completion at the end, so the device works through the queue while the driver is still filling it. Holding the queue lock for the whole batch is what makes those completions unambiguously its own.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The device driver is quite simple and supports just one virtqueue. The file system is extended to support FAT. The extension based mainly on
Hadris FAT.I tested the current approach by using
rusty_demo, which was build as followed:HERMIT_LOG_LEVEL_FILTER="hermit::drivers::blk=debug,hermit::fs=debug,smoltcp,info" cargo build --release -Zbuild-std=std,panic_abort --target=aarch64-unknown-hermit -p rusty_demo --features hermit/pci,hermit/virtio-blk,hermit/loader,hermit/pci-ids,fsAfterwards, I tested the current approach by following command: