Fix vsock test flake

[papertigers]

This fixes #1197.

The test is a simple off by one error which leads to a race condition. We end up waiting for
harness.tx_used_idx() to reach one less descriptor than we should be, which means calling
reset_tx_cursors at the top of the loop resets the QueueWriter too soon. This opens a window for
the poller to not read all the RW packets that we are sending. The log in the test confirms this,
and explains why our read_exact() fails.

Aug 14 02:51:12.719 WARN dropping invalid vsock packet: vsock packet header reported 8192 bytes but the descriptor chain contains 0, component: vsock-test

[papertigers]

Before the fix:

Stress test [   5.027s] iteration 106/3000: 1 test run: 0 passed, 1 failed, 269 skipped                                                                                                    ────────────
      Summary [   7.561s] 106/3000 stress run iterations: 105 passed, 1 failed
         FAIL [   5.025s] [ 106/3000] propolis vsock::poller::test::credit_update_sent_after_flushing_half_buffer
error: test run failed

After the fix:

 Stress test iteration 3000/3000 (00:01:26 elapsed so far, 1 iteration remaining)
        PASS [   0.031s] [3000/3000] propolis vsock::poller::test::credit_update_sent_after_flushing_half_buffer                                                                             Stress test [   0.033s] iteration 3000/3000: 1 test run: 1 passed, 269 skipped
────────────
Summary [  86.578s] 3000/3000 stress run iterations: 3000 passed

[papertigers]

One more follow up, I have been running a stress test for ~4hrs and ~8500 iterations and have not encountered any issues.

[iximeow]

sgtm, one suggestion about how we figure out what to wait for but I'm not sure if I'm missing something and a fixed flake is a fixed flake.

[iximeow]

the diff seems fine but.. if things are quiescent until we publish_chain(); queue_notify(), couldn't we do this at the start of the loop and have a mildly more straightforward check like..

let expected_tx = harness.tx_used_idx() + tx_consumed;

/* do tx */

wait_for_condition(|| harness.tx_used_idx() >= expected_tx, 5000);

and not worry about whatever the state was as we got to the loop?

[papertigers]

I fixed the awkwardness here and I think it reads much more clearly now. If you are good with it, I will merge..

Summary [  95.536s] 3000/3000 stress run iterations: 3000 passed

[iximeow]

yeah, this is a lot nicer, thanks 🙏

[iximeow]

for the sake of less CI flakiness, i'm ok with merging even though header-check fails (fixed by #1208, and only failing because CI now has a newer viona version than Propolis knows about. but that new version is only additive, so ... it's fine!)