Stop test-harness artefacts from losing LAVA results - #554
Open
Ricardo Salveti (ricardosalveti) wants to merge 5 commits into
Open
Stop test-harness artefacts from losing LAVA results#554Ricardo Salveti (ricardosalveti) wants to merge 5 commits into
Ricardo Salveti (ricardosalveti) wants to merge 5 commits into
Conversation
Contributor
Author
|
This is the outcome of an investigation of all the lava jobs executed over the past 8 days. |
Ricardo Salveti (ricardosalveti)
force-pushed
the
claude-findings
branch
from
August 27, 2026 15:17
e0f63fa to
e98feeb
Compare
Ricardo Salveti (ricardosalveti)
marked this pull request as ready for review
August 27, 2026 20:14
The script quiets printk around each signal printf, but printf returns once the line is in the tty buffer, not once the UART has sent it, so printk resumes while the signal is still on the wire and can split it: <<<LAVA_SIGNAL_TESTCASE TEST_CASE_ID=cdsp_remoteproc RESU[ 69.2] LAVA cannot parse that, so a passing test is recorded as a failure. Over 12658 jobs run between 19 and 27 Aug 2026, 62 of 287 recorded test-case failures had no RESULT=FAIL in the log at all, and 36 of 200 sampled jobs carried a spliced signal. Quiet printk across the whole emission instead, sleeping on both sides (LAVA_SIGNAL_DRAIN_SECONDS, default 0.3s) so queued kernel output finishes before the first printf and the signals clear the link before printk resumes. The drain is skipped when printk could not be quieted, the value is validated where it is defaulted, and the signals are read into memory up front so the INT/TERM trap deleting the signal file cannot suppress them. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Ricardo Salveti <ricardo.salveti@oss.qualcomm.com>
display_connected_summary() warns via log_warn when /sys/class/drm is missing, and log() writes to stdout, which callers capture as the summary: a board whose display driver never probed reads as having a connected panel, because "[WARN] ..." is neither empty nor "none". Send the warning to stderr. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ricardo Salveti <ricardo.salveti@oss.qualcomm.com>
display_log_snapshot_and_require_connector() samples /sys/class/drm once, but hot-plug detection is asynchronous, so a board with a panel can still read disconnected seconds after boot. Between 19 and 27 Aug 2026 boards tagged display skipped at rates only a race explains: hamoa-iot-evk-04 42 of 196 runs, lemans-hyd-01 36 of 197, and every skip is silent. Poll instead, bounded by DISPLAY_CONNECTOR_WAIT (default 20s) every DISPLAY_CONNECTOR_POLL (2s), on the monotonic clock so an NTP step cannot cut the wait short. Headless boards reach the same skip and pay the wait once per job rather than once per test in the ten suites sharing the gate: an exhausted wait leaves a tmpfs marker that reduces later checks to a single sample until a connector shows up or the board reboots. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Ricardo Salveti <ricardo.salveti@oss.qualcomm.com>
The boot check compares a state snapshot taken moments after login against running, so an instance still coming up is recorded as a failure; on lemans-evk the second CDSP loses that race. Between 19 and 27 Aug 2026 cdsp_remoteproc failed this way on every board with more than one instance (kaanapali-mtp-06 8 of 17, lemans-02 9 of 89), at rates that rule out a processor that never boots. Re-read the state through the existing wait_remoteproc_state() before deciding, bounded by BOOT_TO/--boot-to (default 30s, 0 restores the single shot), in all four suites sharing the check. The wait runs on get_monotonic_seconds() so an NTP step cannot cut it short, and a non-numeric BOOT_TO falls back to the default instead of silently disabling the wait. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Ricardo Salveti <ricardo.salveti@oss.qualcomm.com>
pw-record has no duration flag on this path, so the recording is ended by the external watchdog, whose clock starts before PipeWire has opened the device: whatever the graph spends coming up is taken out of the recording, and a slow start fails the duration check (duration-too-short:0.724<3.500). Between 19 and 27 Aug 2026 this cost iq-x7181-evk 19 to 29 failures per AudioRecord config out of ~207 runs. Run the watchdog for the requested duration plus AUDIO_RECORD_START_GRACE (default 5s) so a slow start eats the headroom instead of the recording. The duration validated against is unchanged and an explicit --timeout still overrides the watchdog. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Ricardo Salveti <ricardo.salveti@oss.qualcomm.com>
Ricardo Salveti (ricardosalveti)
force-pushed
the
claude-findings
branch
from
August 30, 2026 16:32
e98feeb to
25f2134
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Analysis of all 12,658 meta-qcom LAVA jobs run between 19 and 27 Aug 2026
found that a large share of the red in CI is produced by the harness rather
than by the platforms under test: 62 of 287 recorded test-case failures had
no RESULT=FAIL in the log at all, and 616 jobs died without reporting any
result. This series fixes the six causes that are ours to fix.
quiet window closes before the line has left the UART. 36 of 200 sampled
jobs were affected; cdsp_remoteproc lost 15 of its 51 failures this way.
have a panel skip 18-21% of the time (hamoa-iot-evk-04, lemans-hyd-01).
failing instances that are still coming up (cdsp1 on sa8775p, 9-21%).
truncating recordings below the duration gate.
Each commit carries the measured failure rates it addresses. No test verdicts
change: every fix either lets a passing test be read, or makes a failing test
report as a failure instead of taking the job down with it.
All touched scripts pass
shellcheckandsh -n.