fix(crawler): warn when the body-visibility wait times out - #2145
Conversation
The wait result is discarded when ignore_body_visibility is True (the default), so a page whose body never becomes visible — ng-cloak/v-cloak left behind by an app that failed to bootstrap — costs a flat body_visibility_timeout ms on every crawl while still reporting success=True with an empty error_message. Nothing was logged, so the delay could only be found by instrumenting the pipeline, and body_visibility_timeout (added in #2131) was undiscoverable by the users who most needed it. Warn once when the wait times out and its result is ignored, naming the option and the timeout that applied. The strict path (ignore_body_visibility=False) already raises with visibility details, so it stays quiet. Fixes #2144
Review follow-up on three points: force_verbose — AsyncLogger drops a plain warning when verbose is off (async_logger.py:238), and CrawlerRunConfig.verbose overrides the logger at arun() time. Servers and batch jobs run with verbose off, so the warning was suppressed in exactly the deployments that hit this and can least afford a silent 30s. Verified end to end: with verbose=False the warning was invisible before, and prints now. False positives — csp_compliant_wait also returns False when the page evaluation itself fails (page closed, context destroyed by a redirect), which costs no time. Time the wait and only warn when it actually burned its budget, and report the elapsed time rather than the configured timeout so the number matches the delay being explained. Test — select the warning call by message instead of reading the last call, which only passed because text_mode=True happened to skip the wait_for_images warning. Added coverage for force_verbose and for the early-failure path.
|
Addressed all three review findings in 2 — invisible where it matters most. Confirmed and worse than described: 1 — warning can be false. Fixed by timing the wait and only warning when it actually consumed its budget (>= 90% of 3 — brittle assertion. Now selects the warning call by message instead of reading the last call. Tests: 45 passed (5 in this file, up from 3, plus |
1 participant
Summary
Fixes #2144.
When the body-visibility wait times out, nothing is logged. The crawl reports
success=Truewith an emptyerror_message, andbody_visibility_timeoutms disappear with no indication of where they went — because the wait's result is discarded wheneverignore_body_visibilityisTrue(the default).That silence is what made the delay in #2129 impossible to attribute without instrumenting the pipeline, and what makes
body_visibility_timeout(added in #2131) undiscoverable by exactly the users who need it.This warns once when the wait times out and its result is ignored, naming the option and the timeout that actually applied. The strict path (
ignore_body_visibility=False) already raises with visibility details, so it stays quiet — no double reporting.No config, no behavior change, no new dependency. One log line.
List of files changed and why
crawl4ai/async_crawler_strategy.py— warn after the visibility wait when it timed out and the result is about to be discarded.tests/test_body_visibility_warning.py— new: warns and reports the applied timeout; silent when the body is visible; silent on the strict path.How Has This Been Tested?
Live output on the #2129 repro (two identical pages differing only by
ng-cloakon<body>) — warning on the hidden page, silence on the visible one:tests/test_body_visibility_warning.py+tests/test_config_defaults.py— 43 passed.test_warns_when_wait_times_out_and_result_is_discardedfails.tests/regression/— 320 passed, 2 failed. Both fail identically ondevelopwithout this change:test_cosine_basicis an ImportError from the missing optionaltransformersextra, andtest_soft_404_filters_probesis pre-existing.Checklist: