BasicCrawler finds the throttler with a bare isinstance(self._request_manager, ThrottlingRequestManager) check in three places: the respect_robots_txt_file startup warning (_basic_crawler.py:708), the 429 recording (:1644), and the robots.txt crawl-delay (:1710).
RequestLoader.to_tandem() wraps the manager in a RequestManagerTandem, so await loader.to_tandem(throttler) silently drops the per-domain backoff and the crawl-delay. Both warnings then say the crawler is not using ThrottlingRequestManager, while one is configured.
RequestManagerTandem exposes no accessor for the manager it wraps, so a fix needs new public API. #2158 left this out of scope on purpose.
✍️ Drafted by Claude Code
BasicCrawlerfinds the throttler with a bareisinstance(self._request_manager, ThrottlingRequestManager)check in three places: therespect_robots_txt_filestartup warning (_basic_crawler.py:708), the 429 recording (:1644), and the robots.txt crawl-delay (:1710).RequestLoader.to_tandem()wraps the manager in aRequestManagerTandem, soawait loader.to_tandem(throttler)silently drops the per-domain backoff and the crawl-delay. Both warnings then say the crawler is not usingThrottlingRequestManager, while one is configured.RequestManagerTandemexposes no accessor for the manager it wraps, so a fix needs new public API. #2158 left this out of scope on purpose.✍️ Drafted by Claude Code