Docker: Chain egress proxy through upstream HTTP(S)_PROXY

[SohamKukreti]

Summary

Since 0.9.0, the Docker server's egress pinning proxy dials target IPs directly (asyncio.open_connection), ignoring HTTP_PROXY/HTTPS_PROXY. On hosts that can only reach the internet through a corporate proxy, every crawl fails with Page.goto: net::ERR_TIMED_OUT — this worked in 0.8.x. Reported in discussion #2041.

This PR teaches the pinning proxy to chain through the upstream proxy without weakening the SSRF/rebinding guarantees: validation (resolve_and_pin) still runs locally before the upstream is ever contacted, and the upstream is asked to CONNECT <pinned-ip>:<port> — never a hostname — so it performs no DNS resolution of its own and the pin holds.

Behavior:

• Reads standard HTTP_PROXY / HTTPS_PROXY / NO_PROXY (both cases; scheme-aware selection with fallback); CRAWL4AI_UPSTREAM_PROXY overrides them.
• NO_PROXY supports domain-suffix (.corp.lan) and IP/CIDR (192.168.0.0/16) entries — internal targets should be listed there to dial direct.
• Basic auth via http://user:pass@proxy:port (percent-decoded, UTF-8) sent preemptively; a 407/non-200 from the upstream fails fast with the existing opaque 403 and a server-side log line.
• Plain-HTTP requests are re-issued to the upstream in absolute form against the pinned IP with Connection: close forced, so a reused client connection cannot smuggle unvalidated requests past the pin.
• With no proxy env vars set, the dial path is byte-identical to current behavior — zero change for existing deployments.

Known limitations (documented in the README):

• Proxies that refuse CONNECT-to-an-IP (hostname/domain ACLs) and containers with no DNS at all are not supported yet (a future opt-in hostname mode could cover them).
• NTLM/Kerberos/SPNEGO proxies are not supported — front them with a local translator (e.g. cntlm, px) and point CRAWL4AI_UPSTREAM_PROXY at it.

List of files changed and why

• deploy/docker/egress_proxy.py — upstream detection (upstream_proxy(), _no_proxy_match(), _use_upstream()), the chained _dial() (CONNECT-to-pinned-IP, auth, header drain so upstream headers never leak into the tunnel), absolute-form + Connection: close for chained plain HTTP, startup/refusal log lines.
• deploy/docker/tests/test_security_egress_proxy.py — 3 new tests covering the security contract: chained CONNECT carries only the pinned IP and blocked targets produce zero upstream traffic; plain-HTTP absolute-form + anti-smuggling; env parsing (precedence, scheme selection, credentials, NO_PROXY routing). Plus an autouse fixture clearing proxy env vars so the suite is deterministic on dev machines behind proxies.
• deploy/docker/README.md — "Behind a corporate proxy" note in the Run the Container section (env vars, auth support, limitations).

How Has This Been Tested?

• Full Docker security suite: 319 passed, 1 xfailed (includes the 0.9 hardening contract tests, all unchanged).
• Library regression suite (tests/regression/, non-network): 293 passed; the single failure (test_cosine_basic, missing optional transformers) was verified pre-existing by re-running on the unmodified tree.
• Adversarial pass (32 targeted tests, real sockets, no mocking): env parsing edge cases (malformed/whitespace URLs, non-latin-1 passwords), NO_PROXY matching (anchored suffixes, CIDR, the exact list from [Feature Request]: Support for docker proxy #2041), dead/407/garbage upstreams (fast opaque 403, no hang), IPv6 pin bracketing, 8 concurrent tunnels with no cross-talk, and the smuggling regression.
• End-to-end: real Chromium → pinning proxy → a real forwarding "corporate" proxy → origin server, reproducing the [Feature Request]: Support for docker proxy #2041 topology — page + subresources each arrive as separate validated pinned-IP requests.
• Proxy-auth probe: preemptive Basic accepted first-try by an auth-requiring proxy; wrong/missing creds and NTLM-only proxies fail fast (single attempt, opaque 403, warning logged).

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes