Fix Windows deadlock: decode extension subprocess pipes as UTF-8

[OPTI-james]

Problem

On Windows, extension generations wedge silently mid-run (stuck at ~80%, idle GPU/CPU), and HuggingFace weight downloads stall the same way. Nothing is reported — the run just never finishes.

Root cause

ExtensionProcess launches the worker with bare text=True, so the host decodes the worker's stdout/stderr with the locale codec — cp1252 on Windows. The moment the worker prints something cp1252 can't decode (tqdm's block glyphs, e.g. ▏ = e2 96 8f; byte 0x8f is undefined in cp1252), the stderr reader thread dies:

Exception in thread Thread-12 (_stderr_loop):
  File "...\api\services\extension_process.py", line 213, in _stderr_loop
    ch = stream.read(1)
  File "...\encodings\cp1252.py", line 23, in decode
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 29: character maps to <undefined>

With no reader draining the pipe, the OS pipe buffer fills and the worker blocks forever on its next stderr write — deadlocked inside the diffusion loop with no error surfaced. Any tqdm output triggers it, so on Windows this bites on essentially every generation and every in-worker weight download.

Fix

• Decode both pipes explicitly as UTF-8 with errors="replace" so a stray byte can never kill the reader thread.
• Set PYTHONIOENCODING=utf-8 for the child (via setdefault, so an explicit override is respected) so both ends of the pipe agree on the encoding.

Verification

Reproduced on Windows 11 with the Hunyuan3D 2 Mini extension (RTX 5060 Ti): before the fix, three consecutive runs wedged at "Generating 3D shape…" (one sat for a full hour). After the fix, live tqdm streams through (Diffusion Sampling: 3/50, 10.53it/s) and the same run completes in ~80 s. Full test suite passes (npm test: 23 py + 77 node).

🤖 Generated with Claude Code

[lightningpixel]

Code review

Found 1 issue:

1. This fix duplicates a more complete fix already merged to dev (PR fix(extensions): force UTF-8 stdio on Windows to stop UnicodeEncodeError crashes #272, commit a607358), targeting main instead

PR #272 ("fix(extensions): force UTF-8 stdio on Windows to stop UnicodeEncodeError crashes") already fixed this exact deadlock on dev 9 days ago, touching the same _build_env()/Popen() call in extension_process.py, but via PYTHONUTF8=1 instead of PYTHONIOENCODING=utf-8. It also added child-side reconfiguration (api/services/stdio_utf8.py, called from api/main.py/api/runner.py), an equivalent fix for the Electron process-spawn path (electron/main/process-runner.ts), and fixed the related unicode-arrow prints that originally triggered the crash (api/services/generators/base.py, api/services/generator_registry.py). None of that is on main.

Merging this PR as-is (especially now that its base is being switched to dev) will collide with dev's existing fix on the same lines, and even if merged into main alone, it fixes only one of several places on main still missing the UTF-8 stdio fix.

modly/api/services/extension_process.py

Lines 84 to 92 in 92d4c1b

	# Only set SSL_CERT_FILE if not already provided (preserves corporate/custom certs).
	if "SSL_CERT_FILE" not in env:
	try:
	import certifi
	env["SSL_CERT_FILE"] = certifi.where()
	except ImportError:
	pass
	# Keep the child's stdio UTF-8 so it matches the UTF-8 decode on
	# our side of the pipes (Windows would otherwise pick cp1252).

Compare with the existing fix on dev: https://github.com/lightningpixel/modly/blob/dev/api/services/extension_process.py

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}