fix: resolve /license/validate 502s (stale upstream DNS) + non-JSON error crash - #75
Merged
Merged
Conversation
… lims, policy-engine, hpc-policy-engine Static upstream blocks resolve hostnames once at nginx startup. When auth-service restarted independently of nginx-router, DNS pointed at a stale container IP, causing connection refused / 502 on all /license/, /auth/, /roles/, /users/, /orgs/ requests for ~2 hours (03:43:54–05:41:52). Applies the same resolver + variable-based proxy_pass pattern already used for billing-service to the five affected upstreams. Verified live: nginx -t clean, /license/validate and /auth/validate return real JSON post-recreate, 7/7 heartbeat probes clean over 3.5min watch window, zero connect() failed recurrence. Known gap: pre-fix outage blast radius not fully reconstructable — old container's logs were lost on force-recreate. Follow-up: add persistent/shipped nginx-router logging. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0135NEjymEpbh7fhWwqz9CwL
loginWithLicenseKey, getCurrentUser, and confirmOAuthLink called res.json() unconditionally, so any non-JSON error body (nginx error page, SPA index.html on a routing miss) surfaced as a raw "Unexpected token '<'" SyntaxError instead of a usable message. Adds readJson() helper: never throws, prefers backend-sent error/message/detail fields, falls back to a status-based message. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0135NEjymEpbh7fhWwqz9CwL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the "Unexpected token '<', "<!DOCTYPE "... is not valid JSON" error
reported on webstudio.omnibioai.org's license activation screen.
Root cause (live-reproduced on the production router):
auth-servicerestarted independently of
nginx-router. nginx'supstream auth { server auth-service:8001; }block resolves that hostname once, at startup —after the restart it kept sending traffic to the container's old, dead IP.
Every
/license/,/auth/,/roles/,/users/,/orgs/request 502'dfor ~2 hours (03:43:54–05:41:52) for any real client that hit them.
session.jsthen made the failure mode confusing on top of that: itcalled
res.json()unconditionally, so the raw HTML error page surfacedas an opaque
SyntaxErrorinstead of a readable message.Changes
docker/nginx-router.conf: applies the same resolver + variable-basedproxy_passpattern already used forbilling-servicetoauth,workbench,lims,policy-engine, andhpc-policy-engine, so thesefive upstreams re-resolve DNS per-request instead of once at startup.
rag,gateway,audit,control, and the rest are untouched (stillstatic, out of scope). The
resolverdirective is hoisted to file/httpcontext so it's shared by every
server{}block, includingcontrol.omnibioai.org/lims.omnibioai.org, which previously had noneat all — a straight copy-paste of the variable pattern into those blocks
would have failed to start without this.
src/ui/lib/web/session.js: adds areadJson()helper (neverthrows) and uses it in
loginWithLicenseKey,getCurrentUser, andconfirmOAuthLink. Preference order on failure: knownreasoncode →backend-sent
error/message/detailfield → generic`<action> failed (${res.status})`. Real JSON error bodies from thebackend are still surfaced, not swallowed.
Verification
nginx -tclean against the real Docker network, before and after.docker compose up -d --no-deps --force-recreate nginx-router, then:POST /license/validate→200,application/json,{"valid":false,...,"reason":"invalid_key"}(was502HTML).POST /auth/validate→200,{"valid":false}.probe every 30s (7/7 clean) — zero
connect() failedrecurrence forauth-service.Known gap (not fixed here)
The pre-fix outage's full blast radius isn't reconstructable — the old
container's logs were deleted on
--force-recreate(no log volume, noshipping sidecar, and
cloudflared's journal doesn't log per-requeststatus codes for origin-returned errors). Only one real external client
is confirmed from an incidental capture during diagnosis. Follow-up:
persist or ship
nginx-routerlogs so a postmortem like this isanswerable after a recreate.
🤖 Generated with Claude Code