Skip to content

fix: resolve /license/validate 502s (stale upstream DNS) + non-JSON error crash - #75

Merged
man4ish merged 2 commits into
mainfrom
fix/license-validate-502-stale-upstream-dns
Sep 1, 2026
Merged

fix: resolve /license/validate 502s (stale upstream DNS) + non-JSON error crash#75
man4ish merged 2 commits into
mainfrom
fix/license-validate-502-stale-upstream-dns

Conversation

@man4ish

@man4ish man4ish commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes the "Unexpected token '<', "<!DOCTYPE "... is not valid JSON" error
reported on webstudio.omnibioai.org's license activation screen.

Root cause (live-reproduced on the production router): auth-service
restarted independently of nginx-router. nginx's upstream auth { server auth-service:8001; } block resolves that hostname once, at startup —
after the restart it kept sending traffic to the container's old, dead IP.
Every /license/, /auth/, /roles/, /users/, /orgs/ request 502'd
for ~2 hours (03:43:54–05:41:52) for any real client that hit them.
session.js then made the failure mode confusing on top of that: it
called res.json() unconditionally, so the raw HTML error page surfaced
as an opaque SyntaxError instead of a readable message.

Changes

  • docker/nginx-router.conf: applies the same resolver + variable-based
    proxy_pass pattern already used for billing-service to auth,
    workbench, lims, policy-engine, and hpc-policy-engine, so these
    five upstreams re-resolve DNS per-request instead of once at startup.
    rag, gateway, audit, control, and the rest are untouched (still
    static, out of scope). The resolver directive is hoisted to file/http
    context so it's shared by every server{} block, including
    control.omnibioai.org / lims.omnibioai.org, which previously had none
    at all — a straight copy-paste of the variable pattern into those blocks
    would have failed to start without this.
  • src/ui/lib/web/session.js: adds a readJson() helper (never
    throws) and uses it in loginWithLicenseKey, getCurrentUser, and
    confirmOAuthLink. Preference order on failure: known reason code →
    backend-sent error/message/detail field → generic
    `<action> failed (${res.status})`. Real JSON error bodies from the
    backend are still surfaced, not swallowed.

Verification

  • nginx -t clean against the real Docker network, before and after.
  • docker compose up -d --no-deps --force-recreate nginx-router, then:
    • POST /license/validate200, application/json,
      {"valid":false,...,"reason":"invalid_key"} (was 502 HTML).
    • POST /auth/validate200, {"valid":false}.
  • Watched the nginx error log for ~3.5 min post-recreate with a heartbeat
    probe every 30s (7/7 clean) — zero connect() failed recurrence for
    auth-service.

Known gap (not fixed here)

The pre-fix outage's full blast radius isn't reconstructable — the old
container's logs were deleted on --force-recreate (no log volume, no
shipping sidecar, and cloudflared's journal doesn't log per-request
status codes for origin-returned errors). Only one real external client
is confirmed from an incidental capture during diagnosis. Follow-up:
persist or ship nginx-router logs so a postmortem like this is
answerable after a recreate.

🤖 Generated with Claude Code

man4ish and others added 2 commits September 1, 2026 00:46
… lims,

policy-engine, hpc-policy-engine

Static upstream blocks resolve hostnames once at nginx startup. When
auth-service restarted independently of nginx-router, DNS pointed at a
stale container IP, causing connection refused / 502 on all
/license/, /auth/, /roles/, /users/, /orgs/ requests for ~2 hours
(03:43:54–05:41:52).

Applies the same resolver + variable-based proxy_pass pattern already
used for billing-service to the five affected upstreams. Verified live:
nginx -t clean, /license/validate and /auth/validate return real JSON
post-recreate, 7/7 heartbeat probes clean over 3.5min watch window,
zero connect() failed recurrence.

Known gap: pre-fix outage blast radius not fully reconstructable —
old container's logs were lost on force-recreate. Follow-up: add
persistent/shipped nginx-router logging.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0135NEjymEpbh7fhWwqz9CwL
loginWithLicenseKey, getCurrentUser, and confirmOAuthLink called
res.json() unconditionally, so any non-JSON error body (nginx error
page, SPA index.html on a routing miss) surfaced as a raw
"Unexpected token '<'" SyntaxError instead of a usable message.

Adds readJson() helper: never throws, prefers backend-sent
error/message/detail fields, falls back to a status-based message.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0135NEjymEpbh7fhWwqz9CwL
@man4ish
man4ish merged commit af29452 into main Sep 1, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant