Skip to content

fix: resolve /license/validate 502s (stale upstream DNS) + non-JSON error crash #75

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
man4ish merged 2 commits into from
Sep 1, 2026
Merged

fix: resolve /license/validate 502s (stale upstream DNS) + non-JSON error crash #75

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d003395
fix(nginx): use resolver-based dynamic upstreams for auth, workbench,…
man4ish Sep 1, 2026
52a58ce
fix(web): guard against non-JSON error responses in session.js
man4ish Sep 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 76 additions & 37 deletions docker/nginx-router.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,28 @@
## locations above)
##

# Deferred (request-time, not startup-time) DNS resolution for auth,
# workbench, lims, policy-engine, and hpc-policy-engine -- same reasoning
# billing-service's own `resolver` + `set $billing_upstream ...` pattern
# below documents, generalized to these five and hoisted to http-context
# (this file is `include`d inside nginx.conf's http{} block) so every
# server{} in this file shares one resolver instead of repeating it.
# A static `upstream auth { server auth-service:8001; }` block (the old
# form, still used below for services this pass didn't touch) resolves its
# hostname exactly once, at nginx startup/reload -- if that backend
# container is later recreated and gets a new Docker-network IP (a plain
# restart, a redeploy, anything short of restarting nginx-router itself),
# nginx keeps sending traffic to the old, now-dead IP and every request
# 502s with "connect() failed (111: Connection refused)" until nginx-router
# is recreated too. Confirmed live on this exact router: auth-service
# restarted after nginx-router did and every /license/, /auth/, /roles,
# /users/, /orgs request 502'd for real external traffic until the stale
# IP was flushed. Resolving via variable instead re-resolves per-request
# (cached for `valid=10s`), so a backend restart heals itself within one
# DNS TTL instead of silently 502ing until someone notices and recreates
# the router.
resolver 127.0.0.11 valid=10s;

limit_req_zone $binary_remote_addr zone=auth_limit:10m rate=10r/m;

# X-Forwarded-Proto pass-through -- 2026-07-27 fix for LIMS redirect-loop known-issue.
Expand Down Expand Up @@ -118,13 +140,13 @@ map $http_authorization $control_authorization {
'' $control_cookie_authorization;
}

upstream workbench { server workbench:8000; }
upstream lims { server lims:7000; }
# workbench, lims, auth, policy-engine, and hpc-policy-engine deliberately
# have NO static upstream {} block here anymore -- see the resolver comment
# above. Every location that used to `proxy_pass http://<name>;` now does
# `set $<name>_upstream <service>:<port>; proxy_pass http://$<name>_upstream;`
# instead, same pattern as $billing_upstream below.
upstream rag { server rag:8096; }
upstream gateway { server api-gateway:8080; }
upstream auth { server auth-service:8001; }
upstream policy { server policy-engine:8002; }
upstream hpc { server hpc-policy-engine:8003; }
upstream audit { server security-audit:8004; }
upstream control { server control-center:7070; }
upstream grafana { server grafana:3000; }
Expand Down Expand Up @@ -188,12 +210,14 @@ server {
# nginx startup and refuses to start at all if that name doesn't
# exist -- exactly the crash-loop class of bug already documented for
# web-ui elsewhere in this file. Resolving billing-service by variable
# instead defers DNS lookup to request time (via Docker's embedded
# resolver), so on deployments without billing-service the /billing and
# /entitlements routes below fail per-request with a 502 instead of
# taking the whole router down. Revisit once billing-service has a
# published release image and is added to both release compose files.
resolver 127.0.0.11 valid=10s;
# instead defers DNS lookup to request time (via the `resolver` now
# declared once, http-context, near the top of this file -- shared by
# every server{} block, auth/workbench/lims/policy/hpc's own
# $<name>_upstream locations included), so on deployments without
# billing-service the /billing and /entitlements routes below fail
# per-request with a 502 instead of taking the whole router down.
# Revisit once billing-service has a published release image and is
# added to both release compose files.

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
Expand Down Expand Up @@ -247,7 +271,7 @@ server {
index index.html;
autoindex off;
}
location ^~ /_svc/workbench { rewrite ^/_svc/workbench(/.*)$ $1 break; proxy_pass http://workbench; }
location ^~ /_svc/workbench { rewrite ^/_svc/workbench(/.*)$ $1 break; set $workbench_upstream workbench:8000; proxy_pass http://$workbench_upstream; }
location ^~ /_svc/lims {
# NOTE: this location defines its own proxy_set_header directives, which means
# none of the server-level ones above are inherited (nginx behavior: a location
Expand All @@ -258,7 +282,8 @@ server {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $router_xfp;
rewrite ^/_svc/lims(/.*)$ $1 break;
proxy_pass http://lims;
set $lims_upstream lims:7000;
proxy_pass http://$lims_upstream;
}
# LIMS's session login (django.contrib.auth's LoginView/LogoutView,
# lab_data_manager/urls.py). Bare, not under /_svc/lims -- LIMS has no
Expand All @@ -270,7 +295,8 @@ server {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $router_xfp;
proxy_pass http://lims;
set $lims_upstream lims:7000;
proxy_pass http://$lims_upstream;
}
location = /_svc/rag {
# Internal rewrite, not a client-visible 301 -- this nginx never
Expand Down Expand Up @@ -483,7 +509,8 @@ server {
location ^~ /_svc/auth {
limit_req zone=auth_limit burst=5 nodelay;
rewrite ^/_svc/auth(/.*)$ $1 break;
proxy_pass http://auth;
set $auth_upstream auth-service:8001;
proxy_pass http://$auth_upstream;
}
# Bare /auth/* — OAuth2 login/callback redirect targets registered with
# Google/GitHub/Microsoft are https://webstudio.omnibioai.org/auth/{provider}/callback,
Expand All @@ -494,12 +521,14 @@ server {
# depend on that route.
location ^~ /auth/ {
limit_req zone=auth_limit burst=5 nodelay;
proxy_pass http://auth;
set $auth_upstream auth-service:8001;
proxy_pass http://$auth_upstream;
}
# License key validation endpoint
location ^~ /license/ {
limit_req zone=auth_limit burst=5 nodelay;
proxy_pass http://auth;
set $auth_upstream auth-service:8001;
proxy_pass http://$auth_upstream;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
Expand All @@ -516,7 +545,8 @@ server {
location ^~ /api/license/ {
limit_req zone=auth_limit burst=5 nodelay;
rewrite ^/api/license(/.*)$ $1 break;
proxy_pass http://auth;
set $auth_upstream auth-service:8001;
proxy_pass http://$auth_upstream;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
Expand All @@ -533,11 +563,13 @@ server {
# like /_svc/control needed here).
location ^~ /roles {
limit_req zone=auth_limit burst=5 nodelay;
proxy_pass http://auth;
set $auth_upstream auth-service:8001;
proxy_pass http://$auth_upstream;
}
location ^~ /users/ {
limit_req zone=auth_limit burst=5 nodelay;
proxy_pass http://auth;
set $auth_upstream auth-service:8001;
proxy_pass http://$auth_upstream;
}
# Billing service (billing-service:8005) — org-scoped billing/
# subscription/entitlement data. PR F (v0.7.0 release stabilization):
Expand Down Expand Up @@ -577,7 +609,8 @@ server {
# header comment) rather than discovered later as a "blank page" bug.
location ^~ /orgs {
limit_req zone=auth_limit burst=5 nodelay;
proxy_pass http://auth;
set $auth_upstream auth-service:8001;
proxy_pass http://$auth_upstream;
}
# Workbench (${MACHINE_DIR}/omnibioai, Django) — same bare-route pattern
# as /roles and /users/ above. nginx strips /_svc/workbench before
Expand All @@ -603,36 +636,37 @@ server {
# per its own comment. Distinct from this router's own /_health.
# - /home/ : plugins/home/urls.py, explicit intentional alias
# ("Optional: keep an explicit /home/ alias for the main dashboard").
location ^~ /static/ { proxy_pass http://workbench; }
location ^~ /admin/ { proxy_pass http://workbench; }
location ^~ /static/ { set $workbench_upstream workbench:8000; proxy_pass http://$workbench_upstream; }
location ^~ /admin/ { set $workbench_upstream workbench:8000; proxy_pass http://$workbench_upstream; }
# allauth (google/github/microsoft + password) — path("accounts/",
# include("allauth.urls")). Bare, same reason as /static/ and /admin/
# above: Django has no FORCE_SCRIPT_NAME/prefix-awareness, so it always
# emits bare /accounts/... links regardless of any proxy prefix. Issue
# #5: LIMS's own login used to collide here too -- LIMS moved to
# /lims-accounts/ instead (see below) since only one service can own
# this bare path.
location ^~ /accounts/ { proxy_pass http://workbench; }
location ^~ /accounts/ { set $workbench_upstream workbench:8000; proxy_pass http://$workbench_upstream; }
# Intentionally a full /api/ prefix match, not limited to the specific
# known leaf routes (api/auth/me/, api/run/, api/dev/objects/...) —
# Django 404s on unmatched sub-paths, which is strictly better than the
# SPA catch-all's silent 200. Do not narrow this to explicit leaf
# routes without re-checking that assumption still holds.
location ^~ /api/ { proxy_pass http://workbench; }
location ^~ /api/ { set $workbench_upstream workbench:8000; proxy_pass http://$workbench_upstream; }
# Exact matches only — the real Django routes are literally just
# /health and /health/ (plugins/home/urls.py), nothing else starting
# with /health/. A ^~ prefix match here would also swallow an unrelated
# future path like /health-dashboard.
location = /health { proxy_pass http://workbench; }
location = /health/ { proxy_pass http://workbench; }
location ^~ /home/ { proxy_pass http://workbench; }
location = /health { set $workbench_upstream workbench:8000; proxy_pass http://$workbench_upstream; }
location = /health/ { set $workbench_upstream workbench:8000; proxy_pass http://$workbench_upstream; }
location ^~ /home/ { set $workbench_upstream workbench:8000; proxy_pass http://$workbench_upstream; }
# Django Channels websocket routes (omnibioai/asgi.py +
# plugins/collaboration/routing.py) — /ws/workflow/progress/{id}/ and
# /ws/collaboration/{workspace_id}/. Unlike the plain HTTP routes above,
# these need the Upgrade/Connection handshake headers (same treatment
# as /jupyter/ and /vscode/ below) or the WebSocket handshake fails.
location ^~ /ws/ {
proxy_pass http://workbench;
set $workbench_upstream workbench:8000;
proxy_pass http://$workbench_upstream;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
Expand Down Expand Up @@ -666,19 +700,21 @@ server {
# control-center's actually-served page only calls /report/generate
# and /report/status through the /_svc/control-prefixed apiPath()
# helper, never bare. Matches #6's own closing notes.
location ^~ /dashboard/ { proxy_pass http://workbench; }
location ^~ /ops/ { proxy_pass http://workbench; }
location ^~ /plugins/ { proxy_pass http://workbench; }
location ^~ /dashboard/ { set $workbench_upstream workbench:8000; proxy_pass http://$workbench_upstream; }
location ^~ /ops/ { set $workbench_upstream workbench:8000; proxy_pass http://$workbench_upstream; }
location ^~ /plugins/ { set $workbench_upstream workbench:8000; proxy_pass http://$workbench_upstream; }
location ^~ /_svc/policy {
rewrite ^/_svc/policy/?(.*)$ /$1 break;
proxy_pass http://policy;
set $policy_upstream policy-engine:8002;
proxy_pass http://$policy_upstream;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_buffering off;
}
location ^~ /_svc/hpc {
rewrite ^/_svc/hpc/?(.*)$ /$1 break;
proxy_pass http://hpc;
set $hpc_upstream hpc-policy-engine:8003;
proxy_pass http://$hpc_upstream;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_buffering off;
Expand Down Expand Up @@ -803,7 +839,8 @@ server {
# auth-service.
location ^~ /auth/ {
limit_req zone=auth_limit burst=5 nodelay;
proxy_pass http://auth;
set $auth_upstream auth-service:8001;
proxy_pass http://$auth_upstream;
}

# Everything else: control-center's own backend, bare (no /_svc/control
Expand Down Expand Up @@ -851,7 +888,8 @@ server {
# hostname, which was previously impossible (PR7: 405 on all four).
location ^~ /auth/ {
limit_req zone=auth_limit burst=5 nodelay;
proxy_pass http://auth;
set $auth_upstream auth-service:8001;
proxy_pass http://$auth_upstream;
}

# /auth/verify (PR8's real gateway route) is a separate upstream
Expand All @@ -874,6 +912,7 @@ server {
# second network hop in front of that unchanged container, it does not
# reimplement any of its routing.
location / {
proxy_pass http://lims;
set $lims_upstream lims:7000;
proxy_pass http://$lims_upstream;
}
}
55 changes: 41 additions & 14 deletions src/ui/lib/web/session.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,25 @@ export function authUrl(path) {

let cachedUser = null;

// Reads a fetch Response body as JSON without ever throwing. A backend
// error response isn't guaranteed to be JSON — an nginx 502/504 page, the
// SPA's own index.html on a routing miss (see nginx-router.conf's header
// comment on that failure mode), or a plain-text 500 are all real
// possibilities in front of this same-origin proxy — and calling
// `res.json()` directly on one of those throws a raw
// `SyntaxError: Unexpected token '<', "<!DOCTYPE "... is not valid JSON`
// that callers used to let bubble straight into the UI. Returns the parsed
// body, or null if it was empty or not valid JSON.
async function readJson(res) {
const text = await res.text().catch(() => "");
if (!text) return null;
try {
return JSON.parse(text);
} catch (_) {
return null;
}
}

function notify() {
window.dispatchEvent(new CustomEvent(SESSION_EVENT));
}
Expand Down Expand Up @@ -143,9 +162,18 @@ export async function loginWithLicenseKey(key, email, platform = "web") {
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ key, email, platform }),
});
const data = await res.json();
if (!res.ok || !data.valid) {
throw new Error(LICENSE_ERROR_MESSAGES[data.reason] || "License validation failed");
const data = await readJson(res);
if (!res.ok || !data?.valid) {
// Preference order: a known `reason` code maps to its friendly message
// (unchanged from before); otherwise surface whatever message the
// backend actually sent (a real JSON error body auth-service returned,
// just not one of the LICENSE_ERROR_MESSAGES codes above); otherwise
// — the body wasn't JSON at all (nginx error page, SPA index.html
// fallback, empty body) — fall back to a plain, status-coded message
// instead of leaking the raw HTML/parse failure into the UI.
const reasonMessage = data?.reason && LICENSE_ERROR_MESSAGES[data.reason];
const backendMessage = data?.error || data?.message || data?.detail;
throw new Error(reasonMessage || backendMessage || `License validation failed (${res.status})`);
}
setSession(data.access_token, data.refresh_token);
return getCurrentUser({ force: true });
Expand All @@ -168,8 +196,12 @@ export async function getCurrentUser({ force = false } = {}) {
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ token }),
});
const data = await res.json();
if (!data.valid) {
// readJson (not res.json()) so a non-JSON body (e.g. a transient 502)
// falls through to the same "not valid" branch below instead of
// throwing past clearSession() into the outer catch, which used to
// leave a dead token in localStorage instead of actually clearing it.
const data = await readJson(res);
if (!data?.valid) {
clearSession();
return null;
}
Expand Down Expand Up @@ -216,16 +248,11 @@ export async function confirmOAuthLink(linkToken, password) {
body: JSON.stringify({ link_token: linkToken, password }),
});
if (!res.ok) {
let detail = "Could not confirm the link";
try {
detail = (await res.json())?.detail || detail;
} catch (_) {
// no JSON body
}
throw new Error(detail);
const data = await readJson(res);
throw new Error(data?.detail || `Could not confirm the link (${res.status})`);
}
const data = await res.json();
setSession(data.access_token, data.refresh_token);
const data = await readJson(res);
setSession(data?.access_token, data?.refresh_token);
return getCurrentUser({ force: true });
}

Expand Down
Loading