Skip to content

fix(k8s): guard np_trace_flush when tracing SDK is not loaded - #252

Merged
fedemaleh merged 2 commits into
betafrom
fix/trace-flush-guard
Sep 8, 2026
Merged

fedemaleh merged 2 commits into
betafrom
fix/trace-flush-guard

Conversation

@fedemaleh

Copy link
Copy Markdown
Collaborator

Summary

Deployments were printing this at the exact moment they succeeded:

[deployment] .../k8s/deployment/wait_deployment_active: line 372: np_trace_flush: command not found

The deployment-ready branch in wait_deployment_active is gated on command -v np_scope_progress. That guard is always true: np_scope_progress is one of the np_scope_* wrappers that k8s/logging defines unconditionally, and each wrapper does its own command -v np_trace_* || return 0 internally. So the wrapper existing tells you nothing about whether the tracing SDK (vendor/catalog-tracing-sh/nptrace.sh) was actually sourced — that only happens when NP_API_KEY and NP_TRACE are both set and the vendored file is present.

np_trace_flush comes straight from the SDK, so with tracing off the block ran to completion and then hit an undefined command. wait_deployment_active has no set -e, so it was non-fatal noise rather than a failed deployment — but it read as an error in the operator-facing log right where the deployment reported success.

The fix is the guard already used at k8s/apply_templates:100.

Audit of the other callsites

Checked every np_trace_* reference outside the SDK; no other unguarded one exists:

  • k8s/logging:142,153 (np_scope_step_end, np_scope_step_timeout) — both return early unless _NP_SCOPES_SUBSTEP is set, which only np_scope_step_begin sets, and that one guards on np_trace_adopt.
  • k8s/logging:167 (np_scope_wait_heartbeat) — returns early unless _np_scopes_node succeeds, which guards on np_trace_adopt.
  • k8s/logging:335 (_np_scopes_on_exit) — runs from an EXIT trap registered only inside the branch that sourced the SDK.
  • scheduled_task/logging — same four, same structure.
  • k8s/diagnose/utils/diagnose_utils_np_trace_check_result is a local function, unrelated.

scheduled_task inherits the k8s deployment steps, so it picks this up too.

Test plan

Added a regression test that reproduces the production shape: the np_scope_* wrappers are defined (as they are once load logging runs) while np_trace_flush is not.

The pre-existing suite never caught this because it never defined np_scope_progress at all, so command -v was false and the whole block was skipped — the tests went green without ever reaching the line.

Confirmed the new test fails on the unfixed script and passes after:

bats k8s/deployment/tests/wait_deployment_active.bats   # 32 passed, 0 failed
bats k8s/deployment/tests/apply_templates.bats
bats k8s/deployment/tests/wait_blue_deployment_active.bats

Changelog

Cut as 1.16.3 dated 2026-09-08. 1.16.2 was left untouched.

@sebastiancorrea81 sebastiancorrea81 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@fedemaleh
fedemaleh merged commit 084f672 into beta Sep 8, 2026
3 checks passed
@fedemaleh
fedemaleh deleted the fix/trace-flush-guard branch September 8, 2026 20:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants