Skip to content

fix(shared-mc): retry stranded HEC batches instead of killing sender thread - #26

Open
shiftySenpai wants to merge 7 commits into
splunk:developfrom
shiftySenpai:pr4-hec-retry-fix
Open

fix(shared-mc): retry stranded HEC batches instead of killing sender thread#26
shiftySenpai wants to merge 7 commits into
splunk:developfrom
shiftySenpai:pr4-hec-retry-fix

Conversation

@shiftySenpai

Copy link
Copy Markdown

Summary

Split out of #23 (reliability-fix slice). Stacked on #24 (build/deps) — this branch includes #24's commits, so the diff below will look large until #24 merges, after which GitHub will automatically shrink it to just this PR's commit. Independent of the extended-logging PR.

  • SingleSplunkConnection's HEC sender thread used to die permanently on a single send failure, requiring a server restart to recover. Failures are now caught and the batch is retried instead of killing the thread.

Test plan

  • mvn clean package (not verified in this environment — no local Maven install; please build-check before merging)

Part of the #23 split.

Extract buildHecEnvelope() helper in SingleSplunkConnection using gson
(already a shared-mc dependency) instead of the unmaintained json-simple
1.1 library. Wire format is unchanged: {"event": <message>}. Adds unit
tests covering basic wrapping and quote escaping, and removes the
json-simple dependency from shared-mc/pom.xml and the parent pom's
dependencyManagement.
…'d deps

Adds per-platform MC-version Maven profiles (spigot: 1.21.1 default plus
1.20.1/1.20.4/1.20.6), a shade-plugin build producing
logtosplunk-<mc.version>-<loader>-<loader.version>.jar per module, and marks
server-provided deps (log4j, spigot-api) as provided scope so they aren't
shaded into the plugin jar. Pins kotlin-stdlib to 2.0.21 to clear
CVE-2026-53914 pulled in transitively via splunk-library-javalogging ->
okhttp3 -> kotlin-stdlib:1.6.20. Tunes the OWASP dependency-check plugin to
skip provided-scope CVEs (server operator's responsibility) and not fail on
Sonatype OSS Index 401s (paid-only), while still enforcing the CVSS 7.0 gate.
…der thread

Two related failure modes in SingleSplunkConnection's send loop, both of
which permanently stopped all Splunk logging until the server restarted:

- If sendData() threw for any reason, the sender thread's run() loop wasn't
  catching it, so the whole thread died silently on the first transient
  error (e.g. HEC briefly unreachable at startup).
- If an HTTP execute() call failed, response was never assigned, but the
  finally block unconditionally dereferenced it to close it -- turning
  every send failure into an uncaught NPE. The failed batch was also left
  on the "runway" and, due to an early-return bug, never retried.

Now: send failures are caught and logged per-cycle (thread keeps running),
a failed batch is retried next cycle instead of being stranded, and both
response/httpClient closes are null-guarded.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant