Skip to content

fix: verify that sandbox processes are gone instead of trusting pkill's status - #329

Merged
exKAZUu merged 5 commits into
mainfrom
security-hardening
Aug 24, 2026
Merged

fix: verify that sandbox processes are gone instead of trusting pkill's status#329
exKAZUu merged 5 commits into
mainfrom
security-hardening

Conversation

@exKAZUu

@exKAZUu exKAZUu commented Aug 24, 2026

Copy link
Copy Markdown
Member

Customer Summary

  • When a learner's program is stopped after a run, the judge now double-checks that nothing of it is still running before serving the next request, instead of assuming the stop command worked. This prevents a misbehaving program from lingering into another learner's run on the same server.
  • No visible change for learners; runs that clean up normally behave exactly as before.

Technical Summary

  • killSandboxUserProcesses (src/helpers/sandboxUser.ts) sends the requested signals through sudo pkill best-effort — sudo's exit status 1 is ambiguous (pkill no-match vs. sudo's own auth/fork/exec failures) and a sweep can itself be killed by an in-flight watchdog pkill -KILL — and then verifies the outcome. After a SIGKILL sweep, findSurvivingSandboxUserProcesses lists the sandbox user's processes as the harness user with ps -o pid=,stat=,nlwp=,comm= -u <user> (no privilege needed), ignores only single-threaded zombie rows, re-sends pkill -KILL and waits 100 ms between up to 10 further attempts, and throws only when a live process remains or ps cannot verify (spawn error, signal termination, exit status above 1, or exit 1 with diagnostics).
  • A ['TERM']-only sweep throws only when its sudo could not be spawned, after all requested signals were attempted, so a caller's later SIGKILL sweep is never skipped by a mid-loop throw.

Why

  • Issue fix: terminate sandboxed submissions without spawning a process #306: killSandboxUserProcesses inspected only result.error, so a sudo that started but failed to run pkill (e.g. fork failure under PID-cgroup exhaustion) was reported as success. Checking exit statuses turned out to be unsound (sudo's exit 1 is ambiguous; benign sudo warnings on stderr would have failed every routine sweep), so the function now verifies the post-condition instead.
  • The fork-free termination mechanism proposed in fix: terminate sandboxed submissions without spawning a process #306 (cgroup.kill or a privileged reaper) needs judge-container privilege changes and is intentionally out of scope; a PID-exhausted instance now fails loudly instead of silently.

Testing

  • bun run verify and bun run verify-full passed (the sudo/ps path itself only runs on the Linux judge image with EXERCODE_SANDBOX_USER set).
  • Multi-agent review (review-booster, 5 rounds): all findings fixed; the final round reported no findings.

`killSandboxUserProcesses` inspected only `result.error`, so a `sudo`
that started but could not run `pkill` (e.g. fork failure under PID
exhaustion) was reported as success, leaving the submission's processes
alive for the next request. Fail closed on any exit status other than 0
(processes killed) and 1 (no process matched).

Refs #306

Co-authored-by: WillBooster (Claude Code) <agent@willbooster.com>
@gemini-code-assist

Copy link
Copy Markdown

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request improves the reliability of sandbox process cleanup by tightening the validation of the pkill command's exit status. By ensuring that unexpected exit codes are treated as failures, the system avoids reporting false successes when processes fail to terminate, thereby preventing potential resource leakage between sandbox requests.

Highlights

  • Process Termination Logic: Updated the process termination logic to validate the exit status of the pkill command, ensuring that only status 0 (success) and 1 (no processes found) are treated as successful operations.
  • Error Handling: Added a check to throw an error if pkill exits with a status other than 0 or 1, preventing potential process leakage when sudo or pkill fails unexpectedly.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.


When processes linger and hide, We check every exit with pride. If pkill should fail, We raise up a wail, And keep all the sandbox inside.

Footnotes

  1. Review the Generative AI Prohibited Use Policy, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the process termination logic in src/helpers/sandboxUser.ts to check the exit status of the pkill command, throwing an error if it is neither 0 nor 1. The reviewer suggested enhancing the error handling to distinguish between an exit status and a signal termination when result.status is null.

Comment thread src/helpers/sandboxUser.ts Outdated
exKAZUu and others added 4 commits August 24, 2026 22:31
`sudo` exits with 1 both when `pkill` matched no process and on its own
failures (auth/permission problems, fork/exec failure), so accept exit
status 1 only when stderr is empty — every sudo-side failure reports on
stderr while pkill's no-match is silent. Collect per-signal failures and
throw after the loop so a failed SIGTERM sweep never skips the SIGKILL
sweep, retry a sweep once because a concurrent watchdog `pkill -KILL`
can kill the sweep itself, and name the terminating signal instead of
reporting `exit status null`.

Co-authored-by: WillBooster (Claude Code) <agent@willbooster.com>
…'s status

`sudo`'s exit status and stderr cannot reliably tell "pkill matched
nothing" from sudo's own failures, and benign sudo warnings would have
turned every routine sweep into a cleanup error. Send the signals
best-effort and then verify the outcome as the harness user: after
SIGKILL, list the sandbox user's live (non-zombie) processes with `ps`,
retrying briefly while just-killed processes are reaped, and throw only
when one remains. A SIGTERM-only sweep never throws because survivors
are expected during the caller's grace period.

Co-authored-by: WillBooster (Claude Code) <agent@willbooster.com>
…errors

- Re-issue `pkill -KILL` between survivor checks (up to ~1 s): a child
  forked after pkill scanned /proc was never signalled, and a process
  in uninterruptible sleep dies only once its I/O completes.
- Treat a `ps` that is signal-terminated, exits above 1, or exits 1
  with diagnostics as a failure to verify instead of "no survivors".
- Ignore a zombie row only when it has a single thread (`nlwp`), since
  a thread-group leader that exited via pthread_exit shows as a zombie
  while its other threads keep running.
- A SIGTERM-only sweep reports a sweep that could not be spawned after
  all requested signals were attempted.

Co-authored-by: WillBooster (Claude Code) <agent@willbooster.com>
Co-authored-by: WillBooster (Claude Code) <agent@willbooster.com>
@exKAZUu exKAZUu changed the title fix: treat a pkill that runs but fails as a cleanup failure fix: verify that sandbox processes are gone instead of trusting pkill's status Aug 24, 2026
@exKAZUu exKAZUu self-assigned this Aug 24, 2026
@exKAZUu
exKAZUu merged commit 013576b into main Aug 24, 2026
6 checks passed
@exKAZUu
exKAZUu deleted the security-hardening branch August 24, 2026 23:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant