Skip to content
View Zhanyl-tech's full-sized avatar
🏠
Working from home
🏠
Working from home

Block or report Zhanyl-tech

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. Markdown is supported. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
zhanyl-tech/README.md

Hi there, I'm Zhanyl 👋

I build the systems that allocate scarce, expensive, heterogeneous compute — and the benchmarks that prove whether they actually work.

  • 🔭 Currently building: k8s-gpu-scheduler-lab, a controlled comparison of Kubernetes GPU schedulers on identical traces, and cluster-sre-agent, a multi-agent diagnosis system scored against slurm-rca-bench — built on MCP and LangGraph.
  • 🎓 Education: MS CS (Machine Learning) @ Georgia Tech · CQF (Quantitative Finance) · NVIDIA NCP-AIO
  • Core stack: Python, Go, PyTorch, CUDA, Kubernetes, Slurm
  • 🖥 Platform: NVIDIA BCM · Run:ai · DCGM · MIG · NVLink/NVSwitch · DOCA/BlueField · InfiniBand · Prometheus
  • 📈 Focus: scheduling and resource allocation across Slurm and Kubernetes, GPU cluster reliability, inference infrastructure, and agentic operations

Website · LinkedIn · X


📊 Things I measured that turned out to be wrong

The repos below are ordinary. These are the parts worth reading — each one is a belief I held, tested, and had to discard.

Slurm priority weights barely matter. Testing multifactor policy against a real trace: enabling backfill moved CPU utilisation 72.2% → 83.6% and mean wait 1913.0 → 373.7 min. Sweeping the priority weights everyone tunes moved almost nothing. The real lever was users' --time limits. → slurm-scheduler-lab

The "storage stall halts scheduling" chain does not exist. I built a benchmark scenario around the folk model — filesystem → DB → slurmdbd → slurmctld → scheduling halts. Then I measured it. Accounting goes dark and scheduling keeps running: jobs submitted, started and completed normally throughout, and sinfo never showed a stall. A second storage failure mode (StateSaveLocation unwritable) fails loudly and instantly instead. The scenario now ships documenting the refutation. → slurm-rca-bench

A benchmark can be solved without reading any telemetry. My own suite scored 0.290 for an agent that answers db.mysql to every question and looks at nothing. Adding scenarios whose causes lie elsewhere cut it to 0.145, and a test now fails the build if it climbs back. Publishing a score without that floor tells the reader nothing. → slurm-rca-bench

kubectl rollout restart silently skips Slinky's compute nodes. They're owned by a NodeSet CRD, which rollout restart doesn't understand — so the controller took a rotated auth key and slurmd kept the old one. My rotation script reported success on a cluster that could not run a job. → slinky-gitops


ML platforms, and the agents that operate them — agentic operations over the Slurm control plane, with the open-source tools covering the lifecycle of a GPU allocation


🛠 Open source

Five tools covering the lifecycle of a GPU allocation, plus the benchmark and agent built on top of them. Each is built on one rule: never act on absent evidence.

slurm-rca-bench A public incident-diagnosis benchmark for HPC schedulers. 10 scenarios, 2 deliberately undiagnosable, scored with partial credit against degenerate baselines.
cluster-sre-agent Multi-agent cluster diagnosis, built as five ablatable configs so the dependency graph's contribution is measured rather than asserted.
slurm-scheduler-lab Test Slurm priority and backfill policy against a real sacct trace before it reaches a live controller.
gpu-reaper Reclaim idle GPU allocations, observe-by-default. A telemetry outage can never cancel a job.
ib-slurm-exporter Attribute InfiniBand/RoCE fabric counters to the Slurm job responsible — and refuse to attribute a shared device.
epilog-gpu-validator Drain a node for a persistently faulty GPU between jobs, never for a transient one.
slinky-gitops Slurm on Kubernetes via SchedMD's Slinky, including the auth-key rotation nobody wants to test in production.
research-platform Point-in-time data semantics for quantitative research — as-of queries, feature lineage, and leakage detection. Production Python.

More in progress — a Kubernetes GPU scheduler comparison built on kwok, the remaining agent configurations for cluster-sre-agent, and a CUDA port of a volatility surface calibration. They go public as they get good enough to defend.

✍️ Writing

I publish at zhanyl-tech.github.io — deep dives on HPC and inference, plus shorter lab notes on whatever I'm currently measuring.

♟️ Chess and poker outside of work — both cheaper places to practise reasoning under uncertainty than production is.

Pinned Loading

  1. gpu-reaper gpu-reaper Public

    Detects and reclaims wasted GPU allocations on Slurm clusters — automated capacity recovery with guardrails that fail safe when telemetry is stale.

    Go 1

  2. ib-slurm-exporter ib-slurm-exporter Public

    Correlates InfiniBand/RoCE fabric counters with the Slurm job that owns them — fabric telemetry attribution for diagnosing multi-node training slowdowns.

    Go 1

  3. slurm-scheduler-lab slurm-scheduler-lab Public

    Replays real Slurm job traces against multifactor priority and EASY backfill — a discrete-event simulator for testing scheduling policy before it reaches a production controller.

    Python 1

  4. cluster-sre-agent cluster-sre-agent Public

    Multi-agent diagnosis over an explicit cluster dependency graph — five ablatable configurations, specified before results existed, scored on slurm-rca-bench.

    Python

  5. k8s-gpu-scheduler-lab k8s-gpu-scheduler-lab Public

    A reproducible comparison of Kubernetes GPU schedulers (Kueue, Volcano, KAI) on identical workload traces — kwok-simulated GPU fleet, runs on a laptop.

    Python

  6. slurm-rca-bench slurm-rca-bench Public

    The first public incident-diagnosis benchmark for HPC schedulers — ten reproducible failure scenarios with measured ground truth and degenerate baselines.

    Python