Coding Agent Bench

Reproducible benchmarks for coding agents and models using Harbor

Features

Complete instructions for running popular benchmarks with open models
CLI utility to simplify benchmark runs against self-hosted models
Deployable queue service for scheduling benchmark runs in OpenShift
Leaderboards for popular benchmarks with instructions for reproducing results
Full manifests for deploying open models on OpenShift with vLLM

Features
Leaderboards
CLI Utility
- Prerequisites
- Run a Benchmark
Queue Service
Harbor Command Examples
Deploy models with vLLM
SWE-Bench Acceleration
- Use accelerated images for SWE-bench-verified
- Pre-pull base images
Run with Openshift
- Run Tasks in Openshift (Orchestrate Locally)
- Run Tasks and Orchestrate in Openshift
WIP

Leaderboards

✨ Check out our Coding Agent Leaderboard on HuggingFace ✨

SWE-Bench Verified (pass@1, N=500)

Model	Harness	Score	Cost
Opus 4.8	Claude Code	86.8%	$395
Opus 4.8	OpenCode	83.4%	$320
GPT 5.5	Codex	79.8%	$443
Sonnet 4.6	Claude Code	79.6%	N/A
RedHatAI/Qwen3.6-35B-A3B-NVFP4	Pi	65.0%	$51^†
RedHatAI/Qwen3.6-35B-A3B-NVFP4	Qwen Code	63.8%	$37^†
RedHatAI/Qwen3.6-35B-A3B-NVFP4	Claude Code	63.2%	$48^†
RedHatAI/Qwen3.6-35B-A3B-NVFP4	OpenClaw	58.8%	$33^†
RedHatAI/Qwen3.6-35B-A3B-NVFP4	OpenCode	54.8%	$67^†

SWE-Bench Pro - Ansible Tasks (pass@1, N=96)

Model	Harness	Score	Cost
Opus 4.8	OpenCode	78.1%	$151
Opus 4.8	Claude Code	69.8%	$186
GPT 5.5	Codex	60.4%	$188
GPT 5.5	OpenCode	57.3%	$111
Opus 4.6	Claude Code	51.0%	$172
Sonnet 4.6	Claude Code	50.0%	$184
RedHatAI/Qwen3.6-35B-A3B-NVFP4	Pi	47.9%	$13^†
RedHatAI/Qwen3.6-35B-A3B-NVFP4	Claude Code	45.6%	$10^†
RedHatAI/Qwen3.6-35B-A3B-NVFP4	Qwen Code	43.8%	$9^†
RedHatAI/Qwen3.6-35B-A3B-NVFP4	OpenClaw	40.6%	$9^†
RedHatAI/Qwen3.6-35B-A3B-NVFP4	OpenCode	37.5%	$11^†

Terminal Bench 2.0 (pass@1, N=87)

Model	Harness	Score	Cost
RedHatAI/Qwen3.6-35B-A3B-NVFP4	Pi	36.0%	$11^†
RedHatAI/Qwen3.6-35B-A3B-NVFP4	OpenCode	30.3%	$11^†
RedHatAI/Qwen3.6-35B-A3B-NVFP4	OpenClaw	20.2%	$10^†

More coming soon...

^† - Cost estimates for OSS models are calculated by ($4 per A100 GPU hour × agent benchmark duration).

CLI Utility

The CLI utility will help you configure and run a benchmark jobs with Harbor for self-hosted models. It automatically constructs and runs the Harbor job command for your specified benchmark, agent, self-hosted model.

Prerequisites

Install dependencies with uv
```
uv sync
```
Set up a vLLM server, or other Anthropic- and OpenAI-compatible server
Select a benchmark from among the options in Harbor Hub

Run a Benchmark

The following is the minimal configuration needed to run a job with the CLI:

uv run coding-agent-bench run \
    --agent <agent> \
    --dataset <benchmark-name> \
    --model-name <model-name> \
    --server-url <server-url>

For example, to run swe-bench/swe-bench-verified in Claude Code against a self-hosted model:

uv run coding-agent-bench run \
    --agent claude-code \
    --dataset scale-ai/swe-bench-pro \
    --model-name my-model \
    --server-url http://my.server.url

If you want to see a preview of Harbor command that would be run for a given set of arguments without actually running the job, add the --dry-run flag.

Note

Additional configuration options are available, use uv run coding-agent-bench run --help to see them.

Queue Service

The queue service is a FastAPI application that can be deployed on OpenShift to queue and run benchmarks automatically. Benchmark results are stored to MinIO for later review.

sequenceDiagram
    Requestor->>Queue Service: Request Benchmark Run
    Queue Service->>Queue Service: Wait in Queue
    Queue Service->>Openshift Job: Start Benchmark Job
    Openshift Job->>Harbor Orchestrator Pod: Start Harbor Run
    loop For Each Task
        Harbor Orchestrator Pod->>Task Pod: Run Task Pod
        Task Pod->>Harbor Orchestrator Pod: Save Results
    end
    Harbor Orchestrator Pod->>MinIO: Save Benchmark Results
    Harbor Orchestrator Pod->>Openshift Job: Complete
    Queue Service-->>Openshift Job: Poll for completion
    Openshift Job->>Queue Service: Complete
    Queue Service->>Openshift Job: Cleanup
    Queue Service->>Queue Service: Start Next Job

Set up the service

Log in to your cluster and project:

oc login --server=<server> --token=<token>
oc project <project>

Create the MinIO service for artifact storage:
```
oc apply -f deploy/harbor-minio.yml
```
Note: the default username and password are (minioadmin, minioadmin). You can update this in the deployment file if needed.

Create the orchestrator and task service accounts:

oc apply -f deploy/harbor-orchestrator-sa.yml
oc apply -f deploy/harbor-task-sa.yml

Create a secret file named job-queue-secret with an API_KEY and apply it:

apiVersion: v1
kind: Secret
metadata:
  name:  job-queue-secret
stringData:
  API_KEY: <your-api-key>
type: Opaque

Create the queue service:

oc apply -f deploy/job-queue-service.yml

Get the route for the deployed service:

oc get route job-queue-route --output jsonpath='{.spec.host}'

Check that the application is live by visiting the docs:

export JOB_QUEUE_URL="https://$(oc get route job-queue-route --output jsonpath='{.spec.host}')"
open $JOB_QUEUE_URL/docs

Use the service

Queue up a new benchmark task:

curl -X POST $JOB_QUEUE_URL/jobs -d '{"job_name": "test", "agent": "pi", "dataset": "swe-bench/swe-bench-verified", "model_name": "qwen3.6-27b", "server_url": "<server-url>", "n_tasks": 1}' -H "Content-Type: application/json" -H "X-API-Key: <your-api-key>"

{
    "message":"Job created.",
    "job_id":"b5ef13c8-8909-4bf1-b5b1-43354e9f395c", 
    ... 
}

View the queued/running/completed tasks:

open $JOB_QUEUE_URL/ui

Or list them from the API:

curl $JOB_QUEUE_URL/jobs -H "X-API-Key: <your-api-key>"

Cancel a running or queued job:

curl -X DELETE $JOB_QUEUE_URL/jobs/<job_id> -H "X-API-Key: <your-api-key>"

(Optional) Connect to Nebius

The queue service supports starting and stopping vLLM server instances automatically using Nebius. When enabled, the queue service will create a Nebius VM, SSH into the instance, start the vLLM server for the next model in the queue, run the benchmark job against that model, swap the running model for the next model in the queue, then spin down the Nebius VM when the queue is empty.

To connect the queue service to Nebius, first set up an AI Cloud account.

After setting up the account, install the CLI and login to your account.

Then run the following commands to create a service account in your project:

# Create a service account
export SA_ID=$(nebius iam service-account create \
  --name <service_account_name> \
  --format json | jq -r '.metadata.id')

# Create and attach an authorized key to the service account
nebius iam auth-public-key generate \
  --service-account-id $SA_ID \
  --output ~/.nebius/$SA_ID-credentials.json

Once the service account is created, you can update your job queue secret with the following environment variables needed for Nebius:

apiVersion: v1
kind: Secret
metadata:
  name:  job-queue-secret
stringData:
  API_KEY: <your-api-key>
  NEBIUS_ENABLED: '1'
  NEBIUS_SERVICE_ACCOUNT_CREDS: |
    <service-account-file-content>
  NEBIUS_PARENT_ID: <project-id>
  NEBIUS_TENANT_ID: <tenant-id>
  NEBIUS_SERVICE_ACCOUNT_ID: <service-account-id>
  NEBIUS_SUBNET_ID: <subnet-id>
  NEBIUS_INSTANCE_NAME_PREFIX: job-queue-worker
  NEBIUS_IDLE_TIMEOUT_SECONDS: '600'
  HF_TOKEN: <optional-huggingface-token>
type: Opaque

When creating a job, set server_url to nebius-<resource> to use a managed Nebius instance with the specified GPU resource (e.g. nebius-h200, nebius-b200). Available resources are defined in RESOURCE_CONFIG_REGISTRY.

Harbor Command Examples

Prerequisites:

Install Harbor
Set up a vLLM server, or other Anthropic- and OpenAI-compatible server
Set your benchmark in your environment from among the options in Harbor Hub, e.g.:
```
export BENCHMARK='swe-bench/swe-bench-verified'
```
If you need to filter tasks in your benchmark by name, add the -i flag with your glob pattern to your harbor run command, e.g. -i "*ansible*"

Directory:

Harness	Model Server	Example	Status
Claude Code	vLLM	Link	Validated
Codex	vLLM	Link	Testing
OpenClaw	vLLM	Link	Validated
OpenCode	vLLM	Link	Validated
OpenHands	vLLM	Link	Validated
Pi	vLLM	Link	Validated
Qwen Code	vLLM	Link	Validated
Claude Code	Anthropic	Link	Validated
Claude Code	VertexAI	Link	Validated
Codex	OpenAI	Link	Validated

Note

To use with a locally hosted model (e.g. llama.cpp) use a vLLM example and set SERVER_URL=http://host.docker.internal:<server-port>

Claude Code vLLM

Set the following variables in your environ:

export SERVER_URL=
export MODEL_NAME=

Then run:

harbor run --agent claude-code -d $BENCHMARK \
    --ae ANTHROPIC_BASE_URL=$SERVER_URL \
    --ae ANTHROPIC_API_KEY='sk-no-key-required' \
    --ae ANTHROPIC_MODEL=$MODEL_NAME \
    --ae ANTHROPIC_DEFAULT_OPUS_MODEL=$MODEL_NAME \
    --ae ANTHROPIC_DEFAULT_SONNET_MODEL=$MODEL_NAME \
    --ae ANTHROPIC_DEFAULT_HAIKU_MODEL=$MODEL_NAME

Codex vLLM

Set the following variables in your environ:

export SERVER_URL=
export MODEL_NAME=

Use the utility script to create the config.toml file:

uv run scripts/codex_config_toml.py $MODEL_NAME $SERVER_URL

Then run:

harbor run --agent codex -d $BENCHMARK \
    -m vllm/$MODEL_NAME \
    --ae CODEX_HOME=/root/.codex/ \
    --mounts-json '[ { "type": "bind", "source":"/path/to/coding-agent-bench/config.toml", "target": "/root/.codex/config.toml" } ]'

OpenClaw vLLM

Set the following variables in your environ:

export MODEL_NAME=
export SERVER_URL=
export OPENAI_BASE_URL=$SERVER_URL/v1
export OPENAI_API_KEY='NONE'

Then run:

harbor run --agent openclaw -p $DATASET_DIR/swe-bench-verified \
    -m openai/$MODEL_NAME \
    --agent-kwarg thinking=off \
    --n-concurrent 8

OpenCode vLLM

Set the following variables in your environ:

export MODEL_NAME=

Set the content of your OpenCode config in your environ. Remember to replace the <server-url> with your vLLM server url and the <model-name> with your served model name:

export OPENCODE_CONFIG_CONTENT='{"$schema":"https://opencode.ai/config.json","model":"vllm/<model-name>","provider":{"vllm":{"npm":"@ai-sdk/openai-compatible","name":"vLLM","options":{"baseURL":"<server-url>"},"models":{"<model-name>":{"name":"<model-name>","limit":{"context":196500,"output":65500}}}}}}'

Then run:

harbor run --agent opencode -p $DATASET_DIR/swe-bench-verified \
    -m vllm/$MODEL_NAME \
    --ae "OPENCODE_CONFIG_CONTENT=$OPENCODE_CONFIG_CONTENT"

OpenHands vLLM

Set the following variables in your environ:

export MODEL_NAME=
export SERVER_URL=
export LLM_API_KEY="NONE"

Then run:

harbor run -d $BENCHMARK \
  -a openhands-sdk \
  -m hosted_vllm/$MODEL_NAME \
  --ae HOSTED_VLLM_API_BASE=$SERVER_URL/v1

Pi vLLM

Set the following variables in your environ:

export MODEL_NAME=

Create a models.json file with your vLLM server information:

export PI_MODELS_JSON='{ "providers": { "vllm": { "baseUrl": "<server-url>", "api": "openai-completions", "apiKey": "NONE", "models": [{ "id": "gemma4-26b", "name": "<model-name>", "contextWindow": 262000 }] } } }'
echo $PI_MODELS_JSON > models.json

Then run:

harbor run --agent pi -d $BENCHMARK \
    -m vllm/$MODEL_NAME \
    --ae PI_OFFLINE=1 \
    --ae PI_CODING_AGENT_DIR=/root/.pi/agent \
    --mounts-json '[ { "type": "bind", "source":"/path/to/models.json", "target": "/root/.pi/agent/models.json" } ]'

Qwen Code vLLM

Set the following variables in your environ:

export MODEL_NAME=
export SERVER_URL=
export OPENAI_BASE_URL=$SERVER_URL/v1
export OPENAI_API_KEY='NONE'

Then run:

harbor run --agent qwen-coder -d $BENCHMARK \
    -i $DATASET_PATTERN \
    -m $MODEL_NAME

Claude Code Anthropic

Copy .env.example to .env and set the following variables:

ANTHROPIC_API_KEY=

Then run:

set -a
source .env

harbor run --agent claude-code -d $BENCHMARK \
    -m claude-opus-4-8

Claude Code VertexAI

Set the following variables in your environ:

export CLOUD_ML_REGION=
export ANTHROPIC_VERTEX_PROJECT_ID=
export ANTHROPIC_MODEL=

Then run:

harbor run --agent claude-code -d $BENCHMARK \
    --ae CLAUDE_CODE_USE_VERTEX=1 \
    --ae CLOUD_ML_REGION=$CLOUD_ML_REGION \
    --ae ANTHROPIC_VERTEX_PROJECT_ID=$ANTHROPIC_VERTEX_PROJECT_ID \
    --ae ANTHROPIC_MODEL=$ANTHROPIC_MODEL \
    --ae GOOGLE_APPLICATION_CREDENTIALS='/app/.config/gcloud/application_default_credentials.json' \
    --mounts-json '[ { "type": "bind", "source":"~/.config/gcloud/application_default_credentials.json", "target": "/app/.config/gcloud/application_default_credentials.json" } ]'

Codex OpenAI

Copy .env.example to .env and set the following variables:

OPENAI_API_KEY=

Then run:

set -a
source .env

harbor run --agent codex -d $BENCHMARK \
    -m gpt-5.5

Deploy models with vLLM

Check out deploy/qwen-all-in-one.yml for a sample vLLM deployment of RedHatAI/Qwen3.6-35B-A3B-NVFP4.

Apply to your cluster by running:

oc apply -f deploy/qwen-all-in-one.yml

SWE-Bench Acceleration

Use accelerated images for SWE-bench-verified

Download the SWE-Bench-Verified tasks

harbor download swe-bench/swe-bench-verified

Replace images with the accelerated ones from Epoch AI

uv run scripts/replace_swe_bench_images.py <path-to-dataset>

Pre-pull base images

Download the dataset

harbor download <dataset>

Pull all the base images

uv run scripts/pull_images.py <path-to-dataset>

Run with Openshift

Run Tasks in Openshift (Orchestrate Locally)

Login to your cluster and select a project:

oc login --token=<token> --server=<server>
oc project <project>

Create ServiceAccounts and RoleBindings to run tasks:

oc apply -f deploy/harbor-task-sa.yml

Then in your harbor command, add the flag:

--environment-import-path coding_agent_bench.harbor_envs.openshift:OpenshiftEnvironment

Run Tasks and Orchestrate in Openshift

Login to your cluster and select a project:

oc login --token=<token> --server=<server>
oc project <project>

Create ServiceAccounts and RoleBindings to run tasks and orchestrate:

oc apply -f deploy/harbor-task-sa.yml
oc apply -f deploy/harbor-orchestrator-sa.yml

Create a MinIO deployment to store your job results:

oc apply -f deploy/harbor-minio.yml

Using the CLI, start a job with the --remote flag enabled and set --environment openshift, e.g.:

uv run coding-agent-bench run \
    --agent claude-code \
    --dataset scale-ai/swe-bench-pro \
    --model-name my-model \
    --server-url http://my.server.url \
    --remote \
    --environment openshift

WIP

Run with Podman

Requires podman on PATH with a running Podman machine.

In your harbor command, add the flag:

--environment-import-path coding_agent_bench.harbor_envs.podman:PodmanEnvironment

Run with Gemini and Gemini CLI

export GOOGLE_CLOUD_PROJECT="<your-project>"

harbor run --agent gemini-cli -d $BENCHMARK \
    -m $MODEL_NAME

Run with vLLM and Gemini CLI

harbor run --agent gemini-cli -d $BENCHMARK \
    --ae GOOGLE_GEMINI_BASE_URL=$SERVER_URL \
    --ae GEMINI_MODEL=$MODEL_NAME \
    -m $MODEL_NAME

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
benchmarks		benchmarks
deploy		deploy
scripts		scripts
src/coding_agent_bench		src/coding_agent_bench
.bumpversion.toml		.bumpversion.toml
.env.example		.env.example
.gitignore		.gitignore
Containerfile		Containerfile
README.md		README.md
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

Coding Agent Bench

Features

Table of Contents

Leaderboards

✨ Check out our Coding Agent Leaderboard on HuggingFace ✨

SWE-Bench Verified (pass@1, N=500)

SWE-Bench Pro - Ansible Tasks (pass@1, N=96)

Terminal Bench 2.0 (pass@1, N=87)

CLI Utility

Prerequisites

Run a Benchmark

Queue Service

Set up the service

Use the service

(Optional) Connect to Nebius

Harbor Command Examples

Claude Code vLLM

Codex vLLM

OpenClaw vLLM

OpenCode vLLM

OpenHands vLLM

Pi vLLM

Qwen Code vLLM

Claude Code Anthropic

Claude Code VertexAI

Codex OpenAI

Deploy models with vLLM

SWE-Bench Acceleration

Use accelerated images for SWE-bench-verified

Pre-pull base images

Run with Openshift

Run Tasks in Openshift (Orchestrate Locally)

Run Tasks and Orchestrate in Openshift

WIP

Run with Podman

Run with Gemini and Gemini CLI

Run with vLLM and Gemini CLI

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages