Freight

A distributed CI/CD platform built from scratch to understand how modern build automation systems work internally.

Freight receives GitHub events, creates pipelines, parses pipeline definitions, schedules jobs, dispatches work to independent runner agents, executes jobs inside isolated Docker containers, stores artifacts, manages encrypted secrets, monitors runner health, and provides a live view of every pipeline.

---

Quick Start

Full stack

cp .env.example .env
# fill in SECRET_KEY, GITHUB_WEBHOOK_SECRET, and FERNET_KEY
docker compose up --build

That brings up Postgres, Redis, the Freight server, and two runners. The server applies migrations on boot. Open the dashboard at http://localhost:8000/dashboard and the API docs at http://localhost:8000/docs.

Local development

Run the infrastructure in Docker and the server from a virtualenv:

docker compose up -d postgres redis

python -m venv .venv && .venv/Scripts/activate
pip install -r requirements.txt
alembic upgrade head
uvicorn freight.main:app --reload --port 8000

Start a runner in a second terminal:

python -m runner.main

Trigger a pipeline without waiting for a webhook:

freight run .freight.yml

---

What Freight Does

• Receives GitHub webhooks and verifies their signatures
• Fetches .freight.yml from GitHub at the exact pushed commit
• Builds a job dependency graph and rejects cycles
• Schedules pipeline execution, releasing jobs as their dependencies complete
• Dispatches jobs through Redis with atomic claiming
• Executes jobs inside Docker containers
• Streams job logs to the server as they are produced
• Collects and stores the artifacts a job declares
• Serves artifacts back through the API, the dashboard, and the CLI
• Manages encrypted secrets and injects them into containers at run time
• Detects dead runners by heartbeat and recovers their work
• Retries failed jobs with exponential backoff
• Estimates completion time from historical run durations
• Displays live pipeline progress in a terminal-style dashboard

---

Architecture

GitHub Push
      ↓
Webhook (HMAC verified)
      ↓
Pipeline Parser + DAG validation
      ↓
PostgreSQL (pipeline + job state)
      ↓
Redis (job queue)
      ↓
Freight Runners
      ↓
Docker Containers
      ↓
Artifact Upload
      ↓
Artifact Store
      ↓
Dashboard / CLI / API

The Freight server coordinates the entire execution pipeline. Runner agents execute every job independently inside isolated Docker containers.

---

Tech Stack

Component	Technology
Backend	FastAPI
Database	PostgreSQL
Queue	Redis
Runner	Python
Container Runtime	Docker
Dashboard	FastAPI + Jinja2
CLI	argparse
Artifact Storage	Local Filesystem (MinIO planned)

---

Pipeline Configuration

version: 1

jobs:
  build:
    stage: build
    image: python:3.12
    retries: 2

    script:
      - mkdir -p dist
      - python build.py

    artifacts:
      paths:
        - dist/*
        - reports/**/*.xml

  test:
    stage: test
    image: python:3.12
    needs:
      - build

    script:
      - pytest

Field	Meaning
stage	Grouping label used by the dashboard
image	Docker image the job executes in
script	Commands run sequentially inside the container
needs	Jobs that must complete before this one is released. A bare string is accepted as a single dependency
retries	Extra attempts after a failure. Omitted means a failure is terminal
artifacts.paths	Glob patterns, relative to the workspace, collected after a successful run

---

Artifacts

Artifact declarations travel with the job from the moment a pipeline is parsed, so a runner uploads only the files the pipeline asked for:

.freight.yml → pipeline_parser → Job row → GET /jobs/{id} → runner → POST /jobs/{id}/artifacts → artifact store

Stored paths are always relative to the configured artifact root:

12/48/dist/output.txt

rather than

C:\Users\...\artifacts\12\48\dist\output.txt

An absolute path would bake the uploading machine's filesystem layout into the database, which breaks the moment Freight runs in a container whose artifact volume is mounted elsewhere, and would leak server paths through the API.

The path a runner reports is untrusted: anyone who can push a .freight.yml controls it. Every upload is validated before it is joined onto the artifact root (absolute, drive-qualified, and parent-traversing paths are rejected), and every download re-checks that the resolved file is still inside the root.

Re-uploading the same path for the same job replaces the stored file and updates the existing record. Retried jobs upload their artifacts again, and each attempt should leave one current record rather than a pile of stale duplicates.

Downloading

freight artifacts 48                      # list
freight artifacts 48 --download -o ./out  # download all, keeping structure
freight artifacts 48 -d -a 3 -o ./out     # download one

Downloads reproduce the job's own output tree (out/dist/output.txt), not Freight's storage layout.

The API serves the same files:

GET  /jobs/{job_id}/artifacts                 list
POST /jobs/{job_id}/artifacts                 upload (multipart: path + file)
GET  /jobs/{job_id}/artifacts/{artifact_id}   download

---

Dashboard

Served at /dashboard, on the same origin as the API it reads from.

Page	Shows
/dashboard	Pipeline history with completion bars
/dashboard/pipelines/{id}	Jobs grouped by stage, progress against historical run times, live log tail, artifact downloads
/dashboard/runners	Runner health, heartbeat age, and current job

The dashboard holds no business logic and runs no database queries of its own. Pages are static shells that poll the public JSON API, so a pipeline's status is computed in exactly one place and the dashboard cannot drift out of agreement with the API.

Completion estimates

Freight has no way to know how long a job will take, but it knows how long the same job took the last several times it ran. Averaging recent completed runs of a job with the same name, scoped to the same repository, gives a per-job estimate.

A pipeline's estimate is the longest remaining path through its dependency graph, not the sum of its unfinished jobs, because independent branches run at the same time. Jobs stranded behind a failure contribute nothing, since they will never be released.

A job Freight has never seen produces no estimate, and that renders as unknown rather than as zero.

---

CLI

Installable as a console script:

pip install -e .

Command	Purpose
freight run <path>	Create and schedule a pipeline from a local .freight.yml
freight status <pipeline_id>	Print jobs, progress bars, and estimated finish
freight logs <job_id> [-f]	Print a job's output, optionally following it
freight artifacts <job_id> [-d]	List or download a job's artifacts

Read commands talk to a running server. Set FREIGHT_URL to point them somewhere other than http://127.0.0.1:8000.

freight status exits 2 when the pipeline contains a failed job, so it works as a check in a shell script.

---

Project Structure

freight/            FastAPI server
  core/             config, crypto, retry, heartbeat monitor
  db/               engine, session, Alembic migrations
  models/           one file per ORM table
  schemas/          request and response models
  routers/          HTTP routes, no business logic
  services/         parsing, scheduling, queueing, storage, estimates
runner/             runner agent (claim, execute, stream, upload)
dashboard/          Jinja templates, stylesheet, page routes
cli/                freight command
tests/              pytest suite
artifacts/          artifact storage root (gitignored)
logs/               job log files (gitignored)

Routers never touch the database or Redis directly. They call into freight/services/, which is where the actual behavior lives. That is what lets the CLI, the webhook, and the dashboard share one implementation of each operation instead of three.

---

Core Components

Freight Server

Receives GitHub webhooks, parses pipeline configurations, builds execution graphs, schedules jobs, manages runners, tracks pipeline state, stores artifacts, manages encrypted secrets, and serves both the REST API and the dashboard.

Freight Runner

Registers with the server, waits on Redis for work, atomically claims jobs, checks out source at the pushed commit, launches Docker containers, streams logs, uploads declared artifacts, reports results, and heartbeats continuously.

Queue

Redis distributes jobs across runners. BRPOPLPUSH hands each job to exactly one consumer, and a separate sorted set holds jobs waiting out a retry backoff so they are not claimable before their delay elapses.

Database

PostgreSQL stores pipelines, jobs, runners, artifacts, secrets, and execution history.

---

Fault Tolerance

Runners heartbeat every 10 seconds. A runner silent for 30 seconds is marked dead, and any job it was running is reset to queued and pushed back onto the work queue for another runner to claim. Killing a runner mid-job is a repeatable demo, not a recovery procedure.

Failed jobs are retried according to their retries: budget, each attempt waiting out an exponentially increasing delay capped at 60 seconds. Downstream jobs stay blocked while a retry is pending, since the job has not reached a terminal state.

---

Testing

pytest

The suite runs against the same Postgres and Redis that Freight uses in development, which must be up first:

docker compose up -d postgres redis

That is deliberate. Freight's interesting behavior is concurrency-shaped, and the guarantees under test (a conditional UPDATE that lets exactly one runner claim a job, a Redis list that hands a job to exactly one consumer) only hold on the real engines. A suite running against SQLite and a fake Redis would be testing a system Freight never runs on.

Every test cleans up what it creates, so the suite is safe to run against a database that already holds real pipelines.

File	Covers
test_parser.py	Pipeline parsing, job loading, cycle and dangling-dependency rejection
test_webhook_to_queue.py	Signature verification and webhook through to queue state
test_claim_race.py	Concurrent claims on one job produce exactly one winner
test_heartbeat_timeout.py	Dead-runner detection and job recovery
test_retry.py	Backoff, retry budgets, and downstream blocking
test_artifacts.py	Path validation, storage layout, and download
test_estimates.py	Duration averaging and longest-path completion estimates

---

Notes on Deployment

Runners execute jobs as sibling containers on the host's Docker daemon rather than nesting Docker inside Docker. The daemon resolves bind-mount paths against the host filesystem, so a containerized runner needs its workspace mounted at the same path on both sides. WORKSPACE_ROOT and the matching volume in docker-compose.yml handle this; a runner on the host can leave both at their defaults.

Runner replicas each register under their own container hostname. Pinning RUNNER_NAME in Compose would make every replica reactivate the same runner record, and the fleet would appear to Freight as a single worker.

---

Current Status

Working

• GitHub webhook processing with signature verification
• Pipeline parsing and DAG validation
• PostgreSQL persistence and Alembic migrations
• Redis scheduling with atomic job claiming
• Runner registration and heartbeat monitoring
• Docker execution with source checkout
• Live log streaming
• Encrypted secret storage and injection
• Automatic artifact upload, storage, and download
• Dead-runner detection and job recovery
• Automatic retry with exponential backoff
• Terminal-style dashboard with completion estimates
• CLI for triggering, inspecting, and downloading
• Full Docker Compose stack
• Test suite covering parsing, races, recovery, retries, artifacts, and estimates

Planned

• MinIO artifact backend
• Log streaming over SSE or websockets instead of polling
• Runner autoscaling