Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions guides/workflows/deploy-to-your-compute.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -131,10 +131,10 @@ docker run --rm \

Scale the number of runner containers or virtual machine instances when you want more parallelism. In Kubernetes, increase the `Deployment` replica count. In GCP or AWS, use the scaling controls of the service that runs the container, such as GKE, ECS, EKS, or an auto-scaling VM group. Each runner process connects to the same cluster and claims compatible tasks independently.

As an alternative for local testing or constrained environments, you can run multiple runner processes inside one container or shell session. Use `tilebox parallel` only for that case.
As an alternative for local testing or constrained environments, use `-n` to run multiple independent release runners inside one CLI process.

```bash
tilebox parallel -n 4 -- tilebox runner start --cluster workflow-dev
tilebox runner start -n 3 --cluster workflow-dev
```

Tilebox does not require the runner process to run in Tilebox-managed infrastructure. Use the process manager, scheduler, or container platform that fits your compute environment.
Expand Down
14 changes: 7 additions & 7 deletions guides/workflows/execute-tasks-in-parallel.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: Submit multiple workflow subtasks and process them faster by runnin
icon: arrows-split-up-and-left
---

Use this guide when a workflow can split work into independent tasks. You will create a small workflow that submits 20 sleep subtasks, submit one job, run it with one direct runner, and then run the same workflow with five direct runners in parallel.
Use this guide when a workflow can split work into independent tasks. You will create a small workflow that submits 20 sleep subtasks, submit one job, run it with one direct runner, and then run the same workflow with three direct runners in parallel.

The example is intentionally simple. `time.sleep` stands in for real work such as downloading scenes, processing tiles, calling a model, or writing output files.

Expand Down Expand Up @@ -124,19 +124,19 @@ uv run parallel_workflow.py

The runner executes tasks, but only one after the other, and exits when no more work is available. With one runner, the sleep subtasks don't run in parallel at all.

## Run five direct runners
## Run three direct runners

Submit another job, then start five runner processes for the same workflow file.
Submit another job, then start three runner processes for the same workflow file.

```bash
uv run parallel_workflow.py --submit --count 20 --seconds 5
tilebox parallel -n 5 -- uv run parallel_workflow.py
tilebox parallel -n 3 -- uv run parallel_workflow.py
```

This starts five direct runners. Each process registers the same task classes and asks Tilebox for work. Tilebox assigns queued tasks across the available runners, so multiple `SleepTask` instances run at the same time.
This starts three direct runners. Each process registers the same task classes and asks Tilebox for work. Tilebox assigns queued tasks across the available runners, so multiple `SleepTask` instances run at the same time.

<Tip>
Takeaway: use `tilebox parallel -n 5 -- uv run parallel_workflow.py` to start five local direct runners for the same workflow file.
Takeaway: use `tilebox parallel -n 3 -- uv run parallel_workflow.py` to start three local direct runners for the same workflow file.
</Tip>

## What to expect
Expand All @@ -147,7 +147,7 @@ In the Console, you should see:

- one root task that submits the subtask fan-out
- many `SleepTask(index)` tasks
- multiple tasks running at overlapping times when five runners are active
- multiple tasks running at overlapping times when three runners are active
- logs from each task attached to the same job

For command-line inspection, query logs or spans for the job:
Expand Down
10 changes: 5 additions & 5 deletions workflows/concepts/runners.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -117,14 +117,14 @@ Release runners advertise the task identifiers from workflow releases currently

Start multiple runner processes to execute tasks in parallel. Each runner process claims and executes tasks independently. You can run multiple release runners, multiple direct runners, or a mix of both in the same cluster. This increases parallelism and helps handle large workloads.

To test this, run multiple instances of the runner script in different terminal windows on your local machine, or use the [CLI](/agents-and-ai-tools/tilebox-cli) built-in `parallel` subcommand to start multiple runners in parallel.
For release runners, use `tilebox runner start -n <count>` to run multiple independent runners in one CLI process. For direct runners, run multiple instances of the runner script in different terminal windows or use the [CLI](/agents-and-ai-tools/tilebox-cli) built-in `parallel` subcommand.

```bash
# start multiple release runners in parallel
> tilebox parallel -n 5 -- tilebox runner start --cluster <dev-cluster>
# release runner mode
> tilebox runner start -n 3 --cluster <dev-cluster>

# or direct runner mode
> tilebox parallel -n 5 -- python your_direct_runner.py
# direct runner mode
> tilebox parallel -n 3 -- python your_direct_runner.py
```

## Scaling
Expand Down