Skip to main content

Training & evals

Harbor / Terminal-Bench

Run Harbor trials, including Terminal-Bench, in E2B sandboxes instead of local containers. One flag moves a job to E2B, so you can raise concurrency without loading your machine.

Maintained by
Harbor
How it works
Built into Harbor
Package
harbor[e2b]
Language
Python

Set it up

Get started

Paste the prompt into your coding agent, or follow the steps yourself.

Paste this into your coding agent

Run this project's Harbor evals on E2B sandboxes.

  1. 1.Install Harbor with the e2b extra ("harbor[e2b]"), for example with uv tool install.
  2. 2.Harbor needs the E2B_API_KEY environment variable. If it isn't set, ask me for my API key from https://console.e2b.dev/?tab=keys. Never hardcode it.
  3. 3.Follow the sandbox guide at https://docs.harborframework.com/sandboxes/pre-integrated-sandboxes and run the job with -e e2b, or set environment.type to e2b in the job config.
  4. 4.Run one small job and show me the trial results that came back from the sandboxes.

Or set it up yourself

  1. 1. Install Harbor with E2B

    uv tool install "harbor[e2b]"
  2. 2. Set your API key

    export E2B_API_KEY="e2b_..."
  3. 3. Run Terminal-Bench on E2B

    harbor run \
      -d terminal-bench@2.0 \
      -a codex -m openai/gpt-5.6-sol \
      -e e2b \
      -n 32
  4. 4. Or keep it in a job config

    {
      "n_concurrent_trials": 32,
      "environment": {
        "type": "e2b"
      },
      "agents": [
        {
          "name": "codex",
          "model_name": "openai/gpt-5.6-sol"
        }
      ],
      "datasets": [
        {
          "name": "terminal-bench",
          "version": "2.0"
        }
      ]
    }

Security

What runs in the sandbox

Each Harbor trial runs in its own E2B sandbox, away from the machine running the job.

  • CodeThe agent under test and the task's verifier, inside the environment the task defines.
  • EnvironmentHarbor builds the task's environment into an E2B template once and reuses it across trials.
  • NetworkTasks can turn internet access off or allow only listed hostnames, and change that while a trial runs.
  • LifetimeSandboxes are deleted after each trial by default. -⁠-⁠no-⁠delete keeps them for inspection.

3B+ sandboxes started

Give every agent a machine