#1717th of 42 in Gateways and observability
harbor
Evaluation harness for running agents and models against container-based benchmarks
- Stars
- 6.7k
- License
- Apache-2.0
- Last commit
- Nov 2026
- Last release
- Oct 2026
- Language
- Python
1,281 on 3 Apr 20266,683 on 30 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
Harbor is a Python CLI from the Terminal-Bench team that runs agents such as Claude Code, OpenHands and Codex CLI against benchmark datasets in container environments. It runs locally on Docker or fans out to cloud sandbox providers like Daytona and Modal. It is the official harness for Terminal-Bench 2.0 and can also generate rollouts for RL optimization.
Who it is for: Researchers and engineers benchmarking coding agents and LLMs at scale
Ranked among open-source alternatives to OpenRouter (#10 of 10).
Strengths
- Official harness for Terminal-Bench 2.0; SWE-Bench and Aider Polyglot also listed
- Same `harbor run` command works on local Docker or cloud providers via `--env`
- Parallel runs across sandbox providers including Daytona, Modal, LangSmith and Blaxel
- Supports custom benchmarks and environments, plus rollout generation for RL
Weaknesses
- Needs model provider API keys and, for scale-out, a sandbox provider account
- Local runs depend on Docker; no image or compose file in the repo
- README gives no hardware requirements or full list of supported agents
- Evaluation only; not an agent builder or observability dashboard
What it needs
- no GPU
- Needs Docker, Daytona, Modal
- Models: Provider/model strings via --model, e.g. anthropic/...
Quick answers
Is harbor open source?
Yes. Its code is on GitHub under the Apache-2.0 license.
Does harbor need a GPU?
No, it runs without one.
Also in Gateways and observability
See all 42| Rank | Project | Score |
|---|---|---|
| 1 | OmniRouteLLM gateway routing one OpenAI-compatible endpoint across 372 providers | 96 out of 100 |
| 2 | new-apiGateway that unifies LLM providers behind OpenAI, Anthropic and Gemini APIs | 90 out of 100 |
| 3 | MLflowTracing, evals, prompt registry and AI gateway plus classic ML tracking | 89 out of 100 |
| #4 | LiteLLMOpenAI-format gateway and Python SDK for calling 100+ LLM providers | 87 out of 100 |
| #5 | freellmapiOpenAI-compatible gateway pooling free-tier LLM providers with automatic failover | 86 out of 100 |