#109109th of 168 in ML frameworks and libraries
swe-bench
Benchmark and Docker harness for scoring LLM patches on real GitHub issues
- Stars
- 6.2k
- License
- MIT
- Last commit
- Sep 2026
- Language
- Python
4,691 on 21 Apr 20266,173 on 10 Nov 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
SWE-bench is a dataset and evaluation harness that gives a model a codebase and a GitHub issue, then runs the generated patch in Docker containers to check whether it resolves the issue. It covers the full, Lite, Verified (500 tasks), Multimodal and Multilingual sets, and has a `swebench` CLI for evaluation, image builds, inference and result reports. It also ships SWE-Llama checkpoints and BM25 retrieval datasets.
Who it is for: Researchers and teams benchmarking coding LLMs and agents on real issues
Strengths
- Containerized Docker harness gives reproducible pass/fail grading of patches
- Covers full, Lite, Verified, Multimodal and Multilingual task sets from one `swebench` CLI
- Cloud evaluation available through Modal or the separate sb-cli on AWS
- Includes SWE-Llama 7b/13b weights and BM25 retrieval datasets on Hugging Face
Weaknesses
- Needs about 120GB free disk, 16GB RAM and 8 CPU cores for local runs
- arm64 support is experimental; v5 CLI builds images locally from a task repo
- Results are cached by run_id and instance_id, so changed patches need a new run_id
- Support for creating new SWE-bench instances is temporarily paused
What it needs
- RAM ≥ 16 GB
- Needs Docker, Hugging Face datasets, Modal (optional), sb-cli (optional)
- Models: SWE-Llama 7b, SWE-Llama 13b, API models via mini-SWE-agent
Quick answers
Is swe-bench open source?
Yes. Its code is on GitHub under the MIT license.
Also in ML frameworks and libraries
See all 168| Rank | Project | Score |
|---|---|---|
| 1 | transformersPython library defining open models for text, vision, audio and multimodal tasks | 94 out of 100 |
| 2 | pytorchPython tensor library with GPU support and tape-based autograd | 93 out of 100 |
| 3 | ultralyticsPython library and CLI for training and running YOLO vision models | 93 out of 100 |
| #4 | pytorch-image-modelsPyTorch library of image models, pretrained weights and training scripts | 93 out of 100 |
| #5 | supervisionPython toolkit for loading, annotating and counting computer vision detections | 92 out of 100 |