#199199th of 242 in ML frameworks and libraries
evalai
Host ML challenges with submissions, evaluation workers and public leaderboards
- Stars
- 2.1k
- License
- custom license
- Last commit
- Oct 2026
- Last release
- Apr 2018
- Language
- Python
2,002 on 25 Apr 20262,050 on 14 Nov 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
EvalAI is a Django and Node.js platform for running AI benchmark challenges. Organizers define evaluation phases and dataset splits, participants submit results or Docker images, and workers score them into public or private leaderboards. Organizers can attach their own worker clusters for remote evaluation, and an evalai-cli client covers command-line use.
Who it is for: Researchers and organizers hosting ML benchmark challenges with leaderboards
Strengths
- Arbitrary evaluation phases and dataset splits, with public and private leaderboards
- Remote evaluation: organizers can attach their own worker nodes for heavy challenges
- Participants can submit Docker images evaluated against test environments
- Separate evalai-cli project for submissions from the terminal
Weaknesses
- Last tagged release is from 2018, so there are no recent versioned releases
- Built for challenge hosting, not for evaluating LLM prompts or apps
- Local setup needs Postgres, SQS and Django containers, with workers as an extra profile
- Seeded default accounts use the password 'password' and must be changed
What it needs
- Compose
- Needs Docker, docker-compose, PostgreSQL, SQS, Node.js
- port 8888
What changed
From the release notes. No flag means the note does not say so, not that the upgrade is safe.
Also in ML frameworks and libraries
See all 242| Rank | Project | Score |
|---|---|---|
| 1 | transformersModel-definition library for pretrained text, vision, audio and multimodal models | 94 out of 100 |
| 2 | ultralyticsPython library and CLI for training and running YOLO vision models | 94 out of 100 |
| 3 | pytorch-image-modelsPyTorch library of image models, pretrained weights and training scripts | 94 out of 100 |
| #4 | pytorchPython tensor library with GPU support and tape-based autograd | 93 out of 100 |
| #5 | supervisionPython toolkit for loading, annotating and counting computer vision detections | 92 out of 100 |