#199199th of 242 in ML frameworks and libraries

evalai

Host ML challenges with submissions, evaluation workers and public leaderboards

Stars
2.1k
License
custom license
Last commit
Oct 2026
Last release
Apr 2018
Language
Python
Stars, last 29 weeks+48
25 Apr 2026: 2,002 stars2 May 2026: 2,005 stars9 May 2026: 2,005 stars16 May 2026: 2,006 stars23 May 2026: 2,008 stars30 May 2026: 2,008 stars6 Jun 2026: 2,008 stars13 Jun 2026: 2,012 stars20 Jun 2026: 2,013 stars27 Jun 2026: 2,013 stars4 Jul 2026: 2,017 stars11 Jul 2026: 2,020 stars18 Jul 2026: 2,022 stars25 Jul 2026: 2,024 stars1 Aug 2026: 2,028 stars8 Aug 2026: 2,030 stars15 Aug 2026: 2,032 stars22 Aug 2026: 2,033 stars29 Aug 2026: 2,033 stars5 Sept 2026: 2,033 stars12 Sept 2026: 2,036 stars19 Sept 2026: 2,037 stars26 Sept 2026: 2,040 stars3 Oct 2026: 2,041 stars10 Oct 2026: 2,042 stars17 Oct 2026: 2,044 stars24 Oct 2026: 2,045 stars31 Oct 2026: 2,047 stars7 Nov 2026: 2,050 stars14 Nov 2026: 2,050 stars

2,002 on 25 Apr 20262,050 on 14 Nov 2026

From GitHub's daily star counts.

Checked on GitHub.

Overview

EvalAI is a Django and Node.js platform for running AI benchmark challenges. Organizers define evaluation phases and dataset splits, participants submit results or Docker images, and workers score them into public or private leaderboards. Organizers can attach their own worker clusters for remote evaluation, and an evalai-cli client covers command-line use.

Who it is for: Researchers and organizers hosting ML benchmark challenges with leaderboards

Strengths

  • Arbitrary evaluation phases and dataset splits, with public and private leaderboards
  • Remote evaluation: organizers can attach their own worker nodes for heavy challenges
  • Participants can submit Docker images evaluated against test environments
  • Separate evalai-cli project for submissions from the terminal

Weaknesses

  • Last tagged release is from 2018, so there are no recent versioned releases
  • Built for challenge hosting, not for evaluating LLM prompts or apps
  • Local setup needs Postgres, SQS and Django containers, with workers as an extra profile
  • Seeded default accounts use the password 'password' and must be changed

What it needs

  • Compose
  • Needs Docker, docker-compose, PostgreSQL, SQS, Node.js
  • port 8888

What changed

From the release notes. No flag means the note does not say so, not that the upgrade is safe.

Also in ML frameworks and libraries

See all 242
Also in ML frameworks and libraries
RankProjectScore
1transformersModel-definition library for pretrained text, vision, audio and multimodal models169.3k starsApache-2.094 out of 100
2ultralyticsPython library and CLI for training and running YOLO vision models63.5k starsAGPL-3.094 out of 100
3pytorch-image-modelsPyTorch library of image models, pretrained weights and training scripts37.3k starsApache-2.094 out of 100
#4pytorchPython tensor library with GPU support and tape-based autograd105.8k starscustom license93 out of 100
#5supervisionPython toolkit for loading, annotating and counting computer vision detections Live demo ↗ (opens in a new tab)51.7k starsMIT92 out of 100