#4747th of 56 in ML frameworks and libraries
lm-evaluation-harness
Benchmark harness that runs 60+ academic LLM evals across many model backends
- Stars
- 14.2k
- License
- MIT
- Last commit
- Sep 2026
- Last release
- Aug 2026
- Language
- Python
Missed the latest GitHub check; these facts may be older.
Overview
Command-line and Python framework that runs generative language models through over 60 standard academic benchmarks, with hundreds of subtasks and variants. It loads models via Hugging Face transformers, vLLM, SGLang, NeMo, GPT-NeoX, llama.cpp servers and commercial APIs. Tasks and prompts are defined in YAML with Jinja2 templates, so results stay comparable across papers.
Who it is for: Researchers and engineers benchmarking LLMs on standard academic tasks
Strengths
- Over 60 benchmarks with hundreds of subtasks, using public prompts for reproducible comparisons
- Backends include HF transformers, vLLM, SGLang, NeMo, llama.cpp server and OpenAI-style APIs
- YAML task configs with Jinja2 prompts; plugins register backends, filters and metrics without forking
- Multi-GPU via accelerate data parallelism, model sharding or native PyTorch tensor parallelism
Weaknesses
- No native multi-node evaluation for the hf backend; needs an external server or custom integration
- Base install has no model backend; extras like hf, vllm and api must be added
- llama.cpp gguf backend does not implement loglikelihood_rolling, so perplexity tasks like wikitext fail
- GGUF via hf can hang for hours on tokenizer reconstruction unless a tokenizer path is given
What it needs
- GPU optional
- Needs transformers, vLLM, SGLang, accelerate, PyTorch
- Models: Hugging Face transformers, GGUF, vLLM, SGLang, NVIDIA NeMo
Also in ML frameworks and libraries
See all 56| Rank | Project | Score |
|---|---|---|
| 1 | transformersPython library defining open models for text, vision, audio and multimodal tasks | 95 out of 100 |
| 2 | pytorchPython tensor library with GPU support and tape-based autograd for deep learning | 94 out of 100 |
| 3 | ultralyticsPython library and CLI for training and running YOLO vision models | 92 out of 100 |
| #4 | supervisionPython toolkit for loading, annotating and counting computer vision detections | 89 out of 100 |
| #5 | pytorch-image-modelsPyTorch library of image models, pretrained weights and reference training scripts | 89 out of 100 |