#3131st of 37 in ML frameworks and libraries
datasets
Python library for loading, streaming and preprocessing ML datasets on Apache Arrow
Documentation ↗
(opens in a new tab)Website ↗
(opens in a new tab)Repository on GitHub ↗
(opens in a new tab)
- Stars
- 22k
- License
- Apache-2.0
- Last commit
- Oct 2026
- Last release
- Oct 2026
- Language
- Python
Missed the latest GitHub check; these facts may be older.
Overview
Hugging Face Datasets loads public datasets from the Hub or local files (CSV, JSON, Parquet, Arrow, XML, text, audio, image, video, PDF, NIfTI) with `load_dataset()`. Data is stored as memory-mapped Apache Arrow, so datasets larger than RAM work, and `map()` results are cached and can run in parallel with `num_proc`. A streaming mode iterates over remote data without downloading it.
Who it is for: ML engineers preparing training and evaluation data in Python
Strengths
- Memory-mapped Arrow storage lets datasets exceed available RAM
- Streaming mode with `streaming=True` avoids downloading full datasets
- Converts to NumPy, Pandas, Polars, PyTorch, TensorFlow, JAX and Spark
- Automatic caching of map results; parallel processing via `map(num_proc=N)`
Weaknesses
- Tied to the Hugging Face Hub for the one-line public dataset loading
- Audio, vision, PDF and NIfTI support need optional extras installed
- Built-in similarity search depends on separate FAISS or Elasticsearch setups
- Pinning dataset revisions is left to the user for reproducibility
What it needs
- no GPU
- Needs Hugging Face Hub, Apache Arrow, FAISS (optional), Elasticsearch (optional)
Also in ML frameworks and libraries
See all 37| Rank | Project | Score |
|---|---|---|
| 1 | transformersPython library defining open models for text, vision, audio and multimodal tasks | 95 out of 100 |
| 2 | pytorchPython tensor library with GPU support and tape-based autograd for deep learning | 94 out of 100 |
| 3 | ultralyticsPython library and CLI for training and running YOLO vision models | 90 out of 100 |
| #4 | supervisionPython toolkit for loading, annotating and counting computer vision detections | 87 out of 100 |
| #5 | pytorch-image-modelsPyTorch library of image models, pretrained weights and reference training scripts | 86 out of 100 |