#2020th of 24 in Data and scraping for AI
data-juicer
Operator-based pipelines for cleaning and deduplicating multimodal training data
Live demo ↗
(opens in a new tab)Documentation ↗
(opens in a new tab)Website ↗
(opens in a new tab)Repository on GitHub ↗
(opens in a new tab)
- Stars
- 7.1k
- License
- Apache-2.0
- Last commit
- Oct 2026
- Last release
- Sep 2026
- Language
- Python
Missed the latest GitHub check; these facts may be older.
Overview
Data-Juicer is a Python library and CLI (dj-process) that runs YAML-defined pipelines of 200+ operators over text, image, audio and video data, covering filtering, mapping, deduplication and analysis. It runs locally or on Ray clusters, and the README cites 70B samples in 2h on 50 Ray nodes and 5TB deduplicated in 2.8h on 1280 cores. It also ships LLM-based operators and a natural-language refinement model, Juicer.
Who it is for: ML engineers curating pre-training, fine-tuning, agent and RAG datasets
Strengths
- 200+ operators across text, image, audio, video and multimodal data
- Reproducible YAML recipes, plus a Python API for composing operators
- Ray-based distributed execution with partitioning, checkpoints and OP fusion
- Supports S3 and HDFS I/O, and loads Hugging Face datasets and gz JSONL
Weaknesses
- Many operators need extra dependencies or GPUs; the install is split into extras
- Large-scale performance figures come from big clusters (1280 to 6400 cores)
- Frequent releases with many operator behavior fixes; pin versions for reproducibility
- Docs and several tools are spread across separate repos (hub, agents, sandbox)
What it needs
- GPU optional
- Docker
- Needs Ray (distributed mode), S3 or HDFS (optional), Hugging Face / ModelScope models (optional)
- Models: Hugging Face models, OpenAI-compatible API, LiteLLM, Juicer-35B-A3B
Also in Data and scraping for AI
See all 24| Rank | Project | Score |
|---|---|---|
| 1 | mineruDocument parser that converts PDFs, images and Office files to Markdown and JSON | 81 out of 100 |
| 2 | Crawl4AIPython web crawler that converts pages to LLM-ready Markdown and JSON | 79 out of 100 |
| 3 | doclingPython library and CLI that converts PDFs, Office files and more into structured text | 79 out of 100 |
| #4 | label-studioMulti-user data labeling tool for audio, text, images, video and time series | 74 out of 100 |
| #5 | markitdownPython tool that converts PDF, Office, HTML and audio files to Markdown for LLMs | 71 out of 100 |