#1818th of 24 in Data and scraping for AI
dataflow
Operator-based Python pipelines for generating and cleaning LLM training data
Live demo ↗
(opens in a new tab)Documentation ↗
(opens in a new tab)Repository on GitHub ↗
(opens in a new tab)
- Stars
- 8.3k
- License
- Apache-2.0
- Last commit
- Oct 2026
- Last release
- Sep 2026
- Language
- Python
2,647 on 1 Apr 20268,254 on 21 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
DataFlow is a Python framework that turns noisy sources (PDFs, plain text, low-quality QA pairs) into datasets for pre-training, SFT and RL, using composable operators grouped into pipelines. It ships ready-made pipelines for text, reasoning, Text2SQL, knowledge-base cleaning and agentic RAG, plus a visual WebUI started with `dataflow webui` and an agent that assembles pipelines.
Who it is for: Engineers and researchers building SFT, RL or RAG datasets from raw documents
Strengths
- Pipeline → Operator → Prompt structure, with custom operators packaged via GitHub or PyPI
- Ready-made pipelines for text, reasoning, Text2SQL, knowledge-base cleaning and agentic RAG
- Works with API LLMs or local vLLM inference (`open-dataflow[vllm]`)
- Visual WebUI via `dataflow webui`, plus a DataFlow Agent that builds pipelines
Weaknesses
- Output quality depends on the LLM backend you configure; API use needs a DF_API_KEY
- Local GPU inference needs the vllm extra and a CUDA Docker image (cu124)
- Several pieces (WebUI, Skills, RayOrch, MM) live in separate repos
- Some tutorials are hosted on Bilibili and Feishu, with a Chinese-language focus
What it needs
- GPU optional
- Docker
- Needs OpenAI-compatible LLM API or vLLM, uv
- Models: OpenAI-compatible APIs, vLLM local models
Also in Data and scraping for AI
See all 24| Rank | Project | Score |
|---|---|---|
| 1 | mineruDocument parser that converts PDFs, images and Office files to Markdown and JSON | 81 out of 100 |
| 2 | Crawl4AIPython web crawler that converts pages to LLM-ready Markdown and JSON | 79 out of 100 |
| 3 | doclingPython library and CLI that converts PDFs, Office files and more into structured text | 79 out of 100 |
| #4 | label-studioMulti-user data labeling tool for audio, text, images, video and time series | 74 out of 100 |
| #5 | markitdownPython tool that converts PDF, Office, HTML and audio files to Markdown for LLMs | 71 out of 100 |