#4343rd of 64 in Data and scraping for AIFor data and documents, documents and ocr
docetl
Declarative LLM map-reduce pipelines over documents, with automatic optimization
Live demo ↗
(opens in a new tab)Documentation ↗
(opens in a new tab)Website ↗
(opens in a new tab)Repository on GitHub ↗
(opens in a new tab)
- Stars
- 4.2k
- License
- MIT
- Last commit
- Nov 2026
- Last release
- Jun 2026
- Language
- Python
Missed the latest GitHub check; these facts may be older.
Overview
DocETL runs LLM operators (map, reduce, filter, resolve, split, gather, extract) over large collections of structured or unstructured data, with each step written as a natural-language prompt. Pipelines are defined in Python or YAML and produce tables. An optimizer (MOAR) rewrites prompts, swaps models and decomposes operations to trade accuracy against cost. DocWrangler is a visual playground for developing prompts.
Who it is for: Engineers extracting structured data from large document sets with LLMs
Strengths
- Python API and YAML config both supported, run with `docetl run pipeline.yaml`
- Optimizer rewrites prompts, swaps models and replaces subtasks with code
- Configurable rate limits and per-run cost reporting via pipeline.total_cost
- Backed by peer-reviewed papers (VLDB 2025, UIST 2025) from UC Berkeley
Weaknesses
- Examples assume hosted providers such as OpenAI; local model setup is not shown
- Optimization runs add extra LLM calls, which adds cost up front
- Output is tables; no serving layer or long-running agent runtime
- Prompt-authoring help points to Claude Code skill or hosted prompts
What it needs
- no GPU
- Docker + Compose
- Needs LLM provider API key (OpenAI or other)
- Models: OpenAI, other LLM providers via API key
Also in Data and scraping for AI
See all 64| Rank | Project | Score |
|---|---|---|
| 1 | doclingPython library that converts PDFs, Office files and more into structured text | 93 out of 100 |
| 2 | mineruParses PDFs, images and Office files into Markdown and structured JSON | 91 out of 100 |
| 3 | opendataloader-pdfPDF parser that outputs Markdown, JSON with bounding boxes, and Tagged PDF | 91 out of 100 |
| #4 | agent-reachCLI that gives coding agents read access to login-gated and Chinese web platforms | 89 out of 100 |
| #5 | markitdownPython library and CLI that converts office files, PDFs and audio to Markdown | 87 out of 100 |