#4141st of 64 in Data and scraping for AI
datachain
Python library that indexes cloud files into versioned, typed datasets
Documentation ↗
(opens in a new tab)Website ↗
(opens in a new tab)Repository on GitHub ↗
(opens in a new tab)
- Stars
- 2.8k
- License
- Apache-2.0
- Last commit
- Oct 2026
- Last release
- Oct 2026
- Language
- Python
Missed the latest GitHub check; these facts may be older.
Overview
DataChain indexes files in S3, GCS, Azure or local disk into named, versioned datasets with Pydantic schemas, stored as metadata and file pointers in a local SQLite Dataset DB. Pipelines use map/filter/join/save with checkpoint recovery and delta updates that skip unchanged files. An optional skill lets Claude Code, Cursor, Codex, Copilot and Pi build and query these datasets; distributed compute, MCP and the Knowledge Base need the hosted Studio.
Who it is for: Python engineers building pipelines over unstructured files in cloud storage
Strengths
- Checkpoint recovery resumes a failed UDF from the last uncommitted batch
- delta=True reprocesses only new or changed files and drops results for deleted ones
- Pydantic return types become queryable columns; vectors stored in the same rows for similarity search
- Data stays in your storage; only metadata and file pointers are indexed locally in SQLite
Weaknesses
- Distributed compute, shared registry, MCP endpoint and Knowledge Base require Studio
- Agent skill targets only Claude Code, Cursor, Codex, Copilot and Pi
- Scale claims (hundreds of millions of records) apply to Studio, not the local SQLite DB
- Python-only API; pipelines are written as UDFs with typed signatures
What it needs
- GPU optional
- Needs SQLite (local Dataset DB), S3, GCS or Azure storage (optional), DataChain Studio (optional, for distributed compute and MCP)
- Models: open_clip embeddings (example), LLM-enriched summaries via Studio Knowledge Base
Also in Data and scraping for AI
See all 64| Rank | Project | Score |
|---|---|---|
| 1 | doclingPython library that converts PDFs, Office files and more into structured text | 93 out of 100 |
| 2 | mineruParses PDFs, images and Office files into Markdown and structured JSON | 91 out of 100 |
| 3 | opendataloader-pdfPDF parser that outputs Markdown, JSON with bounding boxes, and Tagged PDF | 91 out of 100 |
| #4 | agent-reachCLI that gives coding agents read access to login-gated and Chinese web platforms | 89 out of 100 |
| #5 | markitdownPython library and CLI that converts office files, PDFs and audio to Markdown | 87 out of 100 |