#4242nd of 60 in Data and scraping for AI
chonkie
Python library that chunks text and code for RAG ingestion pipelines
- Stars
- 4.8k
- License
- MIT
- Last commit
- Aug 2026
- Last release
- Jul 2026
- Language
- Python
3,898 on 22 Apr 20264,833 on 11 Nov 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
Chonkie is a Python library that splits text, code and markdown tables into chunks using token, sentence, recursive, semantic, late, neural and LLM-based strategies. Chunks can be refined with overlap and embeddings, then written to JSON, HuggingFace datasets or vector stores such as Chroma, Qdrant, pgvector, Pinecone and Milvus. It also ships a REST API server (`chonkie serve`) that stores reusable pipelines in SQLite.
Who it is for: Python engineers building RAG ingestion pipelines
Strengths
- Small base install: 505KB wheel, 49MB installed, per the project's own benchmarks
- Many chunkers: token, sentence, recursive, semantic, late, code, neural, LLM-driven, table
- Ten vector DB handshakes and 16 embedding providers, each as an optional extra
- Pipeline API with async support, plus a self-hosted REST server with Docker compose
Weaknesses
- Ingestion only; no retrieval, query or answer generation layer
- Semantic, neural and slumber chunkers need extra installs and embedding or LLM models
- Speed and size comparisons come from the project's own benchmarks
- README advises against the `all` install in production, so extras are picked by hand
What it needs
- GPU optional
- Docker + Compose
- Needs Python, SQLite (API server pipelines), Optional: embedding/LLM providers and vector DBs
- Models: sentence-transformers, Model2Vec, OpenAI, Azure OpenAI, Cohere
- port 8000
What changed
From the release notes. No flag means the note does not say so, not that the upgrade is safe.
Also in Data and scraping for AI
See all 60| Rank | Project | Score |
|---|---|---|
| 1 | doclingPython library that converts PDFs, Office files and more into structured text | 92 out of 100 |
| 2 | mineruParses PDFs, images and Office files into Markdown and structured JSON | 91 out of 100 |
| 3 | opendataloader-pdfPDF parser that outputs Markdown, JSON with bounding boxes, and Tagged PDF | 91 out of 100 |
| #4 | agent-reachCLI that gives coding agents read access to login-gated and Chinese web platforms | 89 out of 100 |
| #5 | markitdownPython library and CLI that converts office files, PDFs and audio to Markdown | 87 out of 100 |