#4242nd of 64 in Data and scraping for AI
mnbvc
Chinese text corpus of 63 TB plus the crawlers and cleaning scripts behind it
Documentation ↗
(opens in a new tab)Website ↗
(opens in a new tab)Repository on GitHub ↗
(opens in a new tab)
- Stars
- 4.3k
- License
- MIT
- Last commit
- Oct 2026
Missed the latest GitHub check; these facts may be older.
Overview
MNBVC is a community-collected corpus of Chinese internet text (news, books, forums, code, Q&A, papers), with a stated target of 253 TB and 63,441 GB collected so far. The repo is mainly a landing page: it links download routes (Verysync P2P, Baidu Netdisk, cleaned subsets on ModelScope and Hugging Face) and many separate repos for encoding detection, deduplication, PDF parsing and per-source cleaning. The data is only roughly processed (html to txt, csv to json).
Who it is for: Teams pretraining or filtering Chinese LLM data who have large storage
Strengths
- Very large Chinese corpus: 63,441 GB collected, target 253 TB
- Covers many text types: forums, novels, papers, code, Q&A, parallel text
- Companion repos for crawling GitHub, Bitbucket and arXiv, and for PDF parsing
- Cleaned subsets are published on Hugging Face and ModelScope
Weaknesses
- Archive data is only roughly processed, with mixed txt, json, jsonl and parquet formats
- No index or classification of archive contents; copyright review was not done
- Full download goes through Verysync P2P or Baidu Netdisk; part1 needs 10 TB+ of disk
- README is a link hub; tools live in many separate repos of varying maintenance
What it needs
- Needs Verysync (P2P sync), Baidu Netdisk, Hugging Face, ModelScope
Also in Data and scraping for AI
See all 64| Rank | Project | Score |
|---|---|---|
| 1 | doclingPython library that converts PDFs, Office files and more into structured text | 93 out of 100 |
| 2 | mineruParses PDFs, images and Office files into Markdown and structured JSON | 91 out of 100 |
| 3 | opendataloader-pdfPDF parser that outputs Markdown, JSON with bounding boxes, and Tagged PDF | 91 out of 100 |
| #4 | agent-reachCLI that gives coding agents read access to login-gated and Chinese web platforms | 89 out of 100 |
| #5 | markitdownPython library and CLI that converts office files, PDFs and audio to Markdown | 87 out of 100 |