#5656th of 75 in ML frameworks and libraries
sentencepiece
Unsupervised BPE and unigram subword tokenizer trained from raw text
- Stars
- 12.1k
- License
- Apache-2.0
- Last commit
- Oct 2026
- Last release
- Jul 2026
- Language
- C++
11,659 on 31 Mar 202612,145 on 20 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
SentencePiece trains BPE or unigram language model tokenizers directly from raw sentences, treating whitespace as the symbol ▁ so detokenization is lossless and language-independent. The core is C++ with a Python module installed via pip. The repo also ships SentencePiece Lite, a zero-dependency C++20 runtime that memory-maps FlatBuffers model files.
Who it is for: Engineers building custom tokenizers for LLMs and translation models
Strengths
- Lossless, reversible tokenization with no language-specific pre-tokenizer, incl. Chinese, Japanese, Thai
- Supports subword regularization (unigram) and BPE-Dropout via enable_sampling, alpha and nbest_size
- Single .model file holds normalization, vocabulary and segmentation; same results in C++ and Python
- README benchmark: 27.41 MB/s single-thread on T5-base unigram vs 3.78 for Hugging Face Fast
Weaknesses
- Tokenizer only; no model training or inference beyond tokenization
- README states it is not an official Google product
- Multi-thread scaling flattens because Python object conversion holds the GIL
- Benchmarks are the project's own, on one dataset and one 24-core machine
What it needs
- no GPU
- Models: BPE, Unigram language model
Also in ML frameworks and libraries
See all 75| Rank | Project | Score |
|---|---|---|
| 1 | transformersPython library defining open models for text, vision, audio and multimodal tasks | 94 out of 100 |
| 2 | pytorchPython tensor library with GPU support and tape-based autograd for deep learning | 94 out of 100 |
| 3 | ultralyticsPython library and CLI for training and running YOLO vision models | 92 out of 100 |
| #4 | pytorch-image-modelsPyTorch library of image models, pretrained weights and reference training scripts | 91 out of 100 |
| #5 | supervisionPython toolkit for loading, annotating and counting computer vision detections | 90 out of 100 |