#4545th of 75 in ML frameworks and libraries
tokenizers
Rust tokenizer library with Python and Node.js bindings
- Stars
- 11.2k
- License
- Apache-2.0
- Last commit
- Oct 2026
- Last release
- Oct 2026
- Language
- Rust
10,499 on 31 Mar 202611,188 on 20 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
Rust tokenization engine that loads tokenizer.json files or Hub tokenizers and encodes and decodes text with BPE, Unigram, WordPiece and WordLevel models. It ships Python and Node.js bindings and is split into sub-crates so serving builds can leave out training and legacy converters. The v1.0.0 release candidate does not yet include all of the 0.23 Python API.
Who it is for: Engineers embedding fast tokenization in serving or training pipelines
Strengths
- Every kernel has a portable scalar path; SIMD (NEON, AVX-512, SSE, wasm SIMD128) is optional
- Sub-crates let serving builds omit training and legacy conversion code
- Bindings for Rust, Python and Node.js; loads from file or Hugging Face Hub
- Byte-exact pre-tokenizer grammars for gpt2, cl100k, o200k, tekken, deepseek and kimi-k2
Weaknesses
- v1.0.0 is a release candidate; Python install needs pip --pre
- Python training, save, offsets, pair input and decode_stream are not back yet
- C, C++, Java and Go bindings are only planned
- Known load failure for ModernBERT and OLMo tokenizers (64-bit hash collision)
What it needs
- no GPU
- Models: BPE, Unigram, WordPiece, WordLevel, tokenizer.json
Also in ML frameworks and libraries
See all 75| Rank | Project | Score |
|---|---|---|
| 1 | transformersPython library defining open models for text, vision, audio and multimodal tasks | 94 out of 100 |
| 2 | pytorchPython tensor library with GPU support and tape-based autograd for deep learning | 94 out of 100 |
| 3 | ultralyticsPython library and CLI for training and running YOLO vision models | 92 out of 100 |
| #4 | pytorch-image-modelsPyTorch library of image models, pretrained weights and reference training scripts | 91 out of 100 |
| #5 | supervisionPython toolkit for loading, annotating and counting computer vision detections | 90 out of 100 |