#168168th of 242 in ML frameworks and libraries
gigatoken
Rust BPE tokenizer for Python, with HuggingFace and tiktoken compatibility
- Stars
- 4.2k
- License
- MIT
- Last commit
- Nov 2026
- Language
- Rust
2 on 25 Apr 20264,157 on 14 Nov 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
Gigatoken is a Rust tokenizer library, installed with pip, that encodes text files using SIMD pretokenization and a pretoken cache across all CPU cores. It accepts HuggingFace model names and can wrap existing HF tokenizers or tiktoken encoders. The README reports 24.5 GB/s on GPT-2 on a 144-core EPYC, against 24.8 MB/s for HF tokenizers.
Who it is for: Engineers tokenizing large text corpora for LLM pretraining or data pipelines
Strengths
- Compatibility mode wraps HF tokenizers or tiktoken with a one-line change
- Built-in bench command with --validate checks output against HF on your tokenizer
- Covers many BPE tokenizers: Llama 3/4, Qwen 2/3/3.5, GPT-OSS, DeepSeek, GLM
- Runs on modern x86 and ARM CPUs
Weaknesses
- Compatibility mode is slower than the native API that gives the headline speedup
- SentencePiece models (Gemma, Mistral, Llama 2) are only about 7-22x faster than HF
- WordPiece is not supported and file sinks are not implemented
- Windows is lightly tested (WSL advised); CJK-heavy data is much slower
What it needs
- no GPU
- Needs Python, pip
- Models: GPT-2, Llama 3/4, Qwen 2/3/3.5, GPT-OSS, DeepSeek V3/R1
Also in ML frameworks and libraries
See all 242| Rank | Project | Score |
|---|---|---|
| 1 | transformersModel-definition library for pretrained text, vision, audio and multimodal models | 94 out of 100 |
| 2 | ultralyticsPython library and CLI for training and running YOLO vision models | 94 out of 100 |
| 3 | pytorch-image-modelsPyTorch library of image models, pretrained weights and training scripts | 94 out of 100 |
| #4 | pytorchPython tensor library with GPU support and tape-based autograd | 93 out of 100 |
| #5 | supervisionPython toolkit for loading, annotating and counting computer vision detections | 92 out of 100 |