#168168th of 242 in ML frameworks and libraries

gigatoken

Rust BPE tokenizer for Python, with HuggingFace and tiktoken compatibility

Stars
4.2k
License
MIT
Last commit
Nov 2026
Language
Rust
Stars, last 29 weeks+4,155
25 Apr 2026: 2 stars2 May 2026: 2 stars9 May 2026: 2 stars16 May 2026: 2 stars23 May 2026: 2 stars30 May 2026: 2 stars6 Jun 2026: 2 stars13 Jun 2026: 2 stars20 Jun 2026: 2 stars27 Jun 2026: 2 stars4 Jul 2026: 2 stars11 Jul 2026: 2 stars18 Jul 2026: 11 stars25 Jul 2026: 3,350 stars1 Aug 2026: 3,839 stars8 Aug 2026: 3,919 stars15 Aug 2026: 3,977 stars22 Aug 2026: 4,028 stars29 Aug 2026: 4,054 stars5 Sept 2026: 4,063 stars12 Sept 2026: 4,076 stars19 Sept 2026: 4,089 stars26 Sept 2026: 4,101 stars3 Oct 2026: 4,108 stars10 Oct 2026: 4,115 stars17 Oct 2026: 4,126 stars24 Oct 2026: 4,135 stars31 Oct 2026: 4,151 stars7 Nov 2026: 4,156 stars14 Nov 2026: 4,157 stars

2 on 25 Apr 20264,157 on 14 Nov 2026

From GitHub's daily star counts.

Checked on GitHub.

Overview

Gigatoken is a Rust tokenizer library, installed with pip, that encodes text files using SIMD pretokenization and a pretoken cache across all CPU cores. It accepts HuggingFace model names and can wrap existing HF tokenizers or tiktoken encoders. The README reports 24.5 GB/s on GPT-2 on a 144-core EPYC, against 24.8 MB/s for HF tokenizers.

Who it is for: Engineers tokenizing large text corpora for LLM pretraining or data pipelines

Strengths

  • Compatibility mode wraps HF tokenizers or tiktoken with a one-line change
  • Built-in bench command with --validate checks output against HF on your tokenizer
  • Covers many BPE tokenizers: Llama 3/4, Qwen 2/3/3.5, GPT-OSS, DeepSeek, GLM
  • Runs on modern x86 and ARM CPUs

Weaknesses

  • Compatibility mode is slower than the native API that gives the headline speedup
  • SentencePiece models (Gemma, Mistral, Llama 2) are only about 7-22x faster than HF
  • WordPiece is not supported and file sinks are not implemented
  • Windows is lightly tested (WSL advised); CJK-heavy data is much slower

What it needs

  • no GPU
  • Needs Python, pip
  • Models: GPT-2, Llama 3/4, Qwen 2/3/3.5, GPT-OSS, DeepSeek V3/R1

Also in ML frameworks and libraries

See all 242
Also in ML frameworks and libraries
RankProjectScore
1transformersModel-definition library for pretrained text, vision, audio and multimodal models169.3k starsApache-2.094 out of 100
2ultralyticsPython library and CLI for training and running YOLO vision models63.5k starsAGPL-3.094 out of 100
3pytorch-image-modelsPyTorch library of image models, pretrained weights and training scripts37.3k starsApache-2.094 out of 100
#4pytorchPython tensor library with GPU support and tape-based autograd105.8k starscustom license93 out of 100
#5supervisionPython toolkit for loading, annotating and counting computer vision detections Live demo ↗ (opens in a new tab)51.7k starsMIT92 out of 100