freetoken

MoE inference engine that runs large open models on consumer NVIDIA GPUs

Documentation ↗ Website ↗ Repository on GitHub ↗

Stars: 14.5k
License: Apache-2.0
Last commit: Oct 2026
Last release: Sep 2026
Language: Python

Missed the latest GitHub check; these facts may be older.

Overview

FreeToken is a Mixture-of-Experts serving engine that splits work between GPU and CPU and keeps experts in a global LRU cache, so large MoE models can run on gaming PCs and laptops. It exposes Anthropic- and OpenAI-compatible APIs for coding agents. It ships as a Python package with a CLI and as a Windows/Linux desktop app with a GUI.

Who it is for: Engineers running large MoE models locally on RTX GPUs

Strengths

Bandwidth-adaptive CPU-GPU co-execution targets MoE models larger than VRAM
Anthropic and OpenAI-compatible APIs work with Claude Code, Codex and OpenCode
Re-allocates VRAM between expert cache and KV memory without engine restart
Supports MXFP4, NVFP4, FP8 and BF16 quantizations

Weaknesses

Focused on MoE models; dense model support is not stated in the README
Native support listed only for NVIDIA RTX 30, 40 and 50 series
No Docker or compose files provided
Uses its own FTW weight format; old checkpoints may need repair

What it needs

GPU required
Needs uv or pip, NVIDIA RTX GPU
Models: DeepSeek-V4-Flash, Qwen3.6-35B-A3B, GLM-5.2, MXFP4, NVFP4

Also in Model serving

See all 33

Also in Model serving
Rank	Project	Score
1	LocalAISelf-hosted inference server with OpenAI-compatible API and pluggable backends49.5k starsMIT	83 out of 100
2	llama.cppC/C++ inference engine serving GGUF models over an OpenAI-compatible API131.2k starsMIT	81 out of 100
3	OllamaLocal model runner with CLI and REST API for open-weight LLMs183.2k starsMIT	80 out of 100
#4	vLLMInference engine for Hugging Face LLMs with PagedAttention and continuous batching93.8k starsApache-2.0	76 out of 100
#5	colibriC inference engine that runs huge MoE models by streaming experts from disk41.9k starsApache-2.0	73 out of 100

How to choose

Pick by hardware first (CPU, NVIDIA, AMD, Apple Silicon) and by model format (GGUF, safetensors).
Throughput engines (continuous batching, paged attention) need GPUs; single-user servers do not.
An OpenAI-compatible API keeps the rest of your stack portable.

Health

Releases: Not checked
Top contributor: Not checked
Security advisories: Not checked
Package advisories: Not checked
OpenSSF Scorecard: Not checked

Not read for this project yet: each refresh reads the facts of a few projects. Bots and coding-agent accounts are left out of the commit counts. Not part of the score: how we check.

License in plain words

Apache-2.0 Permissive

Use it at work: Yes
Change it and rebrand it: Yes
Sell it or host it for others: Yes

Your part: Keep the notices and mark the files you change. It also grants you the contributors’ patent rights.

A plain summary, not legal advice. Read the license on GitHub ↗ (opens in a new tab)

Where to run it

Vast.ai ↗ (opens in a new tab)Rent a GPU by the hour from a marketplace of hosts.

Referral links: if you sign up through one, we get a commission. They never change a rank.

#13 of 33 in Model serving on Archestack, as of 15 Oct 2026.

In your README

The badge updates twice a day.

[![Archestack](https://archestack.com/badge/flashml-org/freetoken.svg)](https://archestack.com/best-of-ai/projects/freetoken/)

How badges work

On your site or blog

A card with the rank, score and signals, in light or dark to match the reader.

<iframe src="https://archestack.com/embed/best-of-ai/freetoken/" title="freetoken on Archestack" width="400" height="270" style="border:0;max-width:100%" loading="lazy"></iframe>

Preview the card

Best of Open-Source AI on GitHub

Each review comes from the project's README and facts we check on GitHub. The fix form opens with freetoken filled in.