22nd of 2 in Training and fine-tuning
minimind
Train a 64M-parameter LLM from scratch in plain PyTorch
- Stars
- 63.7k
- License
- Apache-2.0
- Last commit
- Sep 2026
- Last release
- Oct 2025
- Language
- Python
Checked on GitHub.
Overview
MiniMind is a from-scratch implementation of a small Qwen3-style decoder-only LLM (64M dense, 198M-A64M MoE) with each training stage written in native PyTorch: pretrain, SFT, LoRA, DPO, PPO/GRPO/CISPO, agentic RL and distillation. It ships datasets, a 6,400-token tokenizer, an OpenAI-compatible server and a Streamlit chat UI. The README states one SFT epoch takes about 2 hours on a single 3090.
Who it is for: Engineers learning LLM internals by pretraining and fine-tuning small models
Strengths
- Core algorithms (LoRA, DPO, PPO/GRPO) written in plain PyTorch, not wrapped from trl or peft
- Open datasets for each stage, with mini variants (1.2GB pretrain, 1.6GB SFT) for quick runs
- Weights convert to transformers, llama.cpp, vllm and ollama; ollama and vllm usage is documented
- Checkpoint resume, DDP and DeepSpeed multi-GPU, wandb or swanlab logging
Weaknesses
- Models are tiny (64M), so output quality is far below mainstream LLMs
- README is primarily in Chinese; the English version is a separate file
- Training data is largely synthetic and distilled, with mixed source licenses
- Native MoE training is about 50% slower than dense, per the README
What it needs
- GPU optional
- Needs PyTorch, transformers, streamlit (optional WebUI), swanlab or wandb (optional)
- Models: minimind-3 (64M), minimind-3-moe (198M-A64M), minimind2 series, Transformers format, llama.cpp
Head to head
Also in Training and fine-tuning
See all 2| Rank | Project | Score |
|---|---|---|
| 1 | llamafactoryFine-tuning toolkit for 100+ LLMs with a CLI and Gradio web UI | 74 out of 100 |