mergekit combines pre-trained language models by operating on their weights, driven by a YAML config and the `mergekit-yaml` command. Tensors load lazily, so merges run on CPU or with as little as 8 GB of VRAM. It supports many methods (linear, SLERP, TIES, DARE, DELLA, SCE, Model Stock), plus layer-wise frankenmerges, MoE merging and LoRA extraction.
Who it is for: Practitioners combining open-weight LLMs without retraining
Strengths
Lazy tensor loading: merges run on CPU or with as little as 8 GB VRAM
Many methods: linear, SLERP, TIES, DARE, DELLA, SCE, Model Stock and others
Also covers MoE merging, LoRA extraction, evolutionary merging and tokenizer transplantation
Declarative YAML configs with per-layer and per-tensor parameter gradients
Weaknesses
Command-line only; no bundled UI
No Docker image or compose file provided
Install is from a git clone with pip install -e; no PyPI install documented
Models with different vocabularies need manual tokenizer configuration
What it needs
GPU optional
Needs PyTorch, Hugging Face Hub (optional, for upload)
Models: Llama, Mistral, GPT-NeoX, StableLM, raw PyTorch models