#55th of 7 in Training and fine-tuning
heretic
Removes refusal behavior from transformer language models via automated abliteration
- Stars
- 35.3k
- License
- AGPL-3.0
- Last commit
- Oct 2026
- Last release
- Jun 2026
- Language
- Python
16,859 on 25 Mar 202635,291 on 14 Oct 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
Heretic applies directional ablation (abliteration) to the attention out-projection and MLP down-projection matrices of a transformer model. An Optuna TPE optimizer searches the ablation parameters, co-minimizing refusals and KL divergence from the original model. After a run it can save the model, upload it to Hugging Face, chat with it, or run benchmarks.
Who it is for: Users who want to decensor open-weight LLMs without manual abliteration work
Strengths
- One command, no configuration needed; Optuna TPE tunes ablation parameters automatically
- README reports 3/100 refusals at 0.16 KL on gemma-3-12b-it, versus 0.45 and 1.04 for manual abliterations
- Supports most dense models, several MoE architectures, and some multimodal and hybrid models
- Optional bitsandbytes 4-bit quantization (bnb_4bit) reduces VRAM needs
Weaknesses
- Pure state-space models and some research architectures are not supported out of the box
- AGPL-3.0 license may restrict use in proprietary or networked products
- About 20-30 minutes for a 4B model on an RTX 3090; larger models unknown
- README metrics come from one RTX 5090 setup and may vary by hardware
What it needs
- GPU optional
- Needs Python 3.10+, PyTorch 2.2+, Optuna, bitsandbytes (optional), Hugging Face
- Models: Hugging Face transformer models (dense), MoE, multimodal, hybrid (Qwen3.5), MXFP4 (gpt-oss, needs PyTorch 2.6+)
Also in Training and fine-tuning
See all 7| Rank | Project | Score |
|---|---|---|
| 1 | unslothDesktop and web app to run, fine-tune and export open models | 87 out of 100 |
| 2 | deepspeedPyTorch library for distributed training of large models with ZeRO | 74 out of 100 |
| 3 | llamafactoryFine-tuning toolkit for 100+ LLMs with a CLI and Gradio web UI | 72 out of 100 |
| #4 | minimindTrain a 64M-parameter LLM from scratch in plain PyTorch | 63 out of 100 |
| #6 | nanochatMinimal PyTorch codebase to train and chat with a small GPT-2-class LLM | 49 out of 100 |