#2222nd of 31 in Training and fine-tuning
autoresearch
Agent loop that edits a small GPT training script and keeps changes that lower val_bpb
- Stars
- 98.3k
- License
- no license
- Last commit
- Mar 2026
- Language
- Python
Missed the latest GitHub check; these facts may be older.
Overview
A coding agent (Claude, Codex or similar) is pointed at a single-GPU nanochat-derived training setup and edits train.py in a loop. Each experiment trains for a fixed 5 minutes and is kept or discarded based on validation bits per byte (val_bpb). The human edits program.md, the agent instructions, rather than the Python code.
Who it is for: Researchers with an NVIDIA GPU testing autonomous LLM training runs
Strengths
- Fixed 5-minute budget makes runs comparable: about 12 experiments per hour, around 100 overnight
- Small codebase: agent edits only train.py, so diffs stay reviewable
- val_bpb is vocab-size independent, so architecture changes compare fairly
- Works with any coding agent that can read program.md
Weaknesses
- Requires a single NVIDIA GPU (tested on H100); no CPU, MPS or AMD support in main repo
- Results are not comparable across different hardware because of the fixed time budget
- Default program.md is a bare baseline for one agent; multi-agent setups are left to the user
- README says to disable agent permissions, which is a safety risk without extra sandboxing
What it needs
- GPU required
- Needs uv, PyTorch, Python 3.10+, a coding agent (Claude, Codex or similar)
- Models: GPT (nanochat-style, trained from scratch), Muon + AdamW optimizer
Also in Training and fine-tuning
See all 31| Rank | Project | Score |
|---|---|---|
| 1 | unslothDesktop and web app to run, fine-tune and export open models | 88 out of 100 |
| 2 | deepspeedPyTorch library for distributed training of large models with ZeRO | 84 out of 100 |
| 3 | verlReinforcement learning training library for LLM post-training with PPO and GRPO | 84 out of 100 |
| #4 | peftPython library for LoRA and other parameter-efficient fine-tuning of Transformers models | 84 out of 100 |
| #5 | trlPost-training library with SFT, DPO, GRPO and KTO trainers | 81 out of 100 |