#5353rd of 56 in ML frameworks and libraries
openmythos
PyTorch looped-depth MoE transformer with switchable MLA or GQA attention
- Stars
- 14.9k
- License
- MIT
- Last commit
- May 2026
- Language
- Python
Missed the latest GitHub check; these facts may be older.
Overview
OpenMythos is a PyTorch implementation of a Recurrent-Depth Transformer: a Prelude, a Recurrent Block looped up to max_loop_iters, and a Coda, with a sparse MoE feed-forward using routed and shared experts. It ships configs from 1B to 1T parameters and a DDP training script for a 3B model on FineWeb-Edu. The README calls it a speculative reconstruction unaffiliated with Anthropic, and it mentions no pretrained weights.
Who it is for: Researchers experimenting with looped transformers and MoE architectures
Strengths
- Attention switchable between MLA and GQA, with optional Flash Attention 2 for GQA
- Preset configs from mythos_1b to mythos_1t, installable with pip install open-mythos
- Includes a 3B FineWeb-Edu training script with single-GPU and torchrun multi-GPU modes
- Loop stability constrained by construction, with spectral radius of A checkable in code
Weaknesses
- Theoretical reconstruction based on speculation, not a replica of any real model
- No pretrained weights or benchmark results mentioned in the README
- Flash Attention 2 needs CUDA and build tools
- README example imports mythos_7b, which is not in the listed variants
What it needs
- GPU optional
- Needs PyTorch, flash-attn (optional)
- Models: Recurrent-Depth Transformer, MoE, MLA, GQA
Also in ML frameworks and libraries
See all 56| Rank | Project | Score |
|---|---|---|
| 1 | transformersPython library defining open models for text, vision, audio and multimodal tasks | 95 out of 100 |
| 2 | pytorchPython tensor library with GPU support and tape-based autograd for deep learning | 94 out of 100 |
| 3 | ultralyticsPython library and CLI for training and running YOLO vision models | 92 out of 100 |
| #4 | supervisionPython toolkit for loading, annotating and counting computer vision detections | 89 out of 100 |
| #5 | pytorch-image-modelsPyTorch library of image models, pretrained weights and reference training scripts | 89 out of 100 |