169 out of 100
yue
Lyrics-to-song model that writes an editable melody and chord score first
YuE2 generates full songs with vocals and accompaniment from lyrics and a style prompt. It first plans an editable ABC score (melody and chords), then renders semantic tokens and acoustic latents to 48 kHz stereo audio with an AR–NAR Mixture-of-Transformers and a VAE. The repo also covers zero-shot covers via SheetSage2 transcription, score editing, and an agent skill. Code is Apache-2.0; model weights are CC BY-NC 4.0 with an additional creator permission.
Strengths
- Editable ABC score between prompt and audio lets you inspect and change melody and chords
- Staged Python API: plan, generate_semantic, synthesize, decode; artifacts saved per run
- Zero-shot covers via SheetSage2 transcription with the same checkpoint, no cover fine-tuning
- Reports WildSongBench results against Suno, Mureka and others, with a reproduction guide
Weaknesses
- Needs Linux, Python 3.12 and an NVIDIA BF16 GPU with 24 GB VRAM
- Weights are CC BY-NC 4.0; commercial company use requires a separate license
- Editing renders a new recording; the original waveform is not preserved
- Benchmark numbers are automatic metrics from the authors; gaps between top means are not significant
- GPU required
- Needs Hugging Face model downloads, SheetSage2 (separate environment, for covers), PyTorch
- Models: YuE2-3B, YuE2-Vae, YuE2-Vae-legacy, SheetSage2, MERT-v2