#6666th of 70 in Open models
joycaption
Llama 3.1-based image captioning VLM for building diffusion training datasets
- Stars
- 1.3k
- License
- Apache-2.0
- Last commit
- Feb 2026
- Language
- Jupyter Notebook
1,120 on 21 Apr 20261,301 on 10 Nov 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
JoyCaption is an open-weight vision-language model built on Llama 3.1 that writes captions for images, aimed at preparing diffusion model training data. It supports prompt modes such as descriptive, straightforward, Stable Diffusion and MidJourney prompts, and Danbooru, e621 and Rule34 tag lists. The repo holds usage examples (Transformers, vLLM) and finetuning scripts; the current model is Beta One.
Who it is for: People building image datasets for diffusion model training or finetuning
Strengths
- Covers SFW and NSFW content, unlike the censored captioners it is compared against
- Eleven prompt modes, including booru tag lists and Stable Diffusion prompt styles
- Serves through vLLM with an OpenAI-compatible API
- Includes finetuning scripts and a ComfyUI node with 8-bit and 4-bit support
Weaknesses
- Needs about 17GB VRAM at bfloat16; lower memory requires quantization
- Glitches on 1.5 to 3% of outputs, such as repetition loops
- Tag-list modes (Danbooru, e621, Rule34) have lower accuracy than caption modes
- Not a general instruction follower; left/right and OCR errors are expected
What it needs
- GPU required
- Needs PyTorch, Hugging Face Transformers, vLLM (optional)
- Models: Llama 3.1 LLaVA-style HF checkpoint (bfloat16), 8-bit and 4-bit quantization
Quick answers
Is joycaption open source?
Yes. Its code is on GitHub under the Apache-2.0 license.
Can you run joycaption with Docker?
Yes. Its repository has a Docker setup: a Dockerfile, a Compose file or a published image.
Does joycaption need a GPU?
Yes, it needs a GPU.
Also in Open models
See all 70| Rank | Project | Score |
|---|---|---|
| 1 | ruviewWiFi CSI sensing on ESP32 for presence, breathing, heart rate and pose | 94 out of 100 |
| 2 | layaSingle-pass multilingual classifier for typed choice, score and yes/no decisions | 88 out of 100 |
| 3 | rf-detrReal-time transformer models for object detection, segmentation and keypoints | 79 out of 100 |
| #4 | timesfmPretrained time-series foundation model for zero-shot forecasting from Google Research | 78 out of 100 |
| #5 | yolov5PyTorch YOLOv5 object detection, segmentation and classification models with training code | 76 out of 100 |