#4242nd of 50 in Data and scraping for AI
fastdup
Python library that finds duplicates, outliers and mislabels in image datasets
- Stars
- 1.9k
- License
- custom license
- Last commit
- Aug 2026
- Last release
- Jun 2024
- Language
- Python
1,841 on 21 Apr 20261,916 on 10 Nov 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
fastdup is a pip-installable Python package with a C++ engine that analyzes image and video datasets for duplicates and near-duplicates, outliers, mislabels, broken images and blurry or dark images. It can delete duplicates from disk in one call (default similarity above 0.96, with a dry-run option) and renders static galleries of the findings. It also accepts custom embeddings, such as TIMM or ONNX models like DINOv2.
Who it is for: ML engineers cleaning image and video datasets before training
Strengths
- README claims 400M images processed on a single CPU machine
- Runs locally; data stays on your own host or cloud
- Accepts custom embeddings from TIMM or ONNX models such as DINOv2
- remove_duplicates has a dry_run preview and an adjustable distance threshold
Weaknesses
- License badge says CC BY-NC-ND 4.0, which bars commercial use and derivatives
- Last release was 2024-06-14, while commits continue
- Output is static galleries; hosted Visual Layer Cloud is the upsell for more
- Windows is supported only through WSL2
What it needs
- GPU optional
- Docker
- Models: TIMM, ONNX, DINOv2
Quick answers
Is fastdup open source?
Its code is public on GitHub under its own license terms; read them before you use it at work.
Can you run fastdup with Docker?
Yes. Its repository has a Docker setup: a Dockerfile, a Compose file or a published image.
Does fastdup need a GPU?
No. It runs on a CPU and can use a GPU when you have one.
Also in Data and scraping for AI
See all 50| Rank | Project | Score |
|---|---|---|
| 1 | doclingPython library that converts PDFs, Office files and more into structured text | 92 out of 100 |
| 2 | mineruParses PDFs, images and Office files into Markdown and structured JSON | 91 out of 100 |
| 3 | opendataloader-pdfPDF parser that outputs Markdown, JSON with bounding boxes, and Tagged PDF | 90 out of 100 |
| #4 | agent-reachCLI that gives coding agents read access to login-gated and Chinese web platforms | 88 out of 100 |
| #5 | markitdownPython library and CLI that converts office files, PDFs and audio to Markdown | 87 out of 100 |