omniparser

Screenshot parser that turns GUI screens into labeled elements for vision LLM agents

Live demo ↗ Website ↗ Repository on GitHub ↗

Stars: 25.5k
License: CC-BY-4.0
Last commit: Jul 2026
Last release: Sep 2025
Language: Jupyter Notebook

Missed the latest GitHub check; these facts may be older.

Overview

OmniParser detects interactable regions in UI screenshots with a YOLO-based detector and captions each icon with a Florence-based model. The structured output lets a vision LLM ground its actions on screen elements. The repo ships a demo notebook, a Gradio demo, and OmniTool, which drives a Windows 11 VM with OpenAI, DeepSeek, Qwen or Anthropic models.

Who it is for: Developers building vision-based computer-use or GUI agents

Strengths

Outputs structured, captioned UI elements that any vision-capable LLM can use for grounding
Model weights published on Hugging Face, with V1.5 and V2 checkpoints
Includes a Gradio demo and a hosted Hugging Face Space to try it
OmniTool supports OpenAI, DeepSeek, Qwen 2.5VL and Anthropic Computer Use

Weaknesses

Weights must be fetched manually with huggingface-cli, including a PR revision for the newest detector
Mixed licensing: older Ultralytics-based detectors are AGPL, and the README badge says MIT
OmniTool documentation is marked work in progress
No Docker setup; requires a conda environment with Python 3.12

What it needs

Needs conda, Python 3.12, Hugging Face model weights, Gradio
Models: YOLOv9-E icon detector, Florence icon captioner, OpenAI 4o/o1/o3-mini, DeepSeek R1, Qwen 2.5VL

Also in Open models

See all 10

Also in Open models
Rank	Project	Score
1	llmfitRust tool that checks which LLMs fit your RAM, CPU and GPU37.9k starsMIT	83 out of 100
2	layaSingle-pass multilingual classifier for typed choice, score and yes/no decisions Live demo ↗ (opens in a new tab)32.9k starsApache-2.0	76 out of 100
3	timesfmPretrained time-series foundation model for zero-shot forecasting from Google Research34.5k starsApache-2.0	70 out of 100
#4	unlimited-ocrOCR model that parses long documents and multi-page PDFs in one pass Live demo ↗ (opens in a new tab)27.3k starsMIT	55 out of 100
#5	insightfaceFace detection, recognition and alignment toolbox with ArcFace and RetinaFace models Live demo ↗ (opens in a new tab)29.9k starsno license	52 out of 100

How to choose

Read the license: open weights are not always open for commercial use.
Check the hardware the model needs at the precision you plan to run.
See whether the repo ships inference code or only weights.

Health

Releases: Not checked
Top contributor: Not checked
Security advisories: Not checked
Package advisories: Not checked
OpenSSF Scorecard: Not checked

Not read for this project yet: each refresh reads the facts of a few projects. Bots and coding-agent accounts are left out of the commit counts. Not part of the score: how we check.

License in plain words

CC-BY-4.0 Custom terms

Use it at work: Check
Change it and rebrand it: Check
Sell it or host it for others: Check

Your part: GitHub could not match these terms to a standard license. Read the license before you use it at work, rebrand it or host it for others.

A plain summary, not legal advice. Read the license on GitHub ↗ (opens in a new tab)

#8 of 10 in Open models on Archestack, as of 15 Oct 2026.

In your README

The badge updates twice a day.

[![Archestack](https://archestack.com/badge/microsoft/omniparser.svg)](https://archestack.com/best-of-ai/projects/omniparser/)

How badges work

On your site or blog

A card with the rank, score and signals, in light or dark to match the reader.

<iframe src="https://archestack.com/embed/best-of-ai/omniparser/" title="omniparser on Archestack" width="400" height="270" style="border:0;max-width:100%" loading="lazy"></iframe>

Preview the card

Best of Open-Source AI on GitHub

Each review comes from the project's README and facts we check on GitHub. The fix form opens with omniparser filled in.