#109109th of 118 in Agent frameworks and SDKs
self-operating-computer
CLI that lets multimodal models control a computer through screenshots, mouse and keyboard
- Stars
- 10.3k
- License
- MIT
- Last commit
- Oct 2026
- Last release
- Feb 2025
- Language
- Python
10,111 on 20 Apr 202610,305 on 9 Nov 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
Python framework, installed with pip and run as `operate`, that sends screenshots to a multimodal model and executes the mouse and keyboard actions it returns until an objective is met. It supports GPT-4o, GPT-4.1, o1, Gemini Pro Vision, Claude 3, Qwen-VL and LLaVA via Ollama, with OCR and Set-of-Mark modes for locating click targets. Runs on macOS, Windows and Linux with X server.
Who it is for: Developers trying vision-model computer-use agents on their own machine
Strengths
- Works with hosted models and a local LLaVA via Ollama, selected with the `-m` flag
- OCR and Set-of-Mark modes ground clicks in on-screen text or detected buttons
- Installs with `pip install self-operating-computer`; runs on macOS, Windows and Linux with X server
- Optional voice input for the objective via `--voice`
Weaknesses
- README says LLaVA error rates are very high; local use is only a base to build on
- Last release was 2025-02-28, and the README still lists older models like Claude 3
- README says gpt-4o is required and needs $5 of OpenAI API spend to unlock
- Needs Screen Recording and Accessibility permissions; no sandbox, actions run on the host
What it needs
- GPU optional
- Needs OpenAI API key (default gpt-4o), Ollama (for LLaVA), portaudio (voice mode)
- Models: GPT-4o, GPT-4.1, o1, Gemini Pro Vision, Claude 3
What changed
From the release notes. No flag means the note does not say so, not that the upgrade is safe.
Quick answers
Is self-operating-computer open source?
Yes. Its code is on GitHub under the MIT license.
Does self-operating-computer need a GPU?
No. It runs on a CPU and can use a GPU when you have one.
Also in Agent frameworks and SDKs
See all 118| Rank | Project | Score |
|---|---|---|
| 1 | ppt-masterAgent skill that turns documents into natively editable PowerPoint decks | 95 out of 100 |
| 2 | openai-agents-pythonPython SDK for multi-agent workflows with handoffs, guardrails and tracing | 91 out of 100 |
| 3 | mastraTypeScript framework for building AI agents, workflows and MCP servers | 91 out of 100 |
| #4 | browser-usePython library that lets an LLM agent control a web browser | 90 out of 100 |
| #5 | copilotkitSDK for adding agent chat, generative UI and shared state to apps | 90 out of 100 |