#2727th of 36 in Gateways and observability
gptcache
Python semantic cache for LLM responses using embeddings and vector search
- Stars
- 8.2k
- License
- MIT
- Last commit
- Sep 2026
- Last release
- Aug 2024
- Language
- Python
Missed the latest GitHub check; these facts may be older.
Zilliz is hiring: 11 open roles
Overview
GPTCache is a Python library that caches LLM responses and serves them for exact or semantically similar queries. It embeds each query, looks it up in a vector store, and keeps responses in a SQL cache store. It wraps the OpenAI client and integrates with LangChain and llama_index. The README says no new APIs or models will be added.
Who it is for: Python developers cutting LLM API cost and latency with response caching
Strengths
- Modular: swap embedding model, cache store, vector store and similarity evaluator
- OpenAI adapter needs a few init lines, no changes to existing calls
- Storage options include SQLite, PostgreSQL, MySQL, DynamoDB; vectors in FAISS or Milvus
- Temperature setting can skip the cache; sample benchmark reports hit ratio, latency, recall
Weaknesses
- README says no support will be added for new LLM APIs or models
- Last release 2024-08-01; examples use the older openai.ChatCompletion style
- Semantic matching can return false-positive hits; threshold tuning is left to the user
- README warns the API may change at any time
What it needs
- no GPU
- Needs Python 3.8.1+, SQLite or another SQL database, FAISS or Milvus vector store, OpenAI API key (for OpenAI adapter)
- Models: OpenAI, LangChain, llama.cpp, MiniGPT-4, Dolly
Also in Gateways and observability
See all 36| Rank | Project | Score |
|---|---|---|
| 1 | LiteLLMOpenAI-format gateway and Python SDK for calling 100+ LLM providers | 90 out of 100 |
| 2 | OmniRouteLLM gateway routing one OpenAI-compatible endpoint across 372 providers | 87 out of 100 |
| 3 | new-apiGateway that unifies LLM providers behind OpenAI, Anthropic and Gemini APIs | 83 out of 100 |
| #4 | freellmapiOpenAI-compatible router that pools free LLM provider tiers behind one endpoint | 80 out of 100 |
| #5 | LangfuseTracing, prompt management and evals for LLM apps on ClickHouse | 75 out of 100 |