#2323rd of 50 in Data and scraping for AI
trafilatura
Python library and CLI that extracts main text and metadata from web pages
- Stars
- 7.1k
- License
- Apache-2.0
- Last commit
- Nov 2026
- Last release
- Oct 2026
- Language
- Python
5,698 on 21 Apr 20267,121 on 10 Nov 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
Trafilatura downloads web pages, crawls sitemaps and feeds, and extracts main text, metadata and comments from HTML while dropping headers, footers and boilerplate. It uses its own rule-based extractor with jusText and readability-lxml as fallbacks, and outputs TXT, Markdown, CSV, JSON, HTML, XML or XML-TEI. It runs as a Python package or command-line tool and needs no database.
Who it is for: Developers building text corpora or LLM data pipelines from web pages
Strengths
- Reports top results in ScrapingHub and Bevendorff et al. 2023 extraction benchmarks
- Handles discovery too: sitemaps (TXT, XML), feeds (ATOM, JSON, RSS), URL filtering and deduplication
- Outputs TXT, Markdown, CSV, JSON, HTML, XML and XML-TEI, with metadata fields
- Works on live URLs or already downloaded HTML; no database required
Weaknesses
- Rule-based extraction with fallbacks; no LLM or JavaScript rendering mentioned
- Versions before v1.8.0 are GPLv3+, which matters when pinning old releases
- Not an AI-native tool; LLM or RAG pipeline integration is left to the user
What it needs
- no GPU
- Needs jusText, readability-lxml
Quick answers
Is trafilatura open source?
Yes. Its code is on GitHub under the Apache-2.0 license.
Does trafilatura need a GPU?
No, it runs without one.
Also in Data and scraping for AI
See all 50| Rank | Project | Score |
|---|---|---|
| 1 | doclingPython library that converts PDFs, Office files and more into structured text | 92 out of 100 |
| 2 | mineruParses PDFs, images and Office files into Markdown and structured JSON | 91 out of 100 |
| 3 | opendataloader-pdfPDF parser that outputs Markdown, JSON with bounding boxes, and Tagged PDF | 90 out of 100 |
| #4 | agent-reachCLI that gives coding agents read access to login-gated and Chinese web platforms | 88 out of 100 |
| #5 | markitdownPython library and CLI that converts office files, PDFs and audio to Markdown | 87 out of 100 |