#107107th of 242 in ML frameworks and libraries
spark-nlp
Distributed NLP library running pretrained transformer pipelines on Apache Spark
- Stars
- 4.2k
- License
- Apache-2.0
- Last commit
- Nov 2026
- Last release
- Nov 2026
- Language
- Scala
4,106 on 25 Apr 20264,164 on 14 Nov 2026
From GitHub's daily star counts.
Checked on GitHub.
Overview
Spark NLP is an NLP library built natively on Apache Spark. It provides annotators for tokenization, NER, classification, embeddings, translation, question answering and speech recognition, usable from Python, Scala, Java and R. It ships pretrained pipelines and models and can import models from TensorFlow, ONNX, OpenVINO and llama.cpp GGUF. Version 7.0.0 supports Spark 3.x on Scala 2.12 and Spark 4 on Scala 2.13.
Who it is for: Data engineers running NLP at scale on Spark, Databricks or EMR
Strengths
- Runs NLP pipelines natively inside Spark, so jobs scale across a cluster
- Imports models from TensorFlow, ONNX, OpenVINO and llama.cpp GGUF
- Documented setup for Databricks, EMR, EMR Serverless and GCP Dataproc
- Works fully offline once models and pipelines are downloaded
Weaknesses
- Needs Java and Spark, which is heavy for single-machine use
- Separate Maven artifacts per Spark and Scala version, including a Spark 4.0.0 build
- Apple Silicon and AArch64 builds are marked experimental
- No HTTP API or chat interface included; it is a library
What it needs
- GPU optional
- Needs Apache Spark, Java 8 or 11 (Spark 3.x), Java 17 (Spark 4.x)
- Models: TensorFlow, ONNX, OpenVINO, GGUF (llama.cpp)
What changed
From the release notes. No flag means the note does not say so, not that the upgrade is safe.
Also in ML frameworks and libraries
See all 242| Rank | Project | Score |
|---|---|---|
| 1 | transformersModel-definition library for pretrained text, vision, audio and multimodal models | 94 out of 100 |
| 2 | ultralyticsPython library and CLI for training and running YOLO vision models | 94 out of 100 |
| 3 | pytorch-image-modelsPyTorch library of image models, pretrained weights and training scripts | 94 out of 100 |
| #4 | pytorchPython tensor library with GPU support and tape-based autograd | 93 out of 100 |
| #5 | supervisionPython toolkit for loading, annotating and counting computer vision detections | 92 out of 100 |