#107107th of 242 in ML frameworks and libraries

spark-nlp

Distributed NLP library running pretrained transformer pipelines on Apache Spark

Stars
4.2k
License
Apache-2.0
Last commit
Nov 2026
Last release
Nov 2026
Language
Scala
Stars, last 29 weeks+58
25 Apr 2026: 4,106 stars2 May 2026: 4,110 stars9 May 2026: 4,111 stars16 May 2026: 4,112 stars23 May 2026: 4,112 stars30 May 2026: 4,115 stars6 Jun 2026: 4,116 stars13 Jun 2026: 4,122 stars20 Jun 2026: 4,123 stars27 Jun 2026: 4,125 stars4 Jul 2026: 4,127 stars11 Jul 2026: 4,130 stars18 Jul 2026: 4,134 stars25 Jul 2026: 4,139 stars1 Aug 2026: 4,142 stars8 Aug 2026: 4,146 stars15 Aug 2026: 4,149 stars22 Aug 2026: 4,150 stars29 Aug 2026: 4,152 stars5 Sept 2026: 4,153 stars12 Sept 2026: 4,154 stars19 Sept 2026: 4,155 stars26 Sept 2026: 4,155 stars3 Oct 2026: 4,157 stars10 Oct 2026: 4,158 stars17 Oct 2026: 4,159 stars24 Oct 2026: 4,159 stars31 Oct 2026: 4,160 stars7 Nov 2026: 4,161 stars14 Nov 2026: 4,164 stars

4,106 on 25 Apr 20264,164 on 14 Nov 2026

From GitHub's daily star counts.

Checked on GitHub.

Overview

Spark NLP is an NLP library built natively on Apache Spark. It provides annotators for tokenization, NER, classification, embeddings, translation, question answering and speech recognition, usable from Python, Scala, Java and R. It ships pretrained pipelines and models and can import models from TensorFlow, ONNX, OpenVINO and llama.cpp GGUF. Version 7.0.0 supports Spark 3.x on Scala 2.12 and Spark 4 on Scala 2.13.

Who it is for: Data engineers running NLP at scale on Spark, Databricks or EMR

Strengths

  • Runs NLP pipelines natively inside Spark, so jobs scale across a cluster
  • Imports models from TensorFlow, ONNX, OpenVINO and llama.cpp GGUF
  • Documented setup for Databricks, EMR, EMR Serverless and GCP Dataproc
  • Works fully offline once models and pipelines are downloaded

Weaknesses

  • Needs Java and Spark, which is heavy for single-machine use
  • Separate Maven artifacts per Spark and Scala version, including a Spark 4.0.0 build
  • Apple Silicon and AArch64 builds are marked experimental
  • No HTTP API or chat interface included; it is a library

What it needs

  • GPU optional
  • Needs Apache Spark, Java 8 or 11 (Spark 3.x), Java 17 (Spark 4.x)
  • Models: TensorFlow, ONNX, OpenVINO, GGUF (llama.cpp)

What changed

From the release notes. No flag means the note does not say so, not that the upgrade is safe.

Also in ML frameworks and libraries

See all 242
Also in ML frameworks and libraries
RankProjectScore
1transformersModel-definition library for pretrained text, vision, audio and multimodal models169.3k starsApache-2.094 out of 100
2ultralyticsPython library and CLI for training and running YOLO vision models63.5k starsAGPL-3.094 out of 100
3pytorch-image-modelsPyTorch library of image models, pretrained weights and training scripts37.3k starsApache-2.094 out of 100
#4pytorchPython tensor library with GPU support and tape-based autograd105.8k starscustom license93 out of 100
#5supervisionPython toolkit for loading, annotating and counting computer vision detections Live demo ↗ (opens in a new tab)51.7k starsMIT92 out of 100