Statistics for topic pyspark

RepositoryStats tracks 641,709 Github repositories, of these 118 are tagged with the pyspark topic. The most common primary language for repositories using this topic is Python (54). Other languages include: Jupyter Notebook (32)

Stargazers over time for topic pyspark

90908080707060605050404030302020101000202020202021202120222022202320232024202420252025

Most starred repositories for topic pyspark (view more)

ibis-project
ibis ibis-project
632
5.7k
apache-2.0
81
the portable Python dataframe library
sql mssql mysql trino duckdb impala pandas polars python sqlite pyarrow pyspark bigquery database snowflake clickhouse datafusion postgresql
Created 2015-04-17
9,590 commits to main branch, last one a day ago
microsoft
SynapseML microsoft
842
5.1k
mit
142
Simple and Distributed Machine Learning
ai ml http onnx azure scala spark opencv pyspark synapse big-data lightgbm microsoft databricks apache-spark data-science deep-learning machine-learning model-deployment cognitive-services
Created 2017-06-05
1,676 commits to master branch, last one 2 days ago
JohnSnowLabs
spark-nlp JohnSnowLabs
722
4.0k
apache-2.0
99
State of the Art Natural Language Processing
llm nlp bert onnx spark pyspark llamacpp lemmatizer tensorflow transformers spell-checker entity-extraction language-detection question-answering sentiment-analysis machine-translation text-classification part-of-speech-tagger named-entity-recognition natural-language-processing
Created 2017-09-24
8,652 commits to master branch, last one 2 months ago
apache
linkis apache
1.2k
3.4k
apache-2.0
263
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
sql udf hive jdbc livy spark engine impala linkis presto pyspark storage rest-api scriptis jobserver hive-table thrift-server context-service resource-manager application-manager
Created 2019-07-23
4,237 commits to master branch, last one 20 days ago
AlexIoannides
pyspark-example-project AlexIoannides
758
1.9k
unknown
56
Implementing best practices for PySpark ETL jobs and applications.
etl spark python etl-job pyspark data-science etl-pipeline data-engineering
Created 2017-12-28
36 commits to master branch, last one 3 years ago
uber
petastorm uber
281
1.8k
apache-2.0
37
Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, a...
sysml parquet pyarrow pyspark pytorch tensorflow deep-learning parquet-files machine-learning
Created 2018-06-15
691 commits to master branch, last one about a year ago

Trending repositories for topic pyspark (view more)

Last 3 days (new repositories)
no newly created repositories trending in the last 3 days
Last 3 days (absolute gain)
AlexIoannides
AlexIoannides/pyspark-example-project

Implementing best practices for PySpark ETL jobs and applications.

1,894 (+3)
capitalone
capitalone/datacompy

Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!

550 (+2)
apache-2.0
narwhals-dev
narwhals-dev/narwhals

Lightweight and extensible compatibility layer between dataframe libraries!

933 (+2)
mit
microsoft
microsoft/SynapseML

Simple and Distributed Machine Learning

5,122 (+2)
mit
ibis-project
ibis-project/ibis

the portable Python dataframe library

5,701 (+2)
apache-2.0
Last 3 days (relative gain)
capitalone
capitalone/datacompy

Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!

550 (+0.4%)
apache-2.0
narwhals-dev
narwhals-dev/narwhals

Lightweight and extensible compatibility layer between dataframe libraries!

933 (+0.2%)
mit
kevinschaich
kevinschaich/pyspark-cheatsheet

🐍 Quick reference guide to common patterns & functions in PySpark.

527 (+0.2%)
mit
AlexIoannides
AlexIoannides/pyspark-example-project

Implementing best practices for PySpark ETL jobs and applications.

1,894 (+0.2%)
awesome-spark
awesome-spark/awesome-spark

A curated list of awesome Apache Spark packages and resources.

1,786 (+0.1%)
cc0-1.0
Last week (new repositories)
no newly created repositories trending in the last week
Last week (absolute gain)
ibis-project
ibis-project/ibis

the portable Python dataframe library

5,701 (+11)
apache-2.0
capitalone
capitalone/datacompy

Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!

550 (+10)
apache-2.0
AlexIoannides
AlexIoannides/pyspark-example-project

Implementing best practices for PySpark ETL jobs and applications.

1,894 (+8)
narwhals-dev
narwhals-dev/narwhals

Lightweight and extensible compatibility layer between dataframe libraries!

933 (+7)
mit
davidzajac1
davidzajac1/zillacode

Open Source LeetCode for PySpark, Spark, Pandas and DBT/Snowflake

170 (+3)
apache-2.0
Last week (relative gain)
capitalone
capitalone/datacompy

Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!

550 (+2%)
apache-2.0
davidzajac1
davidzajac1/zillacode

Open Source LeetCode for PySpark, Spark, Pandas and DBT/Snowflake

170 (+2%)
apache-2.0
myamafuj
myamafuj/hadoop-hive-spark-docker

Hadoop-Hive-Spark cluster + Jupyter on Docker

72 (+1%)
mitchelllisle
mitchelllisle/sparkdantic

✨ A Pydantic to PySpark schema library

84 (+1%)
mit
coder2j
coder2j/pyspark-tutorial

PySpark Tutorial for Beginners - Practical Examples in Jupyter Notebook with Spark version 3.4.1. The tutorial covers various topics like Spark Introduction, Spark Installation, Spark RDD Transformati...

109 (+0.9%)
mit
Last month (new repositories)
no newly created repositories trending in the last month
Last month (absolute gain)
ibis-project
ibis-project/ibis

the portable Python dataframe library

5,701 (+73)
apache-2.0
narwhals-dev
narwhals-dev/narwhals

Lightweight and extensible compatibility layer between dataframe libraries!

933 (+49)
mit
AlexIoannides
AlexIoannides/pyspark-example-project

Implementing best practices for PySpark ETL jobs and applications.

1,894 (+37)
apache
apache/incubator-graphar

An open source, standard data file format for graph data storage and retrieval.

274 (+36)
apache-2.0
lakehq
lakehq/sail

LakeSail's computation framework with a mission to unify batch processing, stream processing, and compute-intensive (AI) workloads.

724 (+35)
apache-2.0
Last month (relative gain)
gljoseph
gljoseph/dfw-skytrack

This real-time project integrates flight information from the AviationStack API for DFW Airport and weather data from the National Weather Service API, to provide the latest arrival, departure, and fo...

29 (+26%)
apache
apache/incubator-graphar

An open source, standard data file format for graph data storage and retrieval.

274 (+15%)
apache-2.0
mitchelllisle
mitchelllisle/sparkdantic

✨ A Pydantic to PySpark schema library

84 (+14%)
mit
awslabs
awslabs/amazon-emr-vscode-toolkit

A VS Code Extension to make it easier to manage and develop Spark jobs on EMR

35 (+9%)
apache-2.0
coder2j
coder2j/pyspark-tutorial

PySpark Tutorial for Beginners - Practical Examples in Jupyter Notebook with Spark version 3.4.1. The tutorial covers various topics like Spark Introduction, Spark Installation, Spark RDD Transformati...

109 (+7%)
mit
Last 12-months (new repositories)
Nike-Inc
Nike-Inc/koheesio

Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.

635
apache-2.0
davidzajac1
davidzajac1/zillacode

Open Source LeetCode for PySpark, Spark, Pandas and DBT/Snowflake

170
apache-2.0
ManuelGuerra1987
ManuelGuerra1987/data-engineering-zoomcamp-notes

Detailed notes and homeworks from 2025 Data Engineering Zoomcamp by Datatalks.Club

43
josephmachado
josephmachado/docker_for_data_engineers

Code for blog at: https://www.startdataengineering.com/post/docker-for-de/

36
gljoseph
gljoseph/dfw-skytrack

This real-time project integrates flight information from the AviationStack API for DFW Airport and weather data from the National Weather Service API, to provide the latest arrival, departure, and fo...

29
Last 12-months (absolute gain)
ibis-project
ibis-project/ibis

the portable Python dataframe library

5,701 (+1,578)
apache-2.0
narwhals-dev
narwhals-dev/narwhals

Lightweight and extensible compatibility layer between dataframe libraries!

933 (+823)
mit
lakehq
lakehq/sail

LakeSail's computation framework with a mission to unify batch processing, stream processing, and compute-intensive (AI) workloads.

724 (+723)
apache-2.0
Nike-Inc
Nike-Inc/koheesio

Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.

635 (+634)
apache-2.0
AlexIoannides
AlexIoannides/pyspark-example-project

Implementing best practices for PySpark ETL jobs and applications.

1,894 (+509)
Last 12-months (relative gain)
josephmachado
josephmachado/efficient_data_processing_spark

Code for "Efficient Data Processing in Spark" Course

293 (+7,225%)
narwhals-dev
narwhals-dev/narwhals

Lightweight and extensible compatibility layer between dataframe libraries!

933 (+748%)
mit
gljoseph
gljoseph/dfw-skytrack

This real-time project integrates flight information from the AviationStack API for DFW Airport and weather data from the National Weather Service API, to provide the latest arrival, departure, and fo...

29 (+314%)
coder2j
coder2j/pyspark-tutorial

PySpark Tutorial for Beginners - Practical Examples in Jupyter Notebook with Spark version 3.4.1. The tutorial covers various topics like Spark Introduction, Spark Installation, Spark RDD Transformati...

109 (+289%)
mit
airscholar
airscholar/SparkingFlow

This project demonstrates how to use Apache Airflow to submit jobs to Apache spark cluster in different programming laguages using Python, Scala and Java as an example.

42 (+282%)