Statistics for topic pyspark

RepositoryStats tracks 518,991 Github repositories, of these 97 are tagged with the pyspark topic. The most common primary language for repositories using this topic is Python (43). Other languages include: Jupyter Notebook (25)

Stargazers over time for topic pyspark

Most starred repositories for topic pyspark (view more)

SynapseML microsoft

817

5.0k

mit

146

Simple and Distributed Machine Learning

ai ml http onnx azure scala spark opencv pyspark synapse big-data lightgbm microsoft databricks apache-spark data-science deep-learning machine-learning model-deployment cognitive-services

Created 2017-06-05

1,602 commits to master branch, last one a day ago

ibis ibis-project

537

4.3k

apache-2.0

the portable Python dataframe library

sql dask mssql mysql trino duckdb impala pandas polars python sqlite pyarrow pyspark bigquery database snowflake clickhouse datafusion postgresql sqlalchemy

Created 2015-04-17

8,119 commits to main branch, last one a day ago

spark-nlp JohnSnowLabs

701

3.7k

apache-2.0

100

State of the Art Natural Language Processing

llm nlp bert spark albert pyspark lemmatizer tensorflow transformers spell-checker language-model entity-extraction language-detection question-answering sentiment-analysis machine-translation text-classification part-of-speech-tagger named-entity-recognition natural-language-processing

Created 2017-09-24

8,512 commits to master branch, last one 14 days ago

linkis apache

1.1k

3.2k

apache-2.0

262

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.

sql udf hive jdbc livy spark engine impala linkis presto pyspark storage rest-api scriptis jobserver hive-table thrift-server context-service resource-manager application-manager

Created 2019-07-23

4,152 commits to master branch, last one 2 days ago

petastorm uber

281

1.8k

apache-2.0

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, a...

sysml parquet pyarrow pyspark pytorch tensorflow deep-learning parquet-files machine-learning

Created 2018-06-15

691 commits to master branch, last one 5 months ago

awesome-spark awesome-spark

323

1.6k

cc0-1.0

A curated list of awesome Apache Spark packages and resources.

sparkr awesome pyspark apache-spark

Created 2016-02-01

259 commits to main branch, last one about a year ago

Trending repositories for topic pyspark (view more)

Last 3 days (new repositories)

no newly created repositories trending in the last 3 days

Last 3 days (absolute gain)

ibis-project/ibis

the portable Python dataframe library

4,292 (+12)

apache-2.0

microsoft/SynapseML

Simple and Distributed Machine Learning

4,981 (+4)

mit

awesome-spark/awesome-spark

A curated list of awesome Apache Spark packages and resources.

1,626 (+3)

cc0-1.0

JohnSnowLabs/spark-nlp

State of the Art Natural Language Processing

3,713 (+3)

apache-2.0

kevinschaich/pyspark-cheatsheet

🐍 Quick reference guide to common patterns & functions in PySpark.

349 (+2)

mit

Last 3 days (relative gain)

josephmachado/efficient_data_processing_spark

Code for "Efficient Data Processing in Spark" Course

35 (+3%)

sb-ai-lab/RePlay

A Comprehensive Framework for Building End-to-End Recommendation Systems with State-of-the-Art Models

126 (+0.8%)

apache-2.0

josephmachado/data_engineering_best_practices

Sample project to demonstrate data engineering best practices

136 (+0.7%)

kevinschaich/pyspark-cheatsheet

🐍 Quick reference guide to common patterns & functions in PySpark.

349 (+0.6%)

mit

apache/incubator-graphar

An open source, standard data file format for graph data storage and retrieval.

179 (+0.6%)

apache-2.0

Last week (new repositories)

no newly created repositories trending in the last week

Last week (absolute gain)

ibis-project/ibis

the portable Python dataframe library

4,292 (+25)

apache-2.0

josephmachado/efficient_data_processing_spark

Code for "Efficient Data Processing in Spark" Course

35 (+14)

JohnSnowLabs/spark-nlp

State of the Art Natural Language Processing

3,713 (+8)

apache-2.0

awesome-spark/awesome-spark

A curated list of awesome Apache Spark packages and resources.

1,626 (+5)

cc0-1.0

microsoft/SynapseML

Simple and Distributed Machine Learning

4,981 (+5)

mit

Last week (relative gain)

josephmachado/efficient_data_processing_spark

Code for "Efficient Data Processing in Spark" Course

35 (+67%)

mrugankray/Big-Data-Cluster

The goal of this project is to build a docker cluster that gives access to Hadoop, HDFS, Hive, PySpark, Sqoop, Airflow, Kafka, Flume, Postgres, Cassandra, Hue, Zeppelin, Kadmin, Kafka Control Center ...

41 (+3%)

mit

josephmachado/data_engineering_best_practices

Sample project to demonstrate data engineering best practices

136 (+2%)

sb-ai-lab/RePlay

A Comprehensive Framework for Building End-to-End Recommendation Systems with State-of-the-Art Models

126 (+2%)

apache-2.0

apache/incubator-graphar

An open source, standard data file format for graph data storage and retrieval.

179 (+1%)

apache-2.0

Last month (new repositories)

no newly created repositories trending in the last month

Last month (absolute gain)

ibis-project/ibis

the portable Python dataframe library

4,292 (+234)

apache-2.0

JohnSnowLabs/spark-nlp

State of the Art Natural Language Processing

3,713 (+46)

apache-2.0

josephmachado/efficient_data_processing_spark

Code for "Efficient Data Processing in Spark" Course

35 (+31)

ptyadana/SQL-Data-Analysis-and-Visualization-Projects

SQL data analysis & visualization projects using MySQL, PostgreSQL, SQLite, Tableau, Apache Spark and pySpark.

1,089 (+29)

mit

AlexIoannides/pyspark-example-project

Implementing best practices for PySpark ETL jobs and applications.

1,404 (+22)

Last month (relative gain)

josephmachado/efficient_data_processing_spark

Code for "Efficient Data Processing in Spark" Course

35 (+775%)

Balajirvp/DE-Zoomcamp

Code/Notes for the Data Engineering Zoomcamp by DataTalksClub

25 (+19%)

apache-2.0

coder2j/pyspark-tutorial

PySpark Tutorial for Beginners - Practical Examples in Jupyter Notebook with Spark version 3.4.1. The tutorial covers various topics like Spark Introduction, Spark Installation, Spark RDD Transformati...

31 (+11%)

mit