Statistics for topic pyspark

RepositoryStats tracks 579,129 Github repositories, of these 107 are tagged with the pyspark topic. The most common primary language for repositories using this topic is Python (47). Other languages include: Jupyter Notebook (28)

Stargazers over time for topic pyspark

Most starred repositories for topic pyspark (view more)

ibis ibis-project

595

5.3k

apache-2.0

the portable Python dataframe library

sql mssql mysql trino duckdb impala pandas polars python sqlite pyarrow pyspark bigquery database snowflake clickhouse datafusion postgresql

Created 2015-04-17

8,969 commits to main branch, last one 17 hours ago

SynapseML microsoft

831

5.1k

mit

145

Simple and Distributed Machine Learning

ai ml http onnx azure scala spark opencv pyspark synapse big-data lightgbm microsoft databricks apache-spark data-science deep-learning machine-learning model-deployment cognitive-services

Created 2017-06-05

1,641 commits to master branch, last one 22 days ago

spark-nlp JohnSnowLabs

711

3.9k

apache-2.0

100

State of the Art Natural Language Processing

llm nlp bert onnx spark pyspark llamacpp lemmatizer tensorflow transformers spell-checker entity-extraction language-detection question-answering sentiment-analysis machine-translation text-classification part-of-speech-tagger named-entity-recognition natural-language-processing

Created 2017-09-24

8,617 commits to master branch, last one 13 days ago

linkis apache

1.2k

3.3k

apache-2.0

264

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.

sql udf hive jdbc livy spark engine impala linkis presto pyspark storage rest-api scriptis jobserver hive-table thrift-server context-service resource-manager application-manager

Created 2019-07-23

4,215 commits to master branch, last one a day ago

petastorm uber

284

1.8k

apache-2.0

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, a...

sysml parquet pyarrow pyspark pytorch tensorflow deep-learning parquet-files machine-learning

Created 2018-06-15

691 commits to master branch, last one 11 months ago

awesome-spark awesome-spark

330

1.7k

cc0-1.0

A curated list of awesome Apache Spark packages and resources.

sparkr awesome pyspark apache-spark

Created 2016-02-01

263 commits to main branch, last one 13 days ago

Trending repositories for topic pyspark (view more)

Last 3 days (new repositories)

no newly created repositories trending in the last 3 days

Last 3 days (absolute gain)

josephmachado/efficient_data_processing_spark

Code for "Efficient Data Processing in Spark" Course

238 (+3)

myamafuj/hadoop-hive-spark-docker

Hadoop-Hive-Spark cluster + Jupyter on Docker

61 (+1)

apache/incubator-graphar

An open source, standard data file format for graph data storage and retrieval.

224 (+1)

apache-2.0

lakehq/sail

LakeSail's computation framework with a mission to unify stream processing, batch processing, and compute-intensive (AI) workloads.

369 (+1)

apache-2.0

HariSekhon/DevOps-Python-tools

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML...

774 (+1)

mit

Last 3 days (relative gain)

myamafuj/hadoop-hive-spark-docker

Hadoop-Hive-Spark cluster + Jupyter on Docker

61 (+2%)

josephmachado/efficient_data_processing_spark

Code for "Efficient Data Processing in Spark" Course

238 (+1%)

apache/incubator-graphar

An open source, standard data file format for graph data storage and retrieval.

224 (+0.4%)

apache-2.0

lakehq/sail

LakeSail's computation framework with a mission to unify stream processing, batch processing, and compute-intensive (AI) workloads.

369 (+0.3%)

apache-2.0

HariSekhon/DevOps-Python-tools

774 (+0.1%)

mit

Last week (new repositories)

no newly created repositories trending in the last week

Last week (absolute gain)

josephmachado/efficient_data_processing_spark

Code for "Efficient Data Processing in Spark" Course

238 (+3)

mrpowers-io/tsumugi-spark

SparkConnect Server plugin and protobuf messages for the Amazon Deequ Data Quality Engine.

26 (+1)

apache-2.0

myamafuj/hadoop-hive-spark-docker

Hadoop-Hive-Spark cluster + Jupyter on Docker

61 (+1)

apache/incubator-graphar

An open source, standard data file format for graph data storage and retrieval.

224 (+1)

apache-2.0

lakehq/sail

LakeSail's computation framework with a mission to unify stream processing, batch processing, and compute-intensive (AI) workloads.

369 (+1)

apache-2.0

Last week (relative gain)

mrpowers-io/tsumugi-spark

SparkConnect Server plugin and protobuf messages for the Amazon Deequ Data Quality Engine.

26 (+4%)

apache-2.0

myamafuj/hadoop-hive-spark-docker

Hadoop-Hive-Spark cluster + Jupyter on Docker

61 (+2%)

josephmachado/efficient_data_processing_spark

Code for "Efficient Data Processing in Spark" Course

238 (+1%)

apache/incubator-graphar

An open source, standard data file format for graph data storage and retrieval.

224 (+0.4%)

apache-2.0

lakehq/sail

LakeSail's computation framework with a mission to unify stream processing, batch processing, and compute-intensive (AI) workloads.

369 (+0.3%)

apache-2.0

Last month (new repositories)

no newly created repositories trending in the last month

Last month (absolute gain)

ibis-project/ibis

the portable Python dataframe library

5,266 (+110)

apache-2.0

databrickslabs/dbldatagen

Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POC...

357 (+43)

JohnSnowLabs/spark-nlp

State of the Art Natural Language Processing

3,862 (+34)

apache-2.0

AlexIoannides/pyspark-example-project

Implementing best practices for PySpark ETL jobs and applications.

1,681 (+23)

lakehq/sail

LakeSail's computation framework with a mission to unify stream processing, batch processing, and compute-intensive (AI) workloads.

369 (+19)

apache-2.0

Last month (relative gain)

davidzajac1/zillacode

Open Source LeetCode for PySpark, Spark, Pandas and DBT/Snowflake

97 (+21%)

apache-2.0

airscholar/SparkingFlow

This project demonstrates how to use Apache Airflow to submit jobs to Apache spark cluster in different programming laguages using Python, Scala and Java as an example.

30 (+15%)

databrickslabs/dbldatagen

357 (+14%)

myamafuj/hadoop-hive-spark-docker

Hadoop-Hive-Spark cluster + Jupyter on Docker

61 (+13%)

josephmachado/efficient_data_processing_spark

Code for "Efficient Data Processing in Spark" Course

238 (+8%)

Last 12-months (new repositories)

Nike-Inc/koheesio

Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.

599

apache-2.0

lakehq/sail

LakeSail's computation framework with a mission to unify stream processing, batch processing, and compute-intensive (AI) workloads.

369

apache-2.0

josephmachado/efficient_data_processing_spark

Code for "Efficient Data Processing in Spark" Course

238

davidzajac1/zillacode

Open Source LeetCode for PySpark, Spark, Pandas and DBT/Snowflake

apache-2.0

josephmachado/docker_for_data_engineers

Code for blog at: https://www.startdataengineering.com/post/docker-for-de/

Last 12-months (absolute gain)

ibis-project/ibis

the portable Python dataframe library

5,266 (+2,041)

apache-2.0

Nike-Inc/koheesio

Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.

599 (+598)

apache-2.0

microsoft/SynapseML

Simple and Distributed Machine Learning

5,060 (+452)

mit

AlexIoannides/pyspark-example-project

Implementing best practices for PySpark ETL jobs and applications.

1,681 (+397)

JohnSnowLabs/spark-nlp

State of the Art Natural Language Processing

3,862 (+376)

apache-2.0

Last 12-months (relative gain)

josephmachado/efficient_data_processing_spark

Code for "Efficient Data Processing in Spark" Course

238 (+5,850%)

coder2j/pyspark-tutorial

PySpark Tutorial for Beginners - Practical Examples in Jupyter Notebook with Spark version 3.4.1. The tutorial covers various topics like Spark Introduction, Spark Installation, Spark RDD Transformati...

75 (+1,150%)

mit

opentargets/gentropy

Open Targets python framework for post-GWAS analysis

33 (+560%)

apache-2.0

mitchelllisle/sparkdantic

✨ A Pydantic to PySpark schema library

55 (+450%)

mit

dogukannulu/streaming_data_processing

Create a streaming data, transfer it to Kafka, modify it with PySpark, take it to ElasticSearch and MinIO

57 (+307%)