Trending repositories for topic pyspark
🐍 Quick reference guide to common patterns & functions in PySpark.
Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!
Implementing best practices for PySpark ETL jobs and applications.
Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, a...
Code for "Efficient Data Processing in Spark" Course
Open Source LeetCode for PySpark, Spark, Pandas and DBT/Snowflake
A curated list of awesome Apache Spark packages and resources.
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
🐍 Quick reference guide to common patterns & functions in PySpark.
Open Source LeetCode for PySpark, Spark, Pandas and DBT/Snowflake
Code for "Efficient Data Processing in Spark" Course
Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!
Implementing best practices for PySpark ETL jobs and applications.
Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, a...
A curated list of awesome Apache Spark packages and resources.
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
Implementing best practices for PySpark ETL jobs and applications.
Open Source LeetCode for PySpark, Spark, Pandas and DBT/Snowflake
🐍 Quick reference guide to common patterns & functions in PySpark.
Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!
A curated list of awesome Apache Spark packages and resources.
Code for "Efficient Data Processing in Spark" Course
PySpark Tutorial for Beginners - Practical Examples in Jupyter Notebook with Spark version 3.4.1. The tutorial covers various topics like Spark Introduction, Spark Installation, Spark RDD Transformati...
Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POC...
LakeSail's computation framework with a mission to unify stream processing, batch processing, and compute-intensive (AI) workloads.
Hopsworks - Data-Intensive AI platform with a Feature Store
A simple VS Code devcontainer setup for local PySpark development
This project was a joint effort by Lucas De Oliveira, Chandrish Ambati, and Anish Mukherjee to create a song and playlist embeddings for recommendations in a distributed fashion using a 1M playlist da...
Possibly the fastest DataFrame-agnostic quality check library in town.
An open source, standard data file format for graph data storage and retrieval.
Open Source LeetCode for PySpark, Spark, Pandas and DBT/Snowflake
A simple VS Code devcontainer setup for local PySpark development
PySpark Tutorial for Beginners - Practical Examples in Jupyter Notebook with Spark version 3.4.1. The tutorial covers various topics like Spark Introduction, Spark Installation, Spark RDD Transformati...
Code for "Efficient Data Processing in Spark" Course
🐍 Quick reference guide to common patterns & functions in PySpark.
Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!
This project was a joint effort by Lucas De Oliveira, Chandrish Ambati, and Anish Mukherjee to create a song and playlist embeddings for recommendations in a distributed fashion using a 1M playlist da...
Possibly the fastest DataFrame-agnostic quality check library in town.
Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POC...
An open source, standard data file format for graph data storage and retrieval.
Implementing best practices for PySpark ETL jobs and applications.
LakeSail's computation framework with a mission to unify stream processing, batch processing, and compute-intensive (AI) workloads.
A curated list of awesome Apache Spark packages and resources.
Hopsworks - Data-Intensive AI platform with a Feature Store
LakeSail's computation framework with a mission to unify stream processing, batch processing, and compute-intensive (AI) workloads.
Implementing best practices for PySpark ETL jobs and applications.
A Comprehensive Framework for Building End-to-End Recommendation Systems with State-of-the-Art Models
Open Source LeetCode for PySpark, Spark, Pandas and DBT/Snowflake
SQL data analysis & visualization projects using MySQL, PostgreSQL, SQLite, Tableau, Apache Spark and pySpark.
A curated list of awesome Apache Spark packages and resources.
Code for "Efficient Data Processing in Spark" Course
Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POC...
🐍 Quick reference guide to common patterns & functions in PySpark.
Hopsworks - Data-Intensive AI platform with a Feature Store
Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!
Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.
80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML...
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
LakeSail's computation framework with a mission to unify stream processing, batch processing, and compute-intensive (AI) workloads.
Open Source LeetCode for PySpark, Spark, Pandas and DBT/Snowflake
This project introduces PySpark, a powerful open-source framework for distributed data processing. We explore its architecture, components, and applications for real-time data analysis.
A Comprehensive Framework for Building End-to-End Recommendation Systems with State-of-the-Art Models
A simple VS Code devcontainer setup for local PySpark development
Code for "Efficient Data Processing in Spark" Course
PySpark Tutorial for Beginners - Practical Examples in Jupyter Notebook with Spark version 3.4.1. The tutorial covers various topics like Spark Introduction, Spark Installation, Spark RDD Transformati...
The goal of this project is to build a docker cluster that gives access to Hadoop, HDFS, Hive, PySpark, Sqoop, Airflow, Kafka, Flume, Postgres, Cassandra, Hue, Zeppelin, Kadmin, Kafka Control Center ...
Possibly the fastest DataFrame-agnostic quality check library in town.
Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POC...
This project was a joint effort by Lucas De Oliveira, Chandrish Ambati, and Anish Mukherjee to create a song and playlist embeddings for recommendations in a distributed fashion using a 1M playlist da...
A library that provides useful extensions to Apache Spark and PySpark.
🐍 Quick reference guide to common patterns & functions in PySpark.
Code repository for the "PySpark in Action" book
This repo collects the open-source work of the Analytics Service within NHS Digital Data Services
This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We wil...
Data Engineering examples for Airflow, Prefect, and Mage.ai; dbt for BigQuery, Redshift, ClickHouse, PostgreSQL; Spark/PySpark for Batch processing; and Kafka for Stream processing
Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.
Open Source LeetCode for PySpark, Spark, Pandas and DBT/Snowflake
Code for blog at: https://www.startdataengineering.com/post/docker-for-de/
SparkConnect Server plugin and protobuf messages for the Amazon Deequ Data Quality Engine.
Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.
LakeSail's computation framework with a mission to unify stream processing, batch processing, and compute-intensive (AI) workloads.
Implementing best practices for PySpark ETL jobs and applications.
SQL data analysis & visualization projects using MySQL, PostgreSQL, SQLite, Tableau, Apache Spark and pySpark.
Code for "Efficient Data Processing in Spark" Course
A Comprehensive Framework for Building End-to-End Recommendation Systems with State-of-the-Art Models
A curated list of awesome Apache Spark packages and resources.
🐍 Quick reference guide to common patterns & functions in PySpark.
Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!
Hopsworks - Data-Intensive AI platform with a Feature Store
Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POC...
Possibly the fastest DataFrame-agnostic quality check library in town.
Open Source LeetCode for PySpark, Spark, Pandas and DBT/Snowflake
PySpark Cheat Sheet - example code to help you learn PySpark and develop apps faster
PySpark-Tutorial provides basic algorithms using PySpark
Code for "Efficient Data Processing in Spark" Course
PySpark Tutorial for Beginners - Practical Examples in Jupyter Notebook with Spark version 3.4.1. The tutorial covers various topics like Spark Introduction, Spark Installation, Spark RDD Transformati...
This project introduces PySpark, a powerful open-source framework for distributed data processing. We explore its architecture, components, and applications for real-time data analysis.
Possibly the fastest DataFrame-agnostic quality check library in town.
A simple VS Code devcontainer setup for local PySpark development
Data Engineering examples for Airflow, Prefect, and Mage.ai; dbt for BigQuery, Redshift, ClickHouse, PostgreSQL; Spark/PySpark for Batch processing; and Kafka for Stream processing
A Comprehensive Framework for Building End-to-End Recommendation Systems with State-of-the-Art Models
Sample project to demonstrate data engineering best practices
Code/Notes for the Data Engineering Zoomcamp by DataTalksClub
The goal of this project is to build a docker cluster that gives access to Hadoop, HDFS, Hive, PySpark, Sqoop, Airflow, Kafka, Flume, Postgres, Cassandra, Hue, Zeppelin, Kadmin, Kafka Control Center ...
Create a streaming data, transfer it to Kafka, modify it with PySpark, take it to ElasticSearch and MinIO
An open source, standard data file format for graph data storage and retrieval.
Sparglim✨ makes PySpark App Configurable and Deploy Spark Connect Server Easier!
Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POC...
A Docker Compose template that builds a interactive development environment for PySpark with Jupyter Lab, MinIO as object storage, Hive Metastore, Trino and Kafka
《 Python机器学习及实践：从零开始通往Kaggle竞赛之路（2022年度版）》全书数据和开源代码