Statistics for topic data-pipeline

RepositoryStats tracks 518,325 Github repositories, of these 67 are tagged with the data-pipeline topic. The most common primary language for repositories using this topic is Python (30).

Stargazers over time for topic data-pipeline

Most starred repositories for topic data-pipeline (view more)

airbytehq
airbyte airbytehq
3.7k
14.3k
other
177
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
s3 elt etl data java mssql mysql python bigquery pipeline redshift snowflake postgresql self-hosted data-analysis data-pipeline data-collection data-engineering data-integration change-data-capture
Created 2020-07-27
15,737 commits to master branch, last one 12 hours ago
snowplow
snowplow snowplow
1.2k
6.8k
apache-2.0
265
The enterprise-grade behavioral data engine (web, mobile, server-side, webhooks), running cloud-natively on AWS and GCP
data snowplow analytics data-pipeline data-collection snowplow-events product-analytics snowplow-pipeline marketing-analytics
Created 2012-03-01
6,747 commits to master branch, last one about a month ago
kestra-io
kestra kestra-io
372
6.6k
apache-2.0
61
Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
elt etl data low-code pipeline workflow scheduler reverse-etl data-quality data-pipeline orchestration workflow-engine data-engineering data-integration data-orchestrator data-orchestration
Created 2019-08-24
3,069 commits to develop branch, last one 12 hours ago
apache
flink-cdc apache
1.8k
5.3k
apache-2.0
140
Flink CDC is a streaming data integration tool
cdc elt etl batch flink kafka mysql paimon real-time postgresql distributed data-pipeline data-integration schema-evolution change-data-capture
Created 2020-07-27
962 commits to master branch, last one a day ago
rudderlabs
rudder-server rudderlabs
292
3.9k
other
61
Privacy and Security focused Segment-alternative, in Golang and React
etl golang privacy bigquery redshift security snowflake rudderstack hybrid-cloud customer-data data-pipeline data-warehouse warehouse-first data-integration customer-data-lake segment-alternative data-synchronization warehouse-management customer-data-pipeline customer-data-platform
Created 2019-07-19
5,389 commits to master branch, last one 17 hours ago
superstreamlabs
memphis superstreamlabs
211
3.2k
other
33
Memphis.dev is a highly scalable and effortless data streaming platform
data golang enrichment kubernetes message-bus data-pipeline message-queue microservices data-streaming message-broker messaging-queue schema-registry data-engineering data-stream-processing
Created 2022-02-01
4,804 commits to master branch, last one 10 days ago

Trending repositories for topic data-pipeline (view more)

Last 3 days (new repositories)
no newly created repositories trending in the last 3 days
Last 3 days (absolute gain)
airbytehq
airbytehq/airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

14,270 (+32)
kestra-io
kestra-io/kestra

Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.

6,609 (+23)
apache-2.0
Multiwoven
Multiwoven/multiwoven

🔥🔥🔥 Open Source Alternative to Hightouch, Census, and RudderStack.

650 (+7)
agpl-3.0
superstreamlabs
superstreamlabs/memphis

Memphis.dev is a highly scalable and effortless data streaming platform

3,164 (+5)
feldera
feldera/feldera

Feldera Continuous Analytics Platform

269 (+2)
Last 3 days (relative gain)
josephmachado
josephmachado/efficient_data_processing_spark

Code for "Efficient Data Processing in Spark" Course

35 (+3%)
InfuseAI
InfuseAI/awesome-public-dbt-projects

A curated list of awesome public DBT projects

54 (+2%)
cc0-1.0
pracdata
pracdata/awesome-open-source-data-engineering

A curated list of open source tools used in analytical stacks and data engineering ecosystem

61 (+2%)
Multiwoven
Multiwoven/multiwoven

🔥🔥🔥 Open Source Alternative to Hightouch, Census, and RudderStack.

650 (+1%)
agpl-3.0
airscholar
airscholar/e2e-data-engineering

An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All compone...

110 (+0.9%)
Last week (new repositories)
no newly created repositories trending in the last week
Last week (absolute gain)
airbytehq
airbytehq/airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

14,270 (+66)
kestra-io
kestra-io/kestra

Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.

6,609 (+59)
apache-2.0
apache
apache/flink-cdc

Flink CDC is a streaming data integration tool

5,291 (+21)
apache-2.0
josephmachado
josephmachado/efficient_data_processing_spark

Code for "Efficient Data Processing in Spark" Course

35 (+14)
Multiwoven
Multiwoven/multiwoven

🔥🔥🔥 Open Source Alternative to Hightouch, Census, and RudderStack.

650 (+10)
agpl-3.0
Last week (relative gain)
josephmachado
josephmachado/efficient_data_processing_spark

Code for "Efficient Data Processing in Spark" Course

35 (+67%)
remyxai
remyxai/VQASynth

Compose multimodal datasets 🎹

81 (+7%)
InfuseAI
InfuseAI/awesome-public-dbt-projects

A curated list of awesome public DBT projects

54 (+6%)
cc0-1.0
pracdata
pracdata/awesome-open-source-data-engineering

A curated list of open source tools used in analytical stacks and data engineering ecosystem

61 (+5%)
behnamyazdan
behnamyazdan/ecommerce_realtime_data_pipeline

Ecommerce Realtime Data Pipeline (Data Modeling, Workflow Orchestration, Change Data Capture, Analytical Database and Dashboarding)

27 (+4%)
Last month (new repositories)
no newly created repositories trending in the last month
Last month (absolute gain)
airbytehq
airbytehq/airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

14,270 (+433)
kestra-io
kestra-io/kestra

Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.

6,609 (+366)
apache-2.0
apache
apache/flink-cdc

Flink CDC is a streaming data integration tool

5,291 (+71)
apache-2.0
bruin-data
bruin-data/ingestr

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

2,343 (+53)
mit
Multiwoven
Multiwoven/multiwoven

🔥🔥🔥 Open Source Alternative to Hightouch, Census, and RudderStack.

650 (+36)
agpl-3.0
Last month (relative gain)
josephmachado
josephmachado/efficient_data_processing_spark

Code for "Efficient Data Processing in Spark" Course

35 (+775%)
pracdata
pracdata/awesome-open-source-data-engineering

A curated list of open source tools used in analytical stacks and data engineering ecosystem

61 (+30%)
airscholar
airscholar/e2e-data-engineering

An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All compone...

110 (+21%)
behnamyazdan
behnamyazdan/ecommerce_realtime_data_pipeline

Ecommerce Realtime Data Pipeline (Data Modeling, Workflow Orchestration, Change Data Capture, Analytical Database and Dashboarding)

27 (+17%)
remyxai
remyxai/VQASynth

Compose multimodal datasets 🎹

81 (+14%)
Last 12-months (new repositories)
bruin-data
bruin-data/ingestr

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

2,343
mit
Multiwoven
Multiwoven/multiwoven

🔥🔥🔥 Open Source Alternative to Hightouch, Census, and RudderStack.

650
agpl-3.0
dataflint
dataflint/spark

Performance Observability for Apache Spark

127
apache-2.0
airscholar
airscholar/e2e-data-engineering

An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All compone...

110
remyxai
remyxai/VQASynth

Compose multimodal datasets 🎹

81
Last 12-months (absolute gain)
airbytehq
airbytehq/airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

14,270 (+3,658)
kestra-io
kestra-io/kestra

Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.

6,609 (+3,245)
apache-2.0
bruin-data
bruin-data/ingestr

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

2,343 (+2,342)
mit
apache
apache/flink-cdc

Flink CDC is a streaming data integration tool

5,291 (+1,335)
apache-2.0
Multiwoven
Multiwoven/multiwoven

🔥🔥🔥 Open Source Alternative to Hightouch, Census, and RudderStack.

650 (+647)
agpl-3.0
Last 12-months (relative gain)
feldera
feldera/feldera

Feldera Continuous Analytics Platform

269 (+5,280%)
dataflint
dataflint/spark

Performance Observability for Apache Spark

127 (+2,440%)
apache-2.0
jvalue
jvalue/jayvee

Jayvee is a domain-specific language and runtime for automated processing of data pipelines

76 (+1,420%)
josephmachado
josephmachado/efficient_data_processing_spark

Code for "Efficient Data Processing in Spark" Course

35 (+775%)
digitalghost-dev
digitalghost-dev/premier-league

A Data Engineering project. Repository for backend infrastructure and Streamlit app files for a Premier League Dashboard.

155 (+761%)