14 results found Sort:

apache
seatunnel apache
1.9k
8.2k
apache-2.0
175
SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
cdc elt batch apache offline real-time streaming data-ingestion data-integration high-performance change-data-capture
Created 2017-08-05
4,543 commits to dev branch, last one 2 hours ago
bruin-data
ingestr bruin-data
65
2.7k
mit
19
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
mssql duckdb bigquery snowflake postgresql copy-database data-pipeline data-ingestion data-integration ingestion-pipeline
Created 2024-02-12
614 commits to main branch, last one 19 hours ago
apache
paimon apache
988
2.5k
apache-2.0
73
Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
flink spark paimon big-data table-store data-ingestion streaming-datalake real-time-analytics
Created 2022-01-12
3,598 commits to master branch, last one 4 hours ago
dashbitco
broadway dashbitco
161
2.4k
apache-2.0
46
Concurrent and multi-stage data ingestion and data processing with Elixir
elixir broadway genstage concurrent data-ingestion data-processing
Created 2018-11-05
404 commits to main branch, last one 17 days ago
pravega
pravega pravega
408
2.0k
apache-2.0
106
Pravega - Streaming as a new software defined storage primitive
streaming data-ingestion real-time-data streaming-data distributed-storage
Created 2016-07-11
3,295 commits to master branch, last one 4 months ago
bruin-data
bruin bruin-data
19
616
apache-2.0
6
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
sql python bigquery analytics snowflake data-analysis data-modeling data-platform data-ingestion data-pipelines data-transformation
Created 2023-08-03
1,687 commits to main branch, last one 19 hours ago
CrunchyData
pg_parquet CrunchyData
12
386
other
4
Copy to/from Parquet in S3 from within PostgreSQL
parquet columnar postgresql data-ingestion data-migration
Created 2024-09-04
50 commits to main branch, last one 4 days ago
orbitalapi
orbital orbitalapi
10
300
other
5
Orbital automates integration between data sources (APIs, Databases, Queues and Functions). BFF's, API Composition and ETL pipelines that adapt as your specs change.
bff etl java kotlin taxiql bff-api rest-api typescript api-gateway integration data-piplines microservices api-management data-ingestion api-integration data-engineering semantic-integration
Created 2022-09-26
6,234 commits to develop branch, last one 19 days ago
cuebook
cuelake cuebook
28
285
apache-2.0
12
Use SQL to build ELT pipelines on a data lakehouse.
elt etl sql delta upsert datalake data-lake lakehouse pipelines spark-sql apache-spark data-pipeline data-transfer apache-iceberg data-ingestion data-engineering data-integration zeppelin-notebook incremental-updates
Created 2021-03-11
481 commits to main branch, last one 2 years ago
merantix-momentum
squirrel-core merantix-momentum
8
283
apache-2.0
16
A Python library that enables ML teams to share, load, and transform data in a collaborative, flexible, and efficient way :chestnut:
ai cv ml nlp python dataops pytorch datasets internal data-mesh tensorflow distributed data-science collaboration deep-learning data-ingestion cloud-computing computer-vision machine-learning natural-language-processing
Created 2022-02-11
129 commits to main branch, last one 2 months ago
thedataengineeringbook
thedataengineeringbook thedataengineeringbook
43
111
mit
3
The Data Engineering Book - หนังสือวิศวกรรมข้อมูล ของคนไทย เพื่อคนไทย
book data data-engineer data-pipeline hacktoberfest data-ingestion data-engineering data-integration data-infrastructure
Created 2021-01-07
226 commits to main branch, last one about a year ago
apache
paimon-rust apache
32
107
apache-2.0
14
Apache Paimon Rust The rust implementation of Apache Paimon.
rust paimon big-data table-store data-ingestion streaming-datalake real-time-analytics
Created 2024-07-05
40 commits to main branch, last one 3 months ago
jgperrin
net.jgp.labs.spark jgperrin
47
98
apache-2.0
18
Apache Spark examples exclusively in Java
udf java spark dataframe ingestion data-ingestion
Created 2016-06-26
215 commits to master branch, last one 3 years ago
merantix-momentum
squirrel-datasets-core merantix-momentum
6
42
apache-2.0
14
Squirrel dataset hub
ai cv ml npl python dataops pytorch datasets data-mesh tensorflow distributed data-science collaboration deep-learning data-ingestion cloud-computing computer-vision machine-learning natural-language-processing
This repository has been archived (exclude archived)
Created 2022-02-01
118 commits to main branch, last one about a year ago