11 results found Sort:

apache
seatunnel apache
1.7k
7.6k
apache-2.0
171
SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
cdc elt batch apache offline real-time streaming data-ingestion data-integration high-performance change-data-capture
Created 2017-08-05
4,039 commits to dev branch, last one 2 days ago
bruin-data
ingestr bruin-data
50
2.4k
mit
15
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
mssql duckdb bigquery snowflake postgresql copy-database data-pipeline data-ingestion data-integration ingestion-pipeline
Created 2024-02-12
138 commits to main branch, last one 3 days ago
dashbitco
broadway dashbitco
155
2.3k
apache-2.0
48
Concurrent and multi-stage data ingestion and data processing with Elixir
elixir broadway genstage concurrent data-ingestion data-processing
Created 2018-11-05
395 commits to main branch, last one 8 days ago
apache
paimon apache
834
2.1k
apache-2.0
73
Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
flink spark paimon big-data table-store data-ingestion streaming-datalake real-time-analytics
Created 2022-01-12
2,721 commits to master branch, last one 9 hours ago
pravega
pravega pravega
404
2.0k
apache-2.0
107
Pravega - Streaming as a new software defined storage primitive
streaming data-ingestion real-time-data streaming-data distributed-storage
Created 2016-07-11
3,294 commits to master branch, last one 3 months ago
orbitalapi
orbital orbitalapi
10
287
other
4
Orbital automates integration between data sources (APIs, Databases, Queues and Functions). BFF's, API Composition and ETL pipelines that adapt as your specs change.
bff etl java kotlin taxiql bff-api rest-api typescript api-gateway integration data-piplines microservices api-management data-ingestion api-integration data-engineering semantic-integration
Created 2022-09-26
4,914 commits to develop branch, last one 7 months ago
cuebook
cuelake cuebook
28
283
apache-2.0
12
Use SQL to build ELT pipelines on a data lakehouse.
elt etl sql delta upsert datalake data-lake lakehouse pipelines spark-sql apache-spark data-pipeline data-transfer apache-iceberg data-ingestion data-engineering data-integration zeppelin-notebook incremental-updates
Created 2021-03-11
481 commits to main branch, last one 2 years ago
merantix-momentum
squirrel-core merantix-momentum
8
281
apache-2.0
16
A Python library that enables ML teams to share, load, and transform data in a collaborative, flexible, and efficient way :chestnut:
ai cv ml jax nlp python dataops pytorch datasets data-mesh tensorflow distributed data-science collaboration deep-learning data-ingestion cloud-computing computer-vision machine-learning natural-language-processing
Created 2022-02-11
128 commits to main branch, last one about a month ago
thedataengineeringbook
thedataengineeringbook thedataengineeringbook
44
107
mit
3
The Data Engineering Book - หนังสือวิศวกรรมข้อมูล ของคนไทย เพื่อคนไทย
book data data-engineer data-pipeline hacktoberfest data-ingestion data-engineering data-integration data-infrastructure
Created 2021-01-07
226 commits to main branch, last one 8 months ago
jgperrin
net.jgp.labs.spark jgperrin
44
94
apache-2.0
18
Apache Spark examples exclusively in Java
udf java spark dataframe ingestion data-ingestion
Created 2016-06-26
215 commits to master branch, last one 2 years ago
merantix-momentum
squirrel-datasets-core merantix-momentum
6
42
apache-2.0
14
Squirrel dataset hub
ai cv ml npl python dataops pytorch datasets data-mesh tensorflow distributed data-science collaboration deep-learning data-ingestion cloud-computing computer-vision machine-learning natural-language-processing
Created 2022-02-01
118 commits to main branch, last one about a year ago