15 results found Sort:

apache
seatunnel apache
1.9k
8.4k
apache-2.0
173
SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
cdc elt batch apache offline real-time streaming data-ingestion data-integration high-performance change-data-capture
Created 2017-08-05
4,832 commits to dev branch, last one 3 days ago
bruin-data
ingestr bruin-data
78
2.9k
mit
20
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
mssql duckdb bigquery snowflake postgresql copy-database data-pipeline data-ingestion data-integration ingestion-pipeline
Created 2024-02-12
1,037 commits to main branch, last one 4 days ago
apache
paimon apache
1.0k
2.7k
apache-2.0
74
Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
flink spark paimon big-data table-store data-ingestion streaming-datalake real-time-analytics
Created 2022-01-12
4,075 commits to master branch, last one a day ago
dashbitco
broadway dashbitco
165
2.5k
apache-2.0
44
Concurrent and multi-stage data ingestion and data processing with Elixir
elixir broadway genstage concurrent data-ingestion data-processing
Created 2018-11-05
415 commits to main branch, last one a day ago
pravega
pravega pravega
408
2.0k
apache-2.0
106
Pravega - Streaming as a new software defined storage primitive
streaming data-ingestion real-time-data streaming-data distributed-storage
Created 2016-07-11
3,297 commits to master branch, last one about a month ago
bruin-data
bruin bruin-data
33
915
apache-2.0
8
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
sql python bigquery analytics snowflake data-analysis data-modeling data-platform data-ingestion data-pipelines data-transformation
Created 2023-08-03
2,677 commits to main branch, last one a day ago
CrunchyData
pg_parquet CrunchyData
19
449
other
5
Copy to/from Parquet in S3 or Azure Blob Storage from within PostgreSQL
parquet columnar postgresql data-ingestion data-migration
Created 2024-09-04
66 commits to main branch, last one 3 days ago
orbitalapi
orbital orbitalapi
11
317
other
4
Orbital automates integration between data sources (APIs, Databases, Queues and Functions). BFF's, API Composition and ETL pipelines that adapt as your specs change.
bff etl java kotlin taxiql bff-api rest-api typescript api-gateway integration data-piplines microservices api-management data-ingestion api-integration data-engineering semantic-integration
Created 2022-09-26
6,433 commits to develop branch, last one 22 days ago
cuebook
cuelake cuebook
28
285
apache-2.0
11
Use SQL to build ELT pipelines on a data lakehouse.
elt etl sql delta upsert datalake data-lake lakehouse pipelines spark-sql apache-spark data-pipeline data-transfer apache-iceberg data-ingestion data-engineering data-integration zeppelin-notebook incremental-updates
Created 2021-03-11
481 commits to main branch, last one 3 years ago
merantix-momentum
squirrel-core merantix-momentum
8
281
apache-2.0
16
A Python library that enables ML teams to share, load, and transform data in a collaborative, flexible, and efficient way :chestnut:
ai cv ml nlp python dataops pytorch datasets internal data-mesh tensorflow distributed data-science collaboration deep-learning data-ingestion cloud-computing computer-vision machine-learning natural-language-processing
Created 2022-02-11
130 commits to main branch, last one 2 months ago
apache
paimon-rust apache
35
112
apache-2.0
12
Apache Paimon Rust The rust implementation of Apache Paimon.
rust paimon big-data table-store data-ingestion streaming-datalake real-time-analytics
Created 2024-07-05
40 commits to main branch, last one 6 months ago
thedataengineeringbook
thedataengineeringbook thedataengineeringbook
43
112
mit
2
The Data Engineering Book - หนังสือวิศวกรรมข้อมูล ของคนไทย เพื่อคนไทย
book data data-engineer data-pipeline hacktoberfest data-ingestion data-engineering data-integration data-infrastructure
Created 2021-01-07
226 commits to main branch, last one about a year ago
jgperrin
net.jgp.labs.spark jgperrin
49
101
apache-2.0
17
Apache Spark examples exclusively in Java
udf java spark dataframe ingestion data-ingestion
Created 2016-06-26
215 commits to master branch, last one 3 years ago
unbody-io
unbody unbody-io
7
69
apache-2.0
3
The modular, open-source backend for building AI-native software — powered by knowledge, not static data.
llm rag backend chatbot ai-native agentic-ai etl-pipeline generative-ai data-ingestion knowledge-base developer-tools vector-database data-enhancement supabase-alternative
Created 2025-04-02
93 commits to main branch, last one 2 days ago
merantix-momentum
squirrel-datasets-core merantix-momentum
4
42
apache-2.0
14
Squirrel dataset hub
ai cv ml npl python dataops pytorch datasets data-mesh tensorflow distributed data-science collaboration deep-learning data-ingestion cloud-computing computer-vision machine-learning natural-language-processing
This repository has been archived (exclude archived)
Created 2022-02-01
118 commits to main branch, last one about a year ago