Trending repositories for topic etl
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
An orchestration platform for the development, production, and observation of data assets.
Apache Doris is an easy-to-use, high performance and unified analytics database.
Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming ...
Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.
Ape Data Transfer Suite, written in Rust. Provides ultra-fast data replication between MySQL, PostgreSQL, Redis, MongoDB, Kafka and ClickHouse, ideal for disaster recovery (DR) and migration scenarios...
🍁 Sycamore is an LLM-powered search and analytics platform for unstructured data.
Actively curated list of awesome BI tools. PRs welcome!
Efficient data transformation and modeling framework that is backwards compatible with dbt.
🔮 Instill Core is a full-stack AI infrastructure tool for data, model and pipeline orchestration, designed to streamline every aspect of building versatile AI-first applications
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Ape Data Transfer Suite, written in Rust. Provides ultra-fast data replication between MySQL, PostgreSQL, Redis, MongoDB, Kafka and ClickHouse, ideal for disaster recovery (DR) and migration scenarios...
🍁 Sycamore is an LLM-powered search and analytics platform for unstructured data.
Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
Fast and efficient unstructured data extraction. Written in Rust with bindings for many languages.
A curated list with resources about node-based UIs
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
Actively curated list of awesome BI tools. PRs welcome!
Dataform is a framework for managing SQL based data operations in BigQuery
Efficient data transformation and modeling framework that is backwards compatible with dbt.
🔮 Instill Core is a full-stack AI infrastructure tool for data, model and pipeline orchestration, designed to streamline every aspect of building versatile AI-first applications
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
An orchestration platform for the development, production, and observation of data assets.
Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming ...
Apache Doris is an easy-to-use, high performance and unified analytics database.
Ape Data Transfer Suite, written in Rust. Provides ultra-fast data replication between MySQL, PostgreSQL, Redis, MongoDB, Kafka and ClickHouse, ideal for disaster recovery (DR) and migration scenarios...
Open source data anonymization and synthetic data orchestration for developers. Create high fidelity synthetic data and sync it across your environments.
Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
🧙 Build, run, and manage data pipelines for integrating and transforming data.
Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.
Efficient data transformation and modeling framework that is backwards compatible with dbt.
Fast and efficient unstructured data extraction. Written in Rust with bindings for many languages.
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Ape Data Transfer Suite, written in Rust. Provides ultra-fast data replication between MySQL, PostgreSQL, Redis, MongoDB, Kafka and ClickHouse, ideal for disaster recovery (DR) and migration scenarios...
A curated collection of AI, data engineering, and DevOps projects featuring real-world applications, advanced techniques, and tutorials—ideal for learners and practitioners exploring data science and ...
Fastest open-source tool for replicating Databases to Apache Iceberg or Data Lakehouse. ⚡ Efficient, quick and scalable data ingestion for real-time analytics. Starting with MongoDB
Declarative text based tool for data analysts and engineers to extract, load, transform and orchestrate their data pipelines.
基于python和llm大模型开发的数据处理和任务调度系统。 支持数据源管理，数据模型管理，数据集成，数据查询API接口，低代码自定义数据处理任务模版，单任务及dag任务工作流调度等功能。集成了llm模块实现rag知识库问答，链接各数据源数据进行数据对话问答，交互式数据分析功能。
Context-aware structured outputs. Search your documents or the web for specific data and get it back in JSON or Markdown.
Fast and efficient unstructured data extraction. Written in Rust with bindings for many languages.
Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
Realtime sync data from MySQL/PostgreSQL/MongoDB to Meilisearch
A compute framework for building Search, RAG, Recommendations and Analytics over complex structured & unstructured data.
Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
An orchestration platform for the development, production, and observation of data assets.
Fast and efficient unstructured data extraction. Written in Rust with bindings for many languages.
Ape Data Transfer Suite, written in Rust. Provides ultra-fast data replication between MySQL, PostgreSQL, Redis, MongoDB, Kafka and ClickHouse, ideal for disaster recovery (DR) and migration scenarios...
Open source data anonymization and synthetic data orchestration for developers. Create high fidelity synthetic data and sync it across your environments.
Apache Doris is an easy-to-use, high performance and unified analytics database.
Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming ...
Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.
A compute framework for building Search, RAG, Recommendations and Analytics over complex structured & unstructured data.
Efficient data transformation and modeling framework that is backwards compatible with dbt.
Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
A curated list with resources about node-based UIs
Python Client and Toolkit for DataFrames, Big Data, Machine Learning and ETL in Elasticsearch
Ape Data Transfer Suite, written in Rust. Provides ultra-fast data replication between MySQL, PostgreSQL, Redis, MongoDB, Kafka and ClickHouse, ideal for disaster recovery (DR) and migration scenarios...
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Fastest open-source tool for replicating Databases to Apache Iceberg or Data Lakehouse. ⚡ Efficient, quick and scalable data ingestion for real-time analytics. Starting with MongoDB
Fast and efficient unstructured data extraction. Written in Rust with bindings for many languages.
A curated collection of AI, data engineering, and DevOps projects featuring real-world applications, advanced techniques, and tutorials—ideal for learners and practitioners exploring data science and ...
Arcane Insight is a data analytics project designed to harness the power of SQLMesh & DuckDB to collect, transform, and analyze data from Blizzard’s Hearthstone API. Focused on card statistics and att...
Arquitetura CRM de Baixo Custo com Gen AI, projetada para startups que precisam processar e analisar dados de vendas de forma eficiente.
Declarative text based tool for data analysts and engineers to extract, load, transform and orchestrate their data pipelines.
A compute framework for building Search, RAG, Recommendations and Analytics over complex structured & unstructured data.
基于python和llm大模型开发的数据处理和任务调度系统。 支持数据源管理，数据模型管理，数据集成，数据查询API接口，低代码自定义数据处理任务模版，单任务及dag任务工作流调度等功能。集成了llm模块实现rag知识库问答，链接各数据源数据进行数据对话问答，交互式数据分析功能。
Context-aware structured outputs. Search your documents or the web for specific data and get it back in JSON or Markdown.
This Node-RED custom node lets you consume google sheets spreadsheets on demand.
Use SQL to instantly query Kubernetes API resources. Open source CLI. No DB required.
end-to-end data engineering project to get insights from PyPi using python, duckdb, MotherDuck & Evidence
Steampipe Plugin SDK is a simple abstraction layer to write a Steampipe plugin. Plugins automatically work across all engine types including the Steampipe CLI, Postgres FDW, SQLite extension and the e...
Visual Data Transformation with Python Code Generation. Low-Code Python-based ETL.
Fast and efficient unstructured data extraction. Written in Rust with bindings for many languages.
Radient turns many data types (not just text) into vectors for similarity search, RAG, regression analysis, and more.
Decentralized, account-centric programmable indexing network for web3. Supports blockchain explorers, on-chain portfolios, social graphs, and ZK coprocessors. Open-source EVM-compatible indexer availa...
end-to-end data engineering project to get insights from PyPi using python, duckdb, MotherDuck & Evidence
A curated list of open source tools used in analytics platforms and data engineering ecosystem
Context-aware structured outputs. Search your documents or the web for specific data and get it back in JSON or Markdown.
Build super simple end-to-end data & ETL pipelines for your vector databases and Generative AI applications
Ylem is an open-source platform for real-time data streaming orchestration
Arquitetura CRM de Baixo Custo com Gen AI, projetada para startups que precisam processar e analisar dados de vendas de forma eficiente.
DataForge helps data teams write functional transformation pipelines by leveraging software engineering principles
Fastest open-source tool for replicating Databases to Apache Iceberg or Data Lakehouse. ⚡ Efficient, quick and scalable data ingestion for real-time analytics. Starting with MongoDB
Arcane Insight is a data analytics project designed to harness the power of SQLMesh & DuckDB to collect, transform, and analyze data from Blizzard’s Hearthstone API. Focused on card statistics and att...
This Node-RED custom node lets you consume google sheets spreadsheets on demand.
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
Open source data anonymization and synthetic data orchestration for developers. Create high fidelity synthetic data and sync it across your environments.
An orchestration platform for the development, production, and observation of data assets.
Apache Doris is an easy-to-use, high performance and unified analytics database.
🧙 Build, run, and manage data pipelines for integrating and transforming data.
Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming ...
🔥🔥🔥 Open Source Alternative to Hightouch, Census, and RudderStack - Reverse ETL & Data Activation
Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.
Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage
Efficient data transformation and modeling framework that is backwards compatible with dbt.
Visual Data Transformation with Python Code Generation. Low-Code Python-based ETL.
A curated list with resources about node-based UIs
Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
A compute framework for building Search, RAG, Recommendations and Analytics over complex structured & unstructured data.
🔮 Instill Core is a full-stack AI infrastructure tool for data, model and pipeline orchestration, designed to streamline every aspect of building versatile AI-first applications
A compute framework for building Search, RAG, Recommendations and Analytics over complex structured & unstructured data.
Context-aware structured outputs. Search your documents or the web for specific data and get it back in JSON or Markdown.
A curated list of open source tools used in analytics platforms and data engineering ecosystem
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
Open source data anonymization and synthetic data orchestration for developers. Create high fidelity synthetic data and sync it across your environments.
基于python和llm大模型开发的数据处理和任务调度系统。 支持数据源管理，数据模型管理，数据集成，数据查询API接口，低代码自定义数据处理任务模版，单任务及dag任务工作流调度等功能。集成了llm模块实现rag知识库问答，链接各数据源数据进行数据对话问答，交互式数据分析功能。
Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.
🍁 Sycamore is an LLM-powered search and analytics platform for unstructured data.
This is a repository to demonstrate my details, skills, projects and to keep track of my progression in Data Analytics and Data Science topics.
This Node-RED custom node lets you consume google sheets spreadsheets on demand.
A curated collection of AI, data engineering, and DevOps projects featuring real-world applications, advanced techniques, and tutorials—ideal for learners and practitioners exploring data science and ...
Data Engineering Pilipinas is a community for data engineers, data analysts, data scientists, developers, AI / ML engineers, and users of closed and open source data tools and methods / techniques in ...
Built a real-time streaming pipeline to extract stock data, using Apache Nifi, Debezium, Kafka, and Spark Streaming. Loaded the transformed data into Glue database and created real-time dashboards usi...