Search Results - RepositoryStats

344

23.6k

other

47

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

etl rust kafka python pathway dataflow real-time streaming etl-framework iot-analytics data-analytics data-pipelines data-processing batch-processing stream-processing time-series-analysis machine-learning-algorithms

Created 2022-11-27

1,200 commits to main branch, last one 20 hours ago

Bash-Oneliner onceupon

626

10.4k

mit

113

A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.

bash grep linux shell xargs system xwindow hardware terminal variables one-liners shell-oneliner data-processing oneliner-commands linux-administration

Created 2016-06-14

520 commits to master branch, last one 6 months ago

miller johnkerl

222

9.2k

other

69

Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

Created 2015-05-03

8,990 commits to main branch, last one 13 hours ago

dasel TomWright

144

7.4k

mit

33

Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.

go cli xml json toml yaml query config golang parser update selector devops-tools configuration data-wrangling yaml-processor data-processing data-structures json-processing

Created 2020-09-22

710 commits to master branch, last one 5 months ago

DALI NVIDIA

629

5.3k

apache-2.0

91

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

gpu mxnet paddle python pytorch deep-learning gpu-tensorflow neural-network data-processing audio-processing image-processing machine-learning data-augmentation fast-data-pipeline image-augmentation

Created 2018-06-01

3,838 commits to main branch, last one 17 hours ago

smallpond deepseek-ai

381

4.4k

mit

46

A lightweight data processing framework built on DuckDB and 3FS.

duckdb data-processing

Created 2025-02-24

3 commits to main branch, last one 20 days ago

pandera unionai-oss

329

3.7k

mit

20

A light-weight, flexible, and expressive statistical data testing library

pandas schema testing assertions data-check dataframes validation data-cleaning testing-tools data-assertions data-processing data-validation dataframe-schema pandas-dataframe pandas-validator data-verification pandas-validation hypothesis-testing

Created 2018-11-01

832 commits to main branch, last one 10 hours ago

broadway dashbitco

163

2.5k

apache-2.0

44

Concurrent and multi-stage data ingestion and data processing with Elixir

elixir broadway genstage concurrent data-ingestion data-processing

Created 2018-11-05

413 commits to main branch, last one 7 days ago

texar asyml

372

2.4k

apache-2.0

77

Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/

bert gpt-2 texar xlnet python text-data tensorflow casl-project deep-learning dialog-systems data-processing text-generation machine-learning machine-translation natural-language-processing

Created 2017-07-22

1,719 commits to master branch, last one 4 years ago

DialoGPT microsoft

346

2.4k

mit

55

Large-scale pretraining for dialogue

gpt-2 pytorch dialogpt dialogue text-data transformer data-processing text-generation machine-learning

Created 2019-08-29

83 commits to master branch, last one 2 years ago

numaflow numaproj

129

1.8k

apache-2.0

24

Kubernetes-native platform to run massively parallel data/streaming jobs

k8s pipeline kubernetes map-reduce hacktoberfest data-processing stream-processing

Created 2022-05-20

1,258 commits to main branch, last one a day ago

bytewax bytewax

75

1.7k

apache-2.0

19

Python Stream Processing

rust python dataflow data-science streaming-data data-processing data-engineering machine-learning stream-processing

Created 2022-02-04

2,575 commits to main branch, last one 4 days ago

bonobo python-bonobo

146

1.6k

apache-2.0

57

Extract Transform Load for Python 3.5+

bonobo python3 automation data-processing parallelization extract-transform-load

Created 2016-12-09

981 commits to develop branch, last one 4 years ago

data-science-on-gcp GoogleCloudPlatform

715

1.3k

apache-2.0

91

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

data-science data-analysis data-pipeline cloud-computing data-processing data-engineering machine-learning data-visualization

Created 2017-02-10

547 commits to main branch, last one 2 years ago

pyper pyper-dev

24

1.2k

mit

2

Concurrent Python made simple

data python asyncio threading concurrency data-pipelines data-collection data-processing multiprocessing data-engineering parallel-computing

Created 2024-05-22

115 commits to main branch, last one about a month ago

dolma allenai

130

1.2k

apache-2.0

24

Data and tools for generating and inspecting OLMo pre-training data.

llm nlp data-processing machile-learning large-language-models

Created 2023-06-20

367 commits to main branch, last one 20 days ago

GODEL microsoft

112

865

mit

20

Large-scale pretrained models for goal-directed dialog

pytorch dialogpt dialogue text-data transformer transformers language-model data-processing text-generation dialogue-systems machine-learning pretrained-model conversational-ai language-grounding grounded-generation

Created 2022-05-10

43 commits to main branch, last one about a year ago

DataflowJavaSDK GoogleCloudPlatform

320

856

unknown

126

Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.

big-data data-mining data-science data-analysis data-processing google-cloud-dataflow

This repository has been archived (exclude archived)

Created 2014-12-03

2,332 commits to master branch, last one 6 years ago

NeMo-Curator NVIDIA

116

848

apache-2.0

12

Scalable data pre processing and curation toolkit for LLMs

llm data python llmapps data-prep datarecipes fine-tuning data-quality datacuration data-curation deduplication data-processing data-preparation llm-data-quality fast-data-processing large-language-models semantic-deduplication data-processing-pipelines large-scale-data-processing

Created 2024-03-14

316 commits to main branch, last one 6 hours ago

texar-pytorch asyml

115

745

apache-2.0

24

Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/

bert gpt-2 texar xlnet python pytorch roberta text-data casl-project deep-learning texar-pytorch dialog-systems data-processing text-generation machine-learning machine-translation natural-language-processing

Created 2019-03-08

496 commits to master branch, last one 2 years ago

hstream hstreamdb

55

721

bsd-3-clause

23

HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.

iot sql kafka scale haskell database hstreamdb real-time streaming iot-database streaming-data data-processing materialized-view realtime-database stream-processing financial-analysis streaming-database distributed-systems distributed-database

Created 2020-08-31

1,726 commits to main branch, last one 2 months ago

xidel benibela

43

709

gpl-3.0

28

Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON docu...

Created 2015-06-11

757 commits to master branch, last one about a month ago

synthBTC jofpin

414

683

mit

4

A tool that uses advanced Monte Carlo simulations and Turbit parallel processing to create possible Bitcoin prediction scenarios.

nodejs turbit bitcoin prediction synthetic-data data-processing monte-carlo-simulation

Created 2024-08-02

6 commits to main branch, last one 7 months ago

collapse SebKrantz

35

671

other

8

Advanced and Fast Data Transformation in R

r cran rstats weights weighted panel-data statistics time-series data-science econometrics data-analysis data-processing data-aggregation high-performance data-manipulation data-transformation scientific-computing

Created 2019-02-27

3,309 commits to master branch, last one 3 days ago

text-dedup ChenghaoMou

73

664

apache-2.0

3

All-in-one text de-duplication

nlp de-duplication data-processing text-processing

Created 2021-03-13

363 commits to main branch, last one 10 months ago

awesome-kafka infoslack

163

579

unknown

30

A list about Apache Kafka

kafka apache-kafka apache-spark data-pipeline kafka-streams infrastructure streaming-data data-processing stream-processing

Created 2016-04-29

85 commits to master branch, last one 12 days ago

eternal kousun12

33

544

mit

15

👾~ music, eternal ~ 👾

art glsl midi music webgl webaudio node-based 3d-graphics creative-coding data-processing

Created 2019-01-17

346 commits to master branch, last one 2 years ago

Machine-Learning-Notebooks maykulkarni

246

502

unknown

15

Machine Learning notebooks for refreshing concepts.

python deep-learning data-processing neural-networks machine-learning model-evaluation regression-models clustering-methods classification-trees data-science-notebook deep-learning-tutorial reinforcement-learning python-machine-learning deep-learning-algorithms dimensionality-reduction machine-learning-tutorials machine-learning-algorithms natural-language-processing

Created 2017-10-30

117 commits to master branch, last one 6 years ago

amadeus constellation-rs

26

479

apache-2.0

18

Harmonious distributed data analysis in Rust.

rust data-analysis data-processing stream-processing parallel-computing distributed-computing

Created 2018-10-17

550 commits to master branch, last one 3 years ago

Musoq Puchaczov

21

478

mit

8

SQL Syntax without any database

csv sql csharp dotnet dotnetcore dotnet-core file-system cross-platform query-language data-processing text-processing data-exploration data-analysis-sql ai-assisted-queries plugin-architecture

Created 2018-01-29

550 commits to master branch, last one 28 days ago