Search Results - RepositoryStats

631

10.3k

mit

112

A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.

bash grep linux shell xargs system xwindow hardware terminal variables one-liners shell-oneliner data-processing oneliner-commands linux-administration

Created 2016-06-14

520 commits to master branch, last one 3 months ago

miller johnkerl

219

9.1k

other

71

Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

Created 2015-05-03

8,959 commits to main branch, last one 3 days ago

pathway pathwaycom

158

7.4k

other

29

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

etl rust kafka python pathway dataflow real-time streaming etl-framework iot-analytics data-analytics data-pipelines data-processing batch-processing stream-processing time-series-analysis machine-learning-algorithms

Created 2022-11-27

948 commits to main branch, last one 11 hours ago

dasel TomWright

140

7.3k

mit

32

Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.

go cli xml json toml yaml query config golang parser update selector devops-tools configuration data-wrangling yaml-processor data-processing data-structures json-processing

Created 2020-09-22

710 commits to master branch, last one 2 months ago

DALI NVIDIA

622

5.2k

apache-2.0

92

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

gpu mxnet paddle python pytorch deep-learning gpu-tensorflow neural-network data-processing audio-processing image-processing machine-learning data-augmentation fast-data-pipeline image-augmentation

Created 2018-06-01

3,759 commits to main branch, last one a day ago

pandera unionai-oss

316

3.5k

mit

22

A light-weight, flexible, and expressive statistical data testing library

pandas schema testing assertions data-check dataframes validation data-cleaning testing-tools data-assertions data-processing data-validation dataframe-schema pandas-dataframe pandas-validator data-verification pandas-validation hypothesis-testing

Created 2018-11-01

786 commits to main branch, last one a day ago

broadway dashbitco

161

2.4k

apache-2.0

46

Concurrent and multi-stage data ingestion and data processing with Elixir

elixir broadway genstage concurrent data-ingestion data-processing

Created 2018-11-05

404 commits to main branch, last one 17 days ago

texar asyml

374

2.4k

apache-2.0

79

Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/

bert gpt-2 texar xlnet python text-data tensorflow casl-project deep-learning dialog-systems data-processing text-generation machine-learning machine-translation natural-language-processing

Created 2017-07-22

1,719 commits to master branch, last one 4 years ago

DialoGPT microsoft

341

2.4k

mit

55

Large-scale pretraining for dialogue

gpt-2 pytorch dialogpt dialogue text-data transformer data-processing text-generation machine-learning

Created 2019-08-29

83 commits to master branch, last one 2 years ago

numaflow numaproj

120

1.8k

apache-2.0

22

Kubernetes-native platform to run massively parallel data/streaming jobs

k8s pipeline kubernetes map-reduce hacktoberfest data-processing stream-processing

Created 2022-05-20

1,177 commits to main branch, last one 2 days ago

bytewax bytewax

64

1.6k

apache-2.0

19

Python Stream Processing

rust python dataflow data-science streaming-data data-processing data-engineering machine-learning stream-processing

Created 2022-02-04

2,554 commits to main branch, last one 18 days ago

bonobo python-bonobo

146

1.6k

apache-2.0

58

Extract Transform Load for Python 3.5+

bonobo python3 automation data-processing parallelization extract-transform-load

Created 2016-12-09

981 commits to develop branch, last one 3 years ago

data-science-on-gcp GoogleCloudPlatform

713

1.3k

apache-2.0

92

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

data-science data-analysis data-pipeline cloud-computing data-processing data-engineering machine-learning data-visualization

Created 2017-02-10

547 commits to main branch, last one 2 years ago

dolma allenai

115

1.0k

apache-2.0

23

Data and tools for generating and inspecting OLMo pre-training data.

llm nlp data-processing machile-learning large-language-models

Created 2023-06-20

350 commits to main branch, last one 19 days ago

GODEL microsoft

112

860

mit

21

Large-scale pretrained models for goal-directed dialog

pytorch dialogpt dialogue text-data transformer transformers language-model data-processing text-generation dialogue-systems machine-learning pretrained-model conversational-ai language-grounding grounded-generation

Created 2022-05-10

43 commits to main branch, last one about a year ago

DataflowJavaSDK GoogleCloudPlatform

322

854

unknown

127

Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.

big-data data-mining data-science data-analysis data-processing google-cloud-dataflow

This repository has been archived (exclude archived)

Created 2014-12-03

2,332 commits to master branch, last one 6 years ago

synthBTC jofpin

585

836

mit

3

A tool that uses advanced Monte Carlo simulations and Turbit parallel processing to create possible Bitcoin prediction scenarios.

nodejs turbit bitcoin prediction synthetic-data data-processing monte-carlo-simulation

Created 2024-08-02

6 commits to main branch, last one 4 months ago

texar-pytorch asyml

117

745

apache-2.0

25

Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/

bert gpt-2 texar xlnet python pytorch roberta text-data casl-project deep-learning texar-pytorch dialog-systems data-processing text-generation machine-learning machine-translation natural-language-processing

Created 2019-03-08

496 commits to master branch, last one 2 years ago

hstream hstreamdb

55

713

bsd-3-clause

25

HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.

iot sql kafka scale haskell database hstreamdb real-time streaming iot-database streaming-data data-processing materialized-view realtime-database stream-processing financial-analysis streaming-database distributed-systems distributed-database

Created 2020-08-31

1,725 commits to main branch, last one 2 months ago

xidel benibela

42

691

gpl-3.0

27

Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON docu...

Created 2015-06-11

753 commits to master branch, last one 8 months ago

NeMo-Curator NVIDIA

91

682

apache-2.0

14

Scalable data pre processing and curation toolkit for LLMs

llm data python llmapps data-prep datarecipes fine-tuning data-quality datacuration data-curation deduplication data-processing data-preparation llm-data-quality fast-data-processing large-language-models semantic-deduplication data-processing-pipelines large-scale-data-processing

Created 2024-03-14

232 commits to main branch, last one 23 hours ago

collapse SebKrantz

35

666

other

9

Advanced and Fast Data Transformation in R

r cran rstats weights weighted panel-data statistics time-series data-science econometrics data-analysis data-processing data-aggregation high-performance data-manipulation data-transformation scientific-computing

Created 2019-02-27

3,083 commits to master branch, last one 8 days ago

text-dedup ChenghaoMou

71

635

apache-2.0

4

All-in-one text de-duplication

nlp de-duplication data-processing text-processing

Created 2021-03-13

363 commits to main branch, last one 7 months ago

awesome-kafka infoslack

163

579

unknown

31

A list about Apache Kafka

kafka apache-kafka apache-spark data-pipeline kafka-streams infrastructure streaming-data data-processing stream-processing

Created 2016-04-29

82 commits to master branch, last one 10 months ago

eternal kousun12

32

540

mit

16

👾~ music, eternal ~ 👾

art glsl midi music webgl webaudio node-based 3d-graphics creative-coding data-processing

Created 2019-01-17

346 commits to master branch, last one about a year ago

Machine-Learning-Notebooks maykulkarni

245

496

unknown

16

Machine Learning notebooks for refreshing concepts.

python deep-learning data-processing neural-networks machine-learning model-evaluation regression-models clustering-methods classification-trees data-science-notebook deep-learning-tutorial reinforcement-learning python-machine-learning deep-learning-algorithms dimensionality-reduction machine-learning-tutorials machine-learning-algorithms natural-language-processing

Created 2017-10-30

117 commits to master branch, last one 6 years ago

amadeus constellation-rs

26

474

apache-2.0

19

Harmonious distributed data analysis in Rust.

rust data-analysis data-processing stream-processing parallel-computing distributed-computing

Created 2018-10-17

550 commits to master branch, last one 3 years ago

haupt polyaxon

209

455

agpl-3.0

37

Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon

Created 2016-05-15

497 commits to master branch, last one 2 days ago

Musoq Puchaczov

19

431

mit

8

SQL Syntax without any database

csv sql csharp dotnet dotnetcore dotnet-core file-system cross-platform query-language data-processing text-processing data-exploration data-analysis-sql ai-assisted-queries plugin-architecture

Created 2018-01-29

536 commits to master branch, last one 3 days ago

nonechucks msamogh

27

378

mit

3

Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!

torch pytorch data-cleaning data-pipeline preprocessing data-processing machine-learning data-preprocessing

Created 2018-10-05

30 commits to master branch, last one 3 years ago