Search Results - RepositoryStats

miller johnkerl

202

8.6k

other

68

Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

Created 2015-05-03

8,836 commits to main branch, last one 20 hours ago

Bash-Oneliner onceupon

550

8.2k

mit

92

A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.

bash grep linux shell xargs system xwindow hardware terminal variables one-liners shell-oneliner data-processing oneliner-commands linux-administration

Created 2016-06-14

515 commits to master branch, last one 9 months ago

DALI NVIDIA

606

4.9k

apache-2.0

94

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

gpu mxnet paddle python pytorch deep-learning gpu-tensorflow neural-network data-processing audio-processing image-processing machine-learning data-augmentation fast-data-pipeline image-augmentation

Created 2018-06-01

3,572 commits to main branch, last one 21 hours ago

dasel TomWright

112

4.9k

mit

31

Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.

go cli xml json toml yaml query config golang parser update selector devops-tools configuration data-wrangling yaml-processor data-processing data-structures json-processing

Created 2020-09-22

697 commits to master branch, last one 15 days ago

pandera unionai-oss

283

3.1k

mit

18

A light-weight, flexible, and expressive statistical data testing library

pandas schema testing assertions data-check dataframes validation data-cleaning testing-tools data-assertions data-processing data-validation dataframe-schema pandas-dataframe pandas-validator data-verification pandas-validation hypothesis-testing

Created 2018-11-01

722 commits to main branch, last one a day ago

texar asyml

371

2.4k

apache-2.0

78

Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/

bert gpt-2 texar xlnet python text-data tensorflow casl-project deep-learning dialog-systems data-processing text-generation machine-learning machine-translation natural-language-processing

Created 2017-07-22

1,719 commits to master branch, last one 3 years ago

DialoGPT microsoft

342

2.3k

mit

55

Large-scale pretraining for dialogue

gpt-2 pytorch dialogpt dialogue text-data transformer data-processing text-generation machine-learning

Created 2019-08-29

83 commits to master branch, last one about a year ago

broadway dashbitco

153

2.3k

apache-2.0

48

Concurrent and multi-stage data ingestion and data processing with Elixir

elixir broadway genstage concurrent data-ingestion data-processing

Created 2018-11-05

392 commits to main branch, last one 8 days ago

pathway pathwaycom

87

2.3k

other

20

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

etl rust kafka python pathway dataflow real-time streaming etl-framework iot-analytics data-analytics data-pipelines data-processing batch-processing stream-processing time-series-analysis machine-learning-algorithms

Created 2022-11-27

414 commits to main branch, last one a day ago

bonobo python-bonobo

143

1.6k

apache-2.0

58

Extract Transform Load for Python 3.5+

bonobo python3 automation data-processing parallelization extract-transform-load

Created 2016-12-09

981 commits to develop branch, last one 3 years ago

data-science-on-gcp GoogleCloudPlatform

709

1.3k

apache-2.0

95

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

data-science data-analysis data-pipeline cloud-computing data-processing data-engineering machine-learning data-visualization

Created 2017-02-10

547 commits to main branch, last one about a year ago

bytewax bytewax

56

1.3k

apache-2.0

14

Python Stream Processing

rust python dataflow data-science streaming-data data-processing data-engineering machine-learning stream-processing

Created 2022-02-04

2,345 commits to main branch, last one a day ago

numaflow numaproj

98

940

apache-2.0

17

Kubernetes-native platform to run massively parallel data/streaming jobs

k8s pipeline kubernetes map-reduce hacktoberfest data-processing stream-processing

Created 2022-05-20

914 commits to main branch, last one 16 hours ago

DataflowJavaSDK GoogleCloudPlatform

326

855

unknown

127

Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.

big-data data-mining data-science data-analysis data-processing google-cloud-dataflow

This repository has been archived (exclude archived)

Created 2014-12-03

2,332 commits to master branch, last one 5 years ago

GODEL microsoft

110

841

mit

20

Large-scale pretrained models for goal-directed dialog

pytorch dialogpt dialogue text-data transformer transformers language-model data-processing text-generation dialogue-systems machine-learning pretrained-model conversational-ai language-grounding grounded-generation

Created 2022-05-10

43 commits to main branch, last one about a year ago

dolma allenai

79

811

apache-2.0

17

Data and tools for generating and inspecting OLMo pre-training data.

llm nlp data-processing machile-learning large-language-models

Created 2023-06-20

313 commits to main branch, last one 10 days ago

texar-pytorch asyml

118

744

apache-2.0

24

Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/

bert gpt-2 texar xlnet python pytorch roberta text-data casl-project deep-learning texar-pytorch dialog-systems data-processing text-generation machine-learning machine-translation natural-language-processing

Created 2019-03-08

496 commits to master branch, last one 2 years ago

hstream hstreamdb

56

693

bsd-3-clause

23

HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.

iot sql kafka scale haskell database hstreamdb real-time streaming iot-database streaming-data data-processing materialized-view realtime-database stream-processing financial-analysis streaming-database distributed-systems distributed-database

Created 2020-08-31

1,718 commits to main branch, last one 23 hours ago

xidel benibela

38

655

gpl-3.0

26

Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON docu...

Created 2015-06-11

753 commits to master branch, last one about a month ago

collapse SebKrantz

29

609

other

10

Advanced and Fast Data Transformation in R

r cran rstats weights weighted panel-data statistics time-series data-science econometrics data-analysis data-processing data-aggregation high-performance data-manipulation data-transformation scientific-computing

Created 2019-02-27

2,903 commits to master branch, last one a day ago

awesome-kafka infoslack

160

566

unknown

31

A list about Apache Kafka

kafka apache-kafka apache-spark data-pipeline kafka-streams infrastructure streaming-data data-processing stream-processing

Created 2016-04-29

82 commits to master branch, last one 3 months ago

text-dedup ChenghaoMou

66

518

apache-2.0

4

All-in-one text de-duplication

nlp de-duplication data-processing text-processing

Created 2021-03-13

363 commits to main branch, last one 10 days ago

eternal kousun12

31

512

mit

16

👾~ music, eternal ~ 👾

art glsl midi music webgl webaudio node-based 3d-graphics creative-coding data-processing

Created 2019-01-17

346 commits to master branch, last one about a year ago

amadeus constellation-rs

26

470

apache-2.0

19

Harmonious distributed data analysis in Rust.

rust data-analysis data-processing stream-processing parallel-computing distributed-computing

Created 2018-10-17

550 commits to master branch, last one 3 years ago

Machine-Learning-Notebooks maykulkarni

234

468

unknown

16

Machine Learning notebooks for refreshing concepts.

python deep-learning data-processing neural-networks machine-learning model-evaluation regression-models clustering-methods classification-trees data-science-notebook deep-learning-tutorial reinforcement-learning python-machine-learning deep-learning-algorithms dimensionality-reduction machine-learning-tutorials machine-learning-algorithms natural-language-processing

Created 2017-10-30

117 commits to master branch, last one 5 years ago

haupt polyaxon

213

452

agpl-3.0

37

Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon

Created 2016-05-15

448 commits to master branch, last one 3 days ago

nonechucks msamogh

27

374

mit

3

Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!

torch pytorch data-cleaning data-pipeline preprocessing data-processing machine-learning data-preprocessing

Created 2018-10-05

30 commits to master branch, last one 2 years ago

etl flow-php

21

340

mit

19

PHP - ETL (Extract Transform Load) data processing library

etl flow-php data-processing data-engineering

Created 2020-10-26

853 commits to 1.x branch, last one 6 days ago

fondant ml6team

25

322

apache-2.0

6

Production-ready data processing made easy and shareable

python pipeline fine-tuning data-processing machine-learning foundation-models

Created 2023-03-02

625 commits to main branch, last one 3 days ago

lithops lithops-cloud

97

307

apache-2.0

14

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀

python big-data parallel kubernetes multicloud serverless distributed object-storage cloud-computing data-processing multiprocessing big-data-analytics serverless-computing serverless-functions

Created 2018-04-23

4,279 commits to master branch, last one a day ago