Search Results - RepositoryStats

2.3k

23.4k

apache-2.0

536

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC activ...

s3 fuse hdfs posix seaweedfs kubernetes s3-storage cloud-drive hadoop-hdfs replication blob-storage erasure-coding object-storage tiered-file-system distributed-storage distributed-systems distributed-file-system

Created 2014-07-14

11,222 commits to master branch, last one 3 days ago

BigData-Notes heibaiying

4.3k

16.1k

unknown

449

大数据入门指南 :star:

hdfs hive yarn flume hbase kafka scala spark sqoop storm hadoop azkaban bigdata phoenix big-data mapreduce zookeeper

Created 2019-03-10

607 commits to master branch, last one about a year ago

ceph ceph

6.0k

14.4k

other

657

Ceph is a distributed object, block, and file storage platform

Created 2011-09-01

149,893 commits to main branch, last one 2 days ago

juicefs juicedata

983

11.1k

apache-2.0

114

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

go s3 hdfs posix redis golang bigdata storage filesystem cloud-native object-storage distributed-systems

Created 2021-01-08

3,476 commits to main branch, last one 2 hours ago

God-Of-BigData wangzhiwubigdata

3.2k

9.9k

unknown

332

专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive...

hdfs hive flink flume hbase kafka spark hadoop azkaban bigdata zookeeper

Created 2019-02-14

405 commits to master branch, last one 2 years ago

smart_open piskvorky

382

3.2k

mit

47

Utils for streaming large files (S3, HDFS, gzip, bz2...)

s3 bz2 boto file hdfs python webhdfs streaming gzip-stream hacktoberfest streaming-data

Created 2015-01-02

1,097 commits to develop branch, last one 13 days ago

TileDB TileDB-Inc

185

1.9k

mit

69

The Universal Storage Engine

s3 hdfs arrays tiledb dataframes dense-data s3-storage sparse-data data-science data-analysis sparse-arrays storage-engine scientific-computing

Created 2017-03-31

5,031 commits to dev branch, last one 12 days ago

BigData-Interview water8394

444

1.6k

unknown

52

:dart: :star2:[大数据面试题]分享自己在网络上收集的大数据相关的面试题以及自己的答案总结.目前包含Hadoop/Hive/Spark/Flink/Hbase/Kafka/Zookeeper框架的面试题知识总结

hdfs yarn flink hbase kafka spark hadoop bigdata interview mapreduce interview-questions

Created 2019-08-12

80 commits to master branch, last one 3 years ago

bigdata-growth collabH

369

1.5k

mit

33

大数据知识仓库涉及到数据仓库建模、实时计算、大数据、数据中台、系统设计、Java、算法等。

hdfs hive hudi kudu olap flink hbase kafka spark hadoop bigdata debezium mapreduce bigdatalearning

Created 2020-06-10

1,088 commits to master branch, last one about a month ago

hdfs colinmarc

341

1.4k

mit

36

A native go client for HDFS

go hdfs commandline

Created 2014-10-08

466 commits to master branch, last one 28 days ago

Addax wgzhao

306

1.2k

apache-2.0

34

Addax is a versatile open-source ETL tool that can seamlessly transfer data between various RDBMS and NoSQL databases, making it an ideal solution for data migration.

etl hdfs hive kudu excel mysql trino hadoop impala oracle database influxdb sqlserver clickhouse postgresql

Created 2019-07-17

1,613 commits to master branch, last one 16 days ago

snakebite spotify

216

855

apache-2.0

129

A pure python HDFS client

hdfs python python-hdfs-client

This repository has been archived (exclude archived)

Created 2013-05-07

348 commits to master branch, last one 2 years ago

DevOps-Python-tools HariSekhon

339

784

mit

41

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML...

Created 2015-10-27

5,410 commits to master branch, last one about a month ago

kafka-connect-ui lensesio

130

508

other

25

Web tool for Kafka Connect |

s3 jms hdfs mqtt kafka redis twitter influxdb cassandra documentdb elasticsearch kafka-connect

Created 2016-12-02

197 commits to master branch, last one 3 years ago

CloudEon dromara

108

452

apache-2.0

14

CloudEon uses Kubernetes to install and deploy open-source big data components, enabling the containerized operation of an open-source big data platform. This allows you to reduce your focus on underl...

hdfs yarn doris hadoop bigdata kubernetes cloudnative

Created 2023-01-29

1,007 commits to dev2.0 branch, last one 12 days ago

bigdata_docker fabiogjardim

315

396

unknown

25

Big Data Ecosystem Docker

hue hdfs hive nifi hbase mongo mysql spark hadoop presto metabase zookeeper jupyter-notebook

Created 2020-03-20

82 commits to master branch, last one 2 years ago

storagetapper uber

66

343

mit

22

StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service

s3 cdc etl avro hdfs json kafka mysql msgpack clickhouse postgresql

Created 2017-05-07

250 commits to master branch, last one 2 years ago

Spark-with-Python tirthajyoti

267

341

mit

11

Fundamentals of Spark with Python (using PySpark), code examples

sql hdfs mlib spark apache hadoop python pyspark big-data database analytics dataframe map-reduce apache-spark machine-learning parallel-computing distributed-computing

Created 2018-08-20

73 commits to master branch, last one 4 years ago

juicy-bigdata datawhalechina

41

292

unknown

5

🎉🎉🐳 Datawhale大数据处理导论教程 | 大数据技术方向的开篇课程🎉🎉

hdfs hive hbase spark hadoop bigdata mapreduce

Created 2021-11-18

322 commits to master branch, last one about a year ago

bigdata-file-viewer Eugene-Mark

54

291

gpl-2.0

3

A cross-platform (Windows, MAC, Linux) desktop application to view common bigdata binary format like Parquet, ORC, AVRO, etc. Support local file system, HDFS, AWS S3, Azure Blob Storage ,etc.

orc avro hdfs bigdata parquet

Created 2020-02-05

119 commits to master branch, last one 9 months ago

hdfs mtth

103

271

mit

15

API and command line interface for HDFS

cli hdfs python

Created 2014-03-10

368 commits to master branch, last one 3 months ago

wradlib wradlib

82

270

mit

25

weather radar data processing - python package

hdfs radar python sigmet xarray netcdf4 rainbow weather wradlib hacktoberfest

Created 2016-02-19

1,627 commits to main branch, last one 11 days ago

File-System-Paper hegongshan

22

258

unknown

3

Must-read Papers for File System (FS)

ceph hdfs file-system fragmentation fault-tolerance crash-consistency data-distribution local-file-system kernel-file-system metadata-management many-core-scalability user-space-file-system distributed-file-system

Created 2021-12-01

102 commits to main branch, last one 11 days ago

rumble RumbleDB

82

219

other

27

⛈️ RumbleDB 1.22.0 "Pyrenean oak" 🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to downl...

Created 2017-09-04

6,715 commits to master branch, last one about a month ago

TileDB-Py TileDB-Inc

33

190

mit

29

Python interface to the TileDB storage engine

s3 hdfs array numpy python tiledb storage-manager

Created 2017-05-19

2,040 commits to dev branch, last one 10 days ago

elbencho breuner

25

175

gpl-3.0

16

A distributed storage benchmark for file systems, object stores & block devices with support for GPUs

s3 fio gpu ior hdfs nvme linux mdtest storage windows parallel benchmark live-stats distributed file-systems block-storage deep-learning

Created 2020-06-29

326 commits to master branch, last one 20 days ago

dcos-commons d2iq-archive

171

157

apache-2.0

90

DC/OS SDK is a collection of tools, libraries, and documentation for easy integration of technologies such as Kafka, Cassandra, HDFS, Spark, and TensorFlow with DC/OS.