Search Results - RepositoryStats

2.3k

23.0k

apache-2.0

535

SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, cross-DC activ...

s3 fuse hdfs posix seaweedfs kubernetes s3-storage cloud-drive hadoop-hdfs replication blob-storage erasure-coding object-storage tiered-file-system distributed-storage distributed-systems distributed-file-system

Created 2014-07-14

11,136 commits to master branch, last one 18 hours ago

BigData-Notes heibaiying

4.2k

15.9k

unknown

447

大数据入门指南 :star:

hdfs hive yarn flume hbase kafka scala spark sqoop storm hadoop azkaban bigdata phoenix big-data mapreduce zookeeper

Created 2019-03-10

607 commits to master branch, last one about a year ago

ceph ceph

6.0k

14.2k

other

657

Ceph is a distributed object, block, and file storage platform

Created 2011-09-01

149,357 commits to main branch, last one a day ago

juicefs juicedata

967

10.9k

apache-2.0

113

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

go s3 hdfs posix redis golang bigdata storage filesystem cloud-native object-storage distributed-systems

Created 2021-01-08

3,381 commits to main branch, last one a day ago

God-Of-BigData wangzhiwubigdata

3.2k

9.8k

unknown

332

专注大数据学习面试，大数据成神之路开启。Flink/Spark/Hadoop/Hbase/Hive...

hdfs hive flink flume hbase kafka spark hadoop azkaban bigdata zookeeper

Created 2019-02-14

405 commits to master branch, last one 2 years ago

smart_open piskvorky

383

3.2k

mit

47

Utils for streaming large files (S3, HDFS, gzip, bz2...)

s3 bz2 boto file hdfs python webhdfs streaming gzip-stream hacktoberfest streaming-data

Created 2015-01-02

1,091 commits to develop branch, last one 21 days ago

TileDB TileDB-Inc

185

1.9k

mit

69

The Universal Storage Engine

s3 hdfs arrays tiledb dataframes dense-data s3-storage sparse-data data-science data-analysis sparse-arrays storage-engine scientific-computing

Created 2017-03-31

5,016 commits to dev branch, last one a day ago

BigData-Interview water8394

442

1.6k

unknown

52

:dart: :star2:[大数据面试题]分享自己在网络上收集的大数据相关的面试题以及自己的答案总结.目前包含Hadoop/Hive/Spark/Flink/Hbase/Kafka/Zookeeper框架的面试题知识总结

hdfs yarn flink hbase kafka spark hadoop bigdata interview mapreduce interview-questions

Created 2019-08-12

80 commits to master branch, last one 3 years ago

bigdata-growth collabH

366

1.5k

mit

32

大数据知识仓库涉及到数据仓库建模、实时计算、大数据、数据中台、系统设计、Java、算法等。

hdfs hive hudi kudu olap flink hbase kafka spark hadoop bigdata debezium mapreduce bigdatalearning

Created 2020-06-10

1,088 commits to master branch, last one 7 days ago

hdfs colinmarc

341

1.4k

mit

37

A native go client for HDFS

go hdfs commandline

Created 2014-10-08

465 commits to master branch, last one about a year ago

Addax wgzhao

305

1.2k

apache-2.0

33

Addax is a versatile open-source ETL tool that can seamlessly transfer data between various RDBMS and NoSQL databases, making it an ideal solution for data migration.

etl hdfs hive kudu excel mysql trino hadoop impala oracle database influxdb sqlserver clickhouse postgresql

Created 2019-07-17

1,610 commits to master branch, last one 9 days ago

snakebite spotify

216

855

apache-2.0

128

A pure python HDFS client

hdfs python python-hdfs-client

This repository has been archived (exclude archived)

Created 2013-05-07

348 commits to master branch, last one 2 years ago

DevOps-Python-tools HariSekhon

341

774

mit

41

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML...

Created 2015-10-27

5,403 commits to master branch, last one about a month ago

kafka-connect-ui lensesio

129

506

other

25

Web tool for Kafka Connect |

s3 jms hdfs mqtt kafka redis twitter influxdb cassandra documentdb elasticsearch kafka-connect

Created 2016-12-02

197 commits to master branch, last one 3 years ago

CloudEon dromara

107

442

apache-2.0

14

CloudEon uses Kubernetes to install and deploy open-source big data components, enabling the containerized operation of an open-source big data platform. This allows you to reduce your focus on underl...

hdfs yarn doris hadoop bigdata kubernetes cloudnative

Created 2023-01-29

1,003 commits to dev2.0 branch, last one 20 days ago

bigdata_docker fabiogjardim

312

390

unknown

25

Big Data Ecosystem Docker

hue hdfs hive nifi hbase mongo mysql spark hadoop presto metabase zookeeper jupyter-notebook

Created 2020-03-20

82 commits to master branch, last one 2 years ago

storagetapper uber

66

342

mit

22

StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service

s3 cdc etl avro hdfs json kafka mysql msgpack clickhouse postgresql

Created 2017-05-07

250 commits to master branch, last one 2 years ago

Spark-with-Python tirthajyoti

266

335

mit

11

Fundamentals of Spark with Python (using PySpark), code examples

sql hdfs mlib spark apache hadoop python pyspark big-data database analytics dataframe map-reduce apache-spark machine-learning parallel-computing distributed-computing

Created 2018-08-20

73 commits to master branch, last one 4 years ago

bigdata-file-viewer Eugene-Mark

54

291

gpl-2.0

3

A cross-platform (Windows, MAC, Linux) desktop application to view common bigdata binary format like Parquet, ORC, AVRO, etc. Support local file system, HDFS, AWS S3, Azure Blob Storage ,etc.

orc avro hdfs bigdata parquet

Created 2020-02-05

119 commits to master branch, last one 7 months ago

juicy-bigdata datawhalechina

41

282

unknown

5

🎉🎉🐳 Datawhale大数据处理导论教程 | 大数据技术方向的开篇课程🎉🎉

hdfs hive hbase spark hadoop bigdata mapreduce

Created 2021-11-18

322 commits to master branch, last one about a year ago

hdfs mtth

102

270

mit

15

API and command line interface for HDFS

cli hdfs python

Created 2014-03-10

368 commits to master branch, last one about a month ago

wradlib wradlib

82

269

mit

25

weather radar data processing - python package

hdfs radar python sigmet xarray netcdf4 rainbow weather wradlib hacktoberfest

Created 2016-02-19

1,624 commits to main branch, last one 3 days ago

File-System-Paper hegongshan

19

214

unknown

3

Must-read Papers for File System (FS)

ceph hdfs file-system fragmentation fault-tolerance crash-consistency data-distribution local-file-system kernel-file-system metadata-management many-core-scalability user-space-file-system distributed-file-system

Created 2021-12-01

101 commits to main branch, last one about a month ago

rumble RumbleDB

82

213

other

27

⛈️ RumbleDB 1.22.0 "Pyrenean oak" 🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to downl...

Created 2017-09-04

6,715 commits to master branch, last one 20 days ago

TileDB-Py TileDB-Inc

33

190

mit

29

Python interface to the TileDB storage engine

s3 hdfs array numpy python tiledb storage-manager

Created 2017-05-19

2,027 commits to dev branch, last one a day ago

elbencho breuner

25

171

gpl-3.0

15

A distributed storage benchmark for file systems, object stores & block devices with support for GPUs

s3 fio gpu ior hdfs nvme linux mdtest storage windows parallel benchmark live-stats distributed file-systems block-storage deep-learning

Created 2020-06-29

324 commits to master branch, last one 25 days ago

dcos-commons d2iq-archive

172

157

apache-2.0

92

DC/OS SDK is a collection of tools, libraries, and documentation for easy integration of technologies such as Kafka, Cassandra, HDFS, Spark, and TensorFlow with DC/OS.