SourabhSinghRana / real-time_crypto_data_pipeline_using_kafka

I am using confluent Kafka cluster to produce and consume scraped data. In this project, I've created a real-time data pipeline that utilizes Kafka to scrape, process, and load data onto S3 in JSON format. With a producer-consumer architecture, I ensure that the data is in the right format for loading onto S3 by performing minor transformations

Date Created 2023-05-02 (about a year ago)

Commits 3 (last one about a year ago)

Stargazers 29 (0 this week)

Watchers 3 (0 this week)

Forks 8

License cc0-1.0

Ranking

RepositoryStats indexes 589,134 repositories, of these SourabhSinghRana/real-time_crypto_data_pipeline_using_kafka is ranked #568,525 (3rd percentile) for total stargazers, and #424,515 for total watchers. Github reports the primary language for this repository as Python, for repositories using this language it is ranked #112,686/117,584.

SourabhSinghRana/real-time_crypto_data_pipeline_using_kafka is also tagged with popular topics, for these it's ranked: aws (#2,431/2476), data (#956/983), kafka (#818/840), data-engineering (#284/299), snowflake (#153/158)

All Topics

s3 aws ec2 iam sns data glue kafka athena s3-bucket snow-pipe snowflake eventbridge glue-crawler data-pipeline data-engineering

Star History

Github stargazers over time

Watcher History

Github watchers over time, collection started in '23

Recent Commit History

3 commits on the default branch (main) since jan '22

Yearly Commits

Commits to the default branch (main) per year

Issue History

No issues have been posted

Languages

The only known language in this repository is Python

updated: 2024-11-20 @ 11:56am, id: 635156172 / R_kgDOJdu2zA