golang-collection / go-crawler-distributed

分布式爬虫项目，本项目支持个性化定制页面解析器二次开发，项目整体采用微服务架构，通过消息队列实现消息的异步发送，使用到的框架包括：redigo, gorm, goquery, easyjson, viper, amqp, zap, go-micro，并通过Docker实现容器化部署，中间爬虫节点支持水平拓展。

Date Created 2020-08-17 (4 years ago)

Commits 129 (last one about a year ago)

Stargazers 50 (0 this week)

Watchers 5 (0 this week)

Forks 7

License mit

Ranking

RepositoryStats indexes 624,393 repositories, of these golang-collection/go-crawler-distributed is ranked #486,218 (22nd percentile) for total stargazers, and #321,898 for total watchers. Github reports the primary language for this repository as Go, for repositories using this language it is ranked #22,255/27,323.

golang-collection/go-crawler-distributed is also tagged with popular topics, for these it's ranked: docker (#5,470/6541), go (#5,303/6124), elasticsearch (#699/766), microservice (#546/633), crawler (#519/590), rabbitmq (#411/471)

Other Information

golang-collection/go-crawler-distributed has 2 open pull requests on Github, 5 pull requests have been merged over the lifetime of the repository.

All Topics

go docker crawler go-micro rabbitmq gocrawler microservice elasticsearch

Star History

Github stargazers over time

Watcher History

Github watchers over time, collection started in '23

Recent Commit History

11 commits on the default branch (master) since jan '22

Yearly Commits

Commits to the default branch (master) per year

Issue History

No issues have been posted

Languages

The primary language is Go but there's also others...

updated: 2025-03-07 @ 06:46am, id: 288061828 / R_kgDOESt5hA