adamdehaven / fetchurls

A bash script to spider a site, follow links, and fetch urls (with built-in filtering) into a generated text file.

Date Created 2018-01-13 (7 years ago)

Commits 50 (last one 3 years ago)

Stargazers 129 (0 this week)

Watchers 7 (0 this week)

Forks 44

License mit

Ranking

RepositoryStats indexes 622,366 repositories, of these adamdehaven/fetchurls is ranked #254,530 (59th percentile) for total stargazers, and #274,768 for total watchers. Github reports the primary language for this repository as Shell, for repositories using this language it is ranked #6,566/16,265.

adamdehaven/fetchurls is also tagged with popular topics, for these it's ranked: website (#399/1006), spider (#215/345), shell-script (#113/287)

Other Information

adamdehaven/fetchurls has Github issues enabled, there are 2 open issues and 11 closed issues.

There have been 15 releases, the latest one was published on 2021-12-26 (3 years ago) with the name wget --reject-regex.

Homepage URL: https://www.adamdehaven.com/blog/easily-crawl-a-website-and-fetch-all-urls-with-a-shell-script/

All Topics

urls wget crawl spider website shell-script bash-scripting

Star History

Github stargazers over time

Watcher History

Github watchers over time, collection started in '23

Recent Commit History

0 commits on the default branch (master) since jan '22

No recent commits to this repository

Yearly Commits

Commits to the default branch (master) per year

Issue History

Languages

The only known language in this repository is Shell

updated: 2025-02-13 @ 08:35pm, id: 117303814 / R_kgDOBv3qBg