s0rg/crawley — To efficiently extract unique URLs from web pages and a

Version	Commit	Size	Downloads	Date
latestLatest	HEAD	46.3 KB	0	6d ago

Issues

Downloads

crawley

Crawls web pages and prints any link it can find.

fast html SAX-parser (powered by x/net/html)
js/css lexical parsers (powered by tdewolff/parse) - extract api endpoints from js code and url() properties
small (below 1500 SLOC), idiomatic, more than 80% test covered codebase
grabs most of useful resources urls (pics, videos, audios, forms, etc...)
found urls are streamed to stdout and guranteed to be unique (with fragments omitted)
scan depth (limited by starting host and path, by default - 0) can be configured
can be polite - crawl rules and sitemaps from robots.txt
brute mode - scan html comments for urls (this can lead to bogus results)
make use of HTTP_PROXY / HTTPS_PROXY environment values + handles proxy auth (use HTTP_PROXY="socks5://127.0.0.1:1080/" crawley for socks5)
directory-only scan mode (aka fast-scan)