README
crawley
Crawls web pages and prints any link it can find.
features
- fast html SAX-parser (powered by x/net/html)
- js/css lexical parsers (powered by tdewolff/parse) - extract api endpoints from js code and
url()properties - small (below 1500 SLOC), idiomatic, more than 80% test covered codebase
- grabs most of useful resources urls (pics, videos, audios, forms, etc...)
- found urls are streamed to stdout and guranteed to be unique (with fragments omitted)
- scan depth (limited by starting host and path, by default - 0) can be configured
- can be polite - crawl rules and sitemaps from
robots.txt brutemode - scan html comments for urls (this can lead to bogus results)- make use of
HTTP_PROXY/HTTPS_PROXYenvironment values + handles proxy auth (useHTTP_PROXY="socks5://127.0.0.1:1080/" crawleyfor socks5) - directory-only scan mode (aka
fast-scan)