īit_user said:Cygwin? Given that it's most of the userspace environment + components as you get on a typical Linux distro, it's really quite an easy and painless experience. However, from a practical point of view, you may find that most web servers will block this level of scraping, so you’ll need to tread carefully. Theoretically, Wget can run with an infinite depth level, meaning it’ll never stop trying to go further and deeper with the content it downloads. Depending on the recursive depth you choose, it can also download any pages that are linked to it, as well as the content on those pages, any pages that are linked on those pages, and so on. Instead of only downloading a single file, it’ll instead try to download an entire directory of related files.įor instance, if you specify a web page, it’ll download the content attached to that page (such as images). One of Wget’s most useful features is the ability to download recursively. If you run into trouble with Wget, make sure to limit the number of retries you make and set a wait limit for each download you attempt.
Make sure to use the wget -h or wget -help command to view the full list of options that are available to you.
