Number9 - blog

Wget is a wonderful little utility from the Unix world used for retrieving files from web server. Using Wget is similar to having your own personal web crawler. Give it a few options and it’s off analyzing html files, extracting links in the html page and downloading them in sequence until all links has been downloaded. Of course, you control it all; how deep it crawls; what to download and what not to download.

It has loads of features including:

• Recursive download and ability to set how the traversing depth.
• Resume. Continue where you left off.
• Converting links in html files for offline viewing.
• Support HTTP, HTTPS, and FTP protocols.
• Support Proxy
• Work in the background
• Options to set number of retries
• Option to only download newer files
• Option to limit how much you download
• IPv6 Support

Download:

• Wget for Windows 1.5.3.1 [Direct Link]
• CocoaWget 2.6.3 UB for Mac OS X 10.4+

#
# Download everything from Gooogle.com ^_^
#
wget -np -r -m http:\\www.google.com\

-np don’t go backward and download content from the parent directory. -r recursive; go and download all sub-directory -m mirror, turn on options for mirroring

GNU Wget 1.5.3.1, a non-interactive network retriever.
Usage: wget [OPTION]... [URL]...

Mandatory arguments to long options are mandatory for short options too.

Startup:
  -V,  --version           display the version of Wget and exit.
  -h,  --help              print this help.
  -b,  --background        go to background after startup.
  -e,  --execute=COMMAND   execute a `.wgetrc' command.

Logging and input file:
  -o,  --output-file=FILE     log messages to FILE.
  -a,  --append-output=FILE   append messages to FILE.
  -d,  --debug                print debug output.
  -q,  --quiet                quiet (no output).
  -v,  --verbose              be verbose (this is the default).
  -nv, --non-verbose          turn off verboseness, without being quiet.
  -i,  --input-file=FILE      read URL-s from file.
  -F,  --force-html           treat input file as HTML.

Download:
  -t,  --tries=NUMBER           set number of retries to NUMBER (0 unlimits).
  -O   --output-document=FILE   write documents to FILE.
  -nc, --no-clobber             don't clobber existing files.
  -c,  --continue               restart getting an existing file.
       --dot-style=STYLE        set retrieval display style.
  -N,  --timestamping           don't retrieve files if older than local.
  -S,  --server-response        print server response.
       --spider                 don't download anything.
  -T,  --timeout=SECONDS        set the read timeout to SECONDS.
  -w,  --wait=SECONDS           wait SECONDS between retrievals.
  -Y,  --proxy=on/off           turn proxy on or off.
  -Q,  --quota=NUMBER           set retrieval quota to NUMBER.

Directories:
  -nd  --no-directories            don't create directories.
  -x,  --force-directories         force creation of directories.
  -nH, --no-host-directories       don't create host directories.
  -P,  --directory-prefix=PREFIX   save files to PREFIX/...
       --cut-dirs=NUMBER           ignore NUMBER remote directory components.

HTTP options:
       --http-user=USER      set http user to USER.
       --http-passwd=PASS    set http password to PASS.
  -C,  --cache=on/off        (dis)allow server-cached data (normally allowed).
       --ignore-length       ignore `Content-Length' header field.
       --header=STRING       insert STRING among the headers.
       --proxy-user=USER     set USER as proxy username.
       --proxy-passwd=PASS   set PASS as proxy password.
  -s,  --save-headers        save the HTTP headers to file.
  -U,  --user-agent=AGENT    identify as AGENT instead of Wget/VERSION.

FTP options:
       --retr-symlinks   retrieve FTP symbolic links.
  -g,  --glob=on/off     turn file name globbing on or off.
       --passive-ftp     use the "passive" transfer mode.

Recursive retrieval:
  -r,  --recursive             recursive web-suck -- use with care!.
  -l,  --level=NUMBER          maximum recursion depth (0 to unlimit).
       --delete-after          delete downloaded files.
  -k,  --convert-links         convert non-relative links to relative.
  -m,  --mirror                turn on options suitable for mirroring.
  -nr, --dont-remove-listing   don't remove `.listing' files.

Recursive accept/reject:
  -A,  --accept=LIST                list of accepted extensions.
  -R,  --reject=LIST                list of rejected extensions.
  -D,  --domains=LIST               list of accepted domains.
       --exclude-domains=LIST       comma-separated list of rejected domains.
  -L,  --relative                   follow relative links only.
       --follow-ftp                 follow FTP links from HTML documents.
  -H,  --span-hosts                 go to foreign hosts when recursive.
  -I,  --include-directories=LIST   list of allowed directories.
  -X,  --exclude-directories=LIST   list of excluded directories.
  -nh, --no-host-lookup             don't DNS-lookup hosts.
  -np, --no-parent                  don't ascend to the parent directory.

Mail bug reports and suggestions to