Wget

wget — The non-interactive network downloader.

Usage

• Mirror an entire web site:

wget -m http://www.example.com

• Mirror an entire subdirectory of a web site (with no parent option in case of backlinks):

wget -mk -w 20 -np http://example.com/foo/

• Download all pages from a site and the pages the site links to (one-level deep):

wget -H -r --level=1 -k -p http://www.example.com

• Resume large file download:

wget -c --output-document=Bill_Maher_-_New_Rules_2007-03-15.avi "http://www.youtube.com/watch%3Fv%3DhFjRI5jJ5I4&usg=AL29H23P1UQZRf0yDqRBlwB0jyfSLbzzhg"

• Schedule hourly downloads of a file

wget --output-document=traffic_$(date +\%Y\%m\%d\%H).gif "http://sm3.sitemeter.com/YOUR_CODE"

• Automatically download music (by Jeff Veen):

wget -r -l1 -H -t1 -nd -N -np -A.mp3 -erobots=off -i mp3_sites.txt

where mp3_sites.txt lists your favourite (legal) download sites.

Download multiple files

• Create variable that holds all URLs and then using 'BASH for loop' to download all files:

% URLS="http://www.example.com/foo.tar.gz ftp://ftp.example.org/pub/bar.tar.gz"

• Use for loop as follows:

% for u in $URLS; do wget $u; done

• You can also put a list of the URLs in a file and download using the -i option:

% wget -i download.txt

Automating/scripting download process

#!/bin/sh
# wget-list: manage the list of downloaded files

# invoke wget-list without arguments
while [ `find .wget-list -size +0` ]
 do
  url=`head -n1 .wget-list`
   wget -c $url
   sed -si 1d .wget-list
 done

#/bin/sh
# wget-all: process .wget-list in every subdirectory
# invoke wget-all without arguments

find -name .wget-list -execdir wget-list ';'

#!/bin/sh
# wget-dirs: run wget-all in specified directories
# invoking: wget-dirs <path-to-directory> ...

for dir in $*
  do
      pushd $dir
      wget-all
      popd
  done
wget-all

See also

• curl
• wput
• rsync
• axel
• prozilla

External links

• GNU Wget Manual — last update: 15-Jun-2005
• Geek to Live: Mastering Wget
• wget: your ultimate command line downloader