Wget - Revision history

Christoph: /* Automating/scripting download process */

2021-09-13T07:18:49Z

‎Automating/scripting download process

Christoph: /* Automating/scripting download process */

2021-09-13T07:18:19Z

‎Automating/scripting download process

Christoph: /* Download multiple files */

2021-09-13T07:17:26Z

‎Download multiple files

Christoph: /* Usage */

2021-09-13T07:15:40Z

‎Usage

Christoph at 20:50, 24 May 2015

2015-05-24T20:50:01Z

Christoph: /* See also */

2015-05-22T00:05:18Z

‎See also

Christoph: /* Usage */

2015-05-21T23:54:16Z

‎Usage

Christoph at 23:26, 21 May 2015

2015-05-21T23:26:35Z

Christoph: /* Usage */

2008-05-28T02:08:01Z

‎Usage

Christoph at 01:51, 26 April 2007

2007-04-26T01:51:34Z

@@ Line 93: / Line 93: @@
 # invoke wget-list without arguments
-while [ $(find .wget-list -size +0) ]
+while [ $(find .wget-list -size +0) ]; do
- do
    url=$(head -n1 .wget-list)
-   wget -c ${url}
+  wget -c ${url}
-   sed -si 1d .wget-list
+  sed -si 1d .wget-list
- done
+done
 </pre>

@@ Line 102: / Line 102: @@
 <pre>
-#/bin/sh
+#!/bin/sh
 # wget-all: process .wget-list in every subdirectory
 # invoke wget-all without arguments
@@ Line 114: / Line 114: @@
 # invoking: wget-dirs <path-to-directory> ...
-for dir in $*
+for dir in $*; do
-  do
+  pushd ${dir}
-      pushd ${dir}
+  wget-all
-      wget-all
+  popd
-      popd
+done
-  done
 wget-all
 </pre>

@@ Line 79: / Line 79: @@
 ==Download multiple files==
-* Create variable that holds all URLs and then using 'BASH for loop' to download all files:
+* Create variable that holds all URLs and then using "BASH for loop" to download all files:
-  % URLS="<nowiki>http://www.example.com/foo.tar.gz ftp://ftp.example.org/pub/bar.tar.gz</nowiki>"
+  $ URLS="<nowiki>http://www.example.com/foo.tar.gz ftp://ftp.example.org/pub/bar.tar.gz</nowiki>"
 * Use for loop as follows:
-  % for u in $URLS; do wget $u; done
+  $ for u in $URLS; do wget $u; done
 * You can also put a list of the URLs in a file and download using the <code>-i</code> option:
-  % wget -i download.txt
+  $ wget -i download.txt
 ===Automating/scripting download process===
@@ Line 93: / Line 93: @@
 # invoke wget-list without arguments
-while [ `find .wget-list -size +0` ]
+while [ $(find .wget-list -size +0) ]
   do
-   url=`head -n1 .wget-list`
+   url=$(head -n1 .wget-list)
-    wget -c $url
+    wget -c ${url}
     sed -si 1d .wget-list
   done
@@ Line 116: / Line 116: @@
 for dir in $*
    do
-       pushd $dir
+       pushd ${dir}
        wget-all
        popd

@@ Line 56: / Line 56: @@
 * Limit download speed/rate:
   $ wget --limit-rate=300k <nowiki>https://www.kernel.org/pub/linux/kernel/v4.x/linux-4.0.4.tar.gz</nowiki>
 ==Download multiple files==

@@ Line 35: / Line 35: @@
   #~OR~
   $ wget -r --level=1 -H --timeout=1 -nd -N -np --accept=mp3 -e robots=off -i musicblogs.txt
 * Crawl a website and generate a log file of any broken links:

@@ Line 92: / Line 92: @@
 ==See also==
 *[[curl]]
-*[[wput]]
+*[http://wput.sourceforge.net/ wput] &mdash; tiny wget-like ftp-client for uploading files
 *[[rsync]]
 *[[axel]]

@@ Line 11: / Line 11: @@
 * Background download:
   $ wget -b <nowiki>https://www.kernel.org/pub/linux/kernel/v4.x/linux-4.0.4.tar.gz</nowiki>
 The above command is useful when you initiate a download via a remote machine. This will start downloading in background, so that you can disconnect the terminal once the command is issued.
@@ Line 32: / Line 33: @@
   $ wget -r -l1 -H -t1 -nd -N -np -A.mp3 -erobots=off -i mp3_sites.txt
 where <code>mp3_sites.txt</code> lists your favourite (legal) download sites.
 * Crawl a website and generate a log file of any broken links:
   $ wget --spider -o wget.log -e robots=off --wait 1 -r -p <nowiki>http://www.example.com/</nowiki>
 ==Download multiple files==

@@ Line 2: / Line 2: @@
 ==Usage==
-*Mirror an entire subdirectory of a web site (with no parent option in case of backlinks):
+* Simple download:
-  wget -mk -w 20 -np <nowiki>http://example.com/foo/</nowiki>
+  $ wget <nowiki>http://www.example.com/index.html</nowiki>
-*Download all pages from a site and the pages the site links to (one-level deep):
+* Download a file and store it locally using a different file name:
-  wget -H -r --level=1 -k -p <nowiki>http://www.example.com</nowiki>
+  $ wget -O example.html <nowiki>http://www.example.com/index.html</nowiki>
-*Resume large file download:
+* Background download:
-  wget -c --output-document=Bill_Maher_-_New_Rules_2007-03-15.avi "<nowiki>http://www.youtube.com/watch%3Fv%3DhFjRI5jJ5I4&usg=AL29H23P1UQZRf0yDqRBlwB0jyfSLbzzhg</nowiki>"
+  $ wget -b <nowiki>https://www.kernel.org/pub/linux/kernel/v4.x/linux-4.0.4.tar.gz</nowiki>
-*Schedule hourly downloads of a file
+The above command is useful when you initiate a download via a remote machine. This will start downloading in background, so that you can disconnect the terminal once the command is issued.
- wget --output-document=traffic_$(date +\%Y\%m\%d\%H).gif "<nowiki>http://sm3.sitemeter.com/YOUR_CODE</nowiki>"
-*Automatically download music (by [http://www.veen.com/jeff/archives/000573.html Jeff Veen]):
+* Mirror an entire web site:
-  wget -r -l1 -H -t1 -nd -N -np -A.mp3 -erobots=off -i mp3_sites.txt
+ $ wget -m <nowiki>http://www.example.com</nowiki>
 where <code>mp3_sites.txt</code> lists your favourite (legal) download sites.
 ==Download multiple files==
@@ Line 75: / Line 90: @@
 ==External links==
 *[http://www.gnu.org/software/wget/manual/ GNU Wget Manual] &mdash; last update: 15-Jun-2005
-*[http://www.lifehacker.com/software/top/geek-to-live--mastering-wget-161202.php Geek to Live: Mastering Wget]
+*[http://lifehacker.com/161202/geek-to-live--mastering-wget Geek to Live: Mastering Wget] &mdash; via lifehacker.com
 *[http://www.cyberciti.biz/nixcraft/vivek/blogger/2005/06/linux-wget-your-ultimate-command-line.php wget: your ultimate command line downloader]
 [[Category:Linux Command Line Tools]]