Curl

From Christoph's Personal Wiki
Revision as of 20:35, 27 February 2014 by Christoph (Talk | contribs) (→‎Using cURL for fast downloads)

Jump to: navigation, search

cURL is a command line tool for transferring files with URL syntax, supporting FTP, FTPS, HTTP, HTTPS, TFTP, Telnet, DICT, FILE and LDAP. cURL supports HTTPS certificates, HTTP POST, HTTP PUT, FTP uploading, Kerberos, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM and Negotiate for HTTP and kerberos4 for FTP), file transfer resume, http proxy tunneling and many other features. cURL is open source/free software distributed under MIT License.

The main purpose and use for cURL is to automate unattended file transfers or sequences of operations. It is for example a good tool for simulating a user's actions at a web browser.

Libcurl is the corresponding library/API that users may incorporate into their programs; cURL acts as a stand-alone wrapper to the libcurl library. libcurl is being used to provide URL transfer capabilities to numerous applications, Open Source as well as many commercial ones.

Simple usage

  • Get the main page from firefox's web-server:
curl http://www.firefox.com/
  • Get the README file the user's home directory at funet's ftp-server:
curl ftp://ftp.funet.fi/README
  • Get a web page from a server using port 8000:
curl http://www.weirdserver.com:8000/
  • Get a list of a directory of an FTP site:
curl ftp://cool.haxx.se/
  • Get a gopher document from funet's gopher server:
curl gopher://gopher.funet.fi
  • Get the definition of curl from a dictionary:
curl dict://dict.org/m:curl
  • Fetch two documents at once:
curl ftp://cool.haxx.se/ http://www.weirdserver.com:8000/

Download to a file

  • Get a web page and store in a local file:
curl -o thatpage.html http://www.netscape.com/
  • Get a web page and store in a local file, make the local file get the name of the remote document (if no file name part is specified in the URL, this will fail):
curl -O http://www.netscape.com/index.html
  • Fetch two files and store them with their remote names:
curl -O www.haxx.se/index.html -O curl.haxx.se/download.html

Using cURL for fast downloads

Suppose you want to download the Ubuntu 12.04 (Precise Pangolin; 64-bit) ISO from the following three locations/mirrors: 

url1=http://swtsrv.informatik.uni-mannheim.de/pub/linux/distributions/ubuntu-dvd-release/12.04/release/ubuntu-12.04.3-dvd-amd64.iso
url2=http://www.mirrorservice.org/sites/cdimage.ubuntu.com/cdimage/releases/12.04/release/ubuntu-12.04.3-dvd-amd64.iso
url3=http://mirror.anl.gov/pub/ubuntu-iso/DVDs/ubuntu/12.04/release/ubuntu-12.04.3-dvd-amd64.iso

Get the total size (in bytes) of the ISO: 

ISOURL=http://swtsrv.informatik.uni-mannheim.de/pub/linux/distributions/ubuntu-dvd-release/12.04/release/ubuntu-12.04.3-dvd-amd64.iso
iso_size=`curl -s -I $ISOURL | grep ^Content-Length | cut -d' ' -f2`

The total size of the ISO is 1728053248 bytes (~1.7GB). Using cURL's "--range" option, we can download that ISO in 3 parts from the above 3 different mirrors simultaneously with the following commands: 

curl -r 0-499999999 -o ubuntu-12.04-64bit-iso.part1 $url1 & # 1st 500MB
curl -r 500000000-999999999 -o ubuntu-12.04-64bit-iso.part2 $url2 & # 2nd 500MB
curl -r 1000000000- -o ubuntu-12.04-64bit-iso.part3 $url3 & # remaining bytes

After all three parts have downloaded, `cat` them all together into a single ISO 

cat ubuntu-12.04-64bit-iso.part? > ubuntu-12.04.3-dvd-amd64.iso

Finally, check the integrity of the ISO using the MD5SUM for the original ISO: 

wget -c http://swtsrv.informatik.uni-mannheim.de/pub/linux/distributions/ubuntu-dvd-release/12.04/release/MD5SUMS
grep ubuntu-12.04.3-dvd-amd64.iso MD5SUMS
md5sum ubuntu-12.04.3-dvd-amd64.iso

The two values should be identical. Et voilà! You have downloaded that ISO (potentially) much faster than downloading it as one single ISO.

Note: You could automate the process in a script. You would use the ${iso_size} from above together with the following lines: 

blocksize=`expr 1024 \* 512`
$sum-$(($sum+$blocksize))
curl -\# -r $sum-$(($sum+$blocksize)) -o ubuntu-12.04-64bit-iso.part${num} $url1 &

The "-\#" is to switch from the regular meter to a progress "bar".

See also

External links

Retrieved from "http://wiki.christophchamp.com/index.php?title=Curl&oldid=6006"