Curl
cURL is a command line tool for transferring files with URL syntax, supporting FTP, FTPS, HTTP, HTTPS, TFTP, Telnet, DICT, FILE and LDAP. cURL supports HTTPS certificates, HTTP POST, HTTP PUT, FTP uploading, Kerberos, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM and Negotiate for HTTP and kerberos4 for FTP), file transfer resume, http proxy tunneling and many other features. cURL is open source/free software distributed under MIT License.
The main purpose and use for cURL is to automate unattended file transfers or sequences of operations. It is for example a good tool for simulating a user's actions at a web browser.
Libcurl is the corresponding library/API that users may incorporate into their programs; cURL acts as a stand-alone wrapper to the libcurl library. libcurl is being used to provide URL transfer capabilities to numerous applications, Open Source as well as many commercial ones.
Contents
Simple usage
- Get the main page from firefox's web-server:
curl http://www.firefox.com/
- Get the README file the user's home directory at funet's ftp-server:
curl ftp://ftp.funet.fi/README
- Get a web page from a server using port 8000:
curl http://www.weirdserver.com:8000/
- Get a list of a directory of an FTP site:
curl ftp://cool.haxx.se/
- Get a gopher document from funet's gopher server:
curl gopher://gopher.funet.fi
- Get the definition of curl from a dictionary:
curl dict://dict.org/m:curl
- Fetch two documents at once:
curl ftp://cool.haxx.se/ http://www.weirdserver.com:8000/
Download to a file
- Get a web page and store in a local file:
curl -o thatpage.html http://www.netscape.com/
- Get a web page and store in a local file, make the local file get the name of the remote document (if no file name part is specified in the URL, this will fail):
curl -O http://www.netscape.com/index.html
- Fetch two files and store them with their remote names:
curl -O www.haxx.se/index.html -O curl.haxx.se/download.html
Using cURL for fast downloads
Suppose you want to download the Ubuntu 12.04 (Precise Pangolin; 64-bit) ISO from the following three locations/mirrors:
url1=http://swtsrv.informatik.uni-mannheim.de/pub/linux/distributions/ubuntu-dvd-release/12.04/release/ubuntu-12.04.3-dvd-amd64.iso url2=http://www.mirrorservice.org/sites/cdimage.ubuntu.com/cdimage/releases/12.04/release/ubuntu-12.04.3-dvd-amd64.iso url3=http://mirror.anl.gov/pub/ubuntu-iso/DVDs/ubuntu/12.04/release/ubuntu-12.04.3-dvd-amd64.iso
The total size of the ISO is 1728053248 bytes (~1.7GB). Using cURL's "--range" option, we can download that ISO in 3 parts from the above 3 different mirrors simultaneously with the following commands:
curl -r 0-499999999 -o ubuntu-12.04-64bit-iso.part1 $url1 & curl -r 500000000-9999999999 -o ubuntu-12.04-64bit-iso.part2 $url2 & curl -r 1000000000- -o ubuntu-12.04-64bit-iso.part3 $url3 &
After all three parts have downloaded, `cat` them all together into a single ISO
cat ubuntu-12.04-64bit-iso.part? > ubuntu-12.04.3-dvd-amd64.iso
Finally, check the integrity of the ISO using the MD5SUM for the original ISO:
wget -c http://swtsrv.informatik.uni-mannheim.de/pub/linux/distributions/ubuntu-dvd-release/12.04/release/MD5SUMS grep ubuntu-12.04.3-dvd-amd64.iso MD5SUMS md5sum ubuntu-12.04.3-dvd-amd64.iso
The two values should be identical. Et voilà! You have downloaded that ISO (potentially) much faster than downloading it as one single ISO.
