Curl

From Christoph's Personal Wiki
Revision as of 21:29, 15 April 2015 by Christoph (Talk | contribs) (→‎Simple usage)

Jump to: navigation, search

cURL is a command line tool for transferring files with URL syntax, supporting FTP, FTPS, HTTP, HTTPS, TFTP, Telnet, DICT, FILE and LDAP. cURL supports HTTPS certificates, HTTP POST, HTTP PUT, FTP uploading, Kerberos, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM and Negotiate for HTTP and kerberos4 for FTP), file transfer resume, http proxy tunneling and many other features. cURL is open source/free software distributed under MIT License.

The main purpose and use for cURL is to automate unattended file transfers or sequences of operations. It is for example a good tool for simulating a user's actions at a web browser.

Libcurl is the corresponding library/API that users may incorporate into their programs; cURL acts as a stand-alone wrapper to the libcurl library. libcurl is being used to provide URL transfer capabilities to numerous applications, Open Source as well as many commercial ones.

Simple usage

  • Get the main page from firefox's web-server:
curl http://www.firefox.com/
  • Get the README file the user's home directory at funet's ftp-server:
curl ftp://ftp.funet.fi/README
  • Get a web page from a server using port 8000:
curl http://www.weirdserver.com:8000/
  • Get a list of a directory of an FTP site:
curl ftp://cool.haxx.se/
  • Get a gopher document from funet's gopher server:
curl gopher://gopher.funet.fi
  • Get the definition of curl from a dictionary:
curl dict://dict.org/m:curl
  • Fetch two documents at once:
curl ftp://cool.haxx.se/ http://www.weirdserver.com:8000/
$ curl -I http://xtof.ch/skills
HTTP/1.1 301 Moved Permanently
Date: Wed, 15 Apr 2015 21:24:28 GMT
Server: Apache/2.2.15 (CentOS)
Location: http://wiki.christophchamp.com/index.php/Technical_and_Specialized_Skills
Connection: close
Content-Type: text/html; charset=iso-8859-1

# xtof.ch/skills redirects to http://wiki.christophchamp.com/index.php/Technical_and_Specialized_Skill
# So, use "-L" to follow that redirect:
$ curl -L http://xtof.ch/skills

Miscellaneous examples

  • Check on the amount of time it takes to load a website (lookup/connect/transfer times):
$ for i in `seq 1 3`; do curl -so /dev/null www.example.com \
   -w "time_namelookup: %{time_namelookup}\
       \ttime_connect: %{time_connect}\
       \ttime_starttransfer: %{time_starttransfer}\
       \ttime_total: %{time_total}\n"; done
time_namelookup: 0.004  time_connect: 0.005     time_starttransfer: 0.854       time_total: 0.964
time_namelookup: 0.004  time_connect: 0.005     time_starttransfer: 0.575       time_total: 0.617
time_namelookup: 0.004  time_connect: 0.005     time_starttransfer: 0.550       time_total: 0.555

Download to a file

  • Get a web page and store in a local file:
curl -o thatpage.html http://www.netscape.com/
  • Get a web page and store in a local file, make the local file get the name of the remote document (if no file name part is specified in the URL, this will fail):
curl -O http://www.netscape.com/index.html
  • Fetch two files and store them with their remote names:
curl -O www.haxx.se/index.html -O curl.haxx.se/download.html

Using cURL for fast downloads

Suppose you want to download the Ubuntu 12.04 (Precise Pangolin; 64-bit) ISO from the following three locations/mirrors:

UPDATE: Some of the following links no longer exist. Use 14.04 ISOs or other mirrors instead. E.g.: 

$ curl -sI http://mirror.anl.gov/pub/ubuntu-iso/DVDs/ubuntu/14.04/release/ubuntu-14.04-desktop-amd64+mac.iso|\
    awk '{if ($0 ~ /^Content-Length/){iso_size=$2/1024^2; print iso_size}}'

$ url1=http://swtsrv.informatik.uni-mannheim.de/pub/linux/distributions/ubuntu-dvd-release/12.04/release/ubuntu-12.04.3-dvd-amd64.iso
$ url2=http://www.mirrorservice.org/sites/cdimage.ubuntu.com/cdimage/releases/12.04/release/ubuntu-12.04.3-dvd-amd64.iso
$ url3=http://mirror.anl.gov/pub/ubuntu-iso/DVDs/ubuntu/12.04/release/ubuntu-12.04.3-dvd-amd64.iso

Get the total size (in bytes) of the ISO: 

$ ISOURL=http://swtsrv.informatik.uni-mannheim.de/pub/linux/distributions/ubuntu-dvd-release/12.04/release/ubuntu-12.04.3-dvd-amd64.iso
$ iso_size=`curl -sI $ISOURL | grep ^Content-Length | cut -d' ' -f2`

The total size of the ISO is 1728053248 bytes (~1.7GB). Using cURL's "--range" option, we can download that ISO in 3 parts from the above 3 different mirrors simultaneously with the following commands (do not forget the "&" at the end so each download is backgrounded): 

$ curl -r 0-499999999 -o ubuntu-12.04-64bit-iso.part1 $url1 &         # 1st 500MB
$ curl -r 500000000-999999999 -o ubuntu-12.04-64bit-iso.part2 $url2 & # 2nd 500MB
$ curl -r 1000000000- -o ubuntu-12.04-64bit-iso.part3 $url3 &         # remaining bytes

After all three parts have downloaded, `cat` them all together into a single ISO 

$ cat ubuntu-12.04-64bit-iso.part? > ubuntu-12.04.3-dvd-amd64.iso

Finally, check the integrity of the ISO using the MD5SUM for the original ISO: 

$ wget -c http://swtsrv.informatik.uni-mannheim.de/pub/linux/distributions/ubuntu-dvd-release/12.04/release/MD5SUMS
$ grep ubuntu-12.04.3-dvd-amd64.iso MD5SUMS
$ md5sum ubuntu-12.04.3-dvd-amd64.iso

The two values should be identical. Et voilà! You have downloaded that ISO (potentially) much faster than downloading it as one single ISO.

Note: You could automate the process in a script. You would use the ${iso_size} from above together with the following lines: 

blocksize=`expr 1024 \* 512`
$sum-$(($sum+$blocksize))
curl -\# -r $sum-$(($sum+$blocksize)) -o ubuntu-12.04-64bit-iso.part${num} $url1 &

The "-\#" is to switch from the regular meter to a progress "bar".

See also

External links

Retrieved from "http://wiki.christophchamp.com/index.php?title=Curl&oldid=6298"