Category: Curl

Dec 02

Service Response Time

There are several ways to get this done but I found using curl gives a good quick test of how long a web response was. You can use a formatted text file or just command line.

Command line can be something like this:

# curl -w "\ntime_namelookup:\t%{time_namelookup}\ntime_connect:\t\t%{time_connect}\ntime_appconnect:\t%{time_appconnect}\ntime_pretransfer:\t%{time_pretransfer}\ntime_redirect:\t\t%{time_redirect}\ntime_starttransfer:\t%{time_starttransfer}\n----------\ntime_total:\t\t%{time_total}\n" -o /dev/null -s
time_namelookup:        0.366
time_connect:           0.367
time_appconnect:        0.402
time_pretransfer:       0.402
time_redirect:          0.000
time_starttransfer:     0.408
time_total:             0.408

Or using a text file for formatting:

# curl -w "@curl-format.txt" -o /dev/null -s

            time_namelookup:  0.127
               time_connect:  0.128
            time_appconnect:  0.142
           time_pretransfer:  0.142
              time_redirect:  0.000
         time_starttransfer:  0.150
                 time_total:  0.151

Text file like this:

# cat curl-format.txt 
            time_namelookup:  %{time_namelookup}\n
               time_connect:  %{time_connect}\n
            time_appconnect:  %{time_appconnect}\n
           time_pretransfer:  %{time_pretransfer}\n
              time_redirect:  %{time_redirect}\n
         time_starttransfer:  %{time_starttransfer}\n
                 time_total:  %{time_total}\n

Comments Off on Service Response Time

Feb 07

Curl command line downloads

If you need a command line download on Linux there are several options.  Wget for a simple download is a very good option but I prefer curl as I have had more success when dealing with logins, cookies and uploads than with wget.

Plus if you need to go further and integrate something with python or php the curl libraries are awesome.

Simple download:

$ curl -o myfile.iso ""

With login:

$ curl -o myfile.iso -u user:password ""
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
 Dload  Upload   Total   Spent    Left  Speed
 100 3509M  100 3509M    0     0  6204k      0  0:09:39  0:09:39 --:--:-- 7106k

** Note sometimes to get the correct download string you will need to login into the site with your browser and copy the download location of the link.  Or in some cases actually initiate the download with the browser and then copy the link from the browser download window.

If you are after a more permanent web or scripted solution you can use curl to login and save the cookie.  Then subsequently download the file using the generated cookie.  This requires more experimentation with your particular site.

Login and save a cookie:

$ curl -c cookie.txt -u user:password

List files:

$ curl -b cookie.txt -u user:password

Upload a file:

$ curl -b cookie.txt --upload-file test.rrosso


If you are behind corporate firewalls you might still be able to use curl.
Behind socks5 proxy:

$ curl --socks5 -U rrosso:pwd -c cookie2.txt -u site-user:pwd (login and save a cookie)
$ curl --socks5 -U rrosso:pwd -b cookie2.txt -u site-user:pwd (list files using saved cookie)
$ curl --socks5 -U rrosso:pwd -b cookie2.txt --upload-file test.rrosso

Behind squid proxy:

$ curl -x -U rrosso:pwd -o ARP08110610926072.txt_171317.RECVD -u site-user:pwd

$ curl -x -U rrosso:pwd -c cookie2.txt -u site-user:pwd
Virtual user site-user logged in.

$ curl -x -U rrosso:pwd -b cookie2.txt -u site-user:pwd
total 38

$ curl -x -U rrosso:pwd -b cookie2.txt -u site-user:pwd --upload-file rrosso-test

Comments Off on Curl command line downloads