How to grep the output of cURL?
Solution 1
curl writes the output to stderr, so redirect that and also suppress the progress:
curl -v --silent https://google.com/ 2>&1 | grep expire
The reason why curl
writes the information to stderr is so you can do:
curl <url> | someprgram
without that information clobbering the input of someprogram
Solution 2
It is possible to use --stderr -
as parameter, to redirect the output from stderr (default) to stdout.
With this option you also should use --silent
to suppress the progress bar.
$ curl -v --silent https://google.com/ --stderr - | grep expire
* expire date: 2015-09-01 00:00:00 GMT
Related videos on Youtube
dotancohen
Updated on September 18, 2022Comments
-
dotancohen over 1 year
I need to retrieve the expiry date of an SSL cert. The
curl
application does provide this information:$ curl -v https://google.com/ * Hostname was NOT found in DNS cache * Trying 212.179.180.121... * Connected to google.com (212.179.180.121) port 443 (#0) * successfully set certificate verify locations: * CAfile: none CApath: /etc/ssl/certs * SSLv3, TLS handshake, Client hello (1): * SSLv3, TLS handshake, Server hello (2): * SSLv3, TLS handshake, CERT (11): * SSLv3, TLS handshake, Server key exchange (12): * SSLv3, TLS handshake, Server finished (14): * SSLv3, TLS handshake, Client key exchange (16): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSL connection using ECDHE-ECDSA-AES128-GCM-SHA256 * Server certificate: * subject: C=US; ST=California; L=Mountain View; O=Google Inc; CN=*.google.com * start date: 2014-10-22 13:04:07 GMT * expire date: 2015-01-20 00:00:00 GMT * subjectAltName: google.com matched * issuer: C=US; O=Google Inc; CN=Google Internet Authority G2 * SSL certificate verify ok. > GET / HTTP/1.1 > User-Agent: curl/7.35.0 > Host: google.com > Accept: */* > < HTTP/1.1 302 Found < Cache-Control: private < Content-Type: text/html; charset=UTF-8 < Location: https://www.google.co.il/?gfe_rd=cr&ei=HkxbVMzCM-WkiAbU6YCoCg < Content-Length: 262 < Date: Thu, 06 Nov 2014 10:23:26 GMT * Server GFE/2.0 is not blacklisted < Server: GFE/2.0 < <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8"> <TITLE>302 Moved</TITLE></HEAD><BODY> <H1>302 Moved</H1> The document has moved <A HREF="https://www.google.co.il/?gfe_rd=cr&ei=HkxbVMzCM-WkiAbU6YCoCg">here</A>. </BODY></HTML> * Connection #0 to host google.com left intact
However, when piping the output via
grep
the result is not less information on the screen, but rather much more:$ curl -v https://google.com/ | grep expire * Hostname was NOT found in DNS cache % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 212.179.180.84... * Connected to google.com (212.179.180.84) port 443 (#0) * successfully set certificate verify locations: * CAfile: none CApath: /etc/ssl/certs * SSLv3, TLS handshake, Client hello (1): } [data not shown] * SSLv3, TLS handshake, Server hello (2): { [data not shown] 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* SSLv3, TLS handshake, CERT (11): { [data not shown] * SSLv3, TLS handshake, Server key exchange (12): { [data not shown] * SSLv3, TLS handshake, Server finished (14): { [data not shown] * SSLv3, TLS handshake, Client key exchange (16): } [data not shown] * SSLv3, TLS change cipher, Client hello (1): } [data not shown] * SSLv3, TLS handshake, Finished (20): } [data not shown] * SSLv3, TLS change cipher, Client hello (1): { [data not shown] * SSLv3, TLS handshake, Finished (20): { [data not shown] * SSL connection using ECDHE-ECDSA-AES128-GCM-SHA256 * Server certificate: * subject: C=US; ST=California; L=Mountain View; O=Google Inc; CN=*.google.com * start date: 2014-10-22 13:04:07 GMT * expire date: 2015-01-20 00:00:00 GMT * subjectAltName: google.com matched * issuer: C=US; O=Google Inc; CN=Google Internet Authority G2 * SSL certificate verify ok. > GET / HTTP/1.1 > User-Agent: curl/7.35.0 > Host: google.com > Accept: */* > < HTTP/1.1 302 Found < Cache-Control: private < Content-Type: text/html; charset=UTF-8 < Location: https://www.google.co.il/?gfe_rd=cr&ei=IkxbVMy4K4OBbKuDgKgF < Content-Length: 260 < Date: Thu, 06 Nov 2014 10:23:30 GMT * Server GFE/2.0 is not blacklisted < Server: GFE/2.0 < { [data not shown] 100 260 100 260 0 0 714 0 --:--:-- --:--:-- --:--:-- 714 * Connection #0 to host google.com left intact
I suspect that
curl
detects that it is not printing to a terminal and is thus gives different output, not all of which is recognized bygrep
as beingstdout
and is thus passed through to the terminal. However, the closest thing to this that I could find inman curl
(don't ever google for that!) is this:PROGRESS METER curl normally displays a progress meter during operations, indicating the amount of transferred data, transfer speeds and estimated time left, etc. curl displays this data to the terminal by default, so if you invoke curl to do an operation and it is about to write data to the terminal, it disables the progress meter as otherwise it would mess up the output mixing progress meter and response data. If you want a progress meter for HTTP POST or PUT requests, you need to redirect the response output to a file, using shell redirect (>), -o [file] or similar. It is not the same case for FTP upload as that operation does not spit out any response data to the terminal. If you prefer a progress "bar" instead of the regular meter, -# is your friend.
How can I get just the
expiry
line out of thecurl
output? Furthermore, what should I be reading to understand the situation better?Seems like this would be a good use case for a "stdmeta" file descriptor.
-
Admin over 8 years
-
Admin over 8 years@EladKarako: Thank you. That is a nice answer, but it is not applicable to this situation. Perhaps the Windows shell has no concept of separate output file handlers, but this is a different case in Linux shells.
-
Admin about 5 yearsI feel compelled to throw caution to the wind and Google man curl. I'm certain that's bound to return some worthwhile results.
-
-
astroanu over 8 years-v does the trick of verbose .. thanks
-
Admin over 8 yearsno need for
-v
you will pipe quite a lot text.. especially on multiple redirection and SSL handshakes and certificate exchanging!! you should use either--include
to just add the headers to the response's body, or better yet use--head
, also, you better remove the\r
at the end if you are planning to store the value in a variable or even send it to output later on (see my comment above withContent-Length
example)