Is there any way to cache HTTPS requests in a proxy server?

Question

We are using Squid proxy server in our environment and we want to cache HTTPS requests.

Is there any way to configure Squid or in general a proxy server to cache HTTPS requests?

Solution 1

There is a way to do it, but it's fundamentally against the reasons for using HTTPS.

Here's how you'd do it.

1. Generate a self-signed SSL Certificate for the site you'd like to intercept and cache the requests from.
2. Install and run stunnel on your proxy server, telling it that the certificate it should present is the one generated in stage 1.
3. Have stunnel forward the decrypted requests to squid.
4. You might need to have stunnel on the other side, or openssl_client to re-encrypt the request to the upstream server.

Caveats:

1. Your users will hate you. Every SSL request to that site will present an invalid certificate window.
2. You're exposing yourself to potential lawsuits for doing naughty things. (IANAL)
3. You'll only ever be able to get a self-signed certificate working for this, because of how the PKI web of trust for SSL Certificates is supposed to work. Saying nothing about compromised root CAs.

I'm not going to give you the exact details of how to do this, because a) I think it's somewhat unethical, and b) It's better for you to learn how to do it.

I suggest you research how stunnel and man-in-the-middle attacks work.

Solution 2

Just to explain why this can't be done without MITM - a proxy only sees the DNS name of the server you want to connect to when using encrypted HTTPS. It does not see the URL, nor any response headers. It cannot determine which individual resource you are accessing on a site, whether or not it is cacheable, nor what it's modification times are. All it can see is someone wants something from a remote server using HTTPS.

This means that caching cannot work as the proxy does not know what cached objects to give you, or how to get them in the first place.

Solution 3

No, there are not: they are encrypted... A workaround would be something like a man-in-middle deployment, but that would defeat all the reasons behind https.

Solution 4

Squid has a "hack" for just that:

https://wiki.squid-cache.org/Features/SslPeekAndSplice

Have fun. :)

Solution 5

Zeus (Now Riverbed's) ZTM Traffic Manager can do this as it can translate http and https traffic both ways and cache unencrypted content - it works, we use it, but it's fearsomely expensive - as in the price of a Porsche per server.

View more solutions

Comments

[Tom O'Connor]

This probably belongs on security.se

[Gqqnbig]

Just to clarify. You want to cache requests or responses?

[Supratik]

Are there no workaround to achieve this or force the proxy server to decrypt and cache?

[yrk]

A workaround would be somewhat which resempbles man-in-middle deployment, but that would defeat all the reasons behind https

[phihag]

But it still requires you to install a new root cert on the client, doesn't it? And you still have to trust the proxy, don't you?

[Tom O'Connor]

Actually, if you're on a domain, it's far easier to generate your own CA, and deploy the public certificates for it with Group Policy.

[bbigras]

I disagree that it would defeat all the reasons behind https. If you do this at home and you own the proxy, your data will still use https between your proxy and the web sites.

[yrk]

@brunoqc that's a VPN's job.

[Admin]

Trusting a SSC + MITM is useful for protocol debugging, caching, deep packet inspection and censorship/logging. :/ Other than those reasons, not so good.

[Admin]

If caching https payloads is important for some reason or debugging an https session, MITM is super useful. In fact, this is how Charles works.

[Ben]

@yarek, provided that there is a reason behind that https. Now it feels more like that https have been abusively used in many senarios.