How to get charles proxy work with Android 7 nougat?

Question

Android 7 introduced some changes to the way certificates are handled (http://android-developers.blogspot.com/2016/07/changes-to-trusted-certificate.html) and somehow I cannot make my Charles proxy work any more.

My network_security_config.xml:

<?xml version="1.0" encoding="utf-8"?>
<network-security-config>
    <base-config>
        <trust-anchors>
            <certificates src="system" />
        </trust-anchors>
    </base-config>
    <debug-overrides>
        <trust-anchors>
            <certificates src="user" />
        </trust-anchors>
    </debug-overrides>
</network-security-config>

I'm running in debug mode. But no matter what, I get javax.net.ssl.SSLHandshakeException: java.security.cert.CertPathValidatorException: Trust anchor for certification path not found..

Needless to say, I did install a pfx certificate from Settings -> Security -> Install from storage. The certificate shows in User Credentials but not in Trusted credentials -> User. On my lollipop device, the certificates are listed there.

I'm using okhttp3 as HTTP library.

Any idea what I am doing wrong ?

Solution 1

Based on the troubleshooting thread of comments for the OP, the answer is to install just the proxy's CA cert as trusted, not its cert + private key.

The issue was caused by two factors:

1. Installing not just the MiTM proxy's CA cert but also its private key (thus enabling VPN apps on the device to decrypt/MiTM network traffic from other apps). You don't need the MiTM proxy's private key on the device.

2. Android Nougat change in behavior of the Settings -> Security -> Install from storage flow for files which contain a private key in addition to cert(s). This change in behavior unmasked the above issue.

Prior to Nougat, the Settings -> Security -> Install from storage flow for files containing a private key in addition to certs erroneously installed the certs as trusted for server authentication (e.g., HTTPS, TLS, thus making your MiTM succeed), in addition to being correctly installed as client certs used for authenticating this Android device to servers. In Nougat, the bug was fixed and these certs are no longer installed as trusted for server authentication. This prevents client authentication credentials from affecting (weaking) the security of connections to servers. In your scenario, this prevents your MiTM from succeeding.

What complicates matters is that the Settings -> Security -> Install from storage does not provide an explicit way for the user to specify whether they are installing a client authentication credential (private key + cert chain) or a server authentication trust anchor (just a CA cert -- no private key needed). As a result, the Settings -> Security -> Install from storage flow guesses whether it's dealing with client/user authentication credential or server authentication trust anchor by assuming that, if a private key is specified, it must be a client/user authentication credential. In your case, it incorrectly assumed that you are installing a client/user authentication credential rather than a server authentication trust anchor.

P. S. With regards to your Network Security Config, you should probably configure the app to also trust "system" trust anchors in debug mode (debug-overrides section). Otherwise debug builds of the app won't work unless connections are MiTM'd by a proxy whose CA cert is installed as trusted on the Android device.

Solution 2

The solution is do not use .p12, just navigate with Chrome (with configured proxy on wifi) to http://charlesproxy.com/getssl and install downloaded .pem file.

I had exactly the same problem on my Nexus 5X running Android 7.0. There was previously exported .p12 from Charles 3.11.5 (Help->SSL Proxying->Export Charles Root certificate and Private key). When I tried to install .p12 from phone (Settings->Security->Install from storage) it appears only under "User credentials" and never at "Trusted credentials", and of course SSL with Charles proxy did not work.

The total "how-to" for Android 7.0 would be like that:

1. Configure WiFi + proxy (how Charles requires it). Connect it.
2. On device, navigate with Chrome to http://charlesproxy.com/getssl, accept request for download .pem, then press "open", it launches "Certificate installer" app. Use it to install the certificate as "VPN and apps".
3. Put the attribute android:networkSecurityConfig="@xml/network_security_config" to <application> at Manifest.xml
4. Create res/xml/network_security_config.xml with content from the first post (it is totally correct).
5. Launch Charles and app and have fun.

P.S. Check date/time on the device. It should be correct.

Comments

[CommonsWare]

As a quick test, try adding/moving the user one into the <base-config> and see if that changes matters. It shouldn't, but it'll only take a moment to try.

[Morrison Chang]

Wouldn't you need to add the explicit CA for Charles in developer.android.com/training/articles/security-config.html : 'Trusting Additional CAs' as the Charles CA is a self generated one and wouldn't be in the Android System trust chain?

[mbonnin]

@Morisson Chang: I don't want to embed the Charles CA into the app, I want to be able to add it manually on my development phone like I used to do. From Settings -> Security -> Install from storage

[Alex Klyubin]

1. How are you referencing the network_security_config.xml in your app's AndroidManifest.xml? 2. Would you please link the .pfx file? I presume it contains only the CA cert (no private keys) and thus should be fine to publish. I'm asking because, if the .pfx contains a private key, it'll be assumed to be a client cert file and thus CA from the file won't be installed as trusted for server authentication.

[mbonnin]

@Alex Klyubin: My original .pfx file indeed contained a private key. I just switched to .crt with just one cert inside. No luck so far. I'll try harder, reboot all the things and keep you updated.

[mbonnin]

@AlexKlyubin hurra, it's working now :). I guess I needed to reboot my charles or app or whatever other thing. I was confused because I have installed the same .pfx file on countless android devices before nougat and they were recognized ok. But everything is good now. Could you write an answer so I can approve it ?

[mbonnin]

Regarding the PS: the documentation says Trust anchors specified in debug-overrides are added to all other configurations. So I was under the impression that it added to 'base-config' and not replaced it completely ?

[Alex Klyubin]

Oh, you're right. You don't need to explicitly list "system" trust anchors in debug-overrides.

[David Ferrand]

An important note on this great answer: if your app is targeting API 23 or lower, you don't need steps 3 and 4. In fact you won't be able to compile the network-security-config stuff if you have API 23 or lower.

[Adam Link]

This is a great solution for Android 7.0. Much clearer than the official Charles Proxy information.

[AdamHurwitz]

This answer is unclear. Please refer to @stkent's answer below.

[satyajit]

Likely only me but these steps do not work for me anymore in 7.1.1. I recently did an update

[Dick Lucas]

@satyajit This solution is working for me on Android 7.1.1 on a Google Pixel.

[satyajit]

with or without steps 3 and 4?

[vikramvi]

I'm struck in this weired state where after downloading certificate, tapping on certificate it asks for PIN, even after entering correct PIN multiple times nothing happens ( Motorola 7.0 ). I tried removing PIN on device, restarted, tried same step each time I am taken to "Confirm your PIN" and just gets struck there.

[Alexander Skvortsov]

@vikramvi, Probably something wrong with the phone. Did you try another one?

[StarWind0]

Agree this answer explains what is happening but is very unclear what steps one should do.

[Minji Song]

What if I want to see the proxy log, not from my apps? I can't do the step 3, what should I do?

[mbonnin]

This is most likely due to the nework_security_config.xml (developer.android.com/training/articles/security-config)

[Madhan]

I am facing the same issue. Did you resolve this? @kreker