iOS 11: ATS (App Transport Security) no longer accepts custom anchor certs?

Question

I am leasing a self signed certificate using NSMutableURLRequest and when the certificate is anchored using a custom certificate with SecTrustSetAnchorCertificates IOS 11 fails with the following error message:

refreshPreferences: HangTracerEnabled: 1
refreshPreferences: HangTracerDuration: 500
refreshPreferences: ActivationLoggingEnabled: 0 ActivationLoggingTaskedOffByDA:0
ATS failed system trust
System Trust failed for [1:0x1c417dc40]
TIC SSL Trust Error [1:0x1c417dc40]: 3:0
NSURLSession/NSURLConnection HTTP load failed (kCFStreamErrorDomainSSL, -9802)
Task <721D712D-FDBD-4F52-8C9F-EEEA28104E73>.<1> HTTP load failed (error code: -1200 [3:-9802])
Task <721D712D-FDBD-4F52-8C9F-EEEA28104E73>.<1> finished with error - code: -1200

What used to work for IOS 10 no longer works in IOS 11.

I am aware that IOS 11 no longer supports the following:

• RC4 3DES-CBC AES-CBC
• MD5 SHA-1
• <2048-bit RSA Pub Keys - All TLS connections to servers
• http://
• SSLv3
• TLS 1.0
• TLS 1.1

And the certificate does not use these except for one fingerprint, which is SHA-1, but a SHA-256 fingerprint is also listed.

And by adding the following we can bypass the ATS (App Transport Security) error:

<key>NSAppTransportSecurity</key>
<dict>
        <key>NSExceptionDomains</key>
        <dict>
            <key>mydomain.com</key>
            <dict>
                <!--Include to allow subdomains-->
                <key>NSIncludesSubdomains</key>
                <true/>
                <key>NSExceptionRequiresForwardSecrecy</key>
                <false/>
            </dict>
        </dict>
</dict>

By installing the root / anchor certificate onto the phone itself also works without the need to whitelist the mydomain.com.

Does this mean that ATS no longer supports self-signed certificates?

The following worked in IOS 10:

SecTrustSetAnchorCertificates(serverTrust, (__bridge CFArrayRef)certs);

Using nscurl on a Mac shows many failures, and after installing the root certificate into the "System" Keystore, nscurl succeeds. I did this on macOS 10.12.6.

nscurl --verbose --ats-diagnostics https://

How can I make this work with a custom certificate, but without the need to install certificates or whitelist the domain?

Solution

Some time ago macOS started enforcing a requirement that CA certificates can't also be used as end-entity (eg webserver) certificates. Is it possible that iOS added this requirement between 10 and 11?

If so, the workaround is simple: you create your self-signed CA certificate, and use that certificate to issue the webserver certificate. The CA certificate (basicConstraints: CA=True) is the trust anchor that goes in your trust store; the end-entity certificate (omit basicConstraints; extendedKeyUsage=serverAuth) is presented by the web server. You're just not allowed to use the exact same certificate for both any more.

(This should be a comment but I don't have enough points to comment yet.)