Is it ok to use self-signed certificates for smtp transport?

Question

By "by ok to use" I mean:

• are MTA agents receiving emails from my server going to reject my email
• if not, are they going to make other kind of bad treatment for my emails (marking as spam, unsafe and so...)?

...or is it just better idea to stick with non-encrypted emails?

Solution 1

I ran an MTA with a self-signed certificate for a couple of years, until real ones got cheap enough that I could no longer be bothered to do so, and I didn't have a single rejection because of the unsigned certificate in all that time. I never had a single complaint about a mail being marked as spam because of it, either; if anything, using TLS often seems to mark you out as a non-spamming professional.

In my opinion, it is definitely worth enabling SMTP TLS if you can, whether or not you pay for a third-party-signed certificate.

Edit in response to your comment: It's not that someone couldn't choose to restrict inbound mail on that basis; I've never come across it, is all. A third-party-signed certificate is still useful to prove there is no man-in-the-middle attack happening; but that doesn't seem to be a serious problem in the MTA world at this time. If that starts to change, we could well find that people start to insist on signed certificates.

Security exists to address threats, so if the threat model changes, the range of sane and proportionate security responses will change with it.

Solution 2

Like MadHatter said, at the moment encryption using a self-signed certificate is actually a step up in comparison to many other small time SMTPs if you are talking SMTP-relaying.

There are a few drawbacks, however, but not of the kind you expect. The most important one being your SMTP-clients choking on encryption. There are many small SMTP-clients deployed in off-the-shelf solutions for common business problems that don't like SMTP encryption and fail -- often silently! So, if you are using standard software with built-in mail clients, check them before you switch.

Other than that I'd say, if you are concernced about your mail being marked as spam, consider setting up SPF and DKIM. That helps very much.

Solution 3

The question is certainly somehow dated, but earns a revisit and should receive an updated answer.

Regarding the impact on outbound delivery and spam filtering: According to Google's transparency report on e-mail encryption in transit to/from Google, about 90% of outbound and 93% of inbound e-mails are transmitted using TLS. So with some respect, TLS for e-mail is becoming the norm and considering the high volume of spam, just using TLS as another scoring value in a spam filter is not of much value. As today inbound TLS certificates are usually not being used to authenticate outbound connections, the certificate's contents (metadata, who signed the cert,…) don't impact spam filters on outbound e-mail at all.

Historically, MTAs do have limited options to handle any certificate errors: reject sending the message at all or revert to send the message via plaintext connections. Rejecting the message will result in user support inquiries while reverting to plaintext doesn't provide security benefits over using at least "some" level of encryption. Accordingly, most MTAs have been using an "opportunistic TLS" configuration: prefer using TLS if available, regardless of the exact certificate to validate.

However, there has been some slight shift visible in the industry over the past decade.

Since 2010, the German Association of the Automotive Industry has been recommending using at least opportunistic TLS for e-mail and recommends mandatory TLS for confidential e-mail.

Some e-mail service providers also started introducing optional mandatory outbound TLS, for example German e-mail provider posteo.de did so in 2016.

Posteo's feature can be enabled per user, and if the message cannot be transmitted using an encrypted connection, the sending user will receive an error message. The user then can decide to either (temporarily) disable that feature (e.g. to ask the recipient on upgrading their mail service). Within 2 weeks after the feature has been released, quite a few e-mail services started supporting (opportunistic) TLS.

So not offering TLS at all and just relying on unencrypted mail may result in partial inbound issues with some senders. However, most of those cases at that time did seem to be fine with self-signed certificates.

A research paper on E-Mail security from 2015 discovered 64% of domains do use STARTTLS-supporting mail servers whose certificate did validate with Mozilla's NSS root store - so 36% have been using self-signed, invalid or expired certificates.

More recently, MTA strict transport security did improve the overall trend: if domains hosted on your mail server do make use of MTA strict transport security (RFC 8461), those domains have also agreed to enforce inbound mail via TLS using trusted certificates. RFC 8461's section 4.2 does clearly tell

The certificate presented by the receiving MTA MUST not be expired and MUST chain to a root CA that is trusted by the Sending MTA.

Of course, this only applies to domains using MTA strict transport security.

So after all: As of today, issues with outbound e-mail are unlikely. Regarding inbound e-mail, you're highly encouraged to use proper TLS certificates issued by a public CA, as not doing so may result in some delivery issues. Some of those might be addressable within your domain (like MTA-STS), but others may not be (like sending servers enforcing mandatory outbound TLS).

Comments

[Miloš Đakonović]

Thank you. That I didn't know: that there is no restrictive treatment of self-signed certificates in MTA to MTA traffic, unlike other cases. But, what is real (CA-signed) certs useful for if we can relay on our self-signed?

[quadruplebucky]

Agree 100%. I've never encountered a recipient daemon that wanted to verify my key. And since I'm ready to give to anybody with an MX, I don't care about theirs. This will change.

[Michael Hampton]

I still run an MTA with self-signed certificates. It seems to be quite common.

[hultqvist]

How does using TLS relate to marking as spam? I would assume it is about identifying the receiving end, unless the sender is using its cert in the encrypted connection. Also in what percentage did the sender choose to try TLS at all rather than sending it in the clear?