Amazon EC2/SES SMTP Timeout

Question

I have an issue when trying to send emails from my EC2 instance using SMTP to SES. For some reason I am getting sporadic timeout issues, where I can no longer contact the SMTP host. It is important to note that sending through SES works probably 75% of the time.

I will start with some details. My SES account is working most of the time. The sender email has been verified, and my limits have been increased to 10k/day, 5 emails/second. I am under the impression that I would get a limit-specific error if this was related to my limits. For my SMTP configuration, I am using posfix w/TLS. I have posted a very similar post on the official AWS SES forums, but have not had any success there yet. The information for that post is at the bottom of this post.

Here is an example of a failure from this morning. All of the commands I ran below were run from the EC2 instance that I am trying to send mail from. Sorry for the novel-sized post, I just want to make sure I include everything.

An email I sent to myself was not being delivered:

Jun 25 06:16:36 intranet01 postfix/smtp18832: 9E00C230DA: to=<myemailaddress>, relay=none, delay=150, delays=0.02/0.01/150/0, dsn=4.4.1, status=deferred (connect to email-smtp.us-east-1.amazonaws.comhttp://107.22.187.122:25: Connection timed out)

And when I seen the failure I tried connecting to the email host over port 25. I could not connect:

root@intranet01 sbin# telnet email-smtp.us-east-1.amazonaws.com 25  
Trying 174.129.28.151...  
^C

A few minutes later my deferred email finally went through:

Jun 25 06:23:14 intranet01 postfix/smtp18861: 9E00C230DA: to=<myemailaddress>, relay=email-smtp.us-east-1.amazonaws.comhttp://184.73.218.23:25, delay=548, delays=548/0.02/0.21/0.36, dsn=2.0.0, status=sent (250 Ok 0000013823cf7441-83710873-e946-4c80-8a54-0dd72bae6f30-000000)
Jun 25 06:23:14 intranet01 postfix/qmgr3972: 9E00C230DA: removed

And now I can connect to port 25:

root@intranet01 sbin# telnet email-smtp.us-east-1.amazonaws.com 25  
Trying 107.20.152.208...  
Connected to email-smtp.us-east-1.amazonaws.com.  
Escape character is '^]'.  
220 email-smtp.amazonaws.com ESMTP SimpleEmailService-222567251  
^]

For kicks, I decided to look at the SES host endpoint. It turns out it is just an ELB cname, with A records that point to interfaces in multiple AZ's.

root@intranet01 sbin# dig email-smtp.us-east-1.amazonaws.com  

; <<>> DiG 9.7.3-P3-RedHat-9.7.3-8.P3.17.amzn1 <<>> email-smtp.us-east-1.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8592
;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:  
;email-smtp.us-east-1.amazonaws.com. IN A  

;; ANSWER SECTION:  
email-smtp.us-east-1.amazonaws.com. 54 IN CNAME ses-smtp-prod-335357831.us-east-1.elb.amazonaws.com.  
ses-smtp-prod-335357831.us-east-1.elb.amazonaws.com. 60 IN A 174.129.200.82  
ses-smtp-prod-335357831.us-east-1.elb.amazonaws.com. 60 IN A 184.73.219.75  
ses-smtp-prod-335357831.us-east-1.elb.amazonaws.com. 60 IN A 107.20.152.208  
ses-smtp-prod-335357831.us-east-1.elb.amazonaws.com. 60 IN A 107.20.160.81  
ses-smtp-prod-335357831.us-east-1.elb.amazonaws.com. 60 IN A 107.20.203.50  
ses-smtp-prod-335357831.us-east-1.elb.amazonaws.com. 60 IN A 107.22.229.233 
ses-smtp-prod-335357831.us-east-1.elb.amazonaws.com. 60 IN A 174.129.6.189  
ses-smtp-prod-335357831.us-east-1.elb.amazonaws.com. 60 IN A 174.129.28.151

I tried sending another message and it failed. This time I looped through each of the A record addresses that the SES cname returned. I could not connect to any of them. During this time I also tried connecting from my local machine (not my EC2 instance), and it worked just fine.

root@intranet01 sbin# telnet email-smtp.us-east-1.amazonaws.com 25
Trying 174.129.28.151...
^C
root@intranet01 sbin# telnet email-smtp.us-east-1.amazonaws.com 25
Trying 174.129.6.189...
^C
root@intranet01 sbin# telnet email-smtp.us-east-1.amazonaws.com 25
Trying 107.22.229.233...
^C
root@intranet01 sbin# telnet email-smtp.us-east-1.amazonaws.com 25
Trying 107.20.203.50...
^C
root@intranet01 sbin# telnet email-smtp.us-east-1.amazonaws.com 25
Trying 107.20.160.81...
^C
root@intranet01 sbin# telnet email-smtp.us-east-1.amazonaws.com 25
Trying 107.20.152.208...
^C
root@intranet01 sbin# telnet email-smtp.us-east-1.amazonaws.com 25
Trying 184.73.219.75...
^C
root@intranet01 sbin# telnet email-smtp.us-east-1.amazonaws.com 25
Trying 174.129.200.82...
^C
root@intranet01 sbin# telnet email-smtp.us-east-1.amazonaws.com 25
Trying 174.129.28.151...
^C
root@intranet01 sbin# telnet email-smtp.us-east-1.amazonaws.com 25
Trying 174.129.6.189...
^C
root@intranet01 sbin# telnet email-smtp.us-east-1.amazonaws.com 25
Trying 107.22.229.233...
^C
root@intranet01 sbin# telnet email-smtp.us-east-1.amazonaws.com 25
Trying 107.20.203.50...
^C
root@intranet01 sbin# telnet email-smtp.us-east-1.amazonaws.com 25
Trying 107.20.179.13...
^C
root@intranet01 sbin# telnet email-smtp.us-east-1.amazonaws.com 25
Trying 107.20.160.81...
^C
root@intranet01 sbin# telnet email-smtp.us-east-1.amazonaws.com 25
Trying 184.73.219.75...
^C
root@intranet01 sbin# telnet email-smtp.us-east-1.amazonaws.com 25
Trying 174.129.200.82...
^C

After waiting around 30 seconds I tried again, this time it worked.

root@intranet01 sbin# telnet email-smtp.us-east-1.amazonaws.com 25
Trying 174.129.28.151...
Connected to email-smtp.us-east-1.amazonaws.com.
Escape character is '^]'.
220 email-smtp.amazonaws.com ESMTP SimpleEmailService-222567251
^C^[
^]

telnet>

As I stated earlier, I posted a very similar post on the AWS SES forum. The post can be found below.

https://forums.aws.amazon.com/thread.jspa?threadID=97736&tstart=0

Also, it appears that I am not alone, as I found this other post on the SES forum, looks like the same problem.

https://forums.aws.amazon.com/thread.jspa?threadID=91260&tstart=0

I thought there might be some sort of EC2 SMTP limitation, so I filled out the "sending email from EC2" request form, but it seems silly to me since I am using Amazon's service, as opposed to a 3rd party. So far I have not heard anything back from Amazon after filling out the form.

Does anyone have any ideas? Thanks in advance.

Solution 1

So apparently EC2 has it's own limits. I assumed (incorrectly) that having production access to SES would also mean relaxed SMTP limitations from EC2, but as they are two completely separate products I guess that is not the case.

But as I stated in the last paragraph of my post, you can Request to Remove Email Sending Limitations to have these limits raised. I did that and the problem stopped (it took them around 5 hours to get my limits removed).

---

Update

The EC2 throttling is documented in Connecting to the Amazon SES SMTP Endpoint and actually constrained to port 25, so an alternative and immediate solution is simply using port 587 instead (it's a bit unfortunate that several official SES examples are using port 25 indeed):

Important

Elastic Compute Cloud (EC2) throttles email traffic over port 25 by default. To avoid timeouts when sending email through the SMTP endpoint from EC2, use a different port (587 or 2587) or fill out a Request to Remove Email Sending Limitations to remove the throttle.

Beware that this might be slightly outdated as well, insofar both the AWS Management Console and section Amazon SES SMTP Issues are referring to the more common alternative ports 465 and 587 only:

You are sending to Amazon SES from an Amazon EC2 instance via port 25 and you cannot reach your Amazon SES sending limits or you are receiving time outs — Amazon SES EC2 imposes default sending limits on email sent via port 25 and throttles outbound connections if you attempt to exceed those limits. To remove these limits, submit a Request to Remove Email Sending Limitations. You can also connect to Amazon SES via port 465 or port 587, neither of which is throttled.

Solution 2

If you're running a Django app and suspect Steffen's answer might be the cause - here is a quick litmus test:

In [1]: from django.core.mail.backends.smtp import EmailBackend
In [2]: from django.core.mail import EmailMultiAlternatives
In [3]: message = EmailMultiAlternatives(
   ...:     subject='testing the rate limit',
   ...:     body='this is a test',
   ...:     to=['[email protected]'],
   ...:     from_email='[email protected]',
   ...: )
In [4]: backend_587 = EmailBackend(port=587)
In [5]: backend_25 = EmailBackend(port=25)
In [6]: backend_587.send_messages([message])
Out[6]: 1
In [7]: backend_25.send_messages([message])  # hangs for a long time. Might even timeout

Sending the email from port 25 should hang. Sending the email from 587 should send quickly.

Comments

[BerggreenDK]

yup got our limits raised quickly and without any problems. So now we can send 25K daily as requested.

[Mausimo]

I already have my Email Sending Limitations removed. However, I was still being throttled on port 25 and receiving "the operation has timed out" exceptions. Switching to port 587 solved this. Thank-you!