Too many DNS lookups in an SPF record

Question

My website needs to send out emails with Google Apps, SendGrid and MailChimp services. Google Apps is used to receive and read incoming email to my domain.

I need to set the SPF record for my domain. The following is syntactically correct (not sure about A and MX tokens):

"v=spf1 a mx include:_spf.google.com include:servers.mcsv.net include:sendgrid.net ~all"

But if I test it with http://www.kitterman.com/getspf2.py I get

PermError SPF Permanent Error: Too many DNS lookups

Similar problem as http://www.onlineaspect.com/2009/03/20/too-many-dns-lookups-in-an-spf-record/

How can I optimize/rewrite my SPF record?

Solution 1

So, I've never had to do this before, but based on the article you sent over, this is what I came up with.

We started with:

v=spf1 a mx include:_spf.google.com include:servers.mcsv.net include:sendgrid.net ~all

We get 10 total lookups before we throw the Too many DNS lookups error:

  2 (Initial TXT & SPF Lookups)
  2 (a & mx Lookups)
  1 (_spf.google.com)
  1 (servers.mcsv.net)
 +1 (sendgrid.net)
 -----------------
  7 Lookups

So without even following the included SPF records, we have 7 lookups.

---

Now, let's dive a level deeper.

1. _spf.google.com

The google SPF record evaluates to:

v=spf1 include:_netblocks.google.com include:_netblocks6.google.com ?all

Each of which resolve to the following values:

# _netblocks.google.com
v=spf1 ip4:216.239.32.0/19 ip4:64.233.160.0/19 ip4:66.249.80.0/20 ip4:72.14.192.0/18 ip4:209.85.128.0/17 ip4:66.102.0.0/20 ip4:74.125.0.0/16 ip4:64.18.0.0/20 ip4:207.126.144.0/20 ip4:173.194.0.0/16 ?all

# _netblocks6.google.com
v=spf1 ip6:2607:f8b0:4000::/36 ip6:2a00:1450:4000::/36 ?all

So google gives us 2 more lookups, bringing the total up to 9 Lookups.

2. servers.mcsv.net

Mailchimp is a bit of a doosey because it adds a whole 3 extra lookups:

v=spf1 include:spf1.mcsv.net include:spf2.mcsv.net include:spf.mandrillapp.com ?all

I would imagine that depending on what you're sending through Mailchimp, you might be able to remove one or two of these records (but you'll have to evaluated that yourself).

Anyway, those resolve to the following:

# spf1.mcsv.net
v=spf1 ip4:207.97.237.194/31 ip4:207.97.238.88/29 ip4:207.97.240.168/29 ip4:69.20.10.80/29 ip4:69.20.41.72/27 ip4:74.205.22.1/27 ip4:69.20.90.0/26 ?all

# spf2.mcsv.net
v=spf1 ip4:204.232.163.0/24 ip4:72.26.195.64/27 ip4:74.63.47.96/27 ip4:173.231.138.192/27 ip4:173.231.139.0/24 ip4:173.231.176.0/20 ip4:205.201.128.0/24 ?all

# spf.mandrillapp.com
v=spf1 ip4:205.201.136.0/24 ip4:205.201.137.0/24 ?all

This brings us up to a total of 12 Lookups (Which is two over the limit already).

2. sendgrid.net

SendGrid ends up being the fewest number of additional lookups for us.

v=spf1 ip4:208.115.214.0/24 ip4:74.63.202.0/24 ip4:75.126.200.128/27 ip4:75.126.253.0/24 ip4:67.228.50.32/27 ip4:174.36.80.208/28 ip4:174.36.92.96/27 ip4:69.162.98.0/24 ip4:74.63.194.0/24 ip4:74.63.234.0/24 ip4:74.63.235.0/24 include:sendgrid.biz ~all

So the only additional lookup here is sendgrid.biz, which evaluates to:

v=spf1 ip4:208.115.235.0/24 ip4:74.63.231.0/24 ip4:74.63.247.0/24 ip4:74.63.236.0/24 ip4:208.115.239.0/24 ip4:173.193.132.0/24 ip4:173.193.133.0/24 ip4:208.117.48.0/20 ip4:50.31.32.0/19 ip4:198.37.144.0/20 ~all

This brings our grand total up to 14 lookups.

---

So our grand total is 14 Lookups. We need to get that down to 10. I've outlined a couple of options below, you may need to use more than 1 of them to get it down.

1. Directly include some of the redirected spf records. Now that we know which servers the spf records redirect to, you could cut out the middleman and include them directly. Note: If any of the services end up changing their SPF records, you'll have to go through the process of updating yours manually.

2. Remove some of the services that you're using. Not sure what your use case is for having all of these services, but there's definitely some overlap that you might be able to use. For instance, SendGrid supports (1) transactional outgoing mail, (2) newsletter / marketing emails, and (3) incoming mail. So there may be some reducible redundancy.

3. Remove the MX record if it is redundant. Depending on your setup, the MX lookup can be redundant.

Hope this helps!

Solution 2

Swift's answer is excellent.

A technique which is not mentioned above is to look at whether separate subdomains with their own SPF records can be used for systems that send mail via these different routes.

E.g. if the domain is example.com, have google apps send from addresses like [email protected]. Then there can be a SPF record for gapps.example.com which includes _spf.google.com, and _spf.google.com can be removed from the main example.com SPF record , which reduces lookups by 3.

Solution 3

Have a look at SPF-tools* which help with reassembling the SPF record from the original one that uses includes into a static one containing merely ip4 and ip6 fields. It can be easily coupled together with locally-run DNS server or hosted DNS service using their API to keep everything in sync with the upstream includes.

*I am the author (now along with other contributors) and it is open source under Apache 2.0 license.

Comments

[Robert]

About the option 3: I'm not expert about MX records. Since I use only Google Apps to receive email, can I set the SPF to take MX only from Google and not SendGrid and MailChimp?

[Robert]

This works, for example, but doesn't include MX (so I think I need a MX for Google) v=spf1 a include:_spf.google.com include:servers.mcsv.net include:sendgrid.net ~all

[Swift]

I honestly don't know if MX is required for google. You'd have to consult their documentation for details. I know you need a separate MX record for incoming from SendGrid (not sure why you would need it in the SPF lookup)

[tripleee]

In SPF, the MX entry means to trust whichever host is designated as the MX for your domain. If you don't have a separate MX record for your domain, or if it is already covered by the other SPF rules you have, you don't need to include it. On the other hand, if you do have a designated MX for your domain which handles outgoing mail, not authorizing it to send mail would seem ... counter-productive.

[Xiong Chiamiov]

For anyone looking, here is the relevant section of the RFC limiting MX lookups to 10.

[cherdt]

The relevant section of the RFC for DNS lookups in general is 10.1. Processing Limits

[Steve365]

For anyone reading this now note that RFC 4408 is now obsolete. The current RFC is 7208 and the processing limits can be found here: tools.ietf.org/html/rfc7208#section-4.6.4

[Steve365]

Also note the example above is slightly wrong, the "Initial TXT & SPF Lookups" are not included in the limit of 10 lookups (at least that's how I interpret the RFC...!)

[alexcline]

Looks like spfproxy.org is not in service anymore.

[Mike]

Need to be careful with this. Technically an SPF record longer than 255 characters is also invalid.

[Xiong Chiamiov]

It really seems like we need a modern replacement for SPF that takes into account all these limitations that may have seemed reasonable at the time, but we bump into regularly nowadays.

[Martijn]

Are include= counted as lookups? If they don't have some sort of spf.example.com subdomain specified, they may lookup a lot of TXT records.

[Jose Gallo]

Hi @Swift , how do you know the extra lookups?, I mean, how do you know that _spf.google.com evaluates also _netblocks.google.com include:_netblocks6.google.com and those ones resolve the ips you mention? Thanks indeed

[Magnus]

For others wondering how to find out exactly which lookups each include resolves to, I found the SPF-checker at dmarcanalyzer.com to be very useful. Run the tool on your domain, see what each include itself includes, and edit your DNS record to include those directly instead. I needed this to be able to add Firebase + Gmail to a Hostgator site, which requires including websitewelcome.com. Turns out most includes were duplicated two or more times, resulting in "Too many DNS lookups".