SPF and DKIM help: Do the FAIL reports from DMARC indicate an issue?

Question

I am having trouble determining if my SPF and DKIM are configured properly. Here are key details:

• My domain is mysteryscience.com
• We send mail from google apps, from SendGrid, and from Intercom. All seem to be working properly, although I do hear cases of our emails getting flagged as spam which is why I'm investigating this.
• I have enabled SPF, DKIM, and DMARC
• My SPF record seems to be semantically correct (checked here: http://www.kitterman.com/spf/validate.html)
• My SPF TXT record is: v=spf1 ip4:198.21.0.234 include:_spf.google.com include:spf.mail.intercom.io -all
• 198.21.0.234 is my dedicated IP address for sending through SendGrid (mail.mysteryscience.com is my CNAME forwarding to them)

I have enabled DMARC and I'm reviewing the emails I get from various mail servers. While reviewing my results from Google.com I noticed a bunch of SPF and DKIM fails. It looks like these may have been rejections of legitimate emails I sent, but I'm not sure how to read this file. Here are a few of the results, note the "fail" on a few of the < dkim > and < spf > lines. And here is a dmarcian processed version of the XML file: https://dmarcian.com/dmarc-xml/details/Ybk591jex3JpVBmW/

<record>
<row>
  <source_ip>207.46.163.143</source_ip>
  <count>1</count>
  <policy_evaluated>
    <disposition>none</disposition>
    <dkim>pass</dkim>
    <spf>fail</spf>
  </policy_evaluated>
</row>
<identifiers>
  <header_from>mysteryscience.com</header_from>
</identifiers>
<auth_results>
  <dkim>
    <domain>mysteryscience.com</domain>
    <result>pass</result>
  </dkim>
  <spf>
    <domain>granderie.ca</domain>
    <result>pass</result>
  </spf>
</auth_results>
</record>
<record>
<row>
  <source_ip>209.85.212.178</source_ip>
  <count>1</count>
  <policy_evaluated>
    <disposition>none</disposition>
    <dkim>fail</dkim>
    <spf>pass</spf>
  </policy_evaluated>
</row>
<identifiers>
  <header_from>mysteryscience.com</header_from>
</identifiers>
<auth_results>
  <spf>
    <domain>mysteryscience.com</domain>
    <result>pass</result>
  </spf>
</auth_results>
</record>
<record>
<row>
  <source_ip>2607:f8b0:4001:c05::232</source_ip>
  <count>1</count>
  <policy_evaluated>
    <disposition>none</disposition>
    <dkim>pass</dkim>
    <spf>fail</spf>
  </policy_evaluated>
</row>
<identifiers>
  <header_from>mysteryscience.com</header_from>
</identifiers>
<auth_results>
  <dkim>
    <domain>mysteryscience.com</domain>
    <result>pass</result>
  </dkim>
  <spf>
    <domain>mail.mysteryscience.com</domain>
    <result>fail</result>
  </spf>
</auth_results>
</record>
<record>
<row>
  <source_ip>198.236.20.44</source_ip>
  <count>1</count>
  <policy_evaluated>
    <disposition>none</disposition>
    <dkim>pass</dkim>
    <spf>fail</spf>
  </policy_evaluated>
</row>
<identifiers>
  <header_from>mysteryscience.com</header_from>
</identifiers>
<auth_results>
  <dkim>
    <domain>mysteryscience.com</domain>
    <result>pass</result>
  </dkim>
  <spf>
    <domain>mail.mysteryscience.com</domain>
    <result>fail</result>
  </spf>
</auth_results>
</record>
<record>
<row>
  <source_ip>209.85.212.175</source_ip>
  <count>1</count>
  <policy_evaluated>
    <disposition>none</disposition>
    <dkim>fail</dkim>
    <spf>pass</spf>
  </policy_evaluated>
</row>
<identifiers>
  <header_from>mysteryscience.com</header_from>
</identifiers>
<auth_results>
  <spf>
    <domain>mysteryscience.com</domain>
    <result>pass</result>
  </spf>
</auth_results>
</record>
<record>
<row>
  <source_ip>209.85.215.44</source_ip>
  <count>1</count>
  <policy_evaluated>
    <disposition>none</disposition>
    <dkim>fail</dkim>
    <spf>fail</spf>
  </policy_evaluated>
</row>
<identifiers>
  <header_from>mysteryscience.com</header_from>
</identifiers>
<auth_results>
  <spf>
    <domain>nurturingwisdom.com</domain>
    <result>fail</result>
  </spf>
</auth_results>
</record>
<record>
<row>
  <source_ip>2607:f8b0:4003:c06::236</source_ip>
  <count>2</count>
  <policy_evaluated>
    <disposition>none</disposition>
    <dkim>pass</dkim>
    <spf>fail</spf>
  </policy_evaluated>
</row>
<identifiers>
  <header_from>mysteryscience.com</header_from>
</identifiers>
<auth_results>
  <dkim>
    <domain>mysteryscience.com</domain>
    <result>pass</result>
  </dkim>
  <spf>
    <domain>ssanpete.org</domain>
    <result>none</result>
  </spf>
</auth_results>

Can anyone help me determine if these SPF and DKIM fails are problematic?

Solution 1

I ran some queries like spfquery --mfrom mail.mysteryscience.com -ip 2607:f8b0:4001:c05::232 on the results you provided. It appears you have not configured SPF for mail.mysteryscience.com to allow google to deliver email for that domain. That explains the SPF failures for deliveries from Google. The query above is based on the domains listed in the record.

There are some records that do appear to be Spam, so they should be in the list.

You may have similar problems with email not having appropriate DKIM signatures. Some may be Spam, or you may have delivery paths that do not sign the email with an expected signature.

Solution 2

1. Parse your DMARC XML's somewhere like dmarcian, so your information is human readable
2. Google Outbound gateway goes where? Do you have a gateway configured? If not there's nothing to change.
3. DKIM is failing, are you signing your emails without a public key published? check your DNS.
4. SPF records for subdomains, you only need this if the mail server accepts emails and sends NDR's. a typical subdomain record would be:

mail.example.com. IN A 93.184.216.34

mail.example.com. IN TXT "v=spf1 a -all"

Comments

[peterh]

@MadHatter Although there are undeniable essential similarities, I think this question is much more specialized - and has a much better quality.

[MadHatter]

@peterh you may misunderstand the function of canonical questions on SF; I recommend the first para of the linked document. Nevertheless, unless four others agree with me, this question will stay open, so it's not done and dusted yet.

[Keith Schacht]

I revised my question to help clarify further.

[rubynorails]

Just remove your DMARC record, and watch your problems disappear. See my answer below.

[Keith Schacht]

But deliveries from Google are being sent from mysteryscience.com. It's only deliveries from SendGrid that are being sent from mail.mysteryscience.com. SendGrid controls that SPF (since it's just a CNAME to them). On the IP address you referenced, the "header_from" line a few rows down says "mysteryscience.com" so it seems to confirm what I'm saying. However, I do see a few lines down the SPF fail lists the domain mail.mysteryscience.com. I'm clearly misunderstanding something.

[Keith Schacht]

1. Very helpful, thanks. Here is the dmarcian parsed version of the raw XML I pasted above. I'm having trouble interpreting the results. I see many fails in my raw XML, but it's not obvious if I have actual problems. Can you tell? dmarcian.com/dmarc-xml/details/Ybk591jex3JpVBmW

[Keith Schacht]

2. You're right, I was mis-reading that google gateway comment. I don't have one configured.

[Keith Schacht]

3. You're right, thanks. I had DKIM configured for mail.mysteryscience.com but not for mysteryscience.com. I just added this.

[Keith Schacht]

4. After researching more, I can't add SPF records for subdomains because mail.mysteryscience.com is a CNAME to SendGrid. This delegates the SPF to SendGrid. I don't think this is my issue.

[Keith Schacht]

You mentioned "there are some records that do appear to be spam". How can you tell? Is it possible the 2607:f8b0:4001:c05::232 record is also a spam? Maybe every fail in my XML is legitimate?

[Jacob Evans]

what? Why do you cname mail to sendgrid? I use sendgrid on my blog and I never had to do something like that, I have a dkim sector that is a cname to sendgrid's and include their SPF in my domain, which is all anyone needs to authenticate email.

[Keith Schacht]

Jacob, I'm just following SendGrid's directions on that. If I don't white label, they're claiming the "from" will add "on behalf of" for some email clients because the from domain is different than the mail server sending. This screenshot from their control panel summarizes the difference: dropbox.com/s/is8msl3ly1uhg1e/…

[Jacob Evans]

I was in my SendGrid and saw that, which is why I went back to mandrill