Hotmail/Outlook dumping apparently valid emails

Question

I've written some software that operates a non-commerical contact list for clubs, where the server sends out admin mails to the entire club. The emails are received Ok on Yahoo, gmail, and AOL, but are silently thrown away by hotmail and outlook. They don't end up in the recipient's spam folder, and they're not bounced - they're literally just dumped.

I've spent hours on this, and I'm getting nowhere fast. Any suggestions on how to proceed, and what to look at next? What I know so far is:

1. The emails pass SPF
2. The emails pass DKIM
3. The hostname of the sending machine is griffon.foo.org, and this ends up as the Return-Path in the headers below (presumably sendmail sets the envelope sender to the hostname). The originating IP is identified as 1.2.3.4, and a DNS PTR lookup on 1.2.3.4 returns griffon.foo.org
4. There's no A record for griffon.foo.org, and the A record for foo.org returns 2.3.4.5, not 1.2.3.4. Is this the problem?
5. There's a slight mismatch between the email body From header, and the Return-Path: From is, in general, [email protected]

6. This is a new server, sending out occasional low-volume mails, and I can guarantee that no-one has ever marked an incoming mail as spam. mxtoolbox.com doesn't show 1.2.3.4 on any blacklists, and shows 2.3.4.5 on one list out of 88 (both are on the same machine)

7. The Microsoft servers accept the emails from sendmail, and responds that they are queued for delivery. It then just ignores them.

So it looks like it might be (4), but the MX records for foo.org must point to 2.3.4.5, for complex reasons. Any ideas? I've attached a set of headers received at Yahoo below. Thanks.

From [email protected] Wed Mar 25 17:28:23 2015
X-Apparently-To: [email protected]; Wed, 25 Mar 2015 17:28:32 +0000
Return-Path: <[email protected]>
Received-SPF: pass (domain of griffon.foo.org designates 1.2.3.4 as permitted sender)
X-YMailISG: <CUT>
X-Originating-IP: [1.2.3.4]
Authentication-Results: mta1045.mail.ir2.yahoo.com  from=foo.org; domainkeys=neutral (no sig);  from=foo.org; dkim=pass (ok)
Received: from 127.0.0.1  (EHLO griffon.bar.com) (1.2.3.4)
  by mta1045.mail.ir2.yahoo.com with SMTPS; Wed, 25 Mar 2015 17:28:32 +0000
Received: from griffon.foo.org (localhost.localdomain [127.0.0.1])
    by griffon.bar.com (8.13.8/8.13.8) with ESMTP id t2PHSNVN016920;
    Wed, 25 Mar 2015 17:28:24 GMT
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=foo.org;
    s=default; t=1427304504;
    bh=0TWfPqbvd2nyKtZkN+npokT1bvLtCD0MomG/Bd1zU8g=;
    h=Date:To:From:Subject;
    b=ZcSVHGuzxcBk8He2hzjT6t8gl9hT0cxUsGf7vNcsnF8S+OfNmZ+XuE240+v7JDNe+
     IArMT87aIuLNIp8A1CMOelWvEm7KKdYWKoCy/CDfbe4mcEfu2kWNteai4/5yExPewv
     gFV6OOYAv7uKYd44UjMhuWCZ8qqxD2WZROUeoN2E=
Received: (from me@localhost)
    by griffon.foo.org (8.13.8/8.13.8/Submit) id t2PHSNSu016919;
    Wed, 25 Mar 2015 17:28:23 GMT
Date: Wed, 25 Mar 2015 17:28:23 GMT
Message-Id: <[email protected]>
To: [email protected], [email protected], [email protected],
        [email protected], [email protected]
From: [email protected]
Subject: Test all (31)
Content-Length: 14

Solution

I fixed this by adding a new high-cost MX record for the domain at 1.2.3.4, which pointed to the mail server for 2.3.4.5. This ensures that the reverse DNS matches the forward DNS. This is basically a quick fix to get around the sendmail issue below.

I did find out some useful stuff about Hotmail and Outlook along the way, and what causes mails to be dumped. In my case, the underlying problem was related to how sendmail identifies itself, but this may be useful to anyone else coming this way. Note that DKIM is irrelevant (you can be delivered and get through spam testing without DKIM), and I didn't have to register with Microsoft.

1 - sendmail sets the SMTP 'MAIL FROM' to the user at the current hostname; ie. [email protected] (this is what you see in the Return-Path)

2 - hotmail and outlook will then do a reverse/PTR lookup on the connected IP address, and will silently dump the mail if it doesn't get back to the MAIL FROM domain name

3 - hotmail/outlook will then do an MX lookup on the 'From' server named in the last 'Received' header. If this doesn't match the MAIL FROM domain, the email is dumped

That seems to be it. (3) was causing my problem. sendmail uses gethostbyname to set the 'From' name, although it uses the current hostname to set MAIL FROM. I would have expected gethostbyname just to do a PTR lookup and get the right answer, but it was returning an old FQDN. Don't know why - possibly cacheing issues - but the new MX record fixes it for the moment.

Comments

[Jenny D]

If you're going to munge your headers, please use example.com, example.net etc instead of making up domain names. Unless you actually do own foo.org and bar.com, in which case -congrats on good domain names.

[sebix]

Please also accept your answer. Thanks for your detailed description on what seems to be the problem, there are many questions here regarding these mailservers with special behavior, ignoring standards.

[EML]

Thanks, but note that this isn't a complete answer on how to get mail delivered to Hotmail and Outlook. The reverse/forward DNS test and the MX lookup seem to be the most important, but you can pass this, and have a pass on SPF and DKIM, and have completely innocuous mail contents, and still have your mail silently dumped. I'm having to ask recipients to whitelist me (which always works), and/or move to a mail provider who doesn't think they own the entire mail system.

[thomasrutter]

#2 and #3 in this answer seem to be claiming that Outlook doesn't accept you having email hosted on a service that hosts mail for more than one domain. A sending server can only have one PTR record, but a server that sends mail on behalf of multiple domains will need to use a different MAIL FROM domain for each. Are you sure that it's the domain mentioned in the MAIL FROM command that is important (not HELO)? That is essentially the envelope sender and in the typical case will match the "From:" email header, ie it will be different for the different domains that the server handles mail for.