What diagnosis steps can I do if my emails send, but are not received, not even as spam?

Question

I have a problem where all emails to certain recipients send but never arrive, not even as spam, with no errors, while emails to everyone else work fine. I'm stumped and am looking for ways to diagnose what is happening.

• I sent some emails to a client last week, from Outlook 2016. I've now found that they were never received. I've tried emailing others on their domain and it looks like none of them receive my emails, but others on other domains do
• I've checked my "sent" folder, and they appear identical to all other sent mail. I had no delivery reports or anything similar, and they're in the "sent" folder not "outbox". I've also tried CCing myself in my latest emails to these people - they definitely send.
• I've checked with the client in question, and they never received anything from me, not even in a spam folder. I can receive emails from them, but they receive nothing from me - not even my replies to their emails.
• The first email had two small (500kb) PDF attachments, but the same fate befell follow-up emails that had no attachments. No images or links in any emails. They were normal work-related emails to between one and three individuals who I've had email exchanges with in the past through the same email address and Outlook. They're also in same country as me.
• There were no delivery error replies or anything similar. The first such email was sent 8:55am last Friday, so over 5 days ago, and nothing has been received on my side or theirs.
• Their emails to me arrive fine - and in fact, my first emails to them which failed were replies to their own emails. I've also had emails to this domain received normally as recently as two weeks ago.

• I've tried a variety of test emails to this domain and nothing goes through:

  • Innocent emails that simply say "This is a test email" and similar never arrive
  • Emails from Webmail and my Android mail app don't arrive same as emails from Outlook (and also don't give delivery reports - everything fails silently)
  • Emails sent using my phone's 3G suffer the same fate as emails sent using my WiFi
  • I also created a new email account on the same domain (for example [email protected] to go alongside my usual [email protected]), and it had exactly the same problem (tested using webmail).
  • To test if I might have some kind of messed up SMTP settings that block bounceback delivery reciepts, I sent an email to [email protected] reasoning that it probably doesn't exist. I recieved a perfectly normal "Mail delivery failed: returning message to sender" bounceback - so I am able to receive bouncebacks, I'm just not being sent them from this domain for some reason.
  • Emails to them from, for example, my personal Gmail account, are received fine (so I'm using this in the meantime until this problem is resolved)

The email is from my own domain - I've sent emails to other people from the same email address and the same Outlook and they received fine. Gmail occasionally marks them as spam, which I'm looking into, but other clients seem to have no problems.

Apart from that, I can't see anything to go on. I'm sure there's not enough here to diagnose my problem so I'm asking not for a solution, but for diagnosis steps I can take, for example:

• Is there anything "under the hood" I can look at in Outlook, like a sending report or log?
• Are there any types of server or domain-related logs I should look at which might be relevant? My domain is assigned to an SMTP server on a Centos VPS.
• Are there any types of blacklist or security intervention I should know about and check for that would cause an email to not even get as far as a spam folder?

---

I've seen this question Emails not being received by some people, which is similar but with two differences:

• They're using a mass-mailing system, I'm using regular Outlook, one email at a time.
• The accepted answer blames greylisting - however, my first missing email was from last Friday (five days ago), and apparently greylisting delays emails for between 15 minutes and "a few days".

---

As Tyson suggested, I've tried http://mxtoolbox.com/ but unfortunately it didn't give any clues (at least, not any clues I can see). In case I missed something, here are the results:

Blacklist check

Checking XX.XX.XX.XX against 95 known blacklists...

Listed 0 times with 1 timeouts

[lots of green ticks then at the end of the list:]

TIMEOUT IPrange RBL Project [response time:] 0

So it's not in any known blacklists. I don't know why the IPrange RBL check failed, but I checked manually at http://iprange.net/rbl/lookup/ and I'm not blacklisted there, either.

SMTP check:

So the connection time is a bit slow (I'm not sure why, will look into that), but I don't see why that would cause sent mails to sometimes disappear completely.

http://intodns.com also give solid green ticks for all my domain's MX checks.

---

I've tried browsing log files on the (Centos/Linux) server:

• /var/log/maillog - these are all empty. I believe these are sendmail logs, and I don't currently use sendmail, so this makes sense.
• /var/log/exim/reject.log is full of rejected brute force attempts on dovecot. I have fail2ban and I'll get on to checking my firewall settings etc to see if I can stop them even trying, but I don't think this is related
• /var/log/exim/main.log also contains many rejected brute force attempts, but also contains records of some actual sent emails:

Here's an email to three people on the same domain that failed for all three people (I've edited some of the alphanumeric strings and replaced the IP addresses with TXT.LIKE.TH.IS):

2016-02-12 08:55:41 no host name found for IP address MY.PC'S.IP.ADR
2016-02-12 08:55:49 1aU9Vw-0004vq-EG <= [email protected] H=(MyPCName) [MY.PC'S.IP.ADR] P=esmtpa A=dovecot_login:[email protected] S=1443429 [email protected]
2016-02-12 08:55:51 1aU9Vw-0004vq-EG => [email protected] <[email protected]> R=dnslookup T=remote_smtp H=cluster5.us.messagelabs.com [US.IP.ADR.ESS] X=UNKNOWN:DHE-RSA-AES256-GCM-SHA384:256
2016-02-12 08:55:51 1aU9Vw-0004vq-EG -> [email protected] <[email protected]> R=dnslookup T=remote_smtp H=cluster5.us.messagelabs.com [US.IP.ADR.ESS] X=UNKNOWN:DHE-RSA-AES256-GCM-SHA384:256
2016-02-12 08:55:51 1aU9Vw-0004vq-EG -> [email protected] <[email protected]> R=dnslookup T=remote_smtp H=cluster5.us.messagelabs.com [US.IP.ADR.ESS] X=UNKNOWN:DHE-RSA-AES256-GCM-SHA384:256
2016-02-12 08:55:51 1aU9Vw-0004vq-EG Completed

Here's an email to one person that succeeded (was received by the recipient):

2016-02-12 08:58:20 no host name found for IP address MY.PC'S.IP.ADR
2016-02-12 08:58:23 1aU9YU-0004w0-IN <= [email protected] H=(MyPCName) [MY.PC'S.IP.ADR] P=esmtpa A=dovecot_login:[email protected] S=23133 [email protected]
2016-02-12 08:58:26 1aU9YU-0004w0-IN => [email protected] <[email protected]> R=dnslookup T=remote_smtp H=cluster4.eu.messagelabs.com [UK.IP.ADR.ESS] X=UNKNOWN:DHE-RSA-AES256-GCM-SHA384:256
2016-02-12 08:58:26 1aU9YU-0004w0-IN Completed

I can't see any significant differences between the two. Before and after both is nothing but brute force debris and other emails.

I don't know what the significance of cluster5.us.messagelabs.com or cluster4.eu.messagelabs.com is, but the associated IP addresses are both MessageLabs IP addresses.

Googling messagelabs.com has turned up this blog article, which looks relevant and suggests that (co-incidentally) both my clients are MessageLabs subscribers, but for the important differences that a) unlike the author I don't even get a non-delivery receipt and b) if it was MessageLabs blocking my email, I don't see why they'd block it for one of their customers but not another.

Solution 1

Email troubleshooting can be divided into "sender" and "recipient" issues. Since you are able to send to other people the Sending side is probably working fine. You need to investigate the Recipient side to locate the problem.

Looking at the logs is a good step and can tell you where your messages are getting to and where they are not. Normal email flow goes like this:

1. You send from your email software to your server

2. Your server sends to their server

3. Their server sends to their email client

In this case you can see from the logs that their server seems to be

cluster5.us.messagelabs.com

Messagelabs is an email filtering service that is now owned by Symantec. Message filtering services like this are used to remove all the spam and junk email before the messages are sent to the client software. This means that any messages blocked by messagelabs will not turn up in any spam or junk email folders in the client software. They will just disappear and the recipient will never see any sign of them. On rare occasions they may get a message saying that "a message from [email protected] has been blocked. Contact your IT dept to unblock it."

This sounds very similar to what has happened here. Technically you should get a bounce response from messagelabs like the guy in the link you posted but this is not guaranteed. They may just silently delete your message if they think it is spam. Usually messagelabs will provide an interface for the IT department at their customer where blocked messages can be released. You can ask your contact at the company to check with their IT team for any blocked messages from your email address. At least you can if you have some other way of contacting them!

Other useful general troubleshooting steps: If you didn't have access to the log files you can find out what the server should be for any domain by looking up the "MX records"

For example here: http://mxtoolbox.com/

The MX record is what an email server looks for to find out where they should send your email.

You can then initiate a manual connection to the server listed in the mx record to see if it is accepting email and what error messages you might get. Use a telnet program like Putty: http://www.putty.org/ and telnet to the email server on port 25. Some of the commands you will need are listed here: http://www.yuki-onna.co.uk/email/smtp.html

So now you can connect to their mail server and send an email using your email address as the "From" address and see how the server responds directly. Any email error codes that are returned can be looked up in google or here: http://www.serversmtp.com/en/smtp-error

Once you have checked that you can connect to the server it may tell you why your email is being rejected as spam or for some other reason, but the reason may not be easy to decipher. At this stage I would suggest you ask the messagelabs customer to contact their support number with the error codes (or lack of them) that you received from their server. Since you are not a customer of messagelabs you can't log a problem or ask messagelabs to check the settings on their customer's account. Their customer will have to ask that themselves. This would be similar for any other mail filtering provider.

Hopefully the error code would point you to a particular problem, like your server being listed on a block list or lacking a SPF record and you can fix that yourself because dealing with a mail filtering provider at third hand is never fun. The last problem I had like this took over three months to resolve before the fault was located and messagelabs fixed it.

I will defer to the answer by kubanczyk for details on SPF and DKIM settings because they seem to be much more knowledgeable than I am!

Good luck!

Solution 2

Your outbound SMTP logs indicate that the destination accepted the message. If the destination mail server is kind enough to send a bounceback for any reason, that's all you get. Aside from asking the client (who may not know) what happened to the email, there's not much you can do except guess. You might also be able to look at the transport headers on a message that you received from the client.

Here's a product data sheet for the MessageLabs solution (look at the control actions on page 2)

So this client's mail system is using an enterprise mail security solution, which offers potentially complex policies to block, deny, modify, filter, scan, redirect etc mail based on common factors like:

1. Transport headers (Was this message scanned by another product? Has it been flagged for encryption? Is it signed? Do I trust the source mail system?)
2. Recipients (Who is allowed to email whom?)
3. Subject, attachment restrictions (Is 'V1AGArA' in the subject? Does it contain a .exe?)
4. Restricted keywords in text?
5. Has the text of the message been classified? (Was this message marked as abusive? Does it contain PII?)

The list goes on and on. I am not entirely familiar with MessageLab's offering, but I work with a similar product that the big bank's compliance, governance, risk, and IT security departments love because it allows those departments to filter, audit, archive, review, analyze, categorize, block mail with an extremely granular level of detail. A lot of our clients are legally required to do common things like:

1. Quarantine inbound and outbound mail that can be in potential violation of financial regulations by redirecting the message transparently to the company's legal team to review and approve.
2. Rewrite the participants on inbound or outbound messages based on the content of the message.
3. Block the message from specific mailboxes based on content or keywords
4. Redact and rewrite portions of the message based on document analysis
5. Apply additional restrictions and control actions based on region. One example would be due to ITAR regulations a client of mine had. All e-mail from certain geographical regions needed to have extra deep-content analysis policies applied with large subsets of mail volume requiring manual approval to reach end user mailboxes.

And of course, since anything can and will happen in enterprise email, there's always the possibility of the destination mail server simply discarding your message and forging a 200 OK or 250 COMPLETED response to your relay. It happens... I know some clients who configured mail relays to route mail into a black hole relay to eliminate rogue routing loops. Enterprise mail is always fun :)

Solution 3

Update: This answer describes a way to get a diagnostic report on any email (the emails contents, headers and server setup). While very useful for improving my server settings, unfortunately even after fixing everything raised here, my emails still don't get through. I'll leave it here as others might have more luck than I did.

---

I found several free online email testing services. They generate a one-off email address, you send it an email then click a link, and it gives a report on how various known spam filters would rate that particular email.

They're generally designed for testing newsletters, but suited my purposes fine.

I didn't know which to try, but the first I tried - https://www.mail-tester.com/ - gave useful results.

I used a false [email protected] email account for these, because having a free service to test email accounts for spammyness then selling those email accounts to spam lists is too obvious a business model... :-)

The report gave me useful leads to follow. Here's their verdict:

Not bad. Some inboxes might still refuse you

5/10

And a screenshot of their diagnosis:

(the "message of body contains errors" isn't as bad as it sounds, it's just pointing out there's no unsubscribe link because it's falsly assuming I'm testing a newsletter)

This is what I was looking for: actionable things to try and fix, in the absence of any failed delivery notice.

So I'll now investigate why my email addresses point to the master VPS hostname vps.my-domain.com instead of the mail server hostname mail.my-domain.com, and I'll investigate why the SPF entries - which I set up months ago and which MX tools say are fine - have not fully propogated.

This in particular looks like the root of my particular problem: a quirk of the server configuration that, I imagine, some configurations will consider irrelevant and some will consider fishy:

---

Update...

Bad news... I've fixed all the issues raised in the Mail Tester report (for anyone interested, see my Server Fault questions about the HELO address issue and the SPF propagation issue). My emails now get a perfect 10/10 from Mail Tester...

Wow! Perfect, you can send

10/10

...but my emails still aren't received by this one weird domain. I'm starting to think I must have somehow been added to an organisation-specific blacklist (maybe someone hit the "spam" button by mistake instead of "reply" or "archive"... not sure if that would explain this?).

Solution 4

I'll try my wild guess here. I see you are using SPF and DKIM. So there is also a possibility you use DMARC (especially if your MX happens to be exim).

Now, there is a chance that your mail goes to receives-nothing.org and there you get a bounce message (a return mail indicating the reason why they cannot accept your mail). But, speaking from experience, many organizations, especially the large ones, have severely botched mechanisms to send bounce messages. In particular, Lotus Domino sends flat out wrong DKIM every single time, because it copies your DKIM signature on their message verbatim. Other more subtle errors also tend to happen a lot.

Many bounce messages fail SPF. (Technical reason is that properly they have empty MAIL FROM: <> and they could have a problem with their EHLO.) So if they have a DMARC policy, this leaves them hanging on DKIM only, which also has a great deal of problems. Logically:

DMARC=(SPF or DKIM) and (alignment check of From-header identity)

Therefore try to temporarily disable DMARC, DKIM, SPF checking on incoming mail on your MX. (It means you shouldn't change your DNS records only your exim settings.) Mail them and wait a couple of hours for a bounce, and then re-enable DMARC, DKIM, SPF.

Solution 5

Often the mail-providers have some strange roules in place and remove the mails that match the roules automatically.
We had this problem:
A mail provider has removed our mails based on the street-address as another company (that have sent spam-mails) was located at the same street-address...
So I would:
- send a mail without any content (also without your normal footer) and without attachment to a mailbox, that don't receive your mails
=> If it can be received, some of your normal content is marked as "bad"
In every case I would ask your partner (that don't receive your mails) for the provider-name and then call their hotline.

View more solutions

Comments

[Admin]

Check your outgoing ip for blacklisting on mxtoolbox.com.

[Admin]

@Tyson thanks for the tip, I have done and added the results. If there are any clues there, unfortunately I'm not able to spot them.

[Admin]

@Jonno it seems to be certain recipients who never receive my emails, not even as spam, even when I know they sent, while other recipients always do. I suspect there must be some kind of company security protocol I'm somehow tripping, I'm trying to figure out how to diagnose what it is

[Admin]

@Jonno I've edited to be clearer

[Admin]

This sounds similar to the 500mi email problem...

[Admin]

ibiblio.org/harris/500milemail.html for reference

[Admin]

@TD.512 haha, great story... But unfortunately my circumstances are rather different! That said I am starting to think that timing out might be a factor - but I'd expect to receive error replies in that case?

[Admin]

Also, mine would be the opposite - the emails that work are to an overseas customer over 500 miles away; the ones that fail are to a local customer just up the road...

[Admin]

Could you name the destination domains? Some mail services implement botched "accept and drop without any notice" anti-spam policy.

[Admin]

@Andrzej I'd prefer to keep this anonymous, is there any system I can use to check what a particular recipient's spam policy is? Or is there a way to send a definitely-spammy email to see how it is responded to?

[Admin]

The recipient might be able to do a test with your e-mail address as sender. BTW, have you tried if you can reply to an e-mail received from their side?

[Admin]

@Zina I receive their emails, but my replies to their emails disappear like everything else. I'll add that detail to the question

[Admin]

It looks definitely a problem on the recipient side. You said you are using Outlook and your own domain. What is your e-mail server? Your own or one provided by your hosting provider? Do you have access to it to try to open a SMTP connection from your server to theirs?

[Admin]

@Zina I have full root access to the VPS with the SMTP server, so I certainly could - but I don't know how to! If you could post a rough step-by-step with an indication of what sort of results to look out for, that's exactly the kind of answer I'm looking for! Steps can be general if it depends on implementation details, but just knowing what sort of commands to look for would be a big step forward

[Admin]

If they clear your mail server than they are being denied at the destination domain level. Have you reached out to the web admins on the other side?

[Admin]

Maybe something like this Microsoft Support - XFOR: Telnet to Port 25 to Test SMTP Communication. You should do this from your mail server and I would suggest that you prepare the commands/lines you will use for testing as you cannot use backspace if you type something wrong. With that said it would be better that the recipient could check this (as myself and others suggested). Maybe try it on your mail server with your e-mail in from and to just to see what you should expect.

[Admin]

@Zina perfect, thanks, I'll try that. My problem with asking them to check is, they're a small branch office of a large intentional organisation, their local IT staff know little beyond basic PC maintainence and have no server access, while their HQ wouldn't respond to a 3rd party contractor like me. It'd be a 3 week+ battle to get them to even look at it, even with the help of the local office staff... But if I can diagnose the problem I can make a stronger case

[Admin]

Do you have access to your SMTP server logs? If so - what does the log say about these undelivered e-mails? There should be a log entry saying where the e-mail was sent and what was the result (including response code). It either went directly to the recipient's mail server or to the e-mail relay. So, what does your outgoing SMTP server log says about these undelivered e-mails?

[Admin]

@VL-80 I've added everything relevant from all the logs I can find to the question. I'm not entirely sure what I'm looking for where though (something telling me what to look for where would be a useful answer!)

[Admin]

Have you tried sending from another EMail account... Say Yahoo, GMail, or other, and if so, what was the result? Your logs appear to show the receiving server accepting the sent messages. One can only infer that it then decides that it doesn't like something and silently discards them. Occasionally, such software will notify the intended recipient that a failure occurred. Knowing the destination domain would help with the diagnosis as the MX records of the recipient domain could be inspected. Also, I believe that Outlook can attach a "Delivery Receipt," but a receiving server could ignore it.

[Admin]

Also, do you have reverse-DNS for the sending system properly set up? Are you using SPF records in your DNS and are they correct?

[Admin]

@BillP3rd I believe so - mxlookup and intodns give full marks for my own domain (you can see the mxlookup screenshop above). Here's the mxlookup for the failing recipient, it says something about not allowing relays: mxtoolbox.com/…

[Admin]

even another email account on the same domain: a colleague of mine once lost access to a personal gmail account {through reasons?}, and years later he discovered gmail recipients were not receiving because the google servers were blocking the backup email address he used.

[Admin]

@user568458, have you tried diagnosing the problem on DnsStuff? Their report shows several "problems" with the email aspects of your recipient's domain. (It is not uncommon for email servers to just silently discard any messages that they deem as not passing muster). Are all recipients (both successful and failed) receiving email at the same domain? Everything on the right of the '@' in the address is identical for all recipients?

[Admin]

@user568458, thank you for providing logs. It looks like your e-mails are successfully going out to messagelabs.com. So, I would contact their tech support and ask to verify what is happening with your e-mails. Give them your IP address, log excerpt and all data that you can provide. What is happening to your e-mail after it goes to their server - is sort of black box for you. The only way to find - is to ask them.

[Admin]

In addition to the above comment, you could resend one never-arrived email from another network, for example your home. If it arrives, then this proves that the target server doesn't like you. If it doesn't, then the target server doesn't like the email's contents, so try next a completely innocent "hello" message.

[Admin]

@harrymc I've already tried something similar I think, just realised I hadn't editted it in, see latest edit

[Admin]

Do all these emails have something in common, for example sender-name? Try changing that to something else but "normal" (don't call yourself Mighty Mouse).

[Admin]

@harrymc good point, I'd totally forgotten that I'd created a second email account on the same domain, and it failed too

[Admin]

It's likely that the problem is with the receiving server. Try using a completely different email address. Look for some change that allows it to arrive. It might be that the receiving server keeps a list of allowed addresses - maybe if the recipient sent you an email you would get into the authorized list.

[Admin]

@harrymc their emails to me come through just fine, and they can receive emails from my personal Gmail account. I thought I'd said that in the question but looks like I hadn't, will edit again! :-)

[beeks]

This is irrelevant as this is not an Outlook client problem.

[Random Username]

OP clearly states >>> I sent some emails to a client last week, from Outlook 2016. I've now found that they were never received. I've tried emailing others on their domain and it looks like none of them receive my emails, but others on other domains do >>> Also this >>> Is there anything "under the hood" I can look at in Outlook, like a sending report or log?>>> and this >>> They're using a mass-mailing system, I'm using regular Outlook, one email at a time. >>> ****Reads like an Outlook 2016 problem to me.****

[beeks]

The message left his mail client and the server sent it. Outlook's not at fault.

[user56reinstatemonica8]

I'll look into the thing about port properties. It's not exclusively an Outlook issue, I just editted in results of testing using Webmail and my Android mail app, which suffer the same problem - but I +1ed this answer since something to do with port properties that fixes it for Outlook users might just fix it for me and gives me something else to look into.

[harrymc]

Didn't you report that your emails do go through successfully to another recipient at the same target domain? The above problems should have caused both or none to fail.

[user56reinstatemonica8]

No, there was another recipient on a different domain who received all emails, but whose domain was also managed by MessageLabs (clearly with different settings)

[user56reinstatemonica8]

I think I've ruled out almost all of the above possibilities, including perfecting my email server settings, but my emails still don't get through. Are you aware of such a thing as a configuration where, if one email trips a filter, the entire domain ends up on a private blacklist? Or anything similar?

[user56reinstatemonica8]

You can see a the full email header, with IPs and domains anonimised, over on this Server Fault question - thanks for your help!

[user56reinstatemonica8]

+1 for the experience but I've already tried sending emails that are blank except for "this is a test" type text, as well as replies to theirs that add no footer. I'm 99% sure it's my domain, not the content or individual user, that is being blocked

[FredyWenger]

Then I would ask for the name of the mail-provider from your partner that don't receive your mails and call the mail provider to ask him to the problem and - if he don't see a problem - do a test together with the mail-provider (then, you WILL find out the problem:-)

[user56reinstatemonica8]

This is a great idea, +1, but unfortunately for me (depending on how you look at it) the troublesome domain had manually put my domain on an "always accept" list before I could try this. Any idea how I could get a bounce message like this after such an exception was made?

[kubanczyk]

Simply send to [email protected] to get their bounce message. But why would you manually analyze their bounce messages now, when problem is gone - a lot of effort and a small chance that any improvement happens on their side.

[user56reinstatemonica8]

I'm trying to work out why I started being blocked in the first place, so I can be confident it won't happen again. Also, I'd like to find out why I was never receiving any bouncebacks. Basically I want to reach a point where I can send emails to anyone without worrying that it might have been silently blocked and not arrived.

[user56reinstatemonica8]

Also, I did previously send to a non-existent email address when I was worried I might have accidentally overlooked some setting that was discarding bouncebacks, and I got a bounceback as expected. Would it be true to say, if I get a bounceback from a non-existent email account but don't from this domain, that means this domain never tried to send me a bounceback? Or is it possible their bouncebacks failed for the reasons you describe while the non-existent account bouncebacks succeeded?

[kubanczyk]

I think it's enough of you to check one bounce. If they have a pipeline of smtp servers, it's possible that the first server is detecting the "no such mailbox" situation, and the second server is detecting other type of "I don't like you" situation. It's unlikely, but not impossible, that the two servers have differently configured bounces, one passess and another is lost. It's much more probable that the "I don't like you" situation simply doesn't generate a bounce, so I would really not go deep into the whole thing. Nowadays you can never be confident that your mail is delivered...