Email responses to tracked crm email not tracking correctly

Question

We have a widespread problem with email responses to outgoing crm email, that has been tracked, not getting auto tracked by Outlook + crm addin. Our crm is 2011 online. We have a variety of outlook addin versions among our users. We've been trying to diagnose this for weeks now with the help of MS support but we are not getting anywhere. I've learnt a lot about the ins and outs of how email is supposed to track but I am still confused. If anyone can help us understand the tracking process for incoming email in response to tracked crm outgoing email I would be grateful. Apologies the long question. Here's what we know:

• users on RU5 of the addin don't seem to have the problem
• users on >= RU6 have the problem
• It doesn't happen with 100% of email responses
• we do not use smart matching
• we use the tracking token
• our users run outlook in non-cached mode
• all email servers are exchange

• we do have crm users who have the same email address as contacts

• we do have >1 contacts who have the same email address

• NOTE: these records with the same email addresses were there for a long time before the problem started in March.

In the diagnostics app/synchronization troubleshooting tab, we enable

• Outlook Synchronization
• Automatic E-Mail Tagging

In the addin/set personal options/email tab we enable

• Check incoming e-mail in Outlook and determine whether an e-mail should be linked and saved as MS Dynamics CRM records
• Track = E-mail messages in response to CRM e-mail
• Display Icon = MS Dynamics CRM icon

Allegedly there was a change to the way the outlook addin processes tracking of emails. Apparently the change was from synchronous to asynchronous promotion of email but I have been unable to find any detail of what that means either on the web or so far from MS themselves! I've read varying accounts of which RU the change was in (5/6/7). Again I've been unable to verify which one but if users on RU5 are problem free and >5 have problems it points to RU6.

What would help greatly are answers to the following questions:

How the addin decides if an incoming email should be tracked at all

What does "in response to" mean in the selected options above? i.e. what fields of the email are relevant? If the original tracked email was sent by user A to user B who forwards it on to user C who forwards it back to user A, does that qualify? Or user A tracks/sends to user B/CC user C and user C replies to user A, does that qualify? It seems to me the only way to understand what should get tracked is to know the "rules" and the rules seem to be more closely guarded that the recipe for Coke...

Diagnostics app/Synchronization Troubleshooting tab has an option called "Automatic E-Mail Tagging". What part does this play? What constitutes a "thread"?

Same question for the "Background Track E-Mail" option.

How do these options play together?

If the addin decides an incoming email should be tracked, what happens then?

Trace logs from the addin show errors to do with inserting into the local SQL CE files that it uses but we're not sure what they mean and why they happen

If the processing is now asynchronous, can that lead to timing anomalies? e.g. if the response comes in before the original email has been "promoted" might the addin try to "promote" the response before the original?

Solution

We have made some progress with the situation. Here's what we are currently testing that looks hopeful.

1. unchecking the "Automatic e-mail tagging" option in the addin diagnostics
2. removing office SP1
3. removing an outlook hotfix (will post more details when I have them)

MS are still unwilling/unable to provide clarity on how the various email tracking settings interact (diagnostics/addin crm email tab/server side). If anyone has info on this we'd welcome it. As I find out more I will re-post.

More info Sep 29 2012

An important distinction is between tagging and tracking

Our problems seem to have been caused by having tagging turned on (1 above) Since we turned it off for everybody, our tracking seems to be much more reliable.

I still don't fully understand what tagging means but as far as I know, it refers to the situation where 2 crm users have a copy of the same tracked email and one of them changes some aspect of the email. For example, if user1 sends a tracked/regarding email to user2 and then user1 changes the regarding record of the sent email, if tagging is on then user2's copy of the email will also change. In other words, "tagging" seems to refer to a process whereby crm tries to keep tracked emails between users in sync.

Beware that tagging is a setting in the diagnostics and when an addin is reconfigured for an org, the diagnostic settings revert back to a set of defaults that includes tagging being ON. So every time an add is reconfigured for an org, if you need tagging to be OFF, that has to be done manually, post org re-configuring. As far as I know there is no way currently to override this behavior.

Another side effect of tagging seems to be that it periodically goes through all emails that are either in the outlook inbox and all child folders of the inbox. Believe the same is true for the sent items (i.e. it and any child folders). It seems to do a read operation on the exchange server per email which can result in a high volume of read requests. When we turned tagging OFF, there was a dramatic decrease in read requests to the exchange server, per addin.

On tracking, what we have verified is that when using tracking tokens only (our situation), the addin considers an incoming email to be a "tracked email" if and only if there is a valid tracking token in the subject. Everything else about the email can change but as long as the original tracking token is in the subject, it is considered to be "in response to" the original tracked email.

Another anomaly we noticed was that the following situation is possible (when using tracking tokens only):

1. user1 sends email to user2 or external email address, but does not track before send
2. post email send, user1 tracks the sent email manually
3. a corresponding email activity gets created in crm even though the email itself does not have a tracking token.

On timing issues: we're not 100% sure of this but it does seem possible for a user to receive a tracked email in their inbox that is regarding a record. Before the addin is able to note that the email is tracked (i.e. change icon to 2 heads), if the user tries to click on the regarding record, it can fail.

Due to the nature of our data, we do have multiple records that have the same email address. What seems to happen is that an activityparty record is created for every record instance that has an email address in the form/to/cc/bcc fields of the tracked email. That can lead to a lot of un-necessary activityparty records but the tracking seems to work ok. We have considered trying to intercept the activityparties created but the addin seems to store that data locally per tracked email so it seems like a dangerous approach!

Summary

What we found improved reliability of tracking for us is:

1. Do not use tagging
2. Remember to uncheck tagging if the addin is reconfigured for an org
3. Minimize the amount of emails in the inbox and below
4. Do not modify a tracked sent email