How to hunt down an EAS bug

In the past month I had to troubleshoot a lot of EAS related issues. This is always a complex process and you as an administrator have to collect a lot of data and provide them to your vendors. After providing these, you often feels like a ping-pong ball. Especially when multiple vendors are involved.

Based on two examples I want to explain, how I was able to proof some misbehaviour of EAS clients. Meanwhile both have been acknowledged by the vendor as a bug:

With iOS 10 this feature can trigger a meeting forward to multiple recipients using SmartForward command.

When a user marks an item read or unread, the flag does not get synced to the mailbox.

Update 27.06.2017: There is also a fix in KB4012108 related to this issue. It’s not exactly the issue I’ve found, but a similar one. The details can be found in KB4024649.

For troubleshooting I used the following tools:

Time to Leave

When iOS 10 was introduced, we saw an increase of tickets related to meeting requests. Users complaint about the fact that a meeting got forwarded to all attendees. The person, who forwarded the meeting, hasn’t necessarily to be the organizer. After analyzing a couple of incidents, we could narrow it down to iOS 10 devices. In all cases we could see that these devices issued a SmartForward command:

eas_smartforward_01

Note: As you can see this was done by a client, with one of the latest versions!

Workaround

A possible workaround is to disable this feature:

Settings > Calendar > Default Alert Times > Time to Leave

Read/Unread flag

This case was not as easy as the previous one. Good thing was that I got this reported by a user who was sitting next to me. This made it a lot easier to troubleshoot.

He got two mails within a few minutes and marked them as read. Only one was marked as read on server-side, but his mobile device showed both as read. With this I almost got everything I needed.

Pulling logs

First I pulled all data from IIS logs for the EAS device using my script Get-IISStats.ps1. As we had not yet the debug profile on the device installed, I had to make sure, which client modified the one item. Therefore I pulled the database events for this mailbox using Get-DatabaseEvent. To identify the two mails in the output, I need the value of the property PR_ENTRYID of these items. For this I used MFCMAPI.

Analysis

Here is the one, which was delivered, marked as read and also synced to the server:

DBEvent_EAS_01.png

You can see that it was delivered through Transport and then modified by a mobile client (ClientCategory shows AirSync). But the following was not touched by a mobile client:

DBEvent_EAS_02.png

Here we can see only Transport in ClientCategory.

Now we have the events from the database and we can compare those with the entries in the IIS logs. For this you need to be aware of the following:

Exchange ActiveSync is the only protocol, where all the actions are logged in the IIS logs. You can find the entries in the field cs-uri-query. There is an outdated article, which describes a small portion:

Understand Exchange ActiveSync Reporting Services

It’s definitely worth reading through this article!

I’m looking for client-side actions, which match the timestamps above. My script extracts from the field cs-uri-query the relevant data and you can find the values in the columns EASServerChanges and EASClientChanges. I imported the CSV file into Excel for analysis.

IIS_EAS_01.png

As you can see there is a matching entry for the timestamp 16:11:16. But the value is 0a1c0d0f0e0s0fs, which means one item was changed and not two. I couldn’t find another entry with client-side changes before and afterwards. For the mobile device, both items are marked as read. Thus means client and server are out of sync. Only a full re-sync would get the client back in sync.

Workaround

Currently there is no workaround available.

Conclusion

I believe that the Read/Unread flag issue is not limited to this flag. It seems like a general problem under certain circumstances, which also causes other issues like Calendar items.

As you can see all information is there and you can collect them. It’s only a matter of time and knowledge to analyse and find the correct conclusion. I hope this brief post gave you an idea what capabillities for troubleshooting exist and helps you to identify issues.

Advertisements

11 thoughts on “How to hunt down an EAS bug

  1. Disabling the time to leave notification reportedly does not stop the issue from occurring.
    From what I can find – this is only affecting Exchange 2016 & Exchange Online (which is built on 2016). Curious if either of you (Ingo / Daniel) can confirm which versions of Exchange you’ve witnessed this on.
    Affects not only ios – but android as well. Anything that uses ActiveSync. So I don’t believe it is going to be Apples fix, but Microsoft’s instead. (though in my environment, we’ve only seen it on iOS devices so far)
    Only valid work around is to use Outlook mobile app.
    Some links which may/may not be of assistance: https://support.microsoft.com/en-us/help/4014990/meeting-invites-are-forwarded-by-an-attendee
    https://answers.microsoft.com/en-us/msoffice/forum/msoffice_outlook-mso_win10/outlook-2016-still-sending-repeat-meeting-requests/37acf3ee-d382-4d59-ab23-2a871c3d08f9

    Like

    • Hi Jack, you’re right it’s a pure iOS issue. I’ve seen this also on Exchange 2013 (basically I got aware when we were on this version). Disabling TTL feature mitigates the issue. It depends on the version. Sadly there is no ETA I’m aware of. Same applies to the Read/Unread flag issue, which IMHO causes also a lot of other issues.
      Ciao,
      Ingo

      Like

  2. To be clear – I don’t believe it is just iOS – but ALSO Android, according to reports I’ve seen online (and since Microsoft confirms it is ActiveSync related).
    Also, multiple reports say that disabling TTL on their iPhones has not mitigated the issue.

    Like

    • Well, our Android population is very limited and as mentioned before it’s really depending on the versions. For us it did the trick to disable TTL. Did you check such meetings forwards for SmartForward?

      Like

  3. Pingback: Get-IISStats: Updated version available | The clueless guy

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s