Office 365 Exchange Hybrid: When Autodiscover and legacy PF impacting Outlook performance

I spent last weeks quite some time with Outlook performance issues in an Exchange Hybrid scenario. In addition this is not a normal Hybrid as here multiple Exchange Organizations from different AD Forests without any Trust Relationships are involved.

Thus I’m not talking about the scenario described in TechNet Multi-forest hybrid deployment scenario. It looks more like this:

O365_Hybrid_01

The only connection between the AD Forest is an additional AAD Connect connector. AAD Connect is installed in the forest of the On-Premises Organization B. Filtering is based on OU and attribute, but AAD Connect configuration is not part of this post.

This post is about the following topics:

Both can have a negative impact for performance. I’m describing how you can analyze and (possibly) mitigate the symptoms.

Note: Please keep in mind that every environment is different and there is no solution, which fits every scenario. Please check carefully!

Autodiscover

First you need to understand how Outlook 2016 implemented Autodiscover. You can find details information about this here:

Outlook 2016 Implementation of Autodiscover

Usually the Autodiscover related DNS entries point to your Exchange on-premises infrastructure and will be pointed AFTER migration is completed. When you establish an Exchange Hybrid a coexistence domain will be implemented (usually .mail.onmicrosoft.com), which will be used for mail-flow between Exchange Online and On-Premises and Autodiscover request for migrated mailboxes.

Now assuming a mailbox was migrated, Outlook will get from Exchange On-Premises a redirect address with the value of the coexistence domain address. This looks like this:

O365_Hybrid_02

As next Outlook will try to connect to autodiscover.sap.mail.onmicrosoft.com in order to sent the request to this host on TCP port 443. And here is the problem as O365 is NOT listening on this port. The reason is simple: O365 cannot have ALL their customers domain on a certificate. Therefore you’ll see a delay:

O365_Hybrid_03

In this case the delay is 5.328 seconds. What’s happening during this time? When you capture the traffic you will see that Outlook is trying to connect to each resolved IP address for this host, but the connection is immediately closed (an ACK RESET for a SYN packet):

O365_Hybrid_04

Once all IP addresses have been probed, Outlook connects to autodiscover.sap.mail.onmicrosoft.com on port 80 and get redirected to https://autodiscover-s.outlook.com/autodiscover/autodiscover.xml

O365_Hybrid_05

Here connection is possible and the client get a response.

Mitigation

You can avoid this delay by adding ExcludeHttpsAutoDiscoverDomain with a value of 1 to the registry. The DWORD needs to be added in one of the paths:

HKEY_CURRENT_USER\Software\Microsoft\Office\x.0\Outlook\AutoDiscover
HKEY_CURRENT_USER\Software\Policies\Microsoft\Office\x.0\Outlook\AutoDiscover

There is also another KB article, besides the previous mentioned one:

Unexpected Autodiscover behavior when you have registry settings under the \Autodiscover key

Now you might say that this will affect On-Premises mailboxes. That might be true, but as long as a machine is domain joined it will always use SCP first and will get the response from there. This might not be true for applications like Skype for Business as SCP is not used. At least Skype for Business follows another logic and is not impacted by this setting. But it’s always a good idea to test before implementing!

Legacy PF

When you have multiple Exchange Organizations the chance of having Public Folders is very high. So it is in this scenario. The problem is the design of how a legacy Public Folder migration works. One step is changing the OrganizationConfiguration in Exchange Online. The steps are outlined here:

Step 5: Configure Exchange Online users to access on-premises public folders

Note: The overall process requires much more, but this is for our issue the important part.

With this configuration you tell Exchange Online to include these Public Folder mailboxes in the Autodiscover response for all migrated user mailboxes. The problem is that Outlook evaluates the PrimarySMTPAddress of these PF mailboxes and in order to retrieve the settings an Autodiscover request is triggered.

Now assuming a user Organization B tries to connect to Exchange On-Premises servers in Organization A and no Forest Trust exists. In the past we have seen prompts for credentials as a user from Organization B won’t be able to authenticate. Furthermore we had a case open as user reported a huge performance issue, when switching folders or scrolling in folders. These performance issue were caused by the initial solution we’ve implemented to avoid credential prompts:

We’ve created a PF mailbox in Exchange Online and assigned this empty one to all mailboxes of Organization B, which had no legacy PF. It turned out that Outlook still tries to logon to this PF mailbox and was constantly failing. It was working a few month ago, but looks like some changes in Outlook caused this behavior.

Solution

The solution for avoiding potential prompts for credentials AND this specific performance issue is the following:

  • create a MEU object with an ExternailEmailAddress, which cannot be resolved by a client
  • add this MEU object to the OrganizationConfig
  • assign to all mailboxes the respective PF referral :
    • Mailboxes of Organization A the real PF (e.g.: PFMailbox1)
    • Mailboxes of Organization B, without any PF, the previously created MEU

#create MEU
New-MailUser -Name NPF -Alias npf -ExternalEmailAddress npf@npf -MicrosoftOnlineServicesID npf@yourtenant.onmicrosoft.com -Password (ConvertTo-SecureString -AsPlainText 'Pa$$w0rd' -Force) -MailboxRegion EUR  -DisplayName 'PF Nirvana Object'

#add MEU to OrgConfig
Set-OrganizationConfig -RemotePublicFolderMailboxes @{add="npf"}

#assign PF referrals to mailboxes
Get-Mailbox -Filter {(WindowsEmailAddress -like '*@organization.b.com') -and (RecipientTypeDetails -eq 'UserMailbox')} -ResultSize unlimited | Set-Mailbox -DefaultPublicFolderMailbox npf
Get-Mailbox -Filter {(WindowsEmailAddress -like '*@organization.b.com') -and (RecipientTypeDetails -eq 'UserMailbox')} -ResultSize unlimited | Set-Mailbox -DefaultPublicFolderMailbox PFMailbox1

#connect to MSOLService and disable MEU
Set-MsolUser -UserPrincipalName npf@yourtenant.onmicrosoft.com -BlockCredential $true

Conclusion

I hope this post helps. Keep in mind what I mentioned above: Please perform extensive tests before implementation.

I would like to specially thank the EE from Microsoft, which helped us solving this issue:

Michael E., Robert S., Peter S. and Tobias R.

Thank you!

Advertisements

2 thoughts on “Office 365 Exchange Hybrid: When Autodiscover and legacy PF impacting Outlook performance

  1. You said “Usually the Autodiscover related DNS entries point to your Exchange on-premises infrastructure and will be pointed AFTER migration is completed.”
    Pointed where? We have finished migrating all Production mailboxes so I am interested in what we do at this point.

    Also, is there a way I can verify that SCP is working properly?
    Thanks for the great article.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s