Decommissioning flaw in Exchange

This post is lingering around for a while as I thought it would be fixed some days. Unfortunately it wasn’t fixed as of today, so be aware in case you’re going to decommision your Exchange servers. In particular:

Removing databases.

The need of removing databases can have multiple reasons. In my case it was the next logical step in our journey to Exchange Online.

Symptom

As we were migrating mailboxes to Exchange Online, there was a time, where we could start reducing the number of servers in our datacenters.

So we started decommissioning of the first servers out of approx. 130 in total. First steps taken were the following:

  • moving off any kind of mailboxes from databases in the respective DAG
  • disable circular logging (as we are using Exchange Native Data Protection!)
  • removing all database copies
  • removing active database

So far so good and it looks straight forward. But suddenly we started seeing tickets coming in from service owners of LOB and end-users having connectivity issues.

Troubleshooting

I contacted a few affected folks and cross-checked logs on our servers using Get-IISStats, Get-EWSStats and Collect-Event.

By checking the event logs, I found the first indication:

The event logs across all servers were flodded by these events. And not only Autodiscover and EWS was affected: All HTTP Exchange protocols were affected. The server responded with HTTP 503 error randomly.

Even mailboxes already migrated to Exchange Online were affected as we still have Autodiscover pointing on-premises as long as a single mailbox is there.

Solution

The only solution for this problem is to recycle the application pools on all Exchange servers. This will fix it immediately.

Conclusion

Be careful whenever you’re going to remove databases as the ResourceLocator seems to have an outdated cache and is not re-evaluating. This issue might be fixed in one of the next CUs.

1 thought on “Decommissioning flaw in Exchange

  1. Pingback: The Practical 365 Podcast: Ep 14 - Rumours, Exchange Teams Blog and more

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s