Mac, EXO and Autodiscover

Recently I stumbled across this issue, which was not obvious at the beginning. It all started with seeing high CPU usage of the Autodiscover application pools on my on-premises servers. With high CPU usage I’m talking about 10-15% total usage across ALL servers and this constantly.

Symptom

All server showed an unusual high CPU usage. Further investigation showed that the application pool for Autodiscover was constantly using 10-15% CPU and sometimes spiked for a few seconds up to 50%.

Troubleshooting

I used my script Get-IISStats and parsed the IIS logs. I used the parameter -ClientReport, which collects the statistics about the User-Agents, their number of hits and the requested URI. I parsed the logs of all Exchange servers only for a few hours (we have hourly rotation configured) and I was surprised when I imported the CSV file:

Even though I parsed only a couple of hours, I got for the top User-Agent over 100 million hits!

As you can see, obviously Apple based User-Agents are by far the ones, which sends the most requests to Exchange (look at the ridiculously high numbers!). Further troubleshooting revealed that these clients are sending 15-20 Autodiscover requests per second! First I thought Exchange is causing issues as sending something wrong to clients, but all responses were good.

Mitigation

Yes, currently there is neither a workaround nor a solution (at least I’m aware of). We opened a case with Apple, but similar to the bug in iOS, which causes a lot of ActiveSync request errors NMStolen (have a look at this figure from this post), neither a solution nor at least an idea what’s causing this.

As we cannot block this User-Agent, we tried to find the right workaround and balance to reduce the load on our servers without impact for our users.

As previously mentioned, these clients sending a high amount of requests per second, which can be treated as DDoS attack. Therefore I thought about throttling clients on load balancer level and don’t even let the request going through to our servers at all. I’m not really experienced in writing iRules for F5, which is our vendor, but there are a few examples out in the internet and I added the following code in order to implement throttling (please feel free to share better code!):

when RULE_INIT {
  ... 
  # variable for throttling
  set static::maxRate 3
  set static::windowSecs 3
  set static::timeout 30
}

when HTTP_REQUEST {
    switch -glob -- [string tolower [HTTP::path]] {
    ...
    "/autodiscover*" {
        # Autodiscover
        # throttling for specific User-Agents
        if {[class match [string tolower [HTTP::header "User-Agent"]] contains ThrottledAgents ] } {
            if { [HTTP::header exists X-forwarded-for] } {
                set client_IP_addr [getfield [lindex  [HTTP::header values X-Forwarded-For] 0] "," 1]
            } else {
                set client_IP_addr [IP::client_addr]
            }
            set requestCount [table key -count -subtable $client_IP_addr]
            if { $requestCount < $static::maxRate } {
                incr requestCount 1
                table set -subtable $client_IP_addr $requestCount "ignore" $static::timeout $static::windowSecs
            } else {
                #log local0. "IPAddress $client_IP_addr has exceeded the number of requests allowed."
                HTTP::respond 503 content "Request blockedExceeded requests/sec limit."
                return
            }
        }
        pool AutoD_pool01
    }
    ...
    }
}

Note: This is NOT my complete iRule! This shows only the relevant lines for throttling!

Similar to my other post here for blocking unwanted agents, you need to create a Data Group, which contains the User-Agents in lower-case:

Note: PowerShell is only listed for testing purposes.

I have no knowledge about capabilities of other load balancer vendors. I’m more than happy to add some links if someone shares or write a How-To.

Conclusion

As of today I don’t have got any feedback from Apple support. I’ll keep you posted as soon as I get feedback and maybe when and what needs to be fixed for this. My personal conclusion is pointing to the fact that there is a component on the apple devices, which cannot handle the circumstance that Autodiscover sends a redirect address (<>@<>.mail.onmicrosoft.com) back. This is an absolutely legit response, especially when you have a Hybrid setup. Anyways as long as you have a single mailbox on-premises you need to have Autodiscover pointing to on-premises.

If someone knows more about this issue…I’m more than happy to hear about!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s