Exchange Online migration and TooManyBadItemsPermanetException

I’m sure that a lot of people have seen this issue before when migrating to Exchange Online:

The BadItemLimit was exceeded and therefore the move request failed.

A while a go Ben Winzenz wrote an excellent post on the You Had Me At EHLO blog, where he mentioned that there was a change in Exchange Online and now failed mapping of SIDs will count towards the BadItemLimit.

So far so good, but how do we solve such issues when increasing of bad item limit is not an option and you have to migrate approx. 130.000 mailboxes?

Update 28.08.2018

Due to some issues while removing invalid permissions with Exchange Cmdlets, I enhanced the script. Read more about it here…

Update 03.01.2020

Joshua Bines has also a great script with reporting capabilities(thanks for sharing!). You can find his on GitHub:

https://github.com/JBines/Get-RecipientPermissions.ps1

Introduction

I’m currently involved in a migration project towards Exchange Online. After some pilot testing we also found mailbox, which exceed the BadItemLimit due to invalid permissions. And there are mailboxes with several hundreds of folders and on each permissions set.

How get permissions invalid?

The older a mailbox is the higher the chance that you will find invalid permissions. This is not caused by Exchange or any client. This is just the life-cycle as users will grant other permissions to folders and rarely I’ve seen that users do a clean up. When a user got deleted in AD, these ACL entries won’t get removed and as a result this ACE will be invalid as it’s orphaned.

Challenge

Now that we know what we’re looking for, it might be a good idea to parse the mailboxes for such entries before starting the migration. But how can this be achieved in an acceptable time?

Resolution

Luckily I remembered another migration project, where I had to deal with a similar topic. But at this time sIDHistory was the challenge. You can read about it in the post here. At that time I’ve written a script to parse mailbox folder permissions. You can read about this one here.

With this I already had a working script, which supports multithreading. The only missing part was to get the data using EWS. All permissions are listed in the PermissionSet on a folder. There are two elements:

• Permissions

Containing collection of permission entries

• UnknownEntries

Containing collection unknown permission entries(cannot be resolved against Active Directory)

After adding some functionality, I was able to scan mailboxes multithreaded in a short time.

• Scan mailboxes for invalid entries and store them into a CSV file. In this example I queried all databases from one DAG and piped them into Get-Mailbox Cmdlet and then to the script:

$Invalid=Get-MailboxDatabase dee16dag01* |
Get-Mailbox -ResultSize Unlimited |
C:\Temp\Scripts\Get-MailboxFolderPermissionEWS.PS1 -Impersonate `
-Credentials $cred -MultiThread -SearchUnknownOnly

Here the script is running with the default of 15 threads

• Export the invalid entries into CSV file (just in case or for later processing)

$Invalid | Export-Csv C:\Temp\InvalidEntries.csv `
-NoTypeInformation -Force

Looking into the results, you will find the following:

Field	Description
Mailbox	The mailbox, which has the invalid entry
User	Invalid user name
FolderName	Display name of the folder
FolderType	Type of folder
FolderPath	Path of the folder in the information store
EwsID, StoreID, HexEntryID	The IDs of the folder in different formats converted with ConvertID

Now as we have all the needed information you can easily remove the entries. I also want to highlight that using the StoreID is much faster than using the FolderPath:

$Invalid | %{Remove-MailboxFolderPermission -Identity `
$($_.Mailbox + ':' + $_.StoreID) -User $_.User -Confirm:$false}

Update 28.08.2018

In the last weeks we stumbled across mailbox, where we couldn’t delete the invalid permissions. We see the following error:

There is no existing permission entry found for user: Invalid, User.
+ CategoryInfo : NotSpecified: (:) [Remove-MailboxFolderPermission], UserNotFoundInPermissionEntryException
+ FullyQualifiedErrorId : [Server=EX2016MB01,RequestId=f584f1bd-a671-46ec-b57e-29403b850d89,TimeStamp=8/7/2018 11:27:28 AM] [FailureCategory=Cmdlet-User
NotFoundInPermissionEntryException] B9FB7DAD,Microsoft.Exchange.Management.StoreTasks.RemoveMailboxFolderPermission
+ PSComputerName : EX2016MB01.contoso.local

I did some tests and the permissions could be removed with MFCMAPI without issues. But MFCMAPI is not made to be used in a script. Finally I stumbled about this in the Remarks of UnknownEntries:

You can delete unknown entries from a folder by using the UpdateFolder operation with the SetFolderField element. The unknown entries are deleted when you reset the PermissionSet by using the SetFolderField option of the UpdateFolder operation. Exchange Web Services does not support the deletion of individual entries.

A few hours and a Microsoft support case later I learned that EWS Managed API is unable to perform this action. Therefore I wrote a function, which is using SOAP:

•  POST a GetFolder SOAP request to retrieve the current PermissionSet
• remove from the current PermissionSet the UnknonwEntries
• Post an UpdateFolder with the SetFolderField (which will overwrite the existing PermissionSet)

I had some challenges as there are different kinds of FolderTypes, which needs different SOAP requests. The function is the following:

function Cleanup-FolderPermission
{
    [CmdletBinding()]
    Param(
        [parameter(ValueFromPipelineByPropertyName=$true,Mandatory=$false, Position=0)]
        [Alias('PrimarySmtpAddress')]
        $EmailAddress,
        
        [parameter(Mandatory=$true, Position=1)]
        [System.String]$FolderID,
        
        [parameter(Mandatory=$true, Position=2)]
        [System.String]$ChangeKey,

        [parameter( Mandatory=$false, Position=3)]
        [System.Management.Automation.PsCredential]$Credentials,

        [parameter( Mandatory=$false, Position=4)]
        [System.Boolean]$Impersonate,
        
        [parameter( Mandatory=$true, Position=5)]
        $URL

    )

    Begin
    {
        [System.Boolean]$returnValue = $true
        # SOAP templates
        $GetFolder = "
            <?xml version=`"1.0`" encoding=`"utf-8`"?>
              <soap:Envelope xmlns:xsi=`"http://www.w3.org/2001/XMLSchema-instance`" xmlns:m=`"http://schemas.microsoft.com/exchange/services/2006/messages`" xmlns:t=`"http://schemas.microsoft.com/exchange/services/2006/types`" xmlns:soap=`"http://schemas.xmlsoap.org/soap/envelope/`">
                <soap:Header>
                <t:RequestServerVersion Version=`"Exchange2010_SP2`" />"
        If ($Impersonate)
        {
            $GetFolder += "
                <t:ExchangeImpersonation>
                    <t:ConnectingSID>
                    <t:SmtpAddress>$($EmailAddress)</t:SmtpAddress>
                    </t:ConnectingSID>
                </t:ExchangeImpersonation>"
        }

        $GetFolder += "
            </soap:Header>
              <soap:Body>
              <m:GetFolder>
                  <m:FolderShape>
                  <t:BaseShape>IdOnly</t:BaseShape>
                  <t:AdditionalProperties>
                      <t:FieldURI FieldURI=`"folder:PermissionSet`" />
                  </t:AdditionalProperties>
                  </m:FolderShape>
                  <m:FolderIds>
                  <t:FolderId Id=`"$($FolderID)`" ChangeKey=`"$($ChangeKey)`" />
                  </m:FolderIds>
              </m:GetFolder>
              </soap:Body>
            </soap:Envelope>"

        $Update = "
            <?xml version=`"1.0`" encoding=`"utf-8`"?>
              <soap:Envelope xmlns:soap=`"http://schemas.xmlsoap.org/soap/envelope/`" xmlns:t=`"http://schemas.microsoft.com/exchange/services/2006/types`">
                <soap:Header>
                <t:RequestServerVersion Version=`"Exchange2010_SP2`" />"

        If ($Impersonate)
        {
            $Update += "
                <t:ExchangeImpersonation>
                    <t:ConnectingSID>
                    <t:SmtpAddress>$($EmailAddress)</t:SmtpAddress>
                    </t:ConnectingSID>
                </t:ExchangeImpersonation>"
        }

        $Update += "
            </soap:Header>
              <soap:Body>
                <UpdateFolder xmlns=`"http://schemas.microsoft.com/exchange/services/2006/messages`" xmlns:t=`"http://schemas.microsoft.com/exchange/services/2006/types`">
                  <FolderChanges>
                    <t:FolderChange>
                      <t:FolderId Id=`"$($FolderID)`" ChangeKey=`"$ChangeKey`"/>
                      <t:Updates>
                        <t:SetFolderField>
                        <t:FieldURI FieldURI=`"folder:PermissionSet`"/>
                          <t:FolderType>
                            <t:Bla>
                            </t:Bla>
                          </t:FolderType>
                        </t:SetFolderField>
                      </t:Updates>
                    </t:FolderChange>
                  </FolderChanges>
                </UpdateFolder>
              </soap:Body>
            </soap:Envelope>"

    }
    Process
    {
        try{
            $GetParams = @{
                Uri = $Url
                Method = 'Post'
                Body = $($GetFolder.TrimStart())
                ContentType = 'text/xml'
                Verbose = $false
            }

            $Headers = @{}

            if ($Credentials)
            {
                $CredString = "$($Credentials.UserName.ToString()):$($Credentials.GetNetworkCredential().password.ToString())"
                $encodedCreds = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($CredString))
                $Headers.Add('Authorization',"Basic $($encodedCreds)")
            }
            else
            {
                $GetParams.Add('UseDefaultCredentials',$true)
            }

            #query folder
            $Headers.Add('Accept','application/xml')
            $Headers.Add('Accept-Encoding','gzip,deflate')
            $GetParams.Add('Headers',$Headers)
            [XML]$GetResp = (Invoke-WebRequest @GetParams).Content

            if (-not [System.String]::IsNullOrEmpty($GetResp))
            {
                #query response for PermissionSet
                $root = $GetResp.get_DocumentElement()
                $NamespaceURI = "http://schemas.microsoft.com/exchange/services/2006/types"
                $ns = New-Object System.Xml.XmlNamespaceManager($GetResp.NameTable)
                $ns.AddNameSpace("ns",$NamespaceURI)
                $XPath  = "//PermissionSet"
                $XPath = $XPath -replace "/(?!/)", "/ns:"
                $PermNode = $root.SelectSingleNode($XPath,$ns)

                if ($PermNode)
                {
                    #replace FolderType with the current type
                    $type = $GetResp.Envelope.Body.GetFolderResponse.ResponseMessages.GetFolderResponseMessage.Folders.ChildNodes.LocalName
                    $Update = $Update.Replace('FolderType',$type)
                    #convert to XML
                    [xml]$Update = $Update.TrimStart()
                    #import PermissionSet XML node
                    $Update.Envelope.Body.UpdateFolder.FolderChanges.FolderChange.Updates.SetFolderField.$type.AppendChild($Update.ImportNode($root.SelectSingleNode($XPath,$ns),$true)) | Out-Null
                    #remove helper XML node
                    $updateRoot = $Update.get_DocumentElement()
                    $NamespaceURI = "http://schemas.microsoft.com/exchange/services/2006/types"
                    $ns = New-Object System.Xml.XmlNamespaceManager($update.NameTable)
                    $ns.AddNameSpace("ns",$NamespaceURI)
                    $BlaNode = $updateRoot.SelectSingleNode("//ns:Bla",$ns)
                    $UnknownEntries = $updateRoot.SelectSingleNode("//ns:UnknownEntries",$ns)
                    $BlaNode.get_ParentNode().RemoveChild($BlaNode) | Out-Null
                    $UnknownEntries.get_ParentNode().RemoveChild($UnknownEntries) | Out-Null

                    $UpdateParams = @{
                        Uri = $Url
                        Method = 'Post'
                        Body = $Update
                        ContentType = 'text/xml'
                        Verbose = $false
                    }

                    $UpdateParams.Add('Headers',$Headers)
                    if (-not $Credentials)
                    {
                        $UpdateParams.Add('UseDefaultCredentials',$true)
                    }

                    $UpdatePost = Invoke-WebRequest @UpdateParams
                }
                
            }
        }
        catch{
            Write-Verbose "Error occured while clearing UnknownEntries"
            $returnValue = $false
        }
    }
    End
    {
        $returnValue
    }
}

This function is now part of the script and will be executed, when using the parameter -ClearUnknown

Requirements

In order to run the script you need the following:

• FullAccess or ApplicationImpersonation permission for the mailboxes

Note: I recommend ApplicationImpersonation to avoid throttling!

• Microsoft Exchange Web Services (EWS) installed
• The script could be found now on GitHub:

https://github.com/IngoGege/Get-MailboxFolderPermissionEWS

Conclusion

I hope this post and the script helps to run your migration more smoothly. Feedback is always welcome!