Exchange Online migration and TooManyBadItemsPermanetException

I’m sure that a lot of people have seen this issue before when migrating to Exchange Online:

The BadItemLimit was exceeded and therefore the move request failed.

A while a go Ben Winzenz wrote an excellent post on the You Had Me At EHLO blog, where he mentioned that there was a change in Exchange Online and now failed mapping of SIDs will count towards the BadItemLimit.

So far so good, but how do we solve such issues when increasing of bad item limit is not an option and you have to migrate approx. 130.000 mailboxes?

Update 28.08.2018

Due to some issues while removing invalid permissions with Exchange Cmdlets, I enhanced the script. Read more about it here

Update 03.01.2020

Joshua Bines has also a great script with reporting capabilities(thanks for sharing!). You can find his on GitHub:

https://github.com/JBines/Get-RecipientPermissions.ps1

Introduction

I’m currently involved in a migration project towards Exchange Online. After some pilot testing we also found mailbox, which exceed the BadItemLimit due to invalid permissions. And there are mailboxes with several hundreds of folders and on each permissions set.

How get permissions invalid?

The older a mailbox is the higher the chance that you will find invalid permissions. This is not caused by Exchange or any client. This is just the life-cycle as users will grant other permissions to folders and rarely I’ve seen that users do a clean up. When a user got deleted in AD, these ACL entries won’t get removed and as a result this ACE will be invalid as it’s orphaned.

Challenge

Now that we know what we’re looking for, it might be a good idea to parse the mailboxes for such entries before starting the migration. But how can this be achieved in an acceptable time?

Resolution

Luckily I remembered another migration project, where I had to deal with a similar topic. But at this time sIDHistory was the challenge. You can read about it in the post here. At that time I’ve written a script to parse mailbox folder permissions. You can read about this one here.

With this I already had a working script, which supports multithreading. The only missing part was to get the data using EWS. All permissions are listed in the PermissionSet on a folder. There are two elements:

Containing collection of permission entries

Containing collection unknown permission entries(cannot be resolved against Active Directory)

After adding some functionality, I was able to scan mailboxes multithreaded in a short time.

  • Scan mailboxes for invalid entries and store them into a CSV file. In this example I queried all databases from one DAG and piped them into Get-Mailbox Cmdlet and then to the script:
$Invalid=Get-MailboxDatabase dee16dag01* |
Get-Mailbox -ResultSize Unlimited |
C:\Temp\Scripts\Get-MailboxFolderPermissionEWS.PS1 -Impersonate `
-Credentials $cred -MultiThread -SearchUnknownOnly

Here the script is running with the default of 15 threads

MailboxPermEWS_01
  • Export the invalid entries into CSV file (just in case or for later processing)
$Invalid | Export-Csv C:\Temp\InvalidEntries.csv `
-NoTypeInformation -Force

Looking into the results, you will find the following:

MailboxPermEWS_02
FieldDescription
MailboxThe mailbox, which has the invalid entry
UserInvalid user name
FolderNameDisplay name of the folder
FolderTypeType of folder
FolderPathPath of the folder in the information store
EwsID, StoreID, HexEntryIDThe IDs of the folder in different formats converted with ConvertID

Now as we have all the needed information you can easily remove the entries. I also want to highlight that using the StoreID is much faster than using the FolderPath:

$Invalid | %{Remove-MailboxFolderPermission -Identity `
$($_.Mailbox + ':' + $_.StoreID) -User $_.User -Confirm:$false}

Update 28.08.2018

In the last weeks we stumbled across mailbox, where we couldn’t delete the invalid permissions. We see the following error:

There is no existing permission entry found for user: Invalid, User.
+ CategoryInfo : NotSpecified: (:) [Remove-MailboxFolderPermission], UserNotFoundInPermissionEntryException
+ FullyQualifiedErrorId : [Server=EX2016MB01,RequestId=f584f1bd-a671-46ec-b57e-29403b850d89,TimeStamp=8/7/2018 11:27:28 AM] [FailureCategory=Cmdlet-User
NotFoundInPermissionEntryException] B9FB7DAD,Microsoft.Exchange.Management.StoreTasks.RemoveMailboxFolderPermission
+ PSComputerName : EX2016MB01.contoso.local

I did some tests and the permissions could be removed with MFCMAPI without issues. But MFCMAPI is not made to be used in a script. Finally I stumbled about this in the Remarks of UnknownEntries:

You can delete unknown entries from a folder by using the UpdateFolder operation with the SetFolderField element. The unknown entries are deleted when you reset the PermissionSet by using the SetFolderField option of the UpdateFolder operation. Exchange Web Services does not support the deletion of individual entries.

A few hours and a Microsoft support case later I learned that EWS Managed API is unable to perform this action. Therefore I wrote a function, which is using SOAP:

  •  POST a GetFolder SOAP request to retrieve the current PermissionSet
  • remove from the current PermissionSet the UnknonwEntries
  • Post an UpdateFolder with the SetFolderField (which will overwrite the existing PermissionSet)

I had some challenges as there are different kinds of FolderTypes, which needs different SOAP requests. The function is the following:

function Cleanup-FolderPermission
{
    [CmdletBinding()]
    Param(
        [parameter(ValueFromPipelineByPropertyName=$true,Mandatory=$false, Position=0)]
        [Alias('PrimarySmtpAddress')]
        $EmailAddress,
        
        [parameter(Mandatory=$true, Position=1)]
        [System.String]$FolderID,
        
        [parameter(Mandatory=$true, Position=2)]
        [System.String]$ChangeKey,

        [parameter( Mandatory=$false, Position=3)]
        [System.Management.Automation.PsCredential]$Credentials,

        [parameter( Mandatory=$false, Position=4)]
        [System.Boolean]$Impersonate,
        
        [parameter( Mandatory=$true, Position=5)]
        $URL

    )

    Begin
    {
        [System.Boolean]$returnValue = $true
        # SOAP templates
        $GetFolder = "
            <?xml version=`"1.0`" encoding=`"utf-8`"?>
              <soap:Envelope xmlns:xsi=`"http://www.w3.org/2001/XMLSchema-instance`" xmlns:m=`"http://schemas.microsoft.com/exchange/services/2006/messages`" xmlns:t=`"http://schemas.microsoft.com/exchange/services/2006/types`" xmlns:soap=`"http://schemas.xmlsoap.org/soap/envelope/`">
                <soap:Header>
                <t:RequestServerVersion Version=`"Exchange2010_SP2`" />"
        If ($Impersonate)
        {
            $GetFolder += "
                <t:ExchangeImpersonation>
                    <t:ConnectingSID>
                    <t:SmtpAddress>$($EmailAddress)</t:SmtpAddress>
                    </t:ConnectingSID>
                </t:ExchangeImpersonation>"
        }

        $GetFolder += "
            </soap:Header>
              <soap:Body>
              <m:GetFolder>
                  <m:FolderShape>
                  <t:BaseShape>IdOnly</t:BaseShape>
                  <t:AdditionalProperties>
                      <t:FieldURI FieldURI=`"folder:PermissionSet`" />
                  </t:AdditionalProperties>
                  </m:FolderShape>
                  <m:FolderIds>
                  <t:FolderId Id=`"$($FolderID)`" ChangeKey=`"$($ChangeKey)`" />
                  </m:FolderIds>
              </m:GetFolder>
              </soap:Body>
            </soap:Envelope>"

        $Update = "
            <?xml version=`"1.0`" encoding=`"utf-8`"?>
              <soap:Envelope xmlns:soap=`"http://schemas.xmlsoap.org/soap/envelope/`" xmlns:t=`"http://schemas.microsoft.com/exchange/services/2006/types`">
                <soap:Header>
                <t:RequestServerVersion Version=`"Exchange2010_SP2`" />"

        If ($Impersonate)
        {
            $Update += "
                <t:ExchangeImpersonation>
                    <t:ConnectingSID>
                    <t:SmtpAddress>$($EmailAddress)</t:SmtpAddress>
                    </t:ConnectingSID>
                </t:ExchangeImpersonation>"
        }

        $Update += "
            </soap:Header>
              <soap:Body>
                <UpdateFolder xmlns=`"http://schemas.microsoft.com/exchange/services/2006/messages`" xmlns:t=`"http://schemas.microsoft.com/exchange/services/2006/types`">
                  <FolderChanges>
                    <t:FolderChange>
                      <t:FolderId Id=`"$($FolderID)`" ChangeKey=`"$ChangeKey`"/>
                      <t:Updates>
                        <t:SetFolderField>
                        <t:FieldURI FieldURI=`"folder:PermissionSet`"/>
                          <t:FolderType>
                            <t:Bla>
                            </t:Bla>
                          </t:FolderType>
                        </t:SetFolderField>
                      </t:Updates>
                    </t:FolderChange>
                  </FolderChanges>
                </UpdateFolder>
              </soap:Body>
            </soap:Envelope>"

    }
    Process
    {
        try{
            $GetParams = @{
                Uri = $Url
                Method = 'Post'
                Body = $($GetFolder.TrimStart())
                ContentType = 'text/xml'
                Verbose = $false
            }

            $Headers = @{}

            if ($Credentials)
            {
                $CredString = "$($Credentials.UserName.ToString()):$($Credentials.GetNetworkCredential().password.ToString())"
                $encodedCreds = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($CredString))
                $Headers.Add('Authorization',"Basic $($encodedCreds)")
            }
            else
            {
                $GetParams.Add('UseDefaultCredentials',$true)
            }

            #query folder
            $Headers.Add('Accept','application/xml')
            $Headers.Add('Accept-Encoding','gzip,deflate')
            $GetParams.Add('Headers',$Headers)
            [XML]$GetResp = (Invoke-WebRequest @GetParams).Content

            if (-not [System.String]::IsNullOrEmpty($GetResp))
            {
                #query response for PermissionSet
                $root = $GetResp.get_DocumentElement()
                $NamespaceURI = "http://schemas.microsoft.com/exchange/services/2006/types"
                $ns = New-Object System.Xml.XmlNamespaceManager($GetResp.NameTable)
                $ns.AddNameSpace("ns",$NamespaceURI)
                $XPath  = "//PermissionSet"
                $XPath = $XPath -replace "/(?!/)", "/ns:"
                $PermNode = $root.SelectSingleNode($XPath,$ns)

                if ($PermNode)
                {
                    #replace FolderType with the current type
                    $type = $GetResp.Envelope.Body.GetFolderResponse.ResponseMessages.GetFolderResponseMessage.Folders.ChildNodes.LocalName
                    $Update = $Update.Replace('FolderType',$type)
                    #convert to XML
                    [xml]$Update = $Update.TrimStart()
                    #import PermissionSet XML node
                    $Update.Envelope.Body.UpdateFolder.FolderChanges.FolderChange.Updates.SetFolderField.$type.AppendChild($Update.ImportNode($root.SelectSingleNode($XPath,$ns),$true)) | Out-Null
                    #remove helper XML node
                    $updateRoot = $Update.get_DocumentElement()
                    $NamespaceURI = "http://schemas.microsoft.com/exchange/services/2006/types"
                    $ns = New-Object System.Xml.XmlNamespaceManager($update.NameTable)
                    $ns.AddNameSpace("ns",$NamespaceURI)
                    $BlaNode = $updateRoot.SelectSingleNode("//ns:Bla",$ns)
                    $UnknownEntries = $updateRoot.SelectSingleNode("//ns:UnknownEntries",$ns)
                    $BlaNode.get_ParentNode().RemoveChild($BlaNode) | Out-Null
                    $UnknownEntries.get_ParentNode().RemoveChild($UnknownEntries) | Out-Null

                    $UpdateParams = @{
                        Uri = $Url
                        Method = 'Post'
                        Body = $Update
                        ContentType = 'text/xml'
                        Verbose = $false
                    }

                    $UpdateParams.Add('Headers',$Headers)
                    if (-not $Credentials)
                    {
                        $UpdateParams.Add('UseDefaultCredentials',$true)
                    }

                    $UpdatePost = Invoke-WebRequest @UpdateParams
                }
                
            }
        }
        catch{
            Write-Verbose "Error occured while clearing UnknownEntries"
            $returnValue = $false
        }
    }
    End
    {
        $returnValue
    }
}

This function is now part of the script and will be executed, when using the parameter -ClearUnknown

MailboxPermEWS_03

Requirements

In order to run the script you need the following:

  • FullAccess or ApplicationImpersonation permission for the mailboxes

Note: I recommend ApplicationImpersonation to avoid throttling!

https://github.com/IngoGege/Get-MailboxFolderPermissionEWS

Conclusion

I hope this post and the script helps to run your migration more smoothly. Feedback is always welcome!

14 thoughts on “Exchange Online migration and TooManyBadItemsPermanetException

  1. Pingback: SourcePrincipalMappingException: Tombstoned AccessRights | The clueless guy

  2. When I tried the initial command: Get-MailboxDatabase dee16dag01* |
    Get-Mailbox -ResultSize Unlimited
    i got the error:
    The operation couldn’t be performed because object ‘DatabaseName’ couldn’t be found on ‘DomainControllerName’

    What worked for me is:
    Get-Mailbox -ResultSize Unlimited -Database “dee16dag01*”

    Like

      • My Example is also just an example 🙂
        Thanks for the script! Great work!
        being playing with it for the last few days.
        got some of those messages:
        WARNING: Child script appears to be frozen for user@domain.com, try increasing MaxResultTime
        I’ve increased it and run it once again.

        Like

      • Hi Ofir,
        this warning is thrown, when it takes longer than a certain time for retrieving data. I usually see this when the mailbox has a huge amount of folders or the server is currently very busy.
        Thanks for the laud!
        Ciao,
        Ingo

        Liked by 1 person

  3. This script looks incredibly useful but unfortunately I can’t seem to get it working for me; it locates the unknownentries just fine, but when I attempt to execute the -clearunknown, for every object it returns:
    VERBOSE: Couldn’t clean folder XXXXXXX with UnknownEntry NT
    VERBOSE: Error occured while clearing UnknownEntries

    Any pointers on where to begin troubleshooting? I’ve confirmed the account that is being used has Full Access.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s