Dropbox Upload Failed May Have Been Moved

This blog postal service is a part of a v post series consisting of:

• Exchange Hybrid Migrations: More than Than Merely a Pretty Face up (Part 1)
• Digging Into Hybrid Migration Move Report Data (Role 2)
• Troubleshooting Failed Migrations (Part three - you are hither)
• Troubleshooting Wearisome Migrations (Part 4)
• What to do if a migration is Completed With Warnings (Part v)

Continuing the weblog post series, we arrived at troubleshooting failed migrations.

A 'failed migration' is when the condition of the motion request shows as 'failed', and we take one or more failures logged in the move written report. The move is stopped and needs the administrator's attending to investigate the reason of failure. Sometimes, resuming of the move can help, specially if in that location were some temporary issues on the Exchange Online side that were addressed.

Before getting into troubleshooting, I recommend you bank check the following 'Minimum Requirements'; those are the things we know will break migrations (and we see them do so):

• MRSProxy needs to be enabled and running on Exchange on-premises.
• Exchange Online requires Negotiate (NTLM) authentication for MRSProxy.
• Make sure your migration users are synchronized with AADconnect tool and corresponding mail users are provisioned correctly on the Exchange Online side for corresponding on-premises mailboxes (ExchangeGuid present, allonym, recipient type right, accepted domains for the email addresses and secondary smtp address user@tenant.postal service.onmicrosoft.com).
• Maximum storage quota in Exchange Online for both primary and annal mailbox is 100GB each. Even if y'all have auto-expanding enabled in Substitution Online, as of this writing the maximum mailbox size for migration of main archive is even so 100GB – reference hither.
• Maximum number of items per regular mailbox folder is i million and three meg for the dumpster, reference here
• On-premises migration admin needs to have the minimum required permissions and valid credentials.
• You cannot offboard to Exchange 2010 an Substitution Online Mailbox that has any hold other than Litigation Hold. This is considering Exchange 2010 doesn't know virtually in-place concord which was introduced in Exchange 2013 or nearly organization-wide holds in Role 365. More info on the types of holds can be found here.
• You cannot offboard an Exchange Online archive mailbox which has auto-expanding enabled in Function 365
• You lot cannot offboard a mailbox and a primary archive to Exchange 2007
• You cannot offboard a remote mailbox without ExchangeGuid assault it
• Any network load-balancing for Exchange 2010 MRSProxy servers requires IP persistence (affinity).
• SSL offloading is not supported for MRSProxy.
• For classic hybrid – where we require entering connectivity from Commutation Online to on-bounds Exchange, allow all Substitution Online IP addresses to connect to on-premises EWS / Autodiscover.
• For classic hybrid, pre-authentication for EWS / Autodiscover virtual directories is not supported.(*)
• For classic hybrid, a valid three^rd political party certificate is required for EWS / IIS. Also see this.
• TLS1.two should be enabled in the on-bounds infrastructure.
• If you have an Exchange organization with Exchange 2013/2016 in coexistence with legacy Exchange servers (Exchange 2010), then you need to indicate the MRSproxy namespace to the newer Substitution version in your environment. This is required because, for case, Exchange 2010 cannot proxy to Commutation 2016 in gild to motility an Exchange 2016 mailbox to or from Commutation Online through an Commutation 2010 MRSProxy endpoint.

(*) Strictly speaking, for hybrid migrations and EWS/mrsproxy.svc endpoint, it would piece of work if the device supported NTLM pre-authentication. It would not piece of work for other hybrid scenarios like Complimentary/Busy, which cannot work with pre-authentication on EWS and Autodiscover.

We likewise recommend that you featherbed the network devices such as firewalls and reverse proxies during migrations in order to reduce source network latency and avoid frequent communication transient errors that would result in mailbox locks and slow migrations.

Frequently when troubleshooting Office 365 migrations, the Commutation Admin Eye GUI is helpful and quite verbose regarding the reason of failure, and it many times includes a link to the corresponding documentation folio for more data on specific issue.

Allow me briefly show you some useful info that we can see in the (Classic) Commutation Admin Eye. As a note, at the time of writing this article, the New Commutation Admin Heart doesn't currently show all of this information.

The following can be seen in the above screenshot:

• We accept one migration batch called "Test Hybrid Migration" of type Exchange Remote Move
• Direction of the move is Onboarding (from on-bounds to the cloud)
• The current status Syncing (things are going well then far)
• There is just one migration user in the batch (looking at the Total column)
• The user is non Synced (hasn't reached the Incremental Sync at 95%), not Finalized (hasn't reached the 100% completion) and not Failed (didn't encounter a fatal failure)

After clicking on View details, nosotros also see:

• Who created the batch (crystal@mytenant.onmicrosoft.com),
• When it was created and started (New-MigrationBatch -AutoStart),
• When it should complete (after the initial sync volition be washed)
• There is no last synced time because the status is syncing and no initial sync has been done
• Also, the associated endpoint is the name of my migration endpoint (Get-MigrationEndpoint) through which I am running the batch.

After a little while, the user failed because of the ExchangeGuid missing on the mail user object in Exchange Online:

In such state of affairs, the migration service failed to inject the move request because the user failed validation. This means that we don't have a movement asking for this user and therefore will have no move report.
If you were to click on 'Download the study for this user', y'all would go an empty .txt file.
Let me show you how this failure looks like in PowerShell and what objects are created and available for us to check at that place.
With Get-MigrationBatch command, nosotros tin see the name of the batch, the status, the type and how many users are contained in the batch:

To come across all properties, run Get-MigrationBatch |FL.

Some other attributes values that you lot saw in the Substitution Admin Middle GUI, for this batch were:

CreationDateTime           : vi/1/2020 8:23:35 AM
StartDateTime              : vi/1/2020 8:23:34 AM
LastSyncedDateTime         :
SubmittedByUser            : crystal@<mytenant>.onmicrosoft.com
BatchDirection             : Onboarding
SourceEndpoint             : Hybrid Commutation Miry

If I had multiple batches and I was interested in seeing this detail 1, I would run: Become-MigrationBatch "Test Hybrid Migration" or if I wanted to see all batches that are failed, I would run: Go-MigrationBatch -Condition SyncedWithErrors

Going further with PowerShell, if I want to encounter the migration user contained in that batch, I would do information technology similar this: Become-MigrationUser -BatchId "Test Hybrid Migration". To see all the details on the migration user, I would again append |FL

This mistake is self-explanatory, ExchangeGuid is missing on the user and I can also see it with Become-MailUser command for this migration user:

From the Go-MigrationUser output, I can also come across the RequestGuid is empty, so this also tells me that there is no motility request / move study for this migration user. I can run Get-MoveRequest <user> or Get-MigrationUser -BatchId "Test Hybrid Migration" | Get-MoveRequest to confirm this.

In cases where the error message on the migration user is not so obvious and y'all still don't take a move request created for it, yous can bank check Get-MigrationUserStatistics with DiagnosticInfo verbose switch: Get-MigrationUserStatistics <user identity> -DiagnosticInfo verbose |FL and see if any more details found.
I will at present become through some more than command examples if you want to play around and check simple or more complicated stuff in PowerShell. Also, some things can exist only checked from PowerShell and if you have a movement request created and this is failed or is progressing boring, you tin can see more on analyzing move reports with PowerShell in later office of this weblog series.

To get an overview of migration statistics:

Get-MigrationStatistics

To go all migration users, their status and corresponding batches:

Become-MigrationUser

To get a specific migration user:

Get-MigrationUser <electronic mail address>

To check the error on a specific migration user:

Get-MigrationUser <email address> |FL errorsummary
Get-MigrationUser <e-mail address> |FL

To get all failed migration users:

Get-MigrationUser -Status Failed

To go all failed migration users and their errors:

Go-MigrationUser -Status Failed | FT identity , errorsummary
Get-MigrationUser -Condition Failed | FL identity , errorsummary

To get migration users from a particular batch:

Get-MigrationUser -Batch "Batch Name"

To get all migration batches:

Become-MigrationBatch

To get a particular batch:

Get-MigrationBatch "Batch Proper name"

Checking move requests (specific for hybrid remote moves)

To get all existing movement requests:

Get-MoveRequest

To get move request statistics for a specific move request:

Get-MoveRequestStatistics "User"

Know that there are two main types of failures:

• Transient Exceptions, case DataExportTransientException
• Permanent Exceptions, example StoragePermanentException

Note: For a movement request to be in a Failed state, we would need to have a permanent failure. Besides many transient failures (usually more than 60) will somewhen crusade a permanent failure. Too many transient failures tin too slow down your migration considerably.

To see the failures (transient or permanent), you would run commands similar to these or export the statistics to an XML file (discussed in the afterwards part of this web log series)

To store the movement report in a variable:

$stats = Get-MoveRequestStatistics "Affected User" -IncludeReport

To check all failures and their count:

$stats.written report.Failures | grouping failuretype | Format-Table -AutoSize

To cheque total details of the last failure:

$stats.written report.Failures[-1]

To check the terminal 2 failures:

$stats.Written report.Failures | select -concluding ii

To check the first failure:

$stats.report.Failures[0]

To bank check the offset 3 failures:

$stats.Written report.Failures | select -first three

If there are a lot of failures, yous can create a list of the failures with the PowerShell Alphabetize number associated with each failure past running the post-obit:

$i=0;$stats.report.Failures | % { $_ | Select-Object @{name="alphabetize";expression={$i}},timestamp,failurecode,failuretype,failureside;$i++} | ft

Using this output, you can then easily identify the index number you desire to focus on by enclosing the failure index number in [brackets], example:

$stats.report.Failures[4]

To become failed motion requests:

Get-MoveRequest -MoveStatus Failed

About frequent failures

Here is a list of about often seen failures in hybrid migrations (and when I say 'most frequent' I mean 'most frequent' issues that we come across in back up, not that you lot will meet those errors in every migration). Note that not all are permanent failures, pregnant not all these will crusade your migrations to fail.

• "User is already being moved" – reference hither
• "You tin't use the domain because it's not an accustomed domain for your organization" – reference here
• "Target mailbox doesn't take an smtp proxy matching '.mail.onmicrosoft.com'" – reference here
• "MigrationPermanentException: Cannot observe a recipient that has mailbox GUID" – reference here. Note that another possible scenario for this error is when we cannot find a ComponentShared Mailbox by its GUID on the Exchange Online side. A ComponentShared mailbox is used to host data from other Part 365 workloads like Teams, OneDrive for Business organization and SharePoint. You would check (in Exchange Online PowerShell) these mailbox GUIDs with the control: Get-MailboxLocation -User <SMTP>. If the mailbox GUID in the error belongs to a component shared mailbox, please log a case with Microsoft Support.
• "Yous must specify the PrimaryOnly parameter" – reference here
• "The remote server returned an Error 404" or "HTTP request has exceeded the allotted timeout" – reference here
• "The remote server returned an fault: (403) Forbidden" – reference here
• "Access is denied" – reference here
• "Couldn't switch the mailbox into Sync Source mode" – reference here
• "CommunicationErrorTransientException - The remote endpoint no longer recognizes this sequence. This is almost likely due to an arrest on the remote endpoint. The value of wsrm:Identifier is not a known Sequence identifier. The reliable session was faulted." – reference here
• "The server was unable to process the asking due to an internal error.  For more than data about the error, either turn on IncludeExceptionDetailInFaults ..." – references here and here
• "TooManyBadItemsPermanentException" - Failed to notice a principal from the source forest or target forest – references here and here
• "The data consistency score (Investigate) for this request is as well low" – reference hither. Note that nosotros will have more on Data Consistency Score subsequently in the blog post serial.
• "Exception has been thrown by the target of an invocation." – reference here
• "Transient error CommunicationErrorTransientException has occurred. The system volition retry" – reference here
• "The Mailbox '<username>@contoso.com' isn't enabled for unified messaging." – reference here
• "Failed to catechumen the source mailbox 'Primary (00000000-0000-0000-0000-000000000000)' to mail-enabled user after the move." or "Unable to update Agile Directory information for the source mailbox at the cease of the move." – reference hither
• "Target user <User> already has a main mailbox". Note: pay special attention to the scenario, it matters if you get this mistake in onboarding (move to Exchange Online) or offboarding (move from Exchange Online). For onboarding moves, delight see this, and for offboarding see this. Note on scenario ane pace 7 in that article: information technology is non supported to remote restore a asunder mailbox from Exchange 2010 on-bounds source server version, information technology needs to be minimum Substitution 2013 version.
• "StalledDueTo_Target*" when you movement mailboxes to Commutation Online – reference here. More on this when we will be discussing ho-hum migrations in next office of this weblog post series.
• "MapiExceptionTooComplex: Unable to query table rows. (60 minutes=0x80040117, ec=-2147221225)" – reference hither.
• "Mailbox Replication Proxy Service can't process this request because information technology has reached the maximum number of agile MRS connections allowed" – reference hither.

Migration failures due to Exchange Online mailbox folder limits

I desire to discuss two common migration errors related to mailbox binder quota limits in Exchange Online when onboarding a mailbox to Office 365:

• MapiExceptionFolderHierarchyChildrenCountQuotaExceeded
• MapiExceptionFolderHierarchyDepthQuotaExceeded

Get-go one, FolderHierarchyChildrenCount refers to Maximum number of subfolders per mailbox folder (10,000 subfolders per folder). MapiExceptionFolderHierarchyChildrenCountQuotaExceeded is thrown when a binder has more than 10,000 same level subfolders in it on the source mailbox.
Second ane, FolderHierarchyDepth refers to maximum folder hierarchy depth (300 folders 'deep').
MapiExceptionFolderHierarchyDepthQuotaExceeded is thrown when the source mailbox has a folder hierarchy with more than 300 binder levels depth.
Here is a visual of these 2:

The mailbox folder limits are documented in the above and you can read them with PowerShell (Get-MailboxStatistics) ran against an Exchange Online Mailbox. These particular two folder limits are:

• FolderHierarchyChildrenCountReceiveQuota
• FolderHierarchyDepthReceiveQuota

You can export the mailbox folders to CSV, on the source side with the following commands:

Get-MailboxFolderStatistics <user> -FolderScope NonIpmroot | Consign-Csv -NoTypeInformation PrimaryFolderStats.csv
Go-MailboxFolderStatistics <user> -Archive -FolderScope NonIpmroot | Consign-Csv -NoTypeInformation ArchiveFolderStats.csv

Annotation that NonIpmRoot Folder scope is not bachelor on Commutation 2010 Servers, you volition demand to use MFCMAPI to see system folders.

The solution is to merge affected folders or move them beyond the binder hierarchy. You volition so exist able to drift the mailbox.
However, to avert the warnings that the terminate-user volition receive after migration, you have to make sure you are below warning quotas (that is 9,000 for child folders and 250 for binder depth).

Here is an example of such warning for folder hierarchy children count:

There is another folder limit (maximum number of messages in a mailbox folder) that is quite often seen in hybrid migrations: 1 million items per folder limit and more than rarely the 3 million items per Dumpster (Recoverable Items folder).
These limits are also documented here and you can see read them using Get-MailboxStatistics:

The failure is: MapiExceptionMessagePerFolderCountQuotaExceeded. You lot can export the folders items to CSV using the following commands:

Get-MailboxFolderStatistics <user> -FolderScope NonIpmroot | sort itemsinfolder -Descending | select folderpath, itemsinfolder |Export-Csv -NoTypeInformation PrimaryFolderItems.csv
Get-MailboxFolderStatistics <user> -Archive -FolderScope NonIpmroot | sort itemsinfolder -Descending | select folderpath, itemsinfolder |Export-Csv -NoTypeInformation ArchiveFolderItems.csv
Get-MailboxFolderStatistics <user> -FolderScope RecoverableItems | sort itemsinfolder -Descending | select folderpath, itemsinfolder | Export-Csv -NoTypeInformation PrimaryDumpsterFolderItems.csv
Get-MailboxFolderStatistics <user> -Archive -FolderScope RecoverableItems | sort itemsinfolder -Descending | select folderpath, itemsinfolder |Export-Csv -NoTypeInformation ArchiveDumpsterFolderItems.csv

Solution is to reduce the number of items in the folder (by deleting or moving them to a different folder when possible).
You will need to recreate migration (remove batch or afflicted migration user from migration batch and create new batch with the user) in lodge to successfully gear up this failure.

A few more troubleshooting tips

MoveOptions Parameter

Oftentimes mailbox moves neglect because of decadent items or elements in a mailbox. These mailbox move failures can exist avoided past excluding those (often corrupt) elements from being migrated.

The MoveOptions parameter (previously known as the SkipMoving parameter which is beingness deprecated) can be added to the onboard or offboard asking from PowerShell with the values of:

'SkipFolderRules, SkipFolderACLs, SkipFolderPromotedProperties, SkipFolderViews, SkipFolderRestrictions, SkipContentVerification, SkipPerObjectIndex'.

This volition tell the migration to skip these elements when performing the move. Nosotros recommend yous perform these skips nether the guidance of Microsoft Support.

Yous can review a motility study from a previously failed move attempt and get some clues on what exclusions you should consider making.

For example, this failure below ways that we accept a search folder on the source mailbox where the query (restriction) is too complex and cannot be created on the target.

Sometimes the failure identifies the bodily problematic source folder so you can await more at the DataContext content. You can and then either delete the query on the source mailbox or only skip the migration of the queries (search folders) and then that you tin complete the migration:

Ready-MoveRequest user@contoso.com -MoveOptions @{add="SkipFolderRestrictions"}

Mailbox Integrity checks

If you migrate a mailbox (main mailbox or annal) to Exchange Online and the size is bigger than 10GB, this is considered a large mailbox and the MRS will perform an ISinteg task to ensure integrity of the mailbox that is being moved.

If you suspect that your movement is stuck on ISinteg chore, you can check the movement report in EXO PowerShell and search for all strings containing isinteg keyword:

$stats = Get-MoveRequestStatistics <user> -IncludeReport
$stats.report.Entries | where { [cord] $_ -similar "*IsInteg*" } | % {[string] $_}

If that shows completed, this means there are no bug. Otherwise, you tin can try running the same command MRS is using on your Exchange on-premises environment, in Ems:

New-MailboxRepairRequest <migration user identity> -CorruptionType MessageId

For more info on the New-MailboxRepairRequest cmdlet, you lot tin can cheque here.

Depending on the Substitution Server Version you lot can check and then the status of the repair request.

For Exchange 2013 and later, use this cmdlet:

Get-MailboxRepairRequest -Mailbox <user identity>

For Exchange 2010 version, you would need to look in Event Viewer for the post-obit events:

• Event 10047 when the repair request is started
• Event 10062 when a abuse is detected and repaired
• Result 10048 when the repair completes successfully

Yous can also try to move a mailbox locally from one server to some other, remove the local motion request and and so retry migration of the mailbox to Substitution Online.

Testing MRS service

One utility that can be used for troubleshooting the mailbox move operation is the Test-MRSHealth cmdlet.  Ane thing to realize is that it cannot be tested from Office 365 side since the cmdlet is not bachelor to a tenant administrators. However, at least from my experience, I have never encountered a situation where MRS service would be stopped on the Part 365 side (and was not automatically recovered inside seconds). We tin apply this utility to test the mailbox replication service health on-premises. Too on-premises, you can check if the MRSProxy is enabled on the EWS virtual directories and if EWS application pool is started in IIS manager.

Event Viewer Diagnostic logging

When performing a mailbox move, you can turn up diagnostic logging on the mailbox replication service or other component like asp.net to get better, more granular events in the event log on-premises.

In well-nigh situations, you don't really get useful events in the on-premises event viewer when troubleshooting an Substitution Online remote motility due to the fact that those events would be written in the datacenter. The default event logging can provide yous with enough information on what the issue would be, accept for example consequence 1309 from ASP.NET where the description is self-explanatory: MRSproxy service being disabled.

If yous practise find a relevant event log for the affected Exchange Online remote move in the effect viewer and this is related to MRS, you can turn up diagnostic logging for the MRS service with the post-obit cmdlet:

Get-EventLogLevel 'MSExchange Mailbox Replication*' | Ready-EventLogLevel -Level Skillful

Then reproduce the upshot or look for it to be reproduced once again and then check in the Upshot Viewer logs for any relevant events.

Tracking incoming failed requests from EXO

Peculiarly useful in communication or timeout failures, at that place are 3 main logs to track the MRS requests on the Exchange on-premises servers in gild for you to sympathise if an MRS Substitution Online request reached your Commutation server, or not. These often help united states of america narrow downwards the issue to a almost likely network device (in front of Exchange Server) that could finish the connectedness and not laissez passer information technology to Exchange Servers. Or if the request reaches the Exchange servers, we can run into where this is stuck and get a better agreement on what's the trouble on the Commutation server on-bounds.

Exchange on-premises server logs to track an EXO Incoming MRS request:

• HTTPerr logs: %SystemRoot%\System32\LogFiles\HTTPERR
• IIS logs for Default Web Site (DWS): %SystemDrive%\inetpub\logs\LogFiles\W3SVC1 – UTC Timezone

The name of the IIS logs contains the date of the log, for instance u_ex190930.log is from Sept thirty, 2019.

• HTTPProxy logs for EWS (available in Exchange 2013 or subsequently): %ExchangeInstallPath%Logging\HttpProxy\Ews

The name of the HTTPProxy logs contains the engagement and 60 minutes starting to log, for instance HttpProxy_2019093014-x.LOG (ten^th log from Sept 30, 2019, starting hr fourteen:00 UTC)

Few things to mention here:

• Ever correlate the timestamp of a failure HH:MM:SS in move report with these logs (IIS and HTTPProxy are in UTC timezone)
• A failed request volition never accept 200 Status code (if you see it with 200 in logs, it means you are non looking at the failed ane). Note that for a request that times out, yous might still exist able to encounter it hither with 200 status lawmaking and possibly a college time-taken
• If you encounter the failed request  in HTTPerr logs, this won't probably exist present in IIS logs or HTTPProxy logs – information technology is stuck in front of IIS, check the item reason in HTTPerr logs and bank check for IIS misconfiguration
• If you run into the failed requests in IIS logs , then y'all can do IIS failed request tracing on that status code and bank check further the detailed error in HttpProxy logs

This concludes Part 3 of these blog series. We will be talking about troubleshooting tedious migrations next!

I would like to thank the following persons for contributing to this web log and for their time and patience to read this: Angus Leeming, William Rall, Brad Hughes, Chris Boonham, Ben Winzenz,  Cristian Dimofte, Nicu Simion, Nino Bilic,  Timothy Heeney

mcdonaldmuche1987.blogspot.com

Source: https://techcommunity.microsoft.com/t5/exchange-team-blog/troubleshooting-failed-migrations/ba-p/1746234

Share this post