Exchange 2013 SP1 - Content Index failing

I came across this issue in my own Exchange Environment which includes a DAG and has a second Database copy for each database. However this issue will also occur for anyone moving mailboxes between databases on Exchange Servers such as when migrating from Exchange 2010.

This issue has been documented by Microsoft here:
https://support.microsoft.com/en-us/kb/2807668

Here's the situation:
The mounted database shows a Healthy Content Index state, however the Database copy shows Failed or FailedandSuspended Content Index state. A full re-seed or recreating the content index by deleting it and restarting the Microsoft Exchange Search and Microsoft Exchange Search Host Controller services doesn't fix the issue, leaving whoever has the unfortunate task of troubleshooting the issue quite confused, scratching their head and ready to throw the server out of the nearest window!

If this issue isn't resolved and the Database copy is left in an otherwise Healthy state and if the mounted database or the Exchange server it is hosted on goes down you will end up with a Database copy that will refuse to become active and downtime will result!

So here's the cause of the issue:
When Exchange 2013 is installed you have to run Exchange 2013 setup with the /PrepareAD switch to prepare AD for Exchange 2013. Assuming the installation is successful, one of the groups that should be created is called 'ContentSubmitters'. Well, that wasn't the case with mine, and apparently many others' environments aswell.

TL;DR, here is the fix:
Note: You can do this without requiring any downtime if you do it exactly as below!

1) Create an AD Security Group called ContentSubmitters in the Microsoft Exchange Security Groups OU as a Universal – Security Group,

2) Add Administrators and NetworkService objects on the Security Tab (Make sure Advanced Features in ADUC is enabled) and give them Full Control

Or

Do it via PowerShell:

Add-ADPermission -Id ContentSubmitters -User “Network Service” -Access Rights GenericAll

Add-ADPermission -Id ContentSubmitters -User “Administrators” -Access Rights GenericAll

3) Delete the Content Index from the Mailbox Database folder. This is the directory with a long GUID in the directory with the EDB, by default on the Exchange Program Files directory (As per best practice this should ideally be on a seperate drive from the default Exchange Program Files directory, you did remember to do this, didn't you?).

4) Restart these two services after AD replication has kicked in on all of your Exchange 2013 servers:

Microsoft Exchange Search
Microsoft Exchange Search Host Controller

5) The Content Index will hopefully start to rebuild, however I found that I had to force it.

You can do this via Exchange Control Panel (ECP) navigating to servers > databases > click on the affected database and click Update in the right hand panel under the Content Index State and follow the pop up wizard. This may take some time per affected Database copy but eventually the Content Index should become healthy again!

You can confirm the Content Index state via Powershell using the following command:
Get-MailboxDatabaseCopyStatus

You can force the reseed via Powershell using the following command:
Update-MailboxDatabaseCopy –CatalogOnly

P.S. This issue existed with the RTM version of Exchange 2013, at time of writing this, SP1 has been released and the fix still has not been incorporated into SP1! So this should be a mandatory step after installing an Exchange 2013 environment. I know that I will be making sure this has been done on any Exchange 2013 environment I come accross to prevent future issues!

comments powered by Disqus