locked
Event 1006, MsExchange Mailbox Replication RRS feed

  • Question

  • Hi All,

    I've installed Ms Exchange 2010 with SP1 in Windows Server 2008 R2 SP1 x64. The CAS+HT Role in a host and Mailbox Role in another host. Everything is working fine for a month then problems arised.

    For the last few days i've encountered event 1006, MsExchange Mailbox Replication with below details :

    The Microsoft Exchange Mailbox Replication service was unable to process jobs in a mailbox database.

    Database: Mailbox Database 2058730175
    Error: MapiExceptionMaxThreadsPerMdbExceeded: Unable to open message store. (hr=0x80004005, ec=2700)
    Diagnostic context:
        ......
        Lid: 1750    ---- Remote Context End ----
        Lid: 23354   StoreEc: 0xA8C     
        Lid: 55847   EMSMDBPOOL.EcPoolSessionDoRpc called [length=235]
        Lid: 43559   EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0xA8C][length=226][latency=0]
        Lid: 32881   StoreEc: 0xA8C     
        Lid: 50035  
        Lid: 64625   StoreEc: 0xA8C     
        Lid: 1494    ---- Remote Context Beg ----
        Lid: 26426   ROP: ropLogon [254]
        Lid: 52223   dwParam: 0x33
        Lid: 60415   dwParam: 0x32
        Lid: 46079   StoreEc: 0xA8C     
        Lid: 40271  
        Lid: 12716   StoreEc: 0xA8C     
        Lid: 20794  
        Lid: 28474   StoreEc: 0xA8C     
        Lid: 22330   dwParam: 0x0        Msg: 14.01.0218.015:SVEXCMDN02
        Lid: 1750    ---- Remote Context End ----
        Lid: 23354   StoreEc: 0xA8C     
        Lid: 55847   EMSMDBPOOL.EcPoolSessionDoRpc called [length=235]
        Lid: 43559   EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0xA8C][length=226][latency=0]
        Lid: 32881   StoreEc: 0xA8C     
        Lid: 50035  
        Lid: 64625   StoreEc: 0xA8C     
        Lid: 1494    ---- Remote Context Beg ----
        Lid: 26426   ROP: ropLogon [254]
        Lid: 52223   dwParam: 0x33
        Lid: 60415   dwParam: 0x32
        Lid: 46079   StoreEc: 0xA8C     
        Lid: 40271  
        Lid: 12716   StoreEc: 0xA8C     
        Lid: 20794  
        Lid: 28474   StoreEc: 0xA8C     
        Lid: 22330   dwParam: 0x0        Msg: 14.01.0218.015:SVEXCMDN02
        Lid: 1750    ---- Remote Context End ----
        Lid: 23354   StoreEc: 0xA8C     
        Lid: 25913  
        Lid: 21817   ROP Failure: 0xA8C     
        Lid: 26297  
        Lid: 16585   StoreEc: 0xA8C     
        Lid: 32441  
        Lid: 1706    StoreEc: 0xA8C     
        Lid: 24761  
        Lid: 20665   StoreEc: 0xA8C     
        Lid: 25785  
        Lid: 29881   StoreEc: 0xA8C     

    after that, Ms Exchange not working anymore.

    I had to restart both server ( CAS/HT and Mailbox ) to get Exchange working normal again, but in several hours it would happened again. I've already tried to search forums but still no luck till now. 

    Any hint on what i had to check ? Thanks before. Sorry for my bad english.

    Regards,

    George

    Wednesday, April 10, 2013 2:59 AM

Answers

  • best what we can do is to create a new mailbox database.

    follow this link:

    http://technet.microsoft.com/en-in/library/dd335229%28v=exchg.141%29.aspx

    create few mailbox databases and move the mailbox equally across all the databases.

    move mailbox:

    http://technet.microsoft.com/en-in/library/bb124495%28v=exchg.141%29.aspx

    Regards

    Viman

    • Marked as answer by GeorgeOTJ Wednesday, April 17, 2013 9:38 AM
    Thursday, April 11, 2013 9:31 AM
  • How may mailboxes you have on this database, you can find same by running below cmdlet:

    (get-mailbox -Database "Mailbox Database 2058730175" -ResultSize Unlimited).Count

    Based on the details provided so far, issue seems to be due to high number of connections coming to this database at same time which exchange is not liking and throttling the connections afterwards in which case the registry you've put earlier should do the trick.

    However I will still recommend creating additional mailbox database and lighten some load on this single database such that connections are distributed evenly across, which should fix the issue for long time run.


    AKL

    • Marked as answer by GeorgeOTJ Wednesday, April 17, 2013 9:38 AM
    Thursday, April 11, 2013 4:57 AM

All replies

  • Hi,

    This error occurs because the MailboxReplicationService  caches Mailbox Database information of some databases which was deleted. Restarting the service will clear the cache.

    Maybe it is the simplest way to solve it.

    If it doesn’t work, you can try cmd to check it.

    Get-MailboxDatabase : Can you find the database which you delete?

    Remove-MailboxDatabase -Identity "MyDatabase": You can try to delete database if you can see the database.

    ------------------------------------------------------------------------------------------------------------------

    If this post is useful, please hit the green arrow on the left.

    Thanks


    Wednesday, April 10, 2013 4:13 AM
  • Hi Viman007,

    There is only one Mailbox database used since installation until now. Here is the result of cmd Get-MailboxDatabase :

    [PS] C:\Windows\system32>Get-MailboxDatabase

    Name Server Recovery ReplicationType

    ---- ------ -------- ---------------

    Mailbox Database 2058730175 SVEXCMDN02 False None

    [PS] C:\Windows\system32>

    Regards,

    George

    • Edited by GeorgeOTJ Thursday, April 11, 2013 4:02 AM
    Wednesday, April 10, 2013 4:23 AM
  • Hi,

    the issue might be because of "szMaxRPCThreadsPerMDB" try increasing this value of this.

    http://www.messagingtalk.org/?q=node/389

    ------------------------------------------------------------------------------------------------------------------

    If this post is useful, please hit the green arrow on the left.

    Thanks

    Wednesday, April 10, 2013 5:08 AM
  • Do you have any existing move requests that are not complete?  On the server that is giving you the error, execute the following command:

    Get-MoveRequest

    You might see an invalid move request for the old database which you can remove by using the following (make sure you don't have any valid Move requests in progress before executing the following):

    Get-MoveRequest | Remove-MoveRequest

    OM (MCITP) | Blog

    Wednesday, April 10, 2013 5:30 AM
  • Hi viman007,

    looks promising, i'll try to add the registry key. What value will be good for 500+ Mailbox ?

    Regards,

    George

    Wednesday, April 10, 2013 7:02 AM
  • Hi Om,

    I've executed the command "Get-MoveRequest" on both role CAS/HT Server and role Mailbox Server. Seems that the command was not recognized. Below is the error :

    [PS] C:\Windows\system32>Get-MoveRequest
    The term 'Get-MoveRequest' is not recognized as the name of a cmdlet, function, script file, or operable program. Check
     the spelling of the name, or if a path was included, verify that the path is correct and try again.
    At line:1 char:16
    + Get-MoveRequest <<<<
        + CategoryInfo          : ObjectNotFound: (Get-MoveRequest:String) [], CommandNotFoundException
        + FullyQualifiedErrorId : CommandNotFoundException

    [PS] C:\Windows\system32>

    Regards,

    George

    Wednesday, April 10, 2013 7:04 AM
  • keep the value to

    szMaxRPCThreadsPerMDB = 5000 (decimal)

    if you still see the error increase the value again.

    regards

    Viman

    Wednesday, April 10, 2013 7:10 AM
  • hi viman,

    I've already added the szMaxRPCThreadsPerMDB with value 5000 (decimal) and restart the server today. Let see if the problems still occurs tomorrow. Will let you know the results ASAP. 

    Regards,

    George

    Wednesday, April 10, 2013 11:38 AM
  • Hi Om,

    I've executed the command "Get-MoveRequest" on both role CAS/HT Server and role Mailbox Server. Seems that the command was not recognized. Below is the error :

    [PS] C:\Windows\system32>Get-MoveRequest
    The term 'Get-MoveRequest' is not recognized as the name of a cmdlet, function, script file, or operable program. Check
     the spelling of the name, or if a path was included, verify that the path is correct and try again.
    At line:1 char:16
    + Get-MoveRequest <<<<
        + CategoryInfo          : ObjectNotFound: (Get-MoveRequest:String) [], CommandNotFoundException
        + FullyQualifiedErrorId : CommandNotFoundException

    [PS] C:\Windows\system32>

    Regards,

    George

    Okay, this indicates you never initiated any move request (it requires a new role group, so the command isn't recognized).

    Do you have any production mailboxes on the database Mailbox Database 2058730175?

    Also check if the arbitration mailboxes are on this database.

    get-mailbox -Database “Mailbox Database 2058730175″ –Arbitration

    If yes (and no production mailbox), you can move the arbitration mailboxes to other database and delete this database.

    After that reboot the servers and this should get fixed.


    OM (MCITP) | Blog

    Wednesday, April 10, 2013 12:30 PM
  • Hi Om, Below is the result :

    [PS] C:\Windows\system32>get-mailbox -Database "Mailbox Database 2058730175" -Arbitration

    Name                      Alias                ServerName       ProhibitSendQuota
    ----                      -----                ----------       -----------------
    SystemMailbox{1f05a927... SystemMailbox{1f0... svexcmdn02       unlimited
    SystemMailbox{e0dc1c29... SystemMailbox{e0d... svexcmdn02       unlimited
    FederatedEmail.4c1f4d8... FederatedEmail.4c... svexcmdn02       1 MB (1,048,576 bytes)


    [PS] C:\Windows\system32>

    Also, all mailboxes on Mailbox Database 2058730175 are production mailboxes. Is this normal ?

    Regards,

    George

    • Edited by GeorgeOTJ Thursday, April 11, 2013 4:01 AM
    Thursday, April 11, 2013 4:00 AM
  • How may mailboxes you have on this database, you can find same by running below cmdlet:

    (get-mailbox -Database "Mailbox Database 2058730175" -ResultSize Unlimited).Count

    Based on the details provided so far, issue seems to be due to high number of connections coming to this database at same time which exchange is not liking and throttling the connections afterwards in which case the registry you've put earlier should do the trick.

    However I will still recommend creating additional mailbox database and lighten some load on this single database such that connections are distributed evenly across, which should fix the issue for long time run.


    AKL

    • Marked as answer by GeorgeOTJ Wednesday, April 17, 2013 9:38 AM
    Thursday, April 11, 2013 4:57 AM
  • Hi AKL,

    Below is the cmd result :

    [PS] C:\Windows\system32>(get-mailbox -Database "Mailbox Database 2058730175" -ResultSize Unlimited).Count
    503
    [PS] C:\Windows\system32>

    So, We have 503 Mailboxes in a single database.

    How could we create additional mailbox database and move some mailboxes to the new mailbox database ?

    Regards,

    George

    Thursday, April 11, 2013 9:22 AM
  • Hi viman,

    I've already tested up to:

    szMaxRPCThreadsPerMDB = 20000 (decimal)

    but this afternoon, Exchange server down again and i still see the same error.

    Regards,

    George

    Thursday, April 11, 2013 9:27 AM
  • best what we can do is to create a new mailbox database.

    follow this link:

    http://technet.microsoft.com/en-in/library/dd335229%28v=exchg.141%29.aspx

    create few mailbox databases and move the mailbox equally across all the databases.

    move mailbox:

    http://technet.microsoft.com/en-in/library/bb124495%28v=exchg.141%29.aspx

    Regards

    Viman

    • Marked as answer by GeorgeOTJ Wednesday, April 17, 2013 9:38 AM
    Thursday, April 11, 2013 9:31 AM
  • Hi Viman and AKL,

    OK, i will try create several mailbox databases and move the mailboxes equally.

    Hopefully will fix this problem. I will update you all any progress in this matter. Thanks.

    Regards,

    George

    Thursday, April 11, 2013 9:53 AM
  • Hi All,

    I've setup new mailbox databases and spread the mailboxes across. Now it is currently on progress of moving mailboxes to new databases but it takes a long time. Until now, it is already been almost 40 hours but still stuck on moving only 1 mailbox which is about 6 Gbyte in size. Is this normal ?

    Regards,

    George

    Monday, April 15, 2013 3:36 AM
  • Can you get move request statistics of move request and paste it here? Get-MoveRequestStatistics | fl

    AKL

    Monday, April 15, 2013 3:40 AM
  • Hi AKL,

    I've executed the cmd but it asked for identity, so i run

    Get-MoveRequestStatistics -Identity "xxx@yyy.com" | fl

    But is still running now after 15 minutes ..

    Regards,

    George


    • Edited by GeorgeOTJ Monday, April 15, 2013 4:25 AM
    Monday, April 15, 2013 4:24 AM
  • Can you copy paste the results that you get after running the cmdlet please?

    AKL

    Monday, April 15, 2013 8:12 AM
  • I've stopped and re-run the cmdlet but still no results displayed, seems like it keep running but never finish..

    Hi AKL,

    cmdlet still running.. still no result displayed from 4 hours ago

    Regards,

    George

    • Edited by GeorgeOTJ Monday, April 15, 2013 9:46 AM
    Monday, April 15, 2013 8:21 AM
  • Hi,

    “MapiExceptionMaxThreadsPerMdbExceeded” So please check throttling policy and confirm if RCAMaxConcurrency parameter is low, and try increase it .

    Set-ThrottlingPolicy:

    http://technet.microsoft.com/en-us/library/dd298094(v=exchg.141).aspx


    Wendy Liu
    TechNet Community Support

    Wednesday, April 17, 2013 9:32 AM
  • Hi All,

    FYI, from 13 April 2013 after we created new mailbox databases and spread the mailboxes from old database to new ones, Event 1006 MsExchange Mailbox Replication seems to stop happening.

    The process of moving from old database to new databases is still on progress, and cmdlet result asked by AKL is still not available until today, meanwhile Exchange is running without problem.

    If anything occurs after this, i will update you all again.

    Thank you all for your kindness and attention.

    Regards,

    George

    Wednesday, April 17, 2013 9:37 AM
  • Thanks for your update.

    Wendy Liu
    TechNet Community Support

    Thursday, April 18, 2013 2:20 AM