none
Exchange Server 2010 - Database keeps dismounting RRS feed

  • Question

  • I have a new installation of Exchange Server 2010 (running on Windows Server 2008 R2 - fully patched) with 1 mailbox for testing, and the database will not stay mounted.  It becomes obvious that there is a problem when I reboot the server, and the server hangs indefinately on "Applying Computer Settings", or I get an error from the Outlook client that I do not have permissions to login.  When I login to the exchange console, I see the database is unmounted.  I am unable to start it and get an error listed below.  The only way I have been able to resolve the problem each time and mount the database again, is by re-running 'setup.exe /preparead', and restarting the Exchange services.  When I do this, everything works fine again... but after a few hours, I start having problems again, and the database is unmounted again.  Any ideas on why this keeps happening?  Thanks.  See error below when trying to mount database.

    ----------------------------------------------

    Subject: mydomain.corp/Users/Administrator : Mount-Database Body: Cmdlet Name: Mount-Database Object Modified: mydomain.corp Parameter: Identity = mydomain.corp Caller: mydomain.corp/Users/Administrator Succeeded: False Error: System.InvalidOperationException: Couldn\'t mount the database that you specified. Specified database: mydomain.corp; Error code: An Active Manager operation failed. Error: The database action failed. Error: Operation failed with message: MapiExceptionCallFailed: Unable to mount database. (hr=0x80004005, ec=-2147467259)\n. [Database: mydomain.corp, Server: mxserver1.mydomain.corp]. ---> Microsoft.Exchange.Cluster.Replay.AmDbActionWrapperException: An Active Manager operation failed. Error: The database action failed. Error: Operation failed with message: MapiExceptionCallFailed: Unable to mount database. (hr=0x80004005, ec=-2147467259)\n. ---> Microsoft.Exchange.Data.Storage.AmOperationFailedException: An Active Manager operation failed. Error: Operation failed with message: MapiExceptionCallFailed: Unable to mount database. (hr=0x80004005, ec=-2147467259)\n. ---> Microsoft.Mapi.MapiExceptionCallFailed: MapiExceptionCallFailed: Unable to mount database. (hr=0x80004005, ec=-2147467259)\n\r\n at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException)\r\n at Microsoft.Mapi.ExRpcAdmin.MountDatabase(Guid guidStorageGroup, Guid guidMdb, Int32 ulFlags)\r\n at Microsoft.Exchange.Cluster.ActiveManagerServer.AmStoreHelper.Mount(Guid mdbGuid, MountFlags flags)\r\n at Microsoft.Exchange.Cluster.ActiveManagerServer.ActiveManagerCore.<>c__DisplayClassb.<MountDatabaseDirect>b__2()\r\n at Microsoft.Exchange.Cluster.Replay.FailoverPerformanceTrackerBase`1.RunTimedOperation(TOpCode opCode, Action operation)\r\n at Microsoft.Exchange.Cluster.ActiveManagerServer.ActiveManagerCore.MountDatabaseDirect(Guid mdbGuid, MountFlags storeFlags, AmMountFlags amMountFlags, AmDbActionCode actionCode)\r\n at Microsoft.Exchange.Cluster.ActiveManagerServer.AmRpcServer.<>c__DisplayClass19.<MountDatabaseDirect>b__18()\r\n at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.RunRpcServerOperation(String databaseName, RpcServerOperation rpcOperation)\r\n --- End of inner exception stack trace (Microsoft.Mapi.MapiExceptionCallFailed) ---\r\n --- End of stack trace on server (mxserver1.mydomain.corp) ---\r\n at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.ClientRethrowIfFailed(String databaseName, String serverName, RpcErrorExceptionInfo errorInfo)\r\n at Microsoft.Exchange.Data.Storage.ActiveManager.AmRpcClientHelper.RunRpcOperationWithAuth(AmRpcOperationHint rpcOperationHint, String serverName, String databaseName, NetworkCredential networkCredential, Nullable`1 timeoutMs, InternalRpcOperation rpcOperation)\r\n at Microsoft.Exchange.Data.Storage.ActiveManager.AmRpcClientHelper.MountDatabaseDirectEx(String serverToRpc, Guid dbGuid, AmMountArg mountArg)\r\n at Microsoft.Exchange.Cluster.ActiveManagerServer.AmDbAction.MountDatabaseDirect(AmServerName serverName, AmServerName lastMountedServerName, Guid dbGuid, MountFlags storeFlags, AmMountFlags amFlags, AmDbActionCode actionCode)\r\n at Microsoft.Exchange.Cluster.ActiveManagerServer.AmDbStandaloneAction.<>c__DisplayClass2.<MountInternal>b__0(Object , EventArgs )\r\n at Microsoft.Exchange.Cluster.ActiveManagerServer.AmHelper.HandleKnownExceptions(EventHandler ev)\r\n --- End of inner exception stack trace (Microsoft.Exchange.Data.Storage.AmOperationFailedException) ---\r\n at Microsoft.Exchange.Cluster.ActiveManagerServer.AmDbOperation.Wait(TimeSpan timeout)\r\n at Microsoft.Exchange.Cluster.ActiveManagerServer.ActiveManagerCore.MountDatabase(Guid mdbGuid, MountFlags storeFlags, AmMountFlags amMountFlags, DatabaseMountDialOverride mountDialOverride, AmDbActionCode actionCode)\r\n at Microsoft.Exchange.Cluster.ActiveManagerServer.AmRpcServer.<>c__DisplayClass4.<MountDatabase>b__3()\r\n at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.RunRpcServerOperation(String databaseName, RpcServerOperation rpcOperation)\r\n --- End of stack trace on server (mxserver1.mydomain.corp) ---\r\n at Microsoft.Exchange.Data.Storage.Cluster.HaRpcExceptionWrapperBase`2.ClientRethrowIfFailed(String databaseName, String serverName, RpcErrorExceptionInfo errorInfo)\r\n at Microsoft.Exchange.Data.Storage.ActiveManager.AmRpcClientHelper.RunDatabaseRpcWithReferral(AmRpcOperationHint rpcOperationHint, Database database, String targetServer, InternalRpcOperation rpcOperation)\r\n at Microsoft.Exchange.Management.SystemConfigurationTasks.MountDatabase.RequestMount(MountFlags storeMountFlags)\r\n at Microsoft.Exchange.Management.SystemConfigurationTasks.MountDatabase.InternalProcessRecord()\r\n --- End of inner exception stack trace --- Run Date: 2012-01-29T01:55:08 OriginatingServer: mxserver1 (14.01.0355.001) 
       
       Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException: Cannot open mailbox /o=mydomain/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=mxserver1/cn=Microsoft System Attendant. ---> Microsoft.Mapi.MapiExceptionNetworkError: MapiExceptionNetworkError: Unable to make connection to the server. (hr=0x80040115, ec=-2147221227) Diagnostic context: ...... Lid: 13720 dwParam: 0x6D9 Msg: EEInfo: Flags: 0 Lid: 11672 dwParam: 0x6D9 Msg: EEInfo: NumberOfParameters: 4 Lid: 8856 dwParam: 0x6D9 Msg: EEInfo: prm[0]: Unicode string: ncacn_ip_tcp Lid: 8856 dwParam: 0x6D9 Msg: EEInfo: prm[1]: Unicode string: mxserver1.mydomain.corp Lid: 12952 dwParam: 0x6D9 Msg: EEInfo: prm[2]: Long val: -545057711 Lid: 12952 dwParam: 0x6D9 Msg: EEInfo: prm[3]: Long val: 382312662 Lid: 45169 StoreEc: 0x824 Lid: 44273 Lid: 59431 EMSMDB.EcDoConnectEx called [length=186] Lid: 34855 EMSMDB.EcDoConnectEx returned [ec=0x0][length=82][latency=218] Lid: 25964 StoreEc: 0x824 Lid: 38951 EMSMDB.EcDoDisconnect called [length=8] Lid: 55335 EMSMDB.EcDoDisconnect returned [ec=0x0][length=12][latency=0] Lid: 36081 Lid: 52465 StoreEc: 0x80040115 Lid: 60065 Lid: 33777 StoreEc: 0x80040115 Lid: 59805 Lid: 52209 StoreEc: 0x80040115 Lid: 56583 Lid: 52487 StoreEc: 0x80040115 Lid: 19778 Lid: 27970 StoreEc: 0x80040115 Lid: 17730 Lid: 25922 StoreEc: 0x80040115 at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, SafeExInterfaceHandle iUnknown, Exception innerException) at Microsoft.Mapi.ExRpcConnection.Create(ConnectionCache connectionCache, ExRpcConnectionCreateFlag createFlags, ConnectFlag connectFlags, String serverDn, String userDn, String user, String domain, String password, String httpProxyServerName, Int32 ulConMod, Int32 lcidString, Int32 lcidSort, Int32 cpid, Int32 cReconnectIntervalInMins, Int32 cbRpcBufferSize, Int32 cbAuxBufferSize, Client xropClient, Byte[] clientSessionInfo, TimeSpan connectionTimeout) at Microsoft.Mapi.MapiStore.OpenMapiStore(String serverDn, String userDn, String mailboxDn, Guid guidMailbox, Guid guidMdb, String userName, String domainName, String password, String httpProxyServerName, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, Boolean wantRedirect, String& correctServerDN, ClientIdentityInfo clientIdentity, String applicationId, Client xropClient, Boolean wantWebServices, Byte[] clientSessionInfo, TimeSpan connectionTimeout) at Microsoft.Mapi.MapiStore.OpenMailbox(String serverDn, String userDn, String mailboxDn, String userName, String domainName, String password, ConnectFlag connectFlags, OpenStoreFlag storeFlags, CultureInfo cultureInfo, WindowsIdentity windowsIdentity, String applicationId) at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore) --- End of inner exception stack trace --- at Microsoft.Exchange.Data.Storage.MailboxSession.ForceOpen(MapiStore linkedStore) at Microsoft.Exchange.Data.Storage.MailboxSession.Initialize(MapiStore linkedStore, LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, GenericIdentity auxiliaryIdentity) at Microsoft.Exchange.Data.Storage.MailboxSession.<>c__DisplayClass12.<CreateMailboxSession>b__10(MailboxSession mailboxSession) at Microsoft.Exchange.Data.Storage.MailboxSession.InternalCreateMailboxSession(LogonType logonType, ExchangePrincipal owner, CultureInfo cultureInfo, String clientInfoString, IAccountingObject budget, Action`1 initializeMailboxSession, InitializeMailboxSessionFailure initializeMailboxSessionFailure) at Microsoft.Exchange.Data.Storage.MailboxSession.CreateMailboxSession(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, CultureInfo cultureInfo, String clientInfoString, PropertyDefinition[] mailboxProperties, IList`1 foldersToInit, GenericIdentity auxiliaryIdentity, IAccountingObject budget) at Microsoft.Exchange.Data.Storage.MailboxSession.ConfigurableOpen(ExchangePrincipal mailbox, MailboxAccessInfo accessInfo, CultureInfo cultureInfo, String clientInfoString, LogonType logonType, PropertyDefinition[] mailboxProperties, InitializationFlags initFlags, IList`1 foldersToInit, IAccountingObject budget) at Microsoft.Exchange.Data.Storage.MailboxSession.OpenAsSystemService(ExchangePrincipal mailboxOwner, CultureInfo cultureInfo, String clientInfoString) at Microsoft.Exchange.ProvisioningAgent.MailboxLoggerFactory.XsoMailer.Log(AdminLogMessageData data, LogMessageDelegate logMessage) at Microsoft.Exchange.ProvisioningAgent.AdminLogProvisioningHandler.OnComplete(Boolean succeeded, Exception e)

     

    Sunday, January 29, 2012 2:18 AM

Answers

  • Hi,

    This might help:

    http://support.microsoft.com/kb/925825

    Leif

    • Marked as answer by stealth010 Tuesday, January 31, 2012 10:00 PM
    Sunday, January 29, 2012 3:47 PM
  • On Sun, 29 Jan 2012 15:51:54 +0000, stealth010 wrote:
     
    >
    >
    >Yes, actually this is the article I found that "temporarily" fixes it. I re-run the 'setup /preparead', restart the services and it works... but only for a short while. Few hours later... database is offline again. Trying to figure out why it keeps happening over and over.
    >
    >I also did "Method 2" from this article, thinking that is what I needed to do to fix it... but its still happening.
     
    If you followed "Method 2" and it worked (even once), if you do it
    again does it allow the database to be mounted?
     
    If it does then there may be something that's modifying the domain
    controller security policy and removing the "Exchange Servers" group.
     
    ---
    Rich Matheisen
    MCSE+I, Exchange MVP
     

    --- Rich Matheisen MCSE+I, Exchange MVP
    • Proposed as answer by Leif Pedersen Monday, January 30, 2012 8:27 AM
    • Marked as answer by stealth010 Tuesday, January 31, 2012 10:00 PM
    Sunday, January 29, 2012 4:36 PM
    • Marked as answer by stealth010 Tuesday, January 31, 2012 10:01 PM
    Sunday, January 29, 2012 8:33 PM

All replies

  • Run ExBpa against the server - checking health and permissions. Does it find anything?

    Also, any other errors in the event logs?

     

    Sunday, January 29, 2012 2:41 AM
    Moderator
  • Just ran the BPA health and permissions check. 

    Health check comes back with some items, but nothing that would be related to this problem, I dont think.  The only "Critical Issues" that comes back is "Certificate SAN mismatch" errors... which is to be expected as I havent replaced any of the self-signed certs.

    Permissions check comes back with nothing at all... everything here looks clean... just informational items.

    No other errors in the event logs to speak of... just errors that say the mount operation failed when I attempted to mount the database, with nothing specific in it.

     

    Sunday, January 29, 2012 3:16 AM
  • Did it again.  Here is the actual pop-up error that shows up when I try to re-mount the database.  The error above shows up in the event logs.

     

    --------------------------------------------------------
    Microsoft Exchange Error
    --------------------------------------------------------
    Failed to mount database 'mydomain.corp'.

    mydomain.corp
    Failed
    Error:
    Couldn't mount the database that you specified. Specified database: mydomain.corp; Error code: An Active Manager operation failed. Error: The database action failed. Error: Operation failed with message: MapiExceptionCallFailed: Unable to mount database. (hr=0x80004005, ec=-2147467259)
    . [Database: mydomain.corp, Server: mxserver1.mydomain.corp].

    An Active Manager operation failed. Error: The database action failed. Error: Operation failed with message: MapiExceptionCallFailed: Unable to mount database. (hr=0x80004005, ec=-2147467259)
    . [Database: mydomain.corp, Server: mxserver1.mydomain.corp]

    An Active Manager operation failed. Error: Operation failed with message: MapiExceptionCallFailed: Unable to mount database. (hr=0x80004005, ec=-2147467259)
    . [Server: mxserver1.mydomain.corp]

    MapiExceptionCallFailed: Unable to mount database. (hr=0x80004005, ec=-2147467259)

     

     

    --------------------------------------------------------
    OK
    -------------------------------------------------------- 

    Sunday, January 29, 2012 3:41 PM
  • Hi,

    This might help:

    http://support.microsoft.com/kb/925825

    Leif

    • Marked as answer by stealth010 Tuesday, January 31, 2012 10:00 PM
    Sunday, January 29, 2012 3:47 PM
  • Yes, actually this is the article I found that "temporarily" fixes it.  I re-run the 'setup /preparead', restart the services and it works... but only for a short while.  Few hours later... database is offline again.  Trying to figure out why it keeps happening over and over. 

    I also did "Method 2" from this article, thinking that is what I needed to do to fix it... but its still happening.

    • Marked as answer by stealth010 Tuesday, January 31, 2012 10:08 PM
    • Unmarked as answer by stealth010 Tuesday, January 31, 2012 10:08 PM
    Sunday, January 29, 2012 3:51 PM
  • On Sun, 29 Jan 2012 15:51:54 +0000, stealth010 wrote:
     
    >
    >
    >Yes, actually this is the article I found that "temporarily" fixes it. I re-run the 'setup /preparead', restart the services and it works... but only for a short while. Few hours later... database is offline again. Trying to figure out why it keeps happening over and over.
    >
    >I also did "Method 2" from this article, thinking that is what I needed to do to fix it... but its still happening.
     
    If you followed "Method 2" and it worked (even once), if you do it
    again does it allow the database to be mounted?
     
    If it does then there may be something that's modifying the domain
    controller security policy and removing the "Exchange Servers" group.
     
    ---
    Rich Matheisen
    MCSE+I, Exchange MVP
     

    --- Rich Matheisen MCSE+I, Exchange MVP
    • Proposed as answer by Leif Pedersen Monday, January 30, 2012 8:27 AM
    • Marked as answer by stealth010 Tuesday, January 31, 2012 10:00 PM
    Sunday, January 29, 2012 4:36 PM
  • Everytime this fails, I have checked to make sure that "Exchange Servers" is still in the URA, and it is.  It is not being removed.

    When it happened again today, I also tried doing the following...

    1)  Restart all Exchange services.  They all restarted successfully, but database still could not be mounted.
    2)  Reboot the server.  It then hung indefinately on Applying Computer Settings.
    3)  Rebooted server in Safe Mode, disabled all Exchange services.
    4)  Rebooted, ran 'setup /preparead', and set all Exchange services back to Automatic.
    5)  Rebooted, and everything is fine again... for now...

    So I was able to re-confirm that all that seems to temporarily fix it, is running the '/preparead' again.

    Sunday, January 29, 2012 4:57 PM
  • Do you get an event ID, such as 9519? If yes, do you see anything in the data section of the event ID


    Sukh
    Sunday, January 29, 2012 4:58 PM
  • No...  I dont see a 9518 or 9519 errors in any of the event logs.
    Sunday, January 29, 2012 5:04 PM
  • Can you verify and check your AD replication is working?

    Or if only using a single DC/GC, make sure that it working,

    Also, ensure DNS is working, anything on your dc/gc app logs?


    Sukh
    Sunday, January 29, 2012 5:22 PM
  • All event logs look spotless on the DCs, nothing out of the ordinary to speak of.  Replicaiton appears to be working fine.  DC tests all come back as passed.

    Sunday, January 29, 2012 5:55 PM
  • What does the 2080 event say, seems like you may be missing the SACL right
    Sukh
    Sunday, January 29, 2012 6:02 PM
  • Ah, interesting...

    The 2080 is just informational.  No warnings or errors related to the 2080.

    Process MAD.EXE (PID=6856). Exchange Active Directory Provider has discovered the following servers with the following characteristics:
     (Server name | Roles | Enabled | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version)
    In-site:
    DC01.mydomain.corp        CDG 1 7 7 1 0 0 1 7 1
    DC02.mydomain.corp        CDG 1 7 7 1 0 0 1 7 1

    BUT... I am seeing this.  There are many of these with all the different services listed.  These seem to relate to the time frames associated with the failure of mounting the database.  Here is an example of 2 of them:

    Event ID: 2102 Topology
    Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=6744). All Domain Controller Servers in use are not responding:
    DC01.mydomain.corp
    DC02.mydomain.corp

    Event ID: 2102 Topology
    Process MAD.EXE (PID=6856). All Domain Controller Servers in use are not responding:
    DC01.mydomain.corp
    DC02.mydomain.corp

    Sunday, January 29, 2012 6:15 PM
  • In the Manage Auditing and Security Log do you have both the Exchange Enterprise Servers and Exchange servers listed?
    Sukh
    Sunday, January 29, 2012 7:53 PM
  • I dont have a group called "Exchange Enterprise Servers"... only "Exchange Servers", and that group is part of the Manage Auditing and Security Log.
    Sunday, January 29, 2012 7:59 PM
  • is this a pure Exch 2010 Env?

    And you are checking the default domain controllers poilcy?

     


    Sukh
    Sunday, January 29, 2012 8:03 PM
  • Yes, there is only 1 Exchange server.

    Not sure what you mean about am I checking the default domain controllers policy, but that policy does get applied to the domain contollers, and is mostly unmodified, as the entire environment is pretty much out-of-the-box installation.  The only thing in the default domain controllers policy that has changed, is I added some additional auditing for events, and I changed "Domain Controller:  LDAP server signing requirements" to "Require Signing".

    Sunday, January 29, 2012 8:11 PM
    • Marked as answer by stealth010 Tuesday, January 31, 2012 10:01 PM
    Sunday, January 29, 2012 8:33 PM
  • On Sun, 29 Jan 2012 18:15:49 +0000, stealth010 wrote:
     
    >
    >
    >Ah, interesting...
    >
    >The 2080 is just informational. No warnings or errors related to the 2080.
    >
    >Process MAD.EXE (PID=6856). Exchange Active Directory Provider has discovered the following servers with the following characteristics: (Server name | Roles | Enabled | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version) In-site: DC01.mydomain.corp CDG 1 7 7 1 0 0 1 7 1 DC02.mydomain.corp CDG 1 7 7 1 0 0 1 7 1
     
    The "SACL right" is "0". That's not a "good thing".
     
    The link that Sujh828 gave you should help rectify the problem:
    http://blogs.technet.com/b/richardroddy/archive/2010/06/16/msexchange-adaccess-dsaccess-errors-and-the-manage-auditing-and-security-right.aspx
     
    Running the /preparead sets that correctly, so now you need to find
    out what's removing that right.
     
    ---
    Rich Matheisen
    MCSE+I, Exchange MVP
     

    --- Rich Matheisen MCSE+I, Exchange MVP
    Sunday, January 29, 2012 10:32 PM
  • On Sun, 29 Jan 2012 15:51:54 +0000, stealth010 wrote:
    >
    >
    >Yes, actually this is the article I found that "temporarily" fixes it. I re-run the 'setup /preparead', restart the services and it works... but only for a short while. Few hours later... database is offline again. Trying to figure out why it keeps happening over and over.
    >
    >I also did "Method 2" from this article, thinking that is what I needed to do to fix it... but its still happening.
    If you followed "Method 2" and it worked (even once), if you do it
    again does it allow the database to be mounted?
    If it does then there may be something that's modifying the domain
    controller security policy and removing the "Exchange Servers" group.
    ---
    Rich Matheisen
    MCSE+I, Exchange MVP
    ----------------------------------------------------------------------------------------------


    Rich - You were on to something here.  While the default domain policy is pretty much out-of-the-box, I do apply some additional settings in a separate policy, and apparently I had defined the URA Manage Audit and Security Logs setting with just Administrators and an Auditors group I created.  So that setting was indeed being over-written, and I apparently over-looked it.  An RSOP on the DC revealed this override.  I have since adjusted this to include the "Exchange Servers" group, and will monitor for the next 24hrs or so and let you know if it is resolved.  Thanks for pointing out the SACL right.

    Sukh - Thank you for the link and also pointing out the SACL right.  I have a feeling this will be the answer due to that URA being over-written. 


    • Edited by stealth010 Monday, January 30, 2012 4:34 AM
    Monday, January 30, 2012 4:29 AM
  • It would appear that the problem has been resolved.  It has been 24hrs now, and the database is still online.  I believe it was a combination of

    http://support.microsoft.com/kb/925825  (Method 2)

    -and-

    a separate policy removing the Exchange Servers group from the User Right Assignment - Manage Audit and Security Logs.

    This link that Sukh posted is a detailed description of the problem.

    http://blogs.technet.com/b/richardroddy/archive/2010/06/16/msexchange-adaccess-dsaccess-errors-and-the-manage-auditing-and-security-right.aspx



    Thanks for the help everyone!

    Tuesday, January 31, 2012 10:08 PM