locked
a cluster resource goes offline early RRS feed

  • Question

  • i have a cluster with 15 resource services under an application.

    12 are using the Windows 2008 r2 cluster "generic service" resource type.

    3 are using custome built resource dll.

    failover validation passes for the most part the drives are in warning because they are inuse.

    the cluster logs shows the following message:

     INFO  [RES] mrsClusresMRS <New mrsClusResMRS>: OfflineThread: Process with id 3660 still exists                                                  
     INFO  [RES] mrsClusresMRS <New mrsClusResMRS>: OfflineThread: retrying...                                                                        
     ERR   [RES] mrsClusresMTA <XPR Message Router (mta)>: CheckIsAlive: Verification of the 'MTA' service failed. Error: 1062.                       
     WARN  [RHS] Resource XPR Message Router (mta) IsAlive has indicated failure.                                                                     
     INFO  [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'XPR Message Router (mta)', gen(2) result 1.                                             
     INFO  [RCM] TransitionToState(XPR Message Router (mta)) WaitingToGoOffline-->ProcessingFailure.                                                  
     ERR   [RCM] rcm::RcmResource::HandleFailure: (XPR Message Router (mta))                                                                          
     ERR   [CM] mscs::RegCheckpoint::SaveCheckpoint: ERROR_KEY_DELETED(1018)' because of '::RegSaveKeyEx(handle, file.c_str(), nullptr, options)'     
     INFO  [RCM] Since no top level resource will be offlined by this resource's failure, resetting restartAction to 1                                
     INFO  [RCM] resource XPR Message Router (mta): failure count: 1, restartAction: 1.                                                               
     INFO  [RCM] resource XPRShare S: is waiting for resource XPR Message Router (mta) in state ResWaitingToGoOffline, will not restart               
     INFO  [RCM] resource XPR Message Router (mta) will not be restarting; isLowPriority: false; numDependents: 1, failureCount: 1, restartAction: 1  
     INFO  [RCM] TransitionToState(XPR Message Router (mta)) ProcessingFailure-->[WaitingToTerminate to Failed].                                      
     INFO  [RCM] TransitionToState(XPR Message Router (mta)) [WaitingToTerminate to Failed]-->[Terminating to Failed].                                
     INFO  [RCM] Resource XPR Message Router (mta) stopping completely because it is in a low priority group.                                         
     INFO  [RES] mrsClusresMTA <XPR Message Router (mta)>: Terminate request.                                                                         
     INFO  [RES] mrsClusresMTA <XPR Message Router (mta)>: Terminate: Service died; status = 1062.                                                    
     INFO  [RES] mrsClusresMRS <New mrsClusResMRS>: OfflineThread: ControlService failed with ec 1061                                                 
     INFO  [RES] mrsClusresMRS <New mrsClusResMRS>: OfflineThread: Process with id 3660 still exists                                                  
     INFO  [RES] mrsClusresMRS <New mrsClusResMRS>: OfflineThread: retrying...                                                                        
     INFO  [RCM] HandleMonitorReply: TERMINATERESOURCE for 'XPR Message Router (mta)', gen(3) result 0.                                               
     INFO  [RCM] TransitionToState(XPR Message Router (mta)) [Terminating to Failed]-->Failed.                                       

    the (mta) is a custom resourcetype it's dependencies are the cluster name, cluster drive and a service called infostor (which is another customer resource).

    the (mrs) which is another custom resourcetype depends on (mta). there are 9 generice services that depends on the (mrs). the (mrs) custom resourcetype does not have problem waiting for the dependents to shutdown.

    i'm told that all custom resourcetype built with most the same properties. we are using the default configurable policies and advance policies.

    Wednesday, March 14, 2012 7:05 PM

Answers

  • It just looks like that MTA service is somehow terminated before the cluster gets a chance to try to bring the resource offline. Just before the IsAlive check, you can see that the MTA resource is waiting on the "MRS" resources to go offline:

    INFO  [RCM] 'XPR Message Router (mta)' cannot go offline yet; 'XPR Administrator(mrs)' is in state OfflineCallIssued.

    You never see that OfflineCallIssued state set upon the MTA resource...it just fails an IsAlive check while its in a "WaitingToGoOffline" state. You can even see that the last posted state for MTA is still "WaitingToGoOffline" in this message:

    INFO  [RCM] TransitionToState(XPR Message Router (mta)) WaitingToGoOffline-->ProcessingFailure.

    Is there anything in the event logs showing the MTA service crashing or stopping just before the failure occurs?


    Visit my blog about multi-site clustering

    • Proposed as answer by JohnToner Saturday, March 17, 2012 11:28 PM
    • Marked as answer by Vincent Hu Monday, April 9, 2012 3:46 PM
    Wednesday, March 14, 2012 11:52 PM

All replies

  • When the "mta" resource fails, any resources that depend upon this resource will go offline. This would be the expected behavior.

    Not sure exactly what your question is...in the log message above, the MTA resource is failing an IsAlive check because it does not see the service running:

    C:\>net helpmsg 1062

    The service has not been started.


    Visit my blog about multi-site clustering

    Wednesday, March 14, 2012 8:54 PM
  • i'm trying to find why the mta resource is not being managed correctly.

    should it not successful shutdown when the offline request is sent?

    but your statement about "the service is not running" is interesting. why is the cluster resource not monitoring the shutdown of the service?

    Wednesday, March 14, 2012 9:15 PM
  • The first two lines of the log message above are referring to taking "MRS" resource offline...I don't see any calls to offline the "MTA" resource.

    The first message about the "MTA" resource is regarding an IsAlive check on the "MTA" service, where returns the error 1062...service is not started. This is why this resource failed in this case.

    Do you have "service dependencies" setup outside of cluster? If so, it's possible that one of your other services is killing the MTA service before it gets a chance to take itself offline in the cluster. The cluster is just reacting as it should to a failed IsAlive check.


    Visit my blog about multi-site clustering

    Wednesday, March 14, 2012 9:42 PM
  • i checked all the services and no dependencies are posted other than what is in the cluster resource. let me adde the full cluster log for the shutdown sounds like i might have left too much detail out.

    INFO  [RCM] rcm::RcmApi::OfflineGroup: (XPR2008Cluster)
    INFO  [RCM] rcm::RcmGum::SetGroupPersistentState(XPR2008Cluster,0)
    INFO  [RCM] TransitionToState(XPR Administrator(mrs)) Online-->WaitingToGoOffline.
    INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (XPR2008Cluster, Failed --> Pending)
    INFO  [RCM] Bringing dependent resource 'XPR Internet Mail APL(SmtpApl)' offline before provider resource 'XPR Administrator(mrs)'.
    INFO  [RCM] Bringing dependent resource 'XPR TCP/IP Transport Layer(TcpApl)' offline before provider resource 'XPR Administrator(mrs)'.
    INFO  [RCM] Bringing dependent resource 'XPR Directory Service(DirSvc)' offline before provider resource 'XPR Administrator(mrs)'.
    INFO  [RCM] TransitionToState(XPR TCP/IP Transport Layer(TcpApl)) Online-->WaitingToGoOffline.
    INFO  [RCM] Bringing dependent resource 'stunnel' offline before provider resource 'XPR Administrator(mrs)'.
    INFO  [RCM] TransitionToState(stunnel) Online-->OfflineCallIssued.
    INFO  [RCM] TransitionToState(XPR Directory Service(DirSvc)) Online-->OfflineCallIssued.
    INFO  [RCM] 'XPR TCP/IP Transport Layer(TcpApl)' cannot go offline yet; 'stunnel' is in state OfflineCallIssued.
    INFO  [RCM] TransitionToState(XPR Internet Mail APL(SmtpApl)) Online-->OfflineCallIssued.
    INFO  [RCM] 'XPR Administrator(mrs)' cannot go offline yet; 'XPR Internet Mail APL(SmtpApl)' is in state OfflineCallIssued.
    INFO  [RCM] TransitionToState(XPRShare S:) Online-->WaitingToGoOffline.
    INFO  [RCM] Bringing dependent resource 'XPR Message Router (mta)' offline before provider resource 'XPRShare S:'.
    INFO  [RCM] Bringing dependent resource 'XPR Information Store (infostor)' offline before provider resource 'XPRShare S:'.
    INFO  [RCM] Bringing dependent resource 'File Server Res' offline before provider resource 'XPRShare S:'.
    INFO  [RCM] TransitionToState(XPR Message Router (mta)) Online-->WaitingToGoOffline.
    INFO  [RCM] Bringing dependent resource 'XPR Administrator(mrs)' offline before provider resource 'XPRShare S:'.
    INFO  [RCM] TransitionToState(File Server Res) Online-->WaitingToGoOffline.
    INFO  [RCM] Bringing dependent resource 'XPR License Server(licsvc)' offline before provider resource 'XPRShare S:'.
    INFO  [RCM] Bringing dependent resource 'XPR TCP/IP Transport Layer(TcpApl)' offline before provider resource 'XPRShare S:'.
    INFO  [RCM] TransitionToState(XPR License Server(licsvc)) Online-->WaitingToGoOffline.
    INFO  [RCM] Bringing dependent resource 'XPR Name Locator(nameloc)' offline before provider resource 'XPRShare S:'.
    INFO  [RCM] TransitionToState(XPR Name Locator(nameloc)) Online-->WaitingToGoOffline.
    INFO  [RCM] Bringing dependent resource 'XPR Configuration Service(cfgsvc)' offline before provider resource 'XPRShare S:'.
    INFO  [RCM] TransitionToState(XPR Configuration Service(cfgsvc)) Online-->WaitingToGoOffline.
    INFO  [RCM] Bringing dependent resource 'XPR Status Dispatcher(xmrsvc)' offline before provider resource 'XPRShare S:'.
    INFO  [RCM] 'XPR Status Dispatcher(xmrsvc)' cannot go offline yet; 'XPR Information Store (infostor)' is in state Online.
    INFO  [RCM] TransitionToState(XPR Status Dispatcher(xmrsvc)) Online-->WaitingToGoOffline.
    INFO  [RCM] 'XPR Configuration Service(cfgsvc)' cannot go offline yet; 'XPR Status Dispatcher(xmrsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Name Locator(nameloc)' cannot go offline yet; 'XPR Configuration Service(cfgsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR TCP/IP Transport Layer(TcpApl)' cannot go offline yet; 'stunnel' is in state OfflineCallIssued.
    INFO  [RCM] 'XPR License Server(licsvc)' cannot go offline yet; 'XPR Name Locator(nameloc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Administrator(mrs)' cannot go offline yet; 'XPR Internet Mail APL(SmtpApl)' is in state OfflineCallIssued.
    INFO  [RCM] 'File Server Res' cannot go offline yet; 'XPR License Server(licsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Information Store (infostor)' cannot go offline yet; 'XPR Message Router (mta)' is in state WaitingToGoOffline.
    INFO  [RCM] TransitionToState(XPR Information Store (infostor)) Online-->WaitingToGoOffline.
    INFO  [RCM] 'XPR Message Router (mta)' cannot go offline yet; 'XPR Administrator(mrs)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPRShare S:' cannot go offline yet; 'XPR Message Router (mta)' is in state WaitingToGoOffline.
    INFO  [RCM] TransitionToState(IP Address 172.16.0.119) Online-->WaitingToGoOffline.
    INFO  [RCM] Bringing dependent resource 'XPR2008Cluster' offline before provider resource 'IP Address 172.16.0.119'.
    INFO  [RCM] TransitionToState(XPR2008Cluster) Online-->WaitingToGoOffline.
    INFO  [RCM] Bringing dependent resource 'XPR Information Store (infostor)' offline before provider resource 'IP Address 172.16.0.119'.
    INFO  [RCM] Bringing dependent resource 'File Server Res' offline before provider resource 'IP Address 172.16.0.119'.
    INFO  [RCM] Bringing dependent resource 'XPR Message Router (mta)' offline before provider resource 'IP Address 172.16.0.119'.
    INFO  [RCM] Bringing dependent resource 'XPR License Server(licsvc)' offline before provider resource 'IP Address 172.16.0.119'.
    INFO  [RCM] Bringing dependent resource 'XPR Administrator(mrs)' offline before provider resource 'IP Address 172.16.0.119'.
    INFO  [RCM] Bringing dependent resource 'XPR Name Locator(nameloc)' offline before provider resource 'IP Address 172.16.0.119'.
    INFO  [RCM] Bringing dependent resource 'XPR TCP/IP Transport Layer(TcpApl)' offline before provider resource 'IP Address 172.16.0.119'.
    INFO  [RCM] Bringing dependent resource 'XPR Configuration Service(cfgsvc)' offline before provider resource 'IP Address 172.16.0.119'.
    INFO  [RCM] Bringing dependent resource 'XPR Status Dispatcher(xmrsvc)' offline before provider resource 'IP Address 172.16.0.119'.
    INFO  [RCM] 'XPR Status Dispatcher(xmrsvc)' cannot go offline yet; 'XPR Information Store (infostor)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Configuration Service(cfgsvc)' cannot go offline yet; 'XPR Status Dispatcher(xmrsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR TCP/IP Transport Layer(TcpApl)' cannot go offline yet; 'stunnel' is in state OfflineCallIssued.
    INFO  [RCM] 'XPR Name Locator(nameloc)' cannot go offline yet; 'XPR Configuration Service(cfgsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Administrator(mrs)' cannot go offline yet; 'XPR Internet Mail APL(SmtpApl)' is in state OfflineCallIssued.
    INFO  [RCM] 'XPR License Server(licsvc)' cannot go offline yet; 'XPR Name Locator(nameloc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Message Router (mta)' cannot go offline yet; 'XPR Administrator(mrs)' is in state WaitingToGoOffline.
    INFO  [RCM] 'File Server Res' cannot go offline yet; 'XPR License Server(licsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Information Store (infostor)' cannot go offline yet; 'XPR Message Router (mta)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR2008Cluster' cannot go offline yet; 'XPR Information Store (infostor)' is in state WaitingToGoOffline.
    INFO  [RCM] 'IP Address 172.16.0.119' cannot go offline yet; 'XPR2008Cluster' is in state WaitingToGoOffline.
    INFO  [RCM] TransitionToState(XPR Notification APL(NotApl)) Online-->OfflineCallIssued.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'stunnel', gen(0) result 997.
    INFO  [RCM] TransitionToState(stunnel) OfflineCallIssued-->OfflinePending.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'XPR Directory Service(DirSvc)', gen(0) result 997.
    INFO  [RCM] TransitionToState(XPR Directory Service(DirSvc)) OfflineCallIssued-->OfflinePending.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'XPR Internet Mail APL(SmtpApl)', gen(0) result 997.
    INFO  [RCM] TransitionToState(XPR Internet Mail APL(SmtpApl)) OfflineCallIssued-->OfflinePending.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'XPR Notification APL(NotApl)', gen(0) result 997.
    INFO  [RCM] TransitionToState(XPR Notification APL(NotApl)) OfflineCallIssued-->OfflinePending.
    INFO  [RES] Generic Service <XPR Notification APL(NotApl)>: Service died or not active any more; status = 1062.
    INFO  [RES] Generic Service <XPR Directory Service(DirSvc)>: Service died or not active any more; status = 1062.
    INFO  [RES] Generic Service <XPR Notification APL(NotApl)>: Service is now offline.
    INFO  [RHS] Resource XPR Notification APL(NotApl) has come offline. RHS is about to report resource status to RCM.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'XPR Notification APL(NotApl)', gen(0) result 0.
    INFO  [RCM] TransitionToState(XPR Notification APL(NotApl)) OfflinePending-->OfflineSavingCheckpoints.
    INFO  [RCM] TransitionToState(XPR Notification APL(NotApl)) OfflineSavingCheckpoints-->Offline.
    INFO  [RCM] 'XPR Administrator(mrs)' cannot go offline yet; 'XPR Internet Mail APL(SmtpApl)' is in state OfflinePending.
    INFO  [RCM] 'XPRShare S:' cannot go offline yet; 'XPR Message Router (mta)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Status Dispatcher(xmrsvc)' cannot go offline yet; 'XPR Information Store (infostor)' is in state WaitingToGoOffline.
    INFO  [RCM] 'File Server Res' cannot go offline yet; 'XPR License Server(licsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR TCP/IP Transport Layer(TcpApl)' cannot go offline yet; 'stunnel' is in state OfflinePending.
    INFO  [RCM] 'XPR License Server(licsvc)' cannot go offline yet; 'XPR Name Locator(nameloc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Name Locator(nameloc)' cannot go offline yet; 'XPR Configuration Service(cfgsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Configuration Service(cfgsvc)' cannot go offline yet; 'XPR Status Dispatcher(xmrsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Message Router (mta)' cannot go offline yet; 'XPR Administrator(mrs)' is in state WaitingToGoOffline.
    INFO  [RCM] 'IP Address 172.16.0.119' cannot go offline yet; 'XPR2008Cluster' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR2008Cluster' cannot go offline yet; 'XPR Information Store (infostor)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Information Store (infostor)' cannot go offline yet; 'XPR Message Router (mta)' is in state WaitingToGoOffline.
    INFO  [RES] Generic Service <XPR Directory Service(DirSvc)>: Service is now offline.
    INFO  [RHS] Resource XPR Directory Service(DirSvc) has come offline. RHS is about to report resource status to RCM.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'XPR Directory Service(DirSvc)', gen(0) result 0.
    INFO  [RCM] TransitionToState(XPR Directory Service(DirSvc)) OfflinePending-->OfflineSavingCheckpoints.
    INFO  [RCM] TransitionToState(XPR Directory Service(DirSvc)) OfflineSavingCheckpoints-->Offline.
    INFO  [RCM] 'XPR Administrator(mrs)' cannot go offline yet; 'XPR Internet Mail APL(SmtpApl)' is in state OfflinePending.
    INFO  [RCM] 'XPRShare S:' cannot go offline yet; 'XPR Message Router (mta)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Status Dispatcher(xmrsvc)' cannot go offline yet; 'XPR Information Store (infostor)' is in state WaitingToGoOffline.
    INFO  [RCM] 'File Server Res' cannot go offline yet; 'XPR License Server(licsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR TCP/IP Transport Layer(TcpApl)' cannot go offline yet; 'stunnel' is in state OfflinePending.
    INFO  [RCM] 'XPR License Server(licsvc)' cannot go offline yet; 'XPR Name Locator(nameloc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Name Locator(nameloc)' cannot go offline yet; 'XPR Configuration Service(cfgsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Configuration Service(cfgsvc)' cannot go offline yet; 'XPR Status Dispatcher(xmrsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Message Router (mta)' cannot go offline yet; 'XPR Administrator(mrs)' is in state WaitingToGoOffline.
    INFO  [RCM] 'IP Address 172.16.0.119' cannot go offline yet; 'XPR2008Cluster' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR2008Cluster' cannot go offline yet; 'XPR Information Store (infostor)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Information Store (infostor)' cannot go offline yet; 'XPR Message Router (mta)' is in state WaitingToGoOffline.
    INFO  [RES] Generic Service <stunnel>: Service died or not active any more; status = 1062.
    INFO  [RES] Generic Service <stunnel>: Service is now offline.
    INFO  [RHS] Resource stunnel has come offline. RHS is about to report resource status to RCM.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'stunnel', gen(0) result 0.
    INFO  [RCM] TransitionToState(stunnel) OfflinePending-->OfflineSavingCheckpoints.
    INFO  [RCM] TransitionToState(stunnel) OfflineSavingCheckpoints-->Offline.
    INFO  [RCM] 'XPR Administrator(mrs)' cannot go offline yet; 'XPR Internet Mail APL(SmtpApl)' is in state OfflinePending.
    INFO  [RCM] 'XPRShare S:' cannot go offline yet; 'XPR Message Router (mta)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Status Dispatcher(xmrsvc)' cannot go offline yet; 'XPR Information Store (infostor)' is in state WaitingToGoOffline.
    INFO  [RCM] 'File Server Res' cannot go offline yet; 'XPR License Server(licsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] TransitionToState(XPR TCP/IP Transport Layer(TcpApl)) WaitingToGoOffline-->OfflineCallIssued.
    INFO  [RCM] 'XPR License Server(licsvc)' cannot go offline yet; 'XPR Name Locator(nameloc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Name Locator(nameloc)' cannot go offline yet; 'XPR Configuration Service(cfgsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Configuration Service(cfgsvc)' cannot go offline yet; 'XPR Status Dispatcher(xmrsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Message Router (mta)' cannot go offline yet; 'XPR Administrator(mrs)' is in state WaitingToGoOffline.
    INFO  [RCM] 'IP Address 172.16.0.119' cannot go offline yet; 'XPR2008Cluster' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR2008Cluster' cannot go offline yet; 'XPR Information Store (infostor)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Information Store (infostor)' cannot go offline yet; 'XPR Message Router (mta)' is in state WaitingToGoOffline.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'XPR TCP/IP Transport Layer(TcpApl)', gen(0) result 997.
    INFO  [RCM] TransitionToState(XPR TCP/IP Transport Layer(TcpApl)) OfflineCallIssued-->OfflinePending.
    INFO  [RES] Generic Service <XPR Internet Mail APL(SmtpApl)>: Service died or not active any more; status = 1062.
    INFO  [RES] Generic Service <XPR Internet Mail APL(SmtpApl)>: Service is now offline.
    INFO  [RHS] Resource XPR Internet Mail APL(SmtpApl) has come offline. RHS is about to report resource status to RCM.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'XPR Internet Mail APL(SmtpApl)', gen(0) result 0.
    INFO  [RCM] TransitionToState(XPR Internet Mail APL(SmtpApl)) OfflinePending-->OfflineSavingCheckpoints.
    INFO  [RCM] TransitionToState(XPR Internet Mail APL(SmtpApl)) OfflineSavingCheckpoints-->Offline.
    INFO  [RCM] 'XPR Administrator(mrs)' cannot go offline yet; 'XPR TCP/IP Transport Layer(TcpApl)' is in state OfflinePending.
    INFO  [RCM] 'XPRShare S:' cannot go offline yet; 'XPR Message Router (mta)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Status Dispatcher(xmrsvc)' cannot go offline yet; 'XPR Information Store (infostor)' is in state WaitingToGoOffline.
    INFO  [RCM] 'File Server Res' cannot go offline yet; 'XPR License Server(licsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR License Server(licsvc)' cannot go offline yet; 'XPR Name Locator(nameloc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Name Locator(nameloc)' cannot go offline yet; 'XPR Configuration Service(cfgsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Configuration Service(cfgsvc)' cannot go offline yet; 'XPR Status Dispatcher(xmrsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Message Router (mta)' cannot go offline yet; 'XPR Administrator(mrs)' is in state WaitingToGoOffline.
    INFO  [RCM] 'IP Address 172.16.0.119' cannot go offline yet; 'XPR2008Cluster' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR2008Cluster' cannot go offline yet; 'XPR Information Store (infostor)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Information Store (infostor)' cannot go offline yet; 'XPR Message Router (mta)' is in state WaitingToGoOffline.
    INFO  [RES] Generic Service <XPR TCP/IP Transport Layer(TcpApl)>: Service died or not active any more; status = 1062.
    INFO  [RES] Generic Service <XPR TCP/IP Transport Layer(TcpApl)>: Service is now offline.
    INFO  [RHS] Resource XPR TCP/IP Transport Layer(TcpApl) has come offline. RHS is about to report resource status to RCM.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'XPR TCP/IP Transport Layer(TcpApl)', gen(0) result 0.
    INFO  [RCM] TransitionToState(XPR TCP/IP Transport Layer(TcpApl)) OfflinePending-->OfflineSavingCheckpoints.
    INFO  [RCM] TransitionToState(XPR TCP/IP Transport Layer(TcpApl)) OfflineSavingCheckpoints-->Offline.
    INFO  [RCM] TransitionToState(XPR Administrator(mrs)) WaitingToGoOffline-->OfflineCallIssued.
    INFO  [RCM] 'XPRShare S:' cannot go offline yet; 'XPR Message Router (mta)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Status Dispatcher(xmrsvc)' cannot go offline yet; 'XPR Information Store (infostor)' is in state WaitingToGoOffline.
    INFO  [RCM] 'File Server Res' cannot go offline yet; 'XPR License Server(licsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR License Server(licsvc)' cannot go offline yet; 'XPR Name Locator(nameloc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Name Locator(nameloc)' cannot go offline yet; 'XPR Configuration Service(cfgsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Configuration Service(cfgsvc)' cannot go offline yet; 'XPR Status Dispatcher(xmrsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Message Router (mta)' cannot go offline yet; 'XPR Administrator(mrs)' is in state OfflineCallIssued.
    INFO  [RCM] 'IP Address 172.16.0.119' cannot go offline yet; 'XPR2008Cluster' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR2008Cluster' cannot go offline yet; 'XPR Information Store (infostor)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Information Store (infostor)' cannot go offline yet; 'XPR Message Router (mta)' is in state WaitingToGoOffline.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'XPR Administrator(mrs)', gen(0) result 997.
    INFO  [RCM] TransitionToState(XPR Administrator(mrs)) OfflineCallIssued-->OfflinePending.
    ERR   [RES] mrsClusresMTA <XPR Message Router (mta)>: CheckIsAlive: Verification of the 'MTA' service failed. Error: 1062.
    WARN  [RHS] Resource XPR Message Router (mta) IsAlive has indicated failure.
    INFO  [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'XPR Message Router (mta)', gen(0) result 1.
    INFO  [RCM] TransitionToState(XPR Message Router (mta)) WaitingToGoOffline-->ProcessingFailure.
    ERR   [RCM] rcm::RcmResource::HandleFailure: (XPR Message Router (mta))
    INFO  [RCM] Since no top level resource will be offlined by this resource's failure, resetting restartAction to 1
    INFO  [RCM] resource XPR Message Router (mta): failure count: 0, restartAction: 1.
    INFO  [RCM] resource XPRShare S: is waiting for resource XPR Message Router (mta) in state ResWaitingToGoOffline, will not restart
    INFO  [RCM] resource XPR Message Router (mta) will not be restarting; isLowPriority: false; numDependents: 1, failureCount: 0, restartAction: 1
    INFO  [RCM] TransitionToState(XPR Message Router (mta)) ProcessingFailure-->[WaitingToTerminate to Failed].
    INFO  [RCM] TransitionToState(XPR Message Router (mta)) [WaitingToTerminate to Failed]-->[Terminating to Failed].
    INFO  [RCM] Resource XPR Message Router (mta) stopping completely because it is in a low priority group.
    INFO  [RES] mrsClusresMTA <XPR Message Router (mta)>: Terminate request.
    INFO  [RES] mrsClusresMTA <XPR Message Router (mta)>: Terminate: retrying...
    INFO  [RES] mrsClusresMTA <XPR Message Router (mta)>: Terminate: retrying...
    INFO  [RES] mrsClusresMTA <XPR Message Router (mta)>: Terminate: retrying...
    INFO  [RES] mrsClusresMTA <XPR Message Router (mta)>: Terminate: retrying...
    INFO  [RES] mrsClusresMTA <XPR Message Router (mta)>: Terminate: retrying...
    INFO  [RES] mrsClusresMTA <XPR Message Router (mta)>: Terminate: retrying...
    INFO  [RES] mrsClusresMTA <XPR Message Router (mta)>: Terminate: Service died; status = 1062.
    INFO  [RCM] HandleMonitorReply: TERMINATERESOURCE for 'XPR Message Router (mta)', gen(1) result 0.
    INFO  [RCM] TransitionToState(XPR Message Router (mta)) [Terminating to Failed]-->Failed.
    INFO  [RCM] 'XPRShare S:' cannot go offline yet; 'XPR Information Store (infostor)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Status Dispatcher(xmrsvc)' cannot go offline yet; 'XPR Information Store (infostor)' is in state WaitingToGoOffline.
    INFO  [RCM] 'File Server Res' cannot go offline yet; 'XPR License Server(licsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR License Server(licsvc)' cannot go offline yet; 'XPR Name Locator(nameloc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Name Locator(nameloc)' cannot go offline yet; 'XPR Configuration Service(cfgsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Configuration Service(cfgsvc)' cannot go offline yet; 'XPR Status Dispatcher(xmrsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'IP Address 172.16.0.119' cannot go offline yet; 'XPR2008Cluster' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR2008Cluster' cannot go offline yet; 'XPR Information Store (infostor)' is in state WaitingToGoOffline.
    INFO  [RCM] TransitionToState(XPR Information Store (infostor)) WaitingToGoOffline-->OfflineCallIssued.
    INFO  [RES] mrsclusres <XPR Information Store (infostor)>: Offline request.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'XPR Information Store (infostor)', gen(0) result 997.
    INFO  [RCM] TransitionToState(XPR Information Store (infostor)) OfflineCallIssued-->OfflinePending.
    INFO  [RES] mrsclusres <XPR Information Store (infostor)>: OfflineThread: Process with id 3408 still exists
    INFO  [RES] mrsclusres <XPR Information Store (infostor)>: OfflineThread: retrying...
    INFO  [RES] mrsclusres <XPR Information Store (infostor)>: OfflineThread: ControlService failed with ec 1061
    INFO  [RES] mrsclusres <XPR Information Store (infostor)>: OfflineThread: Process with id 3408 still exists
    INFO  [RES] mrsclusres <XPR Information Store (infostor)>: OfflineThread: retrying...
    INFO  [RES] mrsclusres <XPR Information Store (infostor)>: OfflineThread: ControlService failed with ec 1062
    INFO  [RES] mrsclusres <XPR Information Store (infostor)>: OfflineThread: cannot open process with id 3408 - error 87
    INFO  [RES] mrsclusres <XPR Information Store (infostor)>: OfflineThread: The 'InfoStor' service died or is not active any more; status = 1062.
    INFO  [RES] mrsclusres <XPR Information Store (infostor)>: OfflineThread: Service is now offline.
    INFO  [RHS] Resource XPR Information Store (infostor) has come offline. RHS is about to report resource status to RCM.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'XPR Information Store (infostor)', gen(0) result 0.
    INFO  [RCM] TransitionToState(XPR Information Store (infostor)) OfflinePending-->OfflineSavingCheckpoints.
    INFO  [RCM] TransitionToState(XPR Information Store (infostor)) OfflineSavingCheckpoints-->Offline.
    INFO  [RCM] 'XPRShare S:' cannot go offline yet; 'File Server Res' is in state WaitingToGoOffline.
    INFO  [RCM] TransitionToState(XPR Status Dispatcher(xmrsvc)) WaitingToGoOffline-->OfflineCallIssued.
    INFO  [RCM] 'File Server Res' cannot go offline yet; 'XPR License Server(licsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR License Server(licsvc)' cannot go offline yet; 'XPR Name Locator(nameloc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Name Locator(nameloc)' cannot go offline yet; 'XPR Configuration Service(cfgsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Configuration Service(cfgsvc)' cannot go offline yet; 'XPR Status Dispatcher(xmrsvc)' is in state OfflineCallIssued.
    INFO  [RCM] 'IP Address 172.16.0.119' cannot go offline yet; 'XPR2008Cluster' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR2008Cluster' cannot go offline yet; 'File Server Res' is in state WaitingToGoOffline.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'XPR Status Dispatcher(xmrsvc)', gen(0) result 997.
    INFO  [RCM] TransitionToState(XPR Status Dispatcher(xmrsvc)) OfflineCallIssued-->OfflinePending.
    INFO  [RES] Generic Service <XPR Status Dispatcher(xmrsvc)>: Service died or not active any more; status = 1062.
    INFO  [RES] Generic Service <XPR Status Dispatcher(xmrsvc)>: Service is now offline.
    INFO  [RHS] Resource XPR Status Dispatcher(xmrsvc) has come offline. RHS is about to report resource status to RCM.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'XPR Status Dispatcher(xmrsvc)', gen(0) result 0.
    INFO  [RCM] TransitionToState(XPR Status Dispatcher(xmrsvc)) OfflinePending-->OfflineSavingCheckpoints.
    INFO  [RCM] TransitionToState(XPR Status Dispatcher(xmrsvc)) OfflineSavingCheckpoints-->Offline.
    INFO  [RCM] 'XPRShare S:' cannot go offline yet; 'File Server Res' is in state WaitingToGoOffline.
    INFO  [RCM] 'File Server Res' cannot go offline yet; 'XPR License Server(licsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR License Server(licsvc)' cannot go offline yet; 'XPR Name Locator(nameloc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR Name Locator(nameloc)' cannot go offline yet; 'XPR Configuration Service(cfgsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] TransitionToState(XPR Configuration Service(cfgsvc)) WaitingToGoOffline-->OfflineCallIssued.
    INFO  [RCM] 'IP Address 172.16.0.119' cannot go offline yet; 'XPR2008Cluster' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR2008Cluster' cannot go offline yet; 'File Server Res' is in state WaitingToGoOffline.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'XPR Configuration Service(cfgsvc)', gen(0) result 997.
    INFO  [RCM] TransitionToState(XPR Configuration Service(cfgsvc)) OfflineCallIssued-->OfflinePending.
    INFO  [RES] Generic Service <XPR Configuration Service(cfgsvc)>: Service died or not active any more; status = 1062.
    INFO  [RES] Generic Service <XPR Configuration Service(cfgsvc)>: Service is now offline.
    INFO  [RHS] Resource XPR Configuration Service(cfgsvc) has come offline. RHS is about to report resource status to RCM.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'XPR Configuration Service(cfgsvc)', gen(0) result 0.
    INFO  [RCM] TransitionToState(XPR Configuration Service(cfgsvc)) OfflinePending-->OfflineSavingCheckpoints.
    INFO  [RCM] TransitionToState(XPR Configuration Service(cfgsvc)) OfflineSavingCheckpoints-->Offline.
    INFO  [RCM] 'XPRShare S:' cannot go offline yet; 'File Server Res' is in state WaitingToGoOffline.
    INFO  [RCM] 'File Server Res' cannot go offline yet; 'XPR License Server(licsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR License Server(licsvc)' cannot go offline yet; 'XPR Name Locator(nameloc)' is in state WaitingToGoOffline.
    INFO  [RCM] TransitionToState(XPR Name Locator(nameloc)) WaitingToGoOffline-->OfflineCallIssued.
    INFO  [RCM] 'IP Address 172.16.0.119' cannot go offline yet; 'XPR2008Cluster' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR2008Cluster' cannot go offline yet; 'File Server Res' is in state WaitingToGoOffline.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'XPR Name Locator(nameloc)', gen(0) result 997.
    INFO  [RCM] TransitionToState(XPR Name Locator(nameloc)) OfflineCallIssued-->OfflinePending.
    INFO  [RES] Generic Service <XPR Name Locator(nameloc)>: Service died or not active any more; status = 1062.
    INFO  [RES] Generic Service <XPR Name Locator(nameloc)>: Service is now offline.
    INFO  [RHS] Resource XPR Name Locator(nameloc) has come offline. RHS is about to report resource status to RCM.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'XPR Name Locator(nameloc)', gen(0) result 0.
    INFO  [RCM] TransitionToState(XPR Name Locator(nameloc)) OfflinePending-->OfflineSavingCheckpoints.
    INFO  [RCM] TransitionToState(XPR Name Locator(nameloc)) OfflineSavingCheckpoints-->Offline.
    INFO  [RCM] 'XPRShare S:' cannot go offline yet; 'File Server Res' is in state WaitingToGoOffline.
    INFO  [RCM] 'File Server Res' cannot go offline yet; 'XPR License Server(licsvc)' is in state WaitingToGoOffline.
    INFO  [RCM] TransitionToState(XPR License Server(licsvc)) WaitingToGoOffline-->OfflineCallIssued.
    INFO  [RCM] 'IP Address 172.16.0.119' cannot go offline yet; 'XPR2008Cluster' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR2008Cluster' cannot go offline yet; 'File Server Res' is in state WaitingToGoOffline.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'XPR License Server(licsvc)', gen(0) result 997.
    INFO  [RCM] TransitionToState(XPR License Server(licsvc)) OfflineCallIssued-->OfflinePending.
    INFO  [RES] Generic Service <XPR License Server(licsvc)>: Service died or not active any more; status = 1062.
    INFO  [RES] Generic Service <XPR License Server(licsvc)>: Service is now offline.
    INFO  [RHS] Resource XPR License Server(licsvc) has come offline. RHS is about to report resource status to RCM.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'XPR License Server(licsvc)', gen(0) result 0.
    INFO  [RCM] TransitionToState(XPR License Server(licsvc)) OfflinePending-->OfflineSavingCheckpoints.
    INFO  [RCM] TransitionToState(XPR License Server(licsvc)) OfflineSavingCheckpoints-->Offline.
    INFO  [RCM] 'XPRShare S:' cannot go offline yet; 'File Server Res' is in state WaitingToGoOffline.
    INFO  [RCM] TransitionToState(File Server Res) WaitingToGoOffline-->OfflineCallIssued.
    INFO  [RCM] 'IP Address 172.16.0.119' cannot go offline yet; 'XPR2008Cluster' is in state WaitingToGoOffline.
    INFO  [RCM] 'XPR2008Cluster' cannot go offline yet; 'File Server Res' is in state OfflineCallIssued.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'File Server Res', gen(0) result 997.
    INFO  [RCM] TransitionToState(File Server Res) OfflineCallIssued-->OfflinePending.
    INFO  [RES] File Server <File Server Res>: FileServerDoTerminate: Terminate called... !!!
    INFO  [RHS] Resource File Server Res has come offline. RHS is about to report resource status to RCM.
    INFO  [RES] File Server <File Server Res>: FileServer is now offline.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'File Server Res', gen(0) result 0.
    INFO  [RCM] TransitionToState(File Server Res) OfflinePending-->OfflineSavingCheckpoints.
    INFO  [RCM] TransitionToState(File Server Res) OfflineSavingCheckpoints-->Offline.
    INFO  [RCM] TransitionToState(XPRShare S:) WaitingToGoOffline-->OfflineCallIssued.
    INFO  [RCM] 'IP Address 172.16.0.119' cannot go offline yet; 'XPR2008Cluster' is in state WaitingToGoOffline.
    INFO  [RCM] TransitionToState(XPR2008Cluster) WaitingToGoOffline-->OfflineCallIssued.
    INFO  [RES] Physical Disk <XPRShare S:>: Offline request.
    INFO  [RES] Network Name <XPR2008Cluster>: Taking resource offline...
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'XPRShare S:', gen(0) result 997.
    INFO  [RCM] TransitionToState(XPRShare S:) OfflineCallIssued-->OfflinePending.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'XPR2008Cluster', gen(0) result 997.
    INFO  [RCM] TransitionToState(XPR2008Cluster) OfflineCallIssued-->OfflinePending.
    INFO  [RES] Physical Disk: DriveLetter mask: 0x40000
    INFO  [RES] Network Name <XPR2008Cluster>: TimerQueueTimer rescheduled to fire after 600 secs
    INFO  [RES] Network Name <XPR2008Cluster>: Offline of resource continuing...
    INFO  [RES] Physical Disk: HardDiskpScopeShareCallback: Enter resourceName XPR2008Cluster
    INFO  [RES] Physical Disk: Attempt to REMOVE admin share S$ to/from netname XPR2008CLUSTER, returned 0
    INFO  [RES] Physical Disk <XPRShare S:>: HardDiskpCloseSVIHandles: Exit
    INFO  [RES] Physical Disk: Enter EnumerateDevices: EnumDevice 0
    INFO  [RES] Physical Disk: Exit EnumerateDevices: status 0
    INFO  [RES] Physical Disk: Enter EnumerateDevices: EnumDevice 0
    INFO  [RES] Physical Disk: Exit EnumerateDevices: status 0
    INFO  [RES] Network Name <XPR2008Cluster>: DNS name XPR2008Cluster successful removed from LSA
    INFO  [ClNet] Adapter Local Area Connection* 11 RFC2863 operational status = 1.
    DBG   [ClNet] Created adapter: DeviceGuid:     8D040988-0E89-4BD9-A1CB-23ED95BBBE83
    DBG   [ClNet]                  DeviceName:     Microsoft Failover Cluster Virtual Adapter
    DBG   [ClNet]                  ConnectoidName: Local Area Connection* 11
    DBG   [ClNet]                  Netbios/TCP:    1
    DBG   [ClNet]                  DNS Suffix:
    DBG   [ClNet]                  DnsServer:      fec0:0:0:ffff::1%1
    DBG   [ClNet]                  DnsServer:      fec0:0:0:ffff::2%1
    DBG   [ClNet]                  DnsServer:      fec0:0:0:ffff::3%1
    INFO  [ClNet] Adapter PUBLIC RFC2863 operational status = 1.
    DBG   [ClNet] Created adapter: DeviceGuid:     5AB941FF-2174-4E0A-AEA0-B32021BA49E0
    DBG   [ClNet]                  DeviceName:     Intel(R) PRO/1000 MT Network Connection
    DBG   [ClNet]                  ConnectoidName: PUBLIC
    DBG   [ClNet]                  Netbios/TCP:    1
    DBG   [ClNet]                  DNS Suffix:
    DBG   [ClNet]                  DnsServer:      192.168.1.132
    INFO  [ClNet] Adapter PRIVATE RFC2863 operational status = 1.
    DBG   [ClNet] Created adapter: DeviceGuid:     61698386-DDD2-494B-9730-C761CA40A24E
    DBG   [ClNet]                  DeviceName:     Intel(R) PRO/1000 MT Network Connection #2
    DBG   [ClNet]                  ConnectoidName: PRIVATE
    DBG   [ClNet]                  Netbios/TCP:    1
    DBG   [ClNet]                  DNS Suffix:
    INFO  [ClNet] Adapter isatap.{5AB941FF-2174-4E0A-AEA0-B32021BA49E0} RFC2863 operational status = 2.
    DBG   [ClNet] Created adapter: DeviceGuid:     F52CCEA7-A384-40B9-BD5A-03567F375090
    DBG   [ClNet]                  DeviceName:     Microsoft ISATAP Adapter
    DBG   [ClNet]                  ConnectoidName: isatap.{5AB941FF-2174-4E0A-AEA0-B32021BA49E0}
    DBG   [ClNet]                  Netbios/TCP:    0
    DBG   [ClNet]                  DNS Suffix:
    DBG   [ClNet]                  DnsServer:      192.168.1.132
    INFO  [ClNet] Adapter Local Area Connection* 9 RFC2863 operational status = 2.
    DBG   [ClNet] Created adapter: DeviceGuid:     3F940A69-7E7D-4B9F-AC4D-180DBAF3C1D8
    DBG   [ClNet]                  DeviceName:     Teredo Tunneling Pseudo-Interface
    DBG   [ClNet]                  ConnectoidName: Local Area Connection* 9
    DBG   [ClNet]                  Netbios/TCP:    0
    DBG   [ClNet]                  DNS Suffix:
    INFO  [ClNet] Adapter isatap.{61698386-DDD2-494B-9730-C761CA40A24E} RFC2863 operational status = 2.
    DBG   [ClNet] Created adapter: DeviceGuid:     F631AE92-702F-4106-8457-709CD3CEE9AE
    DBG   [ClNet]                  DeviceName:     Microsoft ISATAP Adapter #2
    DBG   [ClNet]                  ConnectoidName: isatap.{61698386-DDD2-494B-9730-C761CA40A24E}
    DBG   [ClNet]                  Netbios/TCP:    0
    DBG   [ClNet]                  DNS Suffix:
    INFO  [ClNet] Adapter isatap.{8D040988-0E89-4BD9-A1CB-23ED95BBBE83} RFC2863 operational status = 2.
    DBG   [ClNet] Created adapter: DeviceGuid:     41E2EFCF-7289-4ECF-9F0F-552ED553A67D
    DBG   [ClNet]                  DeviceName:     Microsoft ISATAP Adapter #3
    DBG   [ClNet]                  ConnectoidName: isatap.{8D040988-0E89-4BD9-A1CB-23ED95BBBE83}
    DBG   [ClNet]                  Netbios/TCP:    0
    DBG   [ClNet]                  DNS Suffix:
    DBG   [ClNet]                  DnsServer:      fec0:0:0:ffff::1%1
    DBG   [ClNet]                  DnsServer:      fec0:0:0:ffff::2%1
    DBG   [ClNet]                  DnsServer:      fec0:0:0:ffff::3%1
    INFO  [RES] Network Name <XPR2008Cluster>: adapter PUBLIC (1)
    INFO  [RES] Network Name <XPR2008Cluster>: Dynamic DNS disabled on adapter PUBLIC (No IPs on this adapter will be registered in DNS)
    INFO  [RES] Network Name <XPR2008Cluster>: FQDN name XPR2008Cluster.auld17495.com removal with LSA was successful
    WARN  [RES] Physical Disk <XPRShare S:>: OfflineThread: Failed to lock volume \Device\Harddisk2\Partition1, Error 5
    INFO  [RES] Network Name <XPR2008Cluster>: Deleted server name XPR2008CLUSTER from all transports.
    INFO  [RES] Network Name <XPR2008Cluster>: Deleted workstation name XPR2008CLUSTER from transport 0.
    INFO  [RHS] Resource XPR2008Cluster has come offline. RHS is about to report resource status to RCM.
    INFO  [RES] Network Name <XPR2008Cluster>: Resource is now offline
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'XPR2008Cluster', gen(0) result 0.
    INFO  [RCM] TransitionToState(XPR2008Cluster) OfflinePending-->OfflineSavingCheckpoints.
    INFO  [RCM] TransitionToState(XPR2008Cluster) OfflineSavingCheckpoints-->Offline.
    INFO  [RCM] TransitionToState(IP Address 172.16.0.119) WaitingToGoOffline-->OfflineCallIssued.
    INFO  [RES] IP Address <IP Address 172.16.0.119>: Taking resource offline...
    INFO  [RES] IP Address <IP Address 172.16.0.119>: Deleting IP interface 770010AC.
    INFO  [RES] IP Address <IP Address 172.16.0.119>: Address 172.16.0.119 on adapter Intel(R) PRO/1000 MT Network Connection offline.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'IP Address 172.16.0.119', gen(0) result 0.
    INFO  [RCM] TransitionToState(IP Address 172.16.0.119) OfflineCallIssued-->OfflineSavingCheckpoints.
    INFO  [RCM] TransitionToState(IP Address 172.16.0.119) OfflineSavingCheckpoints-->Offline.
    DBG   [NETFTAPI] received NsiDeleteInstance  for 172.16.0.119
    WARN  [NETFTAPI] Failed to query parameters for 172.16.0.119 (status 80070490)
    DBG   [NETFTAPI] Signaled NetftLocalRemove  event for 172.16.0.119
    INFO  [RES] Physical Disk: ReleaseDisk: stop reserve succeeded on device 2 (sig 28d15405)
    INFO  [RHS] Resource XPRShare S: has come offline. RHS is about to report resource status to RCM.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'XPRShare S:', gen(0) result 0.
    INFO  [RCM] TransitionToState(XPRShare S:) OfflinePending-->OfflineSavingCheckpoints.
    INFO  [RCM] TransitionToState(XPRShare S:) OfflineSavingCheckpoints-->Offline.
    INFO  [RES] Generic Service <XPR Administrator(mrs)>: Service died or not active any more; status = 1062.
    INFO  [RES] Generic Service <XPR Administrator(mrs)>: Service is now offline.
    INFO  [RHS] Resource XPR Administrator(mrs) has come offline. RHS is about to report resource status to RCM.
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'XPR Administrator(mrs)', gen(0) result 0.
    INFO  [RCM] TransitionToState(XPR Administrator(mrs)) OfflinePending-->OfflineSavingCheckpoints.
    INFO  [RCM] TransitionToState(XPR Administrator(mrs)) OfflineSavingCheckpoints-->Offline.
    INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (XPR2008Cluster, Pending --> Failed)
    INFO  [CS] PreShutdown notification.
    INFO  [CS] Service Stopping...
    INFO  [CORE] Node quorum state is 'Successfully formed or joined a cluster'. Form/join status with other nodes is as follows:
    INFO  [NODE] Node 1: Farthest reported progress joining with node 2008CLUSTERN2 (id 2) is: Join Succeeded at time 2012/03/14-15:20:04.089: status 0
    INFO  Shutdown lock acquired, proceeding with shutdown
    INFO  [RCM] TransitionToState(Cluster IP Address) Online-->WaitingToGoOffline.
    INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (Cluster Group, Online --> Pending)
    INFO  [RCM] Bringing dependent resource 'Cluster Name' offline before provider resource 'Cluster IP Address'.
    INFO  [RCM] TransitionToState(Cluster Name) Online-->OfflineCallIssued.
    INFO  [RCM] 'Cluster IP Address' cannot go offline yet; 'Cluster Name' is in state OfflineCallIssued.
    INFO  [RCM] rcm::RcmGroup::Offline: deferring offline of quorum resource 'Cluster Disk 1' until all other resources are offline.
    INFO  [RES] Network Name <Cluster Name>: Taking resource offline...
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'Cluster Name', gen(0) result 997.
    INFO  [RCM] TransitionToState(Cluster Name) OfflineCallIssued-->OfflinePending.
    INFO  [RES] Network Name <Cluster Name>: TimerQueueTimer rescheduled to fire after 600 secs
    INFO  [RES] Network Name <Cluster Name>: Offline of resource continuing...
    INFO  [RES] Network Name <Cluster Name>: DNS name 2008CLUSTER successful removed from LSA
    INFO  [ClNet] Adapter Local Area Connection* 11 RFC2863 operational status = 1.
    DBG   [ClNet] Created adapter: DeviceGuid:     8D040988-0E89-4BD9-A1CB-23ED95BBBE83
    DBG   [ClNet]                  DeviceName:     Microsoft Failover Cluster Virtual Adapter
    DBG   [ClNet]                  ConnectoidName: Local Area Connection* 11
    DBG   [ClNet]                  Netbios/TCP:    1
    DBG   [ClNet]                  DNS Suffix:
    DBG   [ClNet]                  DnsServer:      fec0:0:0:ffff::1%1
    DBG   [ClNet]                  DnsServer:      fec0:0:0:ffff::2%1
    DBG   [ClNet]                  DnsServer:      fec0:0:0:ffff::3%1
    INFO  [ClNet] Adapter PUBLIC RFC2863 operational status = 1.
    DBG   [ClNet] Created adapter: DeviceGuid:     5AB941FF-2174-4E0A-AEA0-B32021BA49E0
    DBG   [ClNet]                  DeviceName:     Intel(R) PRO/1000 MT Network Connection
    DBG   [ClNet]                  ConnectoidName: PUBLIC
    DBG   [ClNet]                  Netbios/TCP:    1
    DBG   [ClNet]                  DNS Suffix:
    DBG   [ClNet]                  DnsServer:      192.168.1.132
    INFO  [ClNet] Adapter PRIVATE RFC2863 operational status = 1.
    DBG   [ClNet] Created adapter: DeviceGuid:     61698386-DDD2-494B-9730-C761CA40A24E
    DBG   [ClNet]                  DeviceName:     Intel(R) PRO/1000 MT Network Connection #2
    DBG   [ClNet]                  ConnectoidName: PRIVATE
    DBG   [ClNet]                  Netbios/TCP:    1
    DBG   [ClNet]                  DNS Suffix:
    INFO  [ClNet] Adapter isatap.{5AB941FF-2174-4E0A-AEA0-B32021BA49E0} RFC2863 operational status = 2.
    DBG   [ClNet] Created adapter: DeviceGuid:     F52CCEA7-A384-40B9-BD5A-03567F375090
    DBG   [ClNet]                  DeviceName:     Microsoft ISATAP Adapter
    DBG   [ClNet]                  ConnectoidName: isatap.{5AB941FF-2174-4E0A-AEA0-B32021BA49E0}
    DBG   [ClNet]                  Netbios/TCP:    0
    DBG   [ClNet]                  DNS Suffix:
    DBG   [ClNet]                  DnsServer:      192.168.1.132
    INFO  [ClNet] Adapter Local Area Connection* 9 RFC2863 operational status = 2.
    DBG   [ClNet] Created adapter: DeviceGuid:     3F940A69-7E7D-4B9F-AC4D-180DBAF3C1D8
    DBG   [ClNet]                  DeviceName:     Teredo Tunneling Pseudo-Interface
    DBG   [ClNet]                  ConnectoidName: Local Area Connection* 9
    DBG   [ClNet]                  Netbios/TCP:    0
    DBG   [ClNet]                  DNS Suffix:
    INFO  [ClNet] Adapter isatap.{61698386-DDD2-494B-9730-C761CA40A24E} RFC2863 operational status = 2.
    DBG   [ClNet] Created adapter: DeviceGuid:     F631AE92-702F-4106-8457-709CD3CEE9AE
    DBG   [ClNet]                  DeviceName:     Microsoft ISATAP Adapter #2
    DBG   [ClNet]                  ConnectoidName: isatap.{61698386-DDD2-494B-9730-C761CA40A24E}
    DBG   [ClNet]                  Netbios/TCP:    0
    DBG   [ClNet]                  DNS Suffix:
    INFO  [ClNet] Adapter isatap.{8D040988-0E89-4BD9-A1CB-23ED95BBBE83} RFC2863 operational status = 2.
    DBG   [ClNet] Created adapter: DeviceGuid:     41E2EFCF-7289-4ECF-9F0F-552ED553A67D
    DBG   [ClNet]                  DeviceName:     Microsoft ISATAP Adapter #3
    DBG   [ClNet]                  ConnectoidName: isatap.{8D040988-0E89-4BD9-A1CB-23ED95BBBE83}
    DBG   [ClNet]                  Netbios/TCP:    0
    DBG   [ClNet]                  DNS Suffix:
    DBG   [ClNet]                  DnsServer:      fec0:0:0:ffff::1%1
    DBG   [ClNet]                  DnsServer:      fec0:0:0:ffff::2%1
    DBG   [ClNet]                  DnsServer:      fec0:0:0:ffff::3%1
    INFO  [RES] Network Name <Cluster Name>: adapter PUBLIC (1)
    INFO  [RES] Network Name <Cluster Name>: Dynamic DNS disabled on adapter PUBLIC (No IPs on this adapter will be registered in DNS)
    INFO  [RES] Network Name <Cluster Name>: FQDN name 2008CLUSTER.auld17495.com removal with LSA was successful
    INFO  [RES] Network Name <Cluster Name>: Deleted server name 2008CLUSTER from all transports.
    INFO  [RES] Network Name <Cluster Name>: Deleted workstation name 2008CLUSTER from transport 0.
    INFO  [RHS] Resource Cluster Name has come offline. RHS is about to report resource status to RCM.
    INFO  [RES] Network Name <Cluster Name>: Resource is now offline
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'Cluster Name', gen(0) result 0.
    INFO  [RCM] TransitionToState(Cluster Name) OfflinePending-->OfflineSavingCheckpoints.
    INFO  [RCM] TransitionToState(Cluster Name) OfflineSavingCheckpoints-->Offline.
    INFO  [RCM] TransitionToState(Cluster IP Address) WaitingToGoOffline-->OfflineCallIssued.
    INFO  [RES] IP Address <Cluster IP Address>: Taking resource offline...
    INFO  [RES] IP Address <Cluster IP Address>: Deleting IP interface 760010AC.
    INFO  [RES] IP Address <Cluster IP Address>: Address 172.16.0.118 on adapter Intel(R) PRO/1000 MT Network Connection offline.
    INFO  [RES] IP Address <Cluster IP Address>: All resources offline - cleaning up
    ERR   [RES] IP Address <Cluster IP Address>: WorkerThread: GetClusterNotify failed with status 6.
    INFO  [RES] IP Address <Cluster IP Address>: WorkerThread terminating
    INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'Cluster IP Address', gen(0) result 0.
    INFO  [RCM] TransitionToState(Cluster IP Address) OfflineCallIssued-->OfflineSavingCheckpoints.
    INFO  [RCM] TransitionToState(Cluster IP Address) OfflineSavingCheckpoints-->Offline.
    INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (Cluster Group, Pending --> PartialOnline)
    DBG   [NETFTAPI] received NsiDeleteInstance  for 172.16.0.118
    WARN  [NETFTAPI] Failed to query parameters for 172.16.0.118 (status 80070490)
    DBG   [NETFTAPI] Signaled NetftLocalRemove  event for 172.16.0.118
    INFO  [RCM] rcm::RcmGroup::DeferredQuorumOffline: Quorum resource can now be offlined.  Rest of group is offline.
    INFO  [RCM] TransitionToState(Cluster Disk 1) Online-->OfflineCallIssued.
    INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (Cluster Group, PartialOnline --> Pending)
    INFO  [QUORUM] Node 1: PreOffline for 9df7e926-0d9c-4635-a7e0-cb9902a83df1

    Wednesday, March 14, 2012 10:06 PM
  • It just looks like that MTA service is somehow terminated before the cluster gets a chance to try to bring the resource offline. Just before the IsAlive check, you can see that the MTA resource is waiting on the "MRS" resources to go offline:

    INFO  [RCM] 'XPR Message Router (mta)' cannot go offline yet; 'XPR Administrator(mrs)' is in state OfflineCallIssued.

    You never see that OfflineCallIssued state set upon the MTA resource...it just fails an IsAlive check while its in a "WaitingToGoOffline" state. You can even see that the last posted state for MTA is still "WaitingToGoOffline" in this message:

    INFO  [RCM] TransitionToState(XPR Message Router (mta)) WaitingToGoOffline-->ProcessingFailure.

    Is there anything in the event logs showing the MTA service crashing or stopping just before the failure occurs?


    Visit my blog about multi-site clustering

    • Proposed as answer by JohnToner Saturday, March 17, 2012 11:28 PM
    • Marked as answer by Vincent Hu Monday, April 9, 2012 3:46 PM
    Wednesday, March 14, 2012 11:52 PM
  • the application event log shows no error during the cluster group going offline.

    the system event log shows the mta service entered the stopped state. then an error (mta) in clustered service or application failed.

    it does appear while watching the failover monitor the mta service does terminate before it should. but i don't see how to determine how or why. it should not go offline before it's dependant "mrs".

    any suggestions on how to troubleshoot the termination?

    Thursday, March 15, 2012 2:33 PM
  • Well that tells you that something sent the "stop service" command prematurely to the MTA service and it's unlikely that the MTA service crashing. It suggests that the service was intentionally stopped rather than terminating unexpectedly.

    If you are sure that there are no dependencies which might have caused this service to stop, then you would need to shift the blame towards the custom resource DLL. Its possible that the DLL was written in a manner that it isn't handling the "offline pending" state properly and is prematurely terminating the MTA service. I'm not a programmer so that's pretty much the extent of the assistance I can offer.  

    Good luck.


    Visit my blog about multi-site clustering

    Thursday, March 15, 2012 3:34 PM
  • thank you John,

    i have learn to read the cluster logs better.

    Thursday, March 15, 2012 3:52 PM