none
SQL 2012 Failover Cluster - unable to start because of 'Network Name' failed.

    General discussion

  • Hi all,

    Running a 2012R2 Failover Cluster with SQL 2012. I'm unable to start the SQL 2012 Cluster Role because of the following error;

    Log Name:      System
    Source:        Microsoft-Windows-FailoverClustering
    Event ID:      1069
    Description:
    Cluster resource 'SQL Network Name (SCSQLCL01)' of type 'Network Name' in clustered role 'SQL Server (VMM)' failed.

    Failover cluster manager shows the following;

     

    Observations thus far;

    • Passes all cluster validation tests (no issues)
    • Am sometimes seeing Kerberos errors in the log for both cluster members, but it's not consistent and I cannot pin down the cause;

    The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server scsqlcl01-2$. The target name used was MSServerClusterMgmtAPI/SCSQLCL01CORE.service.local.

    • The cluster computer object has been granted permissions on the cluster
    • All computer objects are created, and DNS entries are present.
    • It sometimes "just works". It comes online without a hitch and I can communicate with the cluster name using the SQL instance no problems

    Any help would be appreciated.

    Thanks.

     

     

     

     

    Friday, August 23, 2013 2:21 PM

All replies

  • You will probably want to take a look at the cluster logs to see why this is failing and then troubleshoot from there. 

    To generate the cluster log, issue the following command:

    Get-ClusterLog -UseLocalTime

    This will generate a cluster.log file in the C:\Windows\Cluster\Reports folder. You can then use the System Event log to look at the time of the failure (1069 event) and line this up with what is happening in the cluster logs.

    FYI, if the UseLocalTime flag is omitted, the cluster.log will in GMT time. This flag just helps you reading these logs and lining up events.

    Hope this helps.


    Visit my blog about multi-site clustering

    Friday, August 23, 2013 8:51 PM
    Moderator
  • John - you legend - thanks for the PS command.

    Ok; found some more info out:

    00000ad8.00000938::2013/08/25-02:37:00.700 WARN  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Unable to get provider's transport name, status 2. Checking if the provider is online..
    00000ad8.00000938::2013/08/25-02:37:00.700 ERR   [RES] Network Name <SQL Network Name (SCSQLCL01)>: Unable to get ip info for the provider resource, status 2
    00000ad8.00000938::2013/08/25-02:37:00.700 ERR   [RES] Network Name <SQL Network Name (SCSQLCL01)>: Online thread Failed: (2)' because of 'Obtaining IP info for SQL Network Name (SCSQLCL01) failed with error 2.'
    00000ad8.00000938::2013/08/25-02:37:00.700 ERR   [RHS] Online for resource SQL Network Name (SCSQLCL01) failed.
    00000640.00000e88::2013/08/25-02:37:00.700 WARN  [RCM] HandleMonitorReply: ONLINERESOURCE for 'SQL Network Name (SCSQLCL01)', gen(3) result 5018/0.
    00000640.00000e88::2013/08/25-02:37:00.700 INFO  [RCM] Res SQL Network Name (SCSQLCL01): OnlinePending -> ProcessingFailure( StateUnknown )

    Now, following this guy who had the exact same problem lead me to a registry entry key that isn't created, doesn't get created, or is overwritten during a stop/start scenario.

    So, I manually created the NbtDeviceName REG_IZ under the Parameters Key, and when it fails (because sometimes it comes online no issue) it also DELETES the NbtDeviceName REG_SZ - and the network name is in a failed state.

    So - do we chalk this one up to 2012R2 Preview, and a fault with SQL 2012 SP1 with failover clustering?

    Saturday, August 24, 2013 3:16 AM
  • HKLM\Cluster\Resources\<GUID>\Parameters

    Present under the \2 Key

    Missing under the \1 Key

    Saturday, August 24, 2013 3:21 AM
  • HI,

    The problem is probaly here  "The cluster computer object has been granted permissions on the cluster " you need to check this. can the cluster object create Items in the AD. and make sure your DNS entries are correct.

    below are some helpfull links. Let us know your status.

    http://robertsmit.wordpress.com/2012/06/26/cluster-network-name-resource-failed-to-create-its-associated-computer-object-in-domain/

    Check also http://social.technet.microsoft.com/Forums/windowsserver/en-US/1eb56fbf-8a73-434b-a0fa-2b6f1c3d7716/unable-to-start-cluster-service-on-windows-2008-sql-node


    Greetings, Robert Smit Follow me @clustermvp http://robertsmit.wordpress.com/ “Please click "Vote As Helpful" if it is helpful for you and Proposed As Answer” Please remember to click “Mark as Answer” on the post that helps you

    Monday, August 26, 2013 9:22 AM
    Moderator
  • Thanks for the links - followed your article in the first one to the letter. The SQL cluster name still wouldn't come online reliably.

    So I rebuilt the servers (from scratch). Deleted all traces of the computer objects in AD and started again.

    • Created a new cluster
    • Delegated permissions to the Computers OU for the Cluster computer object
    • Installed SQL 2012 SP1 on first node
    • Network resource name comes online OK - however, the following is logged
    • The computer object associated with cluster network name resource '' could not be updated.

    • Verified the cluster computer object has appropriate permissions on the SQL cluster computer object (was done when the object was created) - and now the following event is logged;

    Cluster resource 'SQL Network Name (SCSQLCL01)' of type 'Network Name' in clustered role 'SQL Server (VMM)' failed.

    • Granted FULL CONTROL to the SQL cluster computer object -Same even logged, and 50/50 failed/online 
    • Granted Cluster computer object FULL CONTROL to the entire Computer OU (bad) - same event logged and the resources remains failed
    • Granted the SQL cluster computer object full control to the entire COmputers OU (bad) - same event logged

    Any more ideas?

    Have included the log of the last failed attempt in its entirety.

    00000bac.00000ed0::2013/08/30-21:47:13.156 INFO  [RCM] rcm::RcmApi::OfflineResource: (SQL IP Address 1 (SCSQLCL01), 0)
    00000bac.00000ed0::2013/08/30-21:47:13.156 INFO  [GUM] Node 2: executing request locally, gumId:414, my action: /dm/update, # of updates: 1
    00000bac.00000ed0::2013/08/30-21:47:13.157 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.00000ed0::2013/08/30-21:47:13.165 INFO  [GUM] Node 2: executing request locally, gumId:415, my action: /dm/update, # of updates: 1
    00000bac.00000ed0::2013/08/30-21:47:13.165 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.00000ed0::2013/08/30-21:47:13.171 INFO  [RCM] Res SQL IP Address 1 (SCSQLCL01): Online -> WaitingToGoOffline( StateUnknown )
    00000bac.00000ed0::2013/08/30-21:47:13.171 INFO  [RCM] TransitionToState(SQL IP Address 1 (SCSQLCL01)) Online-->WaitingToGoOffline.
    00000bac.00000ed0::2013/08/30-21:47:13.171 INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (SQL Server (VMM), Failed --> Pending)
    00000bac.00000ed0::2013/08/30-21:47:13.172 INFO  [RCM] Bringing dependent resource 'SQL Network Name (SCSQLCL01)' offline before provider resource 'SQL IP Address 1 (SCSQLCL01)'.
    00000bac.00000ed0::2013/08/30-21:47:13.172 INFO  [RCM] Res SQL Network Name (SCSQLCL01): Online -> WaitingToGoOffline( StateUnknown )
    00000bac.00000ed0::2013/08/30-21:47:13.172 INFO  [RCM] TransitionToState(SQL Network Name (SCSQLCL01)) Online-->WaitingToGoOffline.
    00000bac.00000ed0::2013/08/30-21:47:13.172 INFO  [RCM] Res SQL Network Name (SCSQLCL01): WaitingToGoOffline -> OfflineCallIssued( StateUnknown )
    00000bac.00000ed0::2013/08/30-21:47:13.172 INFO  [RCM] TransitionToState(SQL Network Name (SCSQLCL01)) WaitingToGoOffline-->OfflineCallIssued.
    00000bac.000005d8::2013/08/30-21:47:13.172 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.00000ed0::2013/08/30-21:47:13.172 INFO  [RCM] 'SQL IP Address 1 (SCSQLCL01)' cannot go offline yet; 'SQL Network Name (SCSQLCL01)' is in state OfflineCallIssued.
    00000bac.00000ed0::2013/08/30-21:47:13.172 INFO  [RCM] Res SQL IP Address 1 (SCSQLCL01): WaitingToGoOffline -> WaitingToGoOffline( StateUnknown )
    00000bac.000003cc::2013/08/30-21:47:13.172 INFO  [RCM] ignored non-local state Pending for group SQL Server (VMM)
    00000bac.000005d8::2013/08/30-21:47:13.172 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    0000060c.000009f0::2013/08/30-21:47:13.175 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Offline called for resource SQL Network Name (SCSQLCL01)
    00000bac.00000cac::2013/08/30-21:47:13.177 INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'SQL Network Name (SCSQLCL01)', gen(16) result 997/0.
    00000bac.00000cac::2013/08/30-21:47:13.177 INFO  [RCM] Res SQL Network Name (SCSQLCL01): OfflineCallIssued -> OfflinePending( StateUnknown )
    00000bac.00000cac::2013/08/30-21:47:13.177 INFO  [RCM] TransitionToState(SQL Network Name (SCSQLCL01)) OfflineCallIssued-->OfflinePending.
    0000060c.00000e24::2013/08/30-21:47:13.177 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Entering Offline thread
    0000060c.00000e24::2013/08/30-21:47:13.178 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Performing actual offline of resource.
    0000060c.00000e24::2013/08/30-21:47:13.178 INFO  [RES] Network Name: Agent: Closing (fdab3086-42d6-4486-9742-eb1ea1197803,AdminShare)
    0000060c.00000e24::2013/08/30-21:47:13.178 INFO  [RES] Network Name: Agent: Sending request Netname/CloseIndirect to NN:Agent
    0000060c.000009f0::2013/08/30-21:47:13.178 INFO  [RES] Network Name: Agent: OnClose (fdab3086-42d6-4486-9742-eb1ea1197803,AdminShare)
    0000060c.000009f0::2013/08/30-21:47:13.178 INFO  [RES] Network Name: Agent: Sending request Netname/Close to NN:fdab3086-42d6-4486-9742-eb1ea1197803:AdminShare
    0000060c.000009f0::2013/08/30-21:47:13.178 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: AdminShare: Canceling work, state: Closing/Idle
    0000060c.000009f0::2013/08/30-21:47:13.178 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: AdminShare: OnCloseBase, previous state: Initialized/Idle
    00000bac.00001080::2013/08/30-21:47:13.180 INFO  [API] s_ApiUnblockGetNotifyCall: for the HDL( 1c )
    00000bac.00000cac::2013/08/30-21:47:13.181 INFO  [API] s_ApiUnblockGetNotifyCall: for the HDL( 1b )
    0000060c.000009f0::2013/08/30-21:47:13.183 INFO  [RES] Network Name: Agent: OnCloseReply (fdab3086-42d6-4486-9742-eb1ea1197803,AdminShare) result: 0
    0000060c.000009f0::2013/08/30-21:47:13.183 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: SyncReplyHandler Configuration, result: 0
    0000060c.00000e24::2013/08/30-21:47:13.183 INFO  [RES] Network Name: Agent: Closing (fdab3086-42d6-4486-9742-eb1ea1197803,Configuration)
    0000060c.00000e24::2013/08/30-21:47:13.183 INFO  [RES] Network Name: Agent: Sending request Netname/CloseIndirect to NN:Agent
    0000060c.000009f0::2013/08/30-21:47:13.183 INFO  [RES] Network Name: Agent: OnClose (fdab3086-42d6-4486-9742-eb1ea1197803,Configuration)
    0000060c.000009f0::2013/08/30-21:47:13.183 INFO  [RES] Network Name: Agent: Sending request Netname/Close to NN:fdab3086-42d6-4486-9742-eb1ea1197803:Configuration
    0000060c.000009f0::2013/08/30-21:47:13.183 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Configuration: Canceling work, state: Closing/Idle
    0000060c.000009f0::2013/08/30-21:47:13.183 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Configuration: OnCloseBase, previous state: Initialized/Idle
    0000060c.000009f0::2013/08/30-21:47:13.183 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Configuration: Closing... (PreviousState: Initialized, Created: true, Type: Singleton)
    0000060c.000009f0::2013/08/30-21:47:13.183 INFO  [RES] Network Name: Agent: Closing (fdab3086-42d6-4486-9742-eb1ea1197803,AccountAD)
    0000060c.000009f0::2013/08/30-21:47:13.183 INFO  [RES] Network Name: Agent: Sending request Netname/CloseIndirect to NN:Agent
    0000060c.000009f0::2013/08/30-21:47:13.183 INFO  [RES] Network Name: Agent: Closing (fdab3086-42d6-4486-9742-eb1ea1197803,Dns)
    0000060c.000009f0::2013/08/30-21:47:13.183 INFO  [RES] Network Name: Agent: Sending request Netname/CloseIndirect to NN:Agent
    0000060c.000009f0::2013/08/30-21:47:13.183 INFO  [RES] Network Name: Agent: Closing (fdab3086-42d6-4486-9742-eb1ea1197803,Client)
    0000060c.000009f0::2013/08/30-21:47:13.183 INFO  [RES] Network Name: Agent: Sending request Netname/CloseIndirect to NN:Agent
    0000060c.00001318::2013/08/30-21:47:13.183 INFO  [RES] Network Name: Agent: OnClose (fdab3086-42d6-4486-9742-eb1ea1197803,AccountAD)
    0000060c.00001318::2013/08/30-21:47:13.183 INFO  [RES] Network Name: Agent: Sending request Netname/Close to NN:fdab3086-42d6-4486-9742-eb1ea1197803:AccountAD
    0000060c.00001318::2013/08/30-21:47:13.183 INFO  [RES] Network Name: Agent: OnClose (fdab3086-42d6-4486-9742-eb1ea1197803,Dns)
    0000060c.00001318::2013/08/30-21:47:13.183 INFO  [RES] Network Name: Agent: Sending request Netname/Close to NN:fdab3086-42d6-4486-9742-eb1ea1197803:Dns
    0000060c.00000fa0::2013/08/30-21:47:13.183 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: AccountAD: Canceling work, state: Closing/Idle
    0000060c.00001318::2013/08/30-21:47:13.183 INFO  [RES] Network Name: Agent: OnClose (fdab3086-42d6-4486-9742-eb1ea1197803,Client)
    0000060c.00001318::2013/08/30-21:47:13.184 INFO  [RES] Network Name: Agent: Sending request Netname/Close to NN:fdab3086-42d6-4486-9742-eb1ea1197803:Client
    0000060c.00000fa0::2013/08/30-21:47:13.184 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: AccountAD: OnCloseBase, previous state: Initialized/Idle
    0000060c.00001318::2013/08/30-21:47:13.184 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Dns: Canceling work, state: Closing/Idle
    0000060c.00000fa0::2013/08/30-21:47:13.184 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Client: Canceling work, state: Closing/Idle
    0000060c.00001318::2013/08/30-21:47:13.184 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Dns: OnCloseBase, previous state: Initialized/Idle
    0000060c.00000fa0::2013/08/30-21:47:13.184 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Client: OnCloseBase, previous state: Initialized/Idle
    0000060c.00001318::2013/08/30-21:47:13.184 INFO  [RES] Network Name: Agent: OnCloseReply (fdab3086-42d6-4486-9742-eb1ea1197803,Dns) result: 0
    0000060c.00001220::2013/08/30-21:47:13.184 INFO  [RES] Network Name: Agent: OnCloseReply (fdab3086-42d6-4486-9742-eb1ea1197803,AccountAD) result: 0
    0000060c.00000fa0::2013/08/30-21:47:13.184 INFO  [RES] Network Name: [NNLIB] Removing alias 192.168.100.166 for target SCSQLCL01
    0000060c.00001318::2013/08/30-21:47:13.184 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Configuration: Finish Closing NetName (type Singleton) Module Dns, result 0, remaining... 3 (0)
    0000060c.00001318::2013/08/30-21:47:13.184 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Configuration: Finish Closing NetName (type Singleton) Module AccountAD, result 0, remaining... 2 (0)
    0000060c.00000fa0::2013/08/30-21:47:13.187 INFO  [RES] Network Name: [NNLIB] LsaCallAuthenticationPackage success with a request of size 122, result size 0 (status: 0, subStatus: 0)
    0000060c.00000fa0::2013/08/30-21:47:13.187 INFO  [RES] Network Name: [NNLIB] LsaCallAuthenticationPackage success with a request of size 122, result size 0 (status: 0, subStatus: 0)
    0000060c.00000fa0::2013/08/30-21:47:13.187 INFO  [RES] Network Name: [NNLIB] Success on credential operation remove on Account SCSQLCL01$, status: 0 (subStatus: 0)
    0000060c.00000fa0::2013/08/30-21:47:13.188 INFO  [RES] Network Name: [NNLIB] LsaCallAuthenticationPackage success with a request of size 70, result size 0 (status: 0, subStatus: 0)
    0000060c.00000fa0::2013/08/30-21:47:13.188 INFO  [RES] Network Name: [NNLIB] RegisterMachineAlias of SCSQLCL01.service.local, status: 0, subStatus: 0
    0000060c.00000fa0::2013/08/30-21:47:13.188 INFO  [RES] Network Name: [NNLIB] LsaCallAuthenticationPackage success with a request of size 42, result size 0 (status: 0, subStatus: 0)
    0000060c.00000fa0::2013/08/30-21:47:13.188 INFO  [RES] Network Name: [NNLIB] RegisterMachineAlias of SCSQLCL01, status: 0, subStatus: 0
    0000060c.00000fa0::2013/08/30-21:47:13.188 INFO  [RES] Network Name: [NNLIB] LsaCallAuthenticationPackage success with a request of size 42, result size 0 (status: 0, subStatus: 0)
    0000060c.00000fa0::2013/08/30-21:47:13.188 INFO  [RES] Network Name: [NNLIB] RegisterMachineAlias of SCSQLCL01, status: 0, subStatus: 0
    0000060c.00000fa0::2013/08/30-21:47:13.189 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Client: Unregistering netname for multichannel support returned 0
    0000060c.00000fa0::2013/08/30-21:47:13.189 INFO  [RES] Network Name: Agent: OnCloseReply (fdab3086-42d6-4486-9742-eb1ea1197803,Client) result: 0
    0000060c.00000fa0::2013/08/30-21:47:13.189 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Configuration: Finish Closing NetName (type Singleton) Module Client, result 0, remaining... 1 (0)
    0000060c.00000fa0::2013/08/30-21:47:13.189 INFO  [RES] Network Name: Agent: Closing (fdab3086-42d6-4486-9742-eb1ea1197803,Netbios)
    0000060c.00000fa0::2013/08/30-21:47:13.189 INFO  [RES] Network Name: Agent: Sending request Netname/CloseIndirect to NN:Agent
    0000060c.00000fa0::2013/08/30-21:47:13.189 INFO  [RES] Network Name: Agent: OnClose (fdab3086-42d6-4486-9742-eb1ea1197803,Netbios)
    0000060c.00000fa0::2013/08/30-21:47:13.189 INFO  [RES] Network Name: Agent: Sending request Netname/Close to NN:fdab3086-42d6-4486-9742-eb1ea1197803:Netbios
    0000060c.00000fa0::2013/08/30-21:47:13.189 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Netbios: Canceling work, state: Closing/Idle
    0000060c.00000fa0::2013/08/30-21:47:13.189 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Netbios: OnCloseBase, previous state: Initialized/Idle
    0000060c.00000fa0::2013/08/30-21:47:13.191 INFO  [RES] Network Name: [NNLIB] Removed server bits SCSQLCL01 on transport \Device\NetBt_If4
    0000060c.00000fa0::2013/08/30-21:47:13.196 INFO  [RES] Network Name: [NNLIB] Deregistered server name SCSQLCL01 on transport \Device\NetBt_If4
    0000060c.00000fa0::2013/08/30-21:47:13.199 INFO  [RES] Network Name: [NNLIB] Deregistered server name SCSQLCL01 on transport \Device\NetbiosSmb
    0000060c.00000fa0::2013/08/30-21:47:13.199 INFO  [RES] Network Name: [NNLIB] Deleted server name: SCSQLCL01 from all transports
    0000060c.00000fa0::2013/08/30-21:47:13.199 INFO  [RES] Network Name: [NNLIB] Deleting Server Name for server name: SCSQLCL01, ret: 0
    0000060c.00001318::2013/08/30-21:47:13.199 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Netbios: Cleaning up transports
    0000060c.00000fa0::2013/08/30-21:47:13.200 INFO  [RES] Network Name: Agent: OnCloseReply (fdab3086-42d6-4486-9742-eb1ea1197803,Netbios) result: 0
    0000060c.00000fa0::2013/08/30-21:47:13.200 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Configuration: Finish Closing NetName (type Singleton) Module Netbios, result 0, remaining... 0 (0)
    0000060c.00000fa0::2013/08/30-21:47:13.200 INFO  [RES] Network Name: Agent: -- listing 2 instances ---------------------------------------------
    0000060c.00000fa0::2013/08/30-21:47:13.200 INFO  [RES] Network Name: Agent: --- 7 modules for instance: aa748451-309b-4aa0-acea-c6dd23005853:
    0000060c.00000fa0::2013/08/30-21:47:13.200 INFO  [RES] Network Name: Agent: ---- Module: Configuration with states: Offline/Idle (BeingBorn)
    0000060c.000009f0::2013/08/30-21:47:13.200 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Configuration: Closed
    0000060c.00000fa0::2013/08/30-21:47:13.200 INFO  [RES] Network Name: Agent: ---- Module: AccountAD with states: Offline/Idle (BeingBorn)
    0000060c.00000fa0::2013/08/30-21:47:13.200 INFO  [RES] Network Name: Agent: ---- Module: Client with states: Offline/Idle (BeingBorn)
    0000060c.000009f0::2013/08/30-21:47:13.200 INFO  [RES] Network Name: Agent: OnCloseReply (fdab3086-42d6-4486-9742-eb1ea1197803,Configuration) result: 0
    0000060c.00000fa0::2013/08/30-21:47:13.200 INFO  [RES] Network Name: Agent: ---- Module: Identity with states: Initialized/Idle (Alive)
    0000060c.000009f0::2013/08/30-21:47:13.200 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: SyncReplyHandler Configuration, result: 0
    0000060c.00000fa0::2013/08/30-21:47:13.200 INFO  [RES] Network Name: Agent: ---- Module: Dns with states: Offline/Idle (BeingBorn)
    0000060c.00000fa0::2013/08/30-21:47:13.200 INFO  [RES] Network Name: Agent: ---- Module: Netbios with states: Offline/Idle (BeingBorn)
    0000060c.00000fa0::2013/08/30-21:47:13.200 INFO  [RES] Network Name: Agent: ---- Module: AdminShare with states: Offline/Idle (BeingBorn)
    0000060c.00000fa0::2013/08/30-21:47:13.200 INFO  [RES] Network Name: Agent: --- 7 modules for instance: fdab3086-42d6-4486-9742-eb1ea1197803:
    0000060c.00000fa0::2013/08/30-21:47:13.200 INFO  [RES] Network Name: Agent: ---- Module: Configuration with states: Offline/Ending (Alive)
    0000060c.00000fa0::2013/08/30-21:47:13.200 INFO  [RES] Network Name: Agent: ---- Module: AccountAD with states: Closing/Ending (Alive)
    0000060c.00000e24::2013/08/30-21:47:13.200 INFO  [RHS] Resource SQL Network Name (SCSQLCL01) has come offline. RHS is about to report resource status to RCM.
    0000060c.00000fa0::2013/08/30-21:47:13.200 INFO  [RES] Network Name: Agent: ---- Module: Client with states: Closing/Ending (Alive)
    0000060c.00000fa0::2013/08/30-21:47:13.200 INFO  [RES] Network Name: Agent: ---- Module: Identity with states: Initialized/Idle (Alive)
    0000060c.00000fa0::2013/08/30-21:47:13.200 INFO  [RES] Network Name: Agent: ---- Module: Dns with states: Closing/Ending (Alive)
    0000060c.00000fa0::2013/08/30-21:47:13.200 INFO  [RES] Network Name: Agent: ---- Module: Netbios with states: Closing/Ending (Alive)
    0000060c.00000fa0::2013/08/30-21:47:13.200 INFO  [RES] Network Name: Agent: ---- Module: AdminShare with states: Closing/Ending (BeingBorn)
    0000060c.00000fa0::2013/08/30-21:47:13.200 INFO  [RES] Network Name: Agent: -- 0 Zombie modules
    00000bac.00001080::2013/08/30-21:47:13.200 INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'SQL Network Name (SCSQLCL01)', gen(16) result 0/0.
    00000bac.00001080::2013/08/30-21:47:13.200 INFO  [RCM] Res SQL Network Name (SCSQLCL01): OfflinePending -> OfflineSavingCheckpoints( StateUnknown )
    00000bac.00001080::2013/08/30-21:47:13.200 INFO  [RCM] TransitionToState(SQL Network Name (SCSQLCL01)) OfflinePending-->OfflineSavingCheckpoints.
    00000bac.00000cac::2013/08/30-21:47:13.215 INFO  [GUM] Node 2: executing request locally, gumId:416, my action: /dm/update, # of updates: 1
    00000bac.00000cac::2013/08/30-21:47:13.215 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.00000cac::2013/08/30-21:47:13.230 INFO  [GUM] Node 2: executing request locally, gumId:417, my action: /dm/update, # of updates: 1
    00000bac.00000cac::2013/08/30-21:47:13.231 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.00000cac::2013/08/30-21:47:13.246 INFO  [GUM] Node 2: executing request locally, gumId:418, my action: /dm/update, # of updates: 1
    00000bac.00000cac::2013/08/30-21:47:13.246 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.00000cac::2013/08/30-21:47:13.267 INFO  [GUM] Node 2: executing request locally, gumId:419, my action: /dm/update, # of updates: 1
    00000bac.00000cac::2013/08/30-21:47:13.268 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.00000cac::2013/08/30-21:47:13.288 INFO  [GUM] Node 2: executing request locally, gumId:420, my action: /dm/update, # of updates: 1
    00000bac.00000cac::2013/08/30-21:47:13.289 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.00000cac::2013/08/30-21:47:13.305 INFO  [GUM] Node 2: executing request locally, gumId:421, my action: /dm/update, # of updates: 1
    00000bac.00000cac::2013/08/30-21:47:13.306 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.00000cac::2013/08/30-21:47:13.327 INFO  [GUM] Node 2: executing request locally, gumId:422, my action: /dm/update, # of updates: 1
    00000bac.00000cac::2013/08/30-21:47:13.328 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.00000cac::2013/08/30-21:47:13.335 INFO  [RCM] Res SQL Network Name (SCSQLCL01): OfflineSavingCheckpoints -> Offline( StateUnknown )
    00000bac.00000cac::2013/08/30-21:47:13.335 INFO  [RCM] TransitionToState(SQL Network Name (SCSQLCL01)) OfflineSavingCheckpoints-->Offline.
    00000bac.00001080::2013/08/30-21:47:13.335 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.00000cac::2013/08/30-21:47:13.336 WARN  [RCM] SQL IP Address 1 (SCSQLCL01) cannot be hosted on node 2 due to persistent state offline
    00000bac.00000cac::2013/08/30-21:47:13.336 WARN  [RCM] SQL IP Address 1 (SCSQLCL01) cannot be hosted on node 2
    00000bac.00000cac::2013/08/30-21:47:13.336 WARN  [RCM] Possible owners: <vector len='1'>
    00000bac.00000cac::2013/08/30-21:47:13.336 WARN      <item>2</item>
    00000bac.00000cac::2013/08/30-21:47:13.336 WARN  </vector>
    00000bac.00000cac::2013/08/30-21:47:13.336 INFO  [RCM] Res SQL IP Address 1 (SCSQLCL01): WaitingToGoOffline -> OfflineCallIssued( StateUnknown )
    00000bac.00000cac::2013/08/30-21:47:13.336 INFO  [RCM] TransitionToState(SQL IP Address 1 (SCSQLCL01)) WaitingToGoOffline-->OfflineCallIssued.
    00000bac.00000cac::2013/08/30-21:47:13.336 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    0000060c.00000fa0::2013/08/30-21:47:13.338 INFO  [RES] IP Address <SQL IP Address 1 (SCSQLCL01)>: Taking resource offline...
    0000060c.00000fa0::2013/08/30-21:47:13.339 INFO  [RES] IP Address <SQL IP Address 1 (SCSQLCL01)>: Deleting IP interface A664A8C0.
    0000060c.00000fa0::2013/08/30-21:47:13.340 INFO  [RES] IP Address <SQL IP Address 1 (SCSQLCL01)>: Address 192.168.100.166 on adapter Microsoft Hyper-V Network Adapter #3 offline.
    0000060c.00000fa0::2013/08/30-21:47:13.340 INFO  [RES] IP Address <Cluster IP Address>: All resources offline - cleaning up
    0000060c.00000bf4::2013/08/30-21:47:13.341 WARN  [RES] IPv4 Address Resources: Notify port is about to be closed. Exiting...
    00000bac.000005d8::2013/08/30-21:47:13.342 INFO  [API] s_ApiUnblockGetNotifyCall: for the HDL( 1a )
    00000bac.00000cac::2013/08/30-21:47:13.345 INFO  [RCM] HandleMonitorReply: OFFLINERESOURCE for 'SQL IP Address 1 (SCSQLCL01)', gen(0) result 0/0.
    00000bac.00000cac::2013/08/30-21:47:13.345 INFO  [RCM] Res SQL IP Address 1 (SCSQLCL01): OfflineCallIssued -> OfflineSavingCheckpoints( StateUnknown )
    00000bac.00000cac::2013/08/30-21:47:13.345 INFO  [RCM] TransitionToState(SQL IP Address 1 (SCSQLCL01)) OfflineCallIssued-->OfflineSavingCheckpoints.
    00000bac.00000cac::2013/08/30-21:47:13.345 INFO  [RCM] Res SQL IP Address 1 (SCSQLCL01): OfflineSavingCheckpoints -> Offline( StateUnknown )
    00000bac.00000cac::2013/08/30-21:47:13.345 INFO  [RCM] TransitionToState(SQL IP Address 1 (SCSQLCL01)) OfflineSavingCheckpoints-->Offline.
    00000bac.00000cac::2013/08/30-21:47:13.345 INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (SQL Server (VMM), Pending --> Failed)
    00000bac.00000cac::2013/08/30-21:47:13.345 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.00000cac::2013/08/30-21:47:13.346 INFO  [RCM] moved 0 tasks from staging set to task set.  TaskSetSize=0
    00000bac.000003cc::2013/08/30-21:47:13.346 INFO  [RCM] ignored non-local state Failed for group SQL Server (VMM)
    00000bac.00000cac::2013/08/30-21:47:13.346 INFO  [RCM] rcm::RcmPriorityManager::StartGroups: [RCM] done, executed 0 tasks
    00000bac.000005d8::2013/08/30-21:47:13.346 DBG   [NETFTAPI] received NsiDeleteInstance for 192.168.100.166
    00000bac.000005d8::2013/08/30-21:47:13.371 WARN  [NETFTAPI] Failed to query parameters for 192.168.100.166 (status 0x80070490)
    00000bac.000005d8::2013/08/30-21:47:13.371 DBG   [NETFTAPI] Signaled NetftLocalRemove event for 192.168.100.166
    00000bac.00000ed0::2013/08/30-21:47:16.880 INFO  [RCM] rcm::RcmApi::OnlineResource: (SQL Network Name (SCSQLCL01), 0)
    00000bac.00000ed0::2013/08/30-21:47:16.881 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.00000ed0::2013/08/30-21:47:16.889 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.00000ed0::2013/08/30-21:47:16.892 INFO  [GUM] Node 2: executing request locally, gumId:423, my action: /dm/update, # of updates: 1
    00000bac.00000ed0::2013/08/30-21:47:16.893 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.00000ed0::2013/08/30-21:47:16.897 INFO  [GUM] Node 2: executing request locally, gumId:424, my action: /dm/update, # of updates: 1
    00000bac.00000ed0::2013/08/30-21:47:16.897 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.00000ed0::2013/08/30-21:47:16.900 INFO  [RCM-rbtr] giving default token to group SQL Server (VMM)
    00000bac.00000ed0::2013/08/30-21:47:16.900 INFO  [RCM] rcm::RcmResource::Online: bringing SQL Network Name (SCSQLCL01)'s provider resource 'SQL IP Address 1 (SCSQLCL01)' online.
    00000bac.00000ed0::2013/08/30-21:47:16.900 INFO  [RCM-rbtr] giving default token to group SQL Server (VMM)
    00000bac.00000ed0::2013/08/30-21:47:16.900 INFO  [RCM] Res SQL IP Address 1 (SCSQLCL01): Offline -> OnlineCallIssued( StateUnknown )
    00000bac.00000ed0::2013/08/30-21:47:16.900 INFO  [RCM] TransitionToState(SQL IP Address 1 (SCSQLCL01)) Offline-->OnlineCallIssued.
    00000bac.00000ed0::2013/08/30-21:47:16.900 INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (SQL Server (VMM), Failed --> Pending)
    00000bac.000005d8::2013/08/30-21:47:16.900 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.00000ed0::2013/08/30-21:47:16.901 INFO  [RCM-rbtr] giving default token to group SQL Server (VMM)
    00000bac.00000ed0::2013/08/30-21:47:16.901 INFO  [RCM] Res SQL Network Name (SCSQLCL01): Offline -> WaitingToComeOnline( StateUnknown )
    00000bac.00000ed0::2013/08/30-21:47:16.901 INFO  [RCM] TransitionToState(SQL Network Name (SCSQLCL01)) Offline-->WaitingToComeOnline.
    00000bac.00000ed0::2013/08/30-21:47:16.901 INFO  [RCM-rbtr] giving default token to group SQL Server (VMM)
    00000bac.00000ed0::2013/08/30-21:47:16.901 INFO  [RCM-rbtr] giving default token to group SQL Server (VMM)
    00000bac.00000ed0::2013/08/30-21:47:16.901 INFO  [RCM-rbtr] giving default token to group SQL Server (VMM)
    00000bac.00000cac::2013/08/30-21:47:16.901 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.000003cc::2013/08/30-21:47:16.901 INFO  [RCM] ignored non-local state Pending for group SQL Server (VMM)
    0000060c.00000fa0::2013/08/30-21:47:16.902 INFO  [RES] IP Address <SQL IP Address 1 (SCSQLCL01)>: Bringing resource online...
    00000bac.000005d8::2013/08/30-21:47:16.903 INFO  [RCM] HandleMonitorReply: ONLINERESOURCE for 'SQL IP Address 1 (SCSQLCL01)', gen(0) result 997/0.
    00000bac.000005d8::2013/08/30-21:47:16.903 INFO  [RCM] Res SQL IP Address 1 (SCSQLCL01): OnlineCallIssued -> OnlinePending( StateUnknown )
    00000bac.000005d8::2013/08/30-21:47:16.903 INFO  [RCM] TransitionToState(SQL IP Address 1 (SCSQLCL01)) OnlineCallIssued-->OnlinePending.
    0000060c.00000740::2013/08/30-21:47:16.903 INFO  [RES] IP Address <SQL IP Address 1 (SCSQLCL01)>: Online thread running.
    0000060c.00000740::2013/08/30-21:47:16.904 INFO  [RES] IP Address <SQL IP Address 1 (SCSQLCL01)>: Created cluster notify port.
    0000060c.00000740::2013/08/30-21:47:16.904 INFO  [RES] IP Address <SQL IP Address 1 (SCSQLCL01)>: Created cancellation event for cluster notify port.
    0000060c.00000740::2013/08/30-21:47:16.904 INFO  [RES] IP Address <SQL IP Address 1 (SCSQLCL01)>: Created worker thread.
    0000060c.00000318::2013/08/30-21:47:16.905 WARN  [RES] IPv4 Address Resources: WorkerThread running
    00000bac.00000cac::2013/08/30-21:47:16.906 INFO  [GUM] Node 2: executing request locally, gumId:425, my action: /dm/update, # of updates: 1
    00000bac.00000cac::2013/08/30-21:47:16.906 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    0000060c.00000740::2013/08/30-21:47:16.911 INFO  [RES] IP Address <SQL IP Address 1 (SCSQLCL01)>: Successfully deleted NBT interface information from database.
    0000060c.00000740::2013/08/30-21:47:16.911 INFO  [RES] IP Address <SQL IP Address 1 (SCSQLCL01)>: Deleting IP interface A664A8C0.
    0000060c.00000740::2013/08/30-21:47:16.911 WARN  [RES] IP Address <SQL IP Address 1 (SCSQLCL01)>: Failed to delete IP interface A664A8C0, status 87.
    0000060c.00000740::2013/08/30-21:47:16.911 INFO  [RES] IP Address <SQL IP Address 1 (SCSQLCL01)>: could not delete IP address 192.168.100.166, probably it is no longer valid.
    00000bac.000005d8::2013/08/30-21:47:16.913 INFO  [NM] Received request from client address fe80::e5e8:605:788b:dd6f.
    0000060c.00000740::2013/08/30-21:47:16.914 INFO  [RES] IP Address <SQL IP Address 1 (SCSQLCL01)>: Checking for network match: network masks 00FFFFFF=00FFFFFF and addresses A664A8C0^0064A8C0, role 3.
    00000bac.00000cac::2013/08/30-21:47:16.918 INFO  [NM] Received request from client address fe80::e5e8:605:788b:dd6f.
    0000060c.00000740::2013/08/30-21:47:16.918 INFO  [RES] IP Address <SQL IP Address 1 (SCSQLCL01)>: Online: Opened object handle for netinterface 30956718-016c-4427-8a85-c69d21993b14.
    0000060c.00000740::2013/08/30-21:47:16.918 INFO  [RES] IP Address <SQL IP Address 1 (SCSQLCL01)>: Online: Registered notification for netinterface 30956718-016c-4427-8a85-c69d21993b14.
    0000060c.00000740::2013/08/30-21:47:16.929 INFO  [ClNet] Adapter Cloud Management RFC2863 operational status = 1.
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet] Created adapter: DeviceGuid:     8F7BA235-FE15-4474-B7E8-244CEE81FE01
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet]                  DeviceName:     Microsoft Hyper-V Network Adapter #3
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet]                  ConnectoidName: Cloud Management
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet]                  Netbios/TCP:    1
    0000060c.00000740::2013/08/30-21:47:16.929 INFO  [ClNet] Adapter Storage RFC2863 operational status = 1.
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet] Created adapter: DeviceGuid:     0A49F6E2-C930-477C-8C25-B3AEC58BEE97
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet]                  DeviceName:     Microsoft Hyper-V Network Adapter #2
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet]                  ConnectoidName: Storage
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet]                  Netbios/TCP:    1
    0000060c.00000740::2013/08/30-21:47:16.929 INFO  [ClNet] Adapter Cluster Private RFC2863 operational status = 1.
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet] Created adapter: DeviceGuid:     66C03FBD-BCFE-4010-B856-BFAF1BA86D9F
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet]                  DeviceName:     Microsoft Hyper-V Network Adapter
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet]                  ConnectoidName: Cluster Private
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet]                  Netbios/TCP:    1
    0000060c.00000740::2013/08/30-21:47:16.929 INFO  [ClNet] Adapter Local Area Connection* 1 RFC2863 operational status = 1.
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet] Created adapter: DeviceGuid:     C1D0B6A0-0C41-4CED-A64A-69F758F3D4F9
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet]                  DeviceName:     Microsoft Failover Cluster Virtual Adapter
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet]                  ConnectoidName: Local Area Connection* 1
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet]                  Netbios/TCP:    1
    0000060c.00000740::2013/08/30-21:47:16.929 INFO  [ClNet] Adapter isatap.{66C03FBD-BCFE-4010-B856-BFAF1BA86D9F} RFC2863 operational status = 2.
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet] Created adapter: DeviceGuid:     0811F833-8EE2-4790-925C-12EBCF5A556F
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet]                  DeviceName:     Microsoft ISATAP Adapter
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet]                  ConnectoidName: isatap.{66C03FBD-BCFE-4010-B856-BFAF1BA86D9F}
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet]                  Netbios/TCP:    0
    0000060c.00000740::2013/08/30-21:47:16.929 INFO  [ClNet] Adapter isatap.{0A49F6E2-C930-477C-8C25-B3AEC58BEE97} RFC2863 operational status = 2.
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet] Created adapter: DeviceGuid:     0BADC95C-AD79-4D1E-A5C6-2A345A02AA43
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet]                  DeviceName:     Microsoft ISATAP Adapter #2
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet]                  ConnectoidName: isatap.{0A49F6E2-C930-477C-8C25-B3AEC58BEE97}
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet]                  Netbios/TCP:    0
    0000060c.00000740::2013/08/30-21:47:16.929 INFO  [ClNet] Adapter isatap.{8F7BA235-FE15-4474-B7E8-244CEE81FE01} RFC2863 operational status = 2.
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet] Created adapter: DeviceGuid:     6B274C1B-6899-42EE-9809-0019A291AE7C
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet]                  DeviceName:     Microsoft ISATAP Adapter #3
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet]                  ConnectoidName: isatap.{8F7BA235-FE15-4474-B7E8-244CEE81FE01}
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet]                  Netbios/TCP:    0
    0000060c.00000740::2013/08/30-21:47:16.929 INFO  [ClNet] Adapter isatap.{C1D0B6A0-0C41-4CED-A64A-69F758F3D4F9} RFC2863 operational status = 2.
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet] Created adapter: DeviceGuid:     0C87DC00-9131-4D90-B636-5B631BA39542
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet]                  DeviceName:     Microsoft ISATAP Adapter #4
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet]                  ConnectoidName: isatap.{C1D0B6A0-0C41-4CED-A64A-69F758F3D4F9}
    0000060c.00000740::2013/08/30-21:47:16.929 DBG   [ClNet]                  Netbios/TCP:    0
    0000060c.00000740::2013/08/30-21:47:16.929 INFO  [ClNet] Successfully enumerated all adapters and interfaces
    0000060c.00000740::2013/08/30-21:47:16.929 INFO  [RES] IP Address <SQL IP Address 1 (SCSQLCL01)>: Did not write IP interface information to database as it is already present.
    0000060c.00000740::2013/08/30-21:47:16.929 INFO  [RES] IP Address <SQL IP Address 1 (SCSQLCL01)>: Created IP interface A664A8C0.
    00000bac.00000cac::2013/08/30-21:47:16.931 DBG   [NETFTAPI] received NsiAddInstance for 192.168.100.166
    00000bac.00000cac::2013/08/30-21:47:20.272 DBG   [NETFTAPI] received NsiParameterNotification for 192.168.100.166 (IpDadStatePreferred)
    00000bac.00000cac::2013/08/30-21:47:20.278 DBG   [NETFTAPI] Signaled NetftLocalAdd event for 192.168.100.166
    00000bac.00000cac::2013/08/30-21:47:20.279 DBG   [NETFTAPI] Signaled NetftLocalConnect event for 192.168.100.166
    00000bac.000008ec::2013/08/30-21:47:20.279 INFO  [IM] got event: Local endpoint 192.168.100.166:~0~ connected
    0000060c.00000740::2013/08/30-21:47:20.537 INFO  [RES] IP Address <SQL IP Address 1 (SCSQLCL01)>: IpaOnlineThread: ProbePort property is not set.
    0000060c.00000740::2013/08/30-21:47:20.537 INFO  [RHS] Resource SQL IP Address 1 (SCSQLCL01) has come online. RHS is about to report status change to RCM
    00000bac.000005d8::2013/08/30-21:47:20.538 INFO  [RCM] HandleMonitorReply: ONLINERESOURCE for 'SQL IP Address 1 (SCSQLCL01)', gen(0) result 0/0.
    0000060c.00000740::2013/08/30-21:47:20.538 INFO  [RES] IP Address <SQL IP Address 1 (SCSQLCL01)>: IP Address 192.168.100.166 on adapter Microsoft Hyper-V Network Adapter #3 online
    00000bac.000005d8::2013/08/30-21:47:20.538 INFO  [RCM] Res SQL IP Address 1 (SCSQLCL01): OnlinePending -> Online( StateUnknown )
    00000bac.000005d8::2013/08/30-21:47:20.538 INFO  [RCM] TransitionToState(SQL IP Address 1 (SCSQLCL01)) OnlinePending-->Online.
    00000bac.00000cac::2013/08/30-21:47:20.538 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.000005d8::2013/08/30-21:47:20.539 INFO  [RCM-rbtr] giving default token to group SQL Server (VMM)
    00000bac.000005d8::2013/08/30-21:47:20.539 INFO  [RCM-rbtr] giving default token to group SQL Server (VMM)
    00000bac.000005d8::2013/08/30-21:47:20.539 INFO  [RCM] Res SQL Network Name (SCSQLCL01): WaitingToComeOnline -> OnlineCallIssued( StateUnknown )
    00000bac.000005d8::2013/08/30-21:47:20.539 INFO  [RCM] TransitionToState(SQL Network Name (SCSQLCL01)) WaitingToComeOnline-->OnlineCallIssued.
    00000bac.00000cac::2013/08/30-21:47:20.539 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.00001080::2013/08/30-21:47:20.540 INFO  [CM] mscs::RegCheckpoint::RestoreCheckpoint: Restoring checkpoint SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL11.VMM\Cluster for resource SQL Network Name (SCSQLCL01).
    00000bac.00001080::2013/08/30-21:47:20.551 INFO  [CM] mscs::RegCheckpoint::RestoreCheckpoint: Restoring checkpoint SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL11.VMM\MSSQLServer for resource SQL Network Name (SCSQLCL01).
    00000bac.00001080::2013/08/30-21:47:20.555 INFO  [CM] mscs::RegCheckpoint::RestoreCheckpoint: Restoring checkpoint SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL11.VMM\Replication for resource SQL Network Name (SCSQLCL01).
    00000bac.00001080::2013/08/30-21:47:20.557 INFO  [CM] mscs::RegCheckpoint::RestoreCheckpoint: Restoring checkpoint SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL11.VMM\Providers for resource SQL Network Name (SCSQLCL01).
    00000bac.00001080::2013/08/30-21:47:20.558 INFO  [CM] mscs::RegCheckpoint::RestoreCheckpoint: Restoring checkpoint SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL11.VMM\SQLServerSCP for resource SQL Network Name (SCSQLCL01).
    00000bac.00001080::2013/08/30-21:47:20.560 INFO  [CM] mscs::RegCheckpoint::RestoreCheckpoint: Restoring checkpoint SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL11.VMM\CPE for resource SQL Network Name (SCSQLCL01).
    00000bac.00001080::2013/08/30-21:47:20.565 INFO  [CM] mscs::RegCheckpoint::RestoreCheckpoint: Restoring checkpoint SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL11.VMM\SQLServerAgent for resource SQL Network Name (SCSQLCL01).
    0000060c.00000fa0::2013/08/30-21:47:20.571 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Online called for resource SQL Network Name (SCSQLCL01)
    00000bac.00001080::2013/08/30-21:47:20.572 INFO  [RCM] HandleMonitorReply: ONLINERESOURCE for 'SQL Network Name (SCSQLCL01)', gen(16) result 997/0.
    00000bac.00001080::2013/08/30-21:47:20.572 INFO  [RCM] Res SQL Network Name (SCSQLCL01): OnlineCallIssued -> OnlinePending( StateUnknown )
    00000bac.00001080::2013/08/30-21:47:20.572 INFO  [RCM] TransitionToState(SQL Network Name (SCSQLCL01)) OnlineCallIssued-->OnlinePending.
    0000060c.000005d0::2013/08/30-21:47:20.572 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Entering Online thread
    0000060c.000005d0::2013/08/30-21:47:20.573 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Initializing config info
    0000060c.000005d0::2013/08/30-21:47:20.576 INFO  [RES] Network Name: Agent: Initializing (fdab3086-42d6-4486-9742-eb1ea1197803,Identity)
    0000060c.000005d0::2013/08/30-21:47:20.576 INFO  [RES] Network Name: Agent: Sending request Netname/InitializeIndirect to NN:Agent
    0000060c.000005d0::2013/08/30-21:47:20.576 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Performing actual online of resource.
    0000060c.00000fa0::2013/08/30-21:47:20.576 INFO  [RES] Network Name: Agent: OnInitialize (fdab3086-42d6-4486-9742-eb1ea1197803,Identity)
    0000060c.00000fa0::2013/08/30-21:47:20.576 ERR   [RES] Network Name: Agent: InitializeModule, Trying to initialize Module(fdab3086-42d6-4486-9742-eb1ea1197803,Identity) when there is one already in Initialized/Idle state
    0000060c.00000fa0::2013/08/30-21:47:20.576 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: ReplyHandler Identity, result: 183
    0000060c.000005d0::2013/08/30-21:47:20.577 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Obtaining IP info for resource SQL IP Address 1 (SCSQLCL01)
    0000060c.000005d0::2013/08/30-21:47:20.579 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Using provider SQL IP Address 1 (SCSQLCL01), ip address 192.168.100.166, mask 255.255.255.0, prefix length 24
    0000060c.000005d0::2013/08/30-21:47:20.579 WARN  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Unable to get provider's transport name, status 2. Checking if the provider is online..
    0000060c.000005d0::2013/08/30-21:47:20.579 ERR   [RES] Network Name <SQL Network Name (SCSQLCL01)>: Unable to get ip info for the provider resource, status 2
    0000060c.000005d0::2013/08/30-21:47:20.580 ERR   [RES] Network Name <SQL Network Name (SCSQLCL01)>: Online thread Failed: (2)' because of 'Obtaining IP info for SQL Network Name (SCSQLCL01) failed with error 2.'
    0000060c.000005d0::2013/08/30-21:47:20.580 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: All resources offline. Cleaning up.
    0000060c.000005d0::2013/08/30-21:47:20.580 ERR   [RHS] Online for resource SQL Network Name (SCSQLCL01) failed.
    00000bac.00000ed0::2013/08/30-21:47:20.580 WARN  [RCM] HandleMonitorReply: ONLINERESOURCE for 'SQL Network Name (SCSQLCL01)', gen(16) result 5018/0.
    00000bac.00000ed0::2013/08/30-21:47:20.580 INFO  [RCM] Res SQL Network Name (SCSQLCL01): OnlinePending -> ProcessingFailure( StateUnknown )
    00000bac.00000ed0::2013/08/30-21:47:20.580 INFO  [RCM] TransitionToState(SQL Network Name (SCSQLCL01)) OnlinePending-->ProcessingFailure.
    00000bac.00000ed0::2013/08/30-21:47:20.580 ERR   [RCM] rcm::RcmResource::HandleFailure: (SQL Network Name (SCSQLCL01))
    00000bac.00000ed0::2013/08/30-21:47:20.580 INFO  [RCM] resource SQL Network Name (SCSQLCL01): failure count: 6, restartAction: 2 persistentState: 1.
    00000bac.00000ed0::2013/08/30-21:47:20.580 INFO  [RCM] Since no top level resource will be offlined by this resource's failure, resetting restartAction to 1
    00000bac.00000ed0::2013/08/30-21:47:20.580 INFO  [RCM] Res SQL Network Name (SCSQLCL01): ProcessingFailure -> WaitingToTerminate( Failed )
    00000bac.00000ed0::2013/08/30-21:47:20.580 INFO  [RCM] TransitionToState(SQL Network Name (SCSQLCL01)) ProcessingFailure-->[WaitingToTerminate to Failed].
    00000bac.00000ed0::2013/08/30-21:47:20.580 INFO  [RCM] Res SQL Network Name (SCSQLCL01): [WaitingToTerminate to Failed] -> Terminating( Failed )
    00000bac.00000ed0::2013/08/30-21:47:20.580 INFO  [RCM] TransitionToState(SQL Network Name (SCSQLCL01)) [WaitingToTerminate to Failed]-->[Terminating to Failed].
    00000bac.00000ed0::2013/08/30-21:47:20.581 WARN  [RCM] resource 'SQL Network Name (SCSQLCL01)' has exceeded its restart limit.
    00000bac.00000ed0::2013/08/30-21:47:20.581 INFO  [RCM] Will retry online from long delay restart of SQL Network Name (SCSQLCL01) in 3600000 milliseconds.
    00000bac.000012f4::2013/08/30-21:47:20.581 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.00000ed0::2013/08/30-21:47:20.581 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    0000060c.00000fa0::2013/08/30-21:47:20.582 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Terminate called for resource SQL Network Name (SCSQLCL01)
    0000060c.00000fa0::2013/08/30-21:47:20.582 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Entering Offline thread
    0000060c.00000fa0::2013/08/30-21:47:20.582 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Performing actual offline of resource.
    0000060c.00000fa0::2013/08/30-21:47:20.582 INFO  [RES] Network Name: Agent: Closing (fdab3086-42d6-4486-9742-eb1ea1197803,AdminShare)
    0000060c.00000fa0::2013/08/30-21:47:20.582 INFO  [RES] Network Name: Agent: Sending request Netname/CloseIndirect to NN:Agent
    0000060c.00001318::2013/08/30-21:47:20.582 INFO  [RES] Network Name: Agent: OnClose (fdab3086-42d6-4486-9742-eb1ea1197803,AdminShare)
    0000060c.00001318::2013/08/30-21:47:20.582 INFO  [RES] Network Name: Agent: Sending request Netname/Close to NN:fdab3086-42d6-4486-9742-eb1ea1197803:AdminShare
    0000060c.00001318::2013/08/30-21:47:20.582 ERR   [RES] Network Name <SQL Network Name (SCSQLCL01)>: AdminShare: OnCloseBase, Error Already Closing, previous state: Closing/Ending
    0000060c.00001318::2013/08/30-21:47:20.582 INFO  [RES] Network Name: Agent: OnCloseReply (fdab3086-42d6-4486-9742-eb1ea1197803,AdminShare) result: 1904
    0000060c.00001318::2013/08/30-21:47:20.582 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: SyncReplyHandler Configuration, result: 1904
    0000060c.00000fa0::2013/08/30-21:47:20.583 INFO  [RES] Network Name: Agent: Closing (fdab3086-42d6-4486-9742-eb1ea1197803,Configuration)
    0000060c.00000fa0::2013/08/30-21:47:20.583 INFO  [RES] Network Name: Agent: Sending request Netname/CloseIndirect to NN:Agent
    0000060c.000009f0::2013/08/30-21:47:20.583 INFO  [RES] Network Name: Agent: OnClose (fdab3086-42d6-4486-9742-eb1ea1197803,Configuration)
    0000060c.000009f0::2013/08/30-21:47:20.583 INFO  [RES] Network Name: Agent: Sending request Netname/Close to NN:fdab3086-42d6-4486-9742-eb1ea1197803:Configuration
    0000060c.000009f0::2013/08/30-21:47:20.583 WARN  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Configuration: Already canceled, state: Closing/Ending
    0000060c.000009f0::2013/08/30-21:47:20.583 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: Configuration: OnCloseBase, previous state: Offline/Ending
    0000060c.000009f0::2013/08/30-21:47:20.583 INFO  [RES] Network Name: Agent: OnCloseReply (fdab3086-42d6-4486-9742-eb1ea1197803,Configuration) result: 0
    0000060c.000009f0::2013/08/30-21:47:20.583 INFO  [RES] Network Name <SQL Network Name (SCSQLCL01)>: SyncReplyHandler Configuration, result: 0
    00000bac.00001080::2013/08/30-21:47:20.583 INFO  [RCM] HandleMonitorReply: TERMINATERESOURCE for 'SQL Network Name (SCSQLCL01)', gen(17) result 0/0.
    00000bac.00001080::2013/08/30-21:47:20.583 INFO  [RCM] Res SQL Network Name (SCSQLCL01): [Terminating to Failed] -> Failed( StateUnknown )
    00000bac.00001080::2013/08/30-21:47:20.583 INFO  [RCM] TransitionToState(SQL Network Name (SCSQLCL01)) [Terminating to Failed]-->Failed.
    00000bac.00001080::2013/08/30-21:47:20.583 INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (SQL Server (VMM), Pending --> Failed)
    00000bac.00001080::2013/08/30-21:47:20.584 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.000012f4::2013/08/30-21:47:20.584 INFO  [RCM] moved 0 tasks from staging set to task set.  TaskSetSize=0
    00000bac.000012f4::2013/08/30-21:47:20.584 INFO  [RCM] rcm::RcmPriorityManager::StartGroups: [RCM] done, executed 0 tasks
    00000bac.000003cc::2013/08/30-21:47:20.584 INFO  [RCM] ignored non-local state Failed for group SQL Server (VMM)
    00000bac.000005d8::2013/08/30-21:47:20.593 INFO  [GUM] Node 2: executing request locally, gumId:426, my action: /dm/update, # of updates: 1
    00000bac.000005d8::2013/08/30-21:47:20.593 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.000005d8::2013/08/30-21:47:20.606 INFO  [GUM] Node 2: executing request locally, gumId:427, my action: /dm/update, # of updates: 1
    00000bac.000005d8::2013/08/30-21:47:20.607 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.000005d8::2013/08/30-21:47:20.624 INFO  [GUM] Node 2: executing request locally, gumId:428, my action: /dm/update, # of updates: 1
    00000bac.000005d8::2013/08/30-21:47:20.625 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.000005d8::2013/08/30-21:47:20.635 INFO  [GUM] Node 2: executing request locally, gumId:429, my action: /dm/update, # of updates: 1
    00000bac.000005d8::2013/08/30-21:47:20.635 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.000005d8::2013/08/30-21:47:20.648 INFO  [GUM] Node 2: executing request locally, gumId:430, my action: /dm/update, # of updates: 1
    00000bac.000005d8::2013/08/30-21:47:20.648 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.000005d8::2013/08/30-21:47:20.658 INFO  [GUM] Node 2: executing request locally, gumId:431, my action: /dm/update, # of updates: 1
    00000bac.000005d8::2013/08/30-21:47:20.659 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message
    00000bac.000005d8::2013/08/30-21:47:20.669 INFO  [GUM] Node 2: executing request locally, gumId:432, my action: /dm/update, # of updates: 1
    00000bac.000005d8::2013/08/30-21:47:20.669 INFO  [GEM] Node 2: Sending 1 messages as a batched GEM message


    Thursday, August 29, 2013 11:56 AM
  • hi matthew,

    so your SQL server is named SCSQLCL01

    but the windows cluster is creating and registering the SCSQLCL01 so what you should do is

    check your cluster  if the cluster name is correctly running ! do propertys on the name and you can 3 OK

    this object needs active directory access on creating objects. this means the cluster name needs full access on the SCSQLCL01 object.

    So basicly the cluster object not the SQL ( SCSQLCL01) needs full access on SCSQLCL01 you will see this will work.

    http://robertsmit.wordpress.com/2012/06/26/cluster-network-name-resource-failed-to-create-its-associated-computer-object-in-domain/

    let me know your status. and did you run cluster validation this is also needed for SQL setup.


    Greetings, Robert Smit Follow me @clustermvp http://robertsmit.wordpress.com/ “Please click "Vote As Helpful" if it is helpful for you and Proposed As Answer” Please remember to click “Mark as Answer” on the post that helps you

    Thursday, August 29, 2013 1:35 PM
    Moderator
  • Not quite - but close

    Nodes:
    SCSQLCL01-1
    SCSQLCL01-2

    Cluster Core name:

    SCSQLCL01CORE (when I created the cluster, this was the name I gave it)

    SQL Cluster Name:

    SCSQLCL01 (specified during SQL setup)

    Alas - the properties of the SQL cluster name object:

    Friday, August 30, 2013 2:01 AM
  • HI,

    But with these settings the SQL node will not get online ?

    And If you are looking in the cluster resoures from scsqlcl01 what part is not starting.

    scsqlcl01 netbios name or the IP adress 192.168.100.166

    Did you run cluster validation ?


    Greetings, Robert Smit Follow me @clustermvp http://robertsmit.wordpress.com/ “Please click "Vote As Helpful" if it is helpful for you and Proposed As Answer” Please remember to click “Mark as Answer” on the post that helps you

    Friday, August 30, 2013 8:20 AM
    Moderator
  • Hi Robert!

    Yes - that's correct - it's sometimes works, sometimes doesn't.

    It's the NETBIOS name SCSQLCL01 - the IP address ALWAYS comes online OK.

    Passes all cluster validation tests OK as well.

    Very strange isn't it? I have several other clusters in the domain NOT running SQL, all of which have no issues moving their cluster NETBIOS name between nodes.

    Monday, September 09, 2013 6:27 AM