Sign in
 
Exchange Server TechCenter > Exchange Server Forums > High availability/Disaster Recovery > Issue with CCR setup in Exchange 2007 SP1
Ask a questionAsk a question
Search Forums:
 

QuestionIssue with CCR setup in Exchange 2007 SP1

  • Monday, October 19, 2009 9:15 AMAl Man Users MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote
    Hi,

    We are running an Exchange 2007 SP1 CCR setup.  We are having an issue with our failover testing.

    When carrying out a manual failover all works fine but if we shut down Node 1, the cluster shows Node 2 as the 'Current Owner' but the status stays on Pending with all databases in an offline state and this never changes.  During this time the following error occurs.

    Event ID – 1177

    Node – Node2

    The Cluster service is shutting down because quorum was lost. This could be due to the loss of network connectivity between some or all nodes in the cluster, or a failover of the witness disk.

    Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapter. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.


    Is this the error that stops the server from coming online correctly?  If so where should I be look to resolve this problem?

    Any help is very much appreciated.

    Alex
     

All Replies

  • Monday, October 19, 2009 1:15 PMDaWy Users MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote
    As you mention your problem is the famous "split brain syndrome", if you want to know more of the same search it in the following article:

    http://technet.microsoft.com/en-us/library/bb124521.aspx

    Therefore, you should verify that you have successfully created the File Share Witness, for this check in detail the following article:

    http://technet.microsoft.com/en-us/library/bb124922.aspx


    Regards
    Dario Woitasen | MCITP: Enterprise Messaging Administrator | MCTS: Microsoft Exchange Server 2010, Configuration
     
  • Monday, October 19, 2009 3:53 PMjader3rd Users MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote
    Do you have a disk witness or a file share witness for your cluster?
     
  • Tuesday, October 20, 2009 2:09 AMJames-LuoMSFT, ModeratorUsers MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote

    Check info:

    1.      Please describe the exchange topology, and the FSW placement

    2.      Please ensure that both nodes are able to communicate fine and also able to communicate with the FSW server

    3.      Please ensure that the FSW folder is excluded from Anti-virus software (Please disable the Anti-virus software on the nodes, FSW server, and try to reproduce the issue)

    4.      Have you referred this article to configure the FSW quorum?

    5.      Please refer this article to check the network configuration

    6.      Please also install KB 950179 on the nodes if you didn’t install already

    7.      Please run ExBPA against the CMS for health check

     
  • Tuesday, October 20, 2009 12:14 PMAl Man Users MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote
    It's a file share witness
     
  • Tuesday, October 20, 2009 12:30 PMAl Man Users MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote
    I've checked all settings in accordance with these documents and I can't see anything obviously wrong.
     
  • Tuesday, October 20, 2009 1:04 PMDaWy Users MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote
    Al, please follow the instructions provided by James and also test the following. Replay the problem by shutting off the node 1, then from node 2 with the cluster service account trying to access by UNC the File Share Witness.
     
    Dario Woitasen | MCITP: Enterprise Messaging Administrator | MCTS: Microsoft Exchange Server 2010, Configuration
     
  • Tuesday, October 20, 2009 4:13 PMjader3rd Users MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote
    Are both of the nodes enabled as a possible owner for the IP address resources? Are both of the nodes enabled as possible owners for the FSW resource?
     
  • Thursday, October 22, 2009 11:33 AMAl Man Users MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote

    Hi, sorry for the slow reply, crazy day yesterday!!

    James, regarding your points

    1. We have 2 Exchange 2007 SP1 servers running CCR with the FSW running on the Hub Server on the Primary Subnet with the Primary Exchange Cluster Node, the Exchange Secondary Cluster Node is running on a different subnet. 

    All servers only have 1 NIC in each.

    Primary Exchange Cluster Node and Hub Server exist in the same blade chassis.

    2. I have checked all server for communications with each other and the FSW.  All is fine.

    3. I had tested with AV off on the 2 clusters, however I had not disabled on the hub server.  I have added an exclusion and test with all 3 without AV at the weekend.

    4. Yes this is the article we used.

    5. Has been done.

    6. Installed.

    7. Done, no issues.

     
  • Thursday, October 22, 2009 11:52 AMAl Man Users MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote
    Jader,

    Both nodes are enabled as possible owners of the IP address resources.

    Both nodes are enabled as possible owners of the quorum.
     
  • Thursday, October 22, 2009 11:53 AMAl Man Users MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote
    Can I confirm one thing, although I believe it is correct.

    Should a controlled shut down of node 1 result in a successful failover to node 2 if all was working correctly?

    Alex.
     
  • Thursday, October 22, 2009 12:25 PMDaWy Users MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote
    Sure Al, a controlled or uncontrolled shutdown of one of your nodes should result in a succesful failover to the other node, if everything is properly configured and the other node is operating as the File Share Witness.
    Dario Woitasen | MCITP: Enterprise Messaging Administrator | MCTS: Microsoft Exchange Server 2010, Configuration
     
  • Thursday, October 22, 2009 4:00 PMAl Man Users MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote

    Dawy,

    Thanks, back to the drawing board then....can't figure this one out!!

    Al.

     
  • Friday, October 23, 2009 3:20 AMJames-LuoMSFT, ModeratorUsers MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote

    Check info:

    1.      Please verify the NIC has been upgraded to latest version

    2.      Please disable IPv6 (KB 952842), TCP Chimney offload, Receive Side Scaling and the Network DMA (KB 951037) on the nodes

    3.      Have you verified the permissions on the FSW folder? Please ensure that cluster service account and admin have full access permission on the folder

    4.      What’s the anti-virus software on the nodes? Is it from Trend Micro?

    5.      Could you restart the cluster service on the node 2? What’s the result?

    6.      Please use ClusDiag.exe to troubleshoot the cluster

     
  • Monday, October 26, 2009 9:02 AMAl Man Users MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote
    1.  Hi, yes updated to latest.

    2. These are disabled.

    3. Yes rights have been verified.

    4. Yes it is Trendmicro Officescan.

    5. I have not done yet.

    6. Working on this.

    One thing I did notice was that the second node was set as a possible owner of the IP address of Node 1.  Which could never happen as on another subnet.  Could this cause the issue?  I would suppose the setup should be.

    Cluster Ownership - Node 1 or Node 2.

    Node 1 IP - Only ClusterServer1 can be Owner
    Node 2 IP - Only ClusterServer2 can be Owner

    Could anyone advise.

    Thanks
    Alex
     
  • Monday, October 26, 2009 9:24 AMJames-LuoMSFT, ModeratorUsers MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote

    Please uninstall the TrendMicro product from nodes, and test the issue again. I have seen several cases that appear the same symptom due to the TrendMicro software on them

     
  • Monday, October 26, 2009 10:27 AMAl Man Users MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote
    James,

    Sorry, we had already removed from the Cluster Nodes, but it's still on the FSW server but with the FSW excluded, sorry I should have said.

    Do you think the mis-config I mentioned above could be effecting it?

    Alex.
     
  • Tuesday, October 27, 2009 1:55 AMJames-LuoMSFT, ModeratorUsers MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote

    Do you talk about the resources in the cluster groups? If yes, both nodes should be the possible owner

     
  • Tuesday, October 27, 2009 2:36 PMAl Man Users MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote
    James,

    If in 'Service and Applications' you go the Server Name.

    Under this you have the 2 IP Addresses (Because our servers are seperate subnets)

    If I do a properties on the IP address of our primary server 2 possible owners are set on the 'advanced policies' tab.  The primary server and the secondary server on our DR site.  But the secondary server exists on another subnet so could never be the owner of this IP.

    When I do the same on the IP address of the secondary server only 1 possible owner is set, the server on the DR.

    I am sure this is wrong I just wonder whether it could cause the issues we are having.

    Alex.
     
  • Wednesday, October 28, 2009 2:56 AMJames-LuoMSFT, ModeratorUsers MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote

    Quote: “When I do the same on the IP address of the secondary server only 1 possible owner is set, the server on the DR”

    Per my knowledge, possible owner should include both nodes

     
  • Friday, October 30, 2009 3:00 PMCapecol Users MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote

    it is recommend that you create a CNAME record in DNS for the server hosting the share, instead of the actual server name. When creating the share for the file share witness use the fully-qualified domain name for the CNAME record instead of the server name, as this practice assists with site resilience.

    check this:http://msexchangeteam.com/archive/2007/04/25/438185.aspx

    Capecol MCSA - MCTS Exchange Server 2007
     
  • Monday, November 09, 2009 8:21 AMJames-LuoMSFT, ModeratorUsers MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote
    Any update on this thread?
    James Luo
    TechNet Subscriber Support (http://technet.microsoft.com/en-us/subscriptions/ms788697.aspx)
    If you have any feedback on our support, please contact tngfb@microsoft.com
     
  • Monday, November 09, 2009 4:49 PMMichel de Rooij Users MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote
    1177 means arbitration failed. Check http://technet.microsoft.com/en-us/library/cc773498(WS.10).aspx
    Also make sure no firewall (Windows, hardware) is active.

    Note: Possible owners should include both.
     
  • Saturday, November 14, 2009 1:58 PMAl Man Users MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote
    We are currently simulating a DR failover. 

    We have found out more information, something in particular we noticed is that the cluster service will start but after a short period it stops.  I ran a cluster log, please see below - hopefully you will be able to see whats going wrong.

    -------------------------------------

    000015ec.000019e4::2009/11/14-13:10:08.337 INFO  -----------------------------+ LOG BEGIN +-----------------------------
    000015ec.000019e4::2009/11/14-13:10:08.337 INFO  [CS] Starting clussvc as a service
    000015ec.000019e4::2009/11/14-13:10:08.337 INFO  [CS] cluster service logging level is 2
    000015ec.000016f8::2009/11/14-13:10:08.368 DBG   [NETFTAPI] received NsiInitialNotification
    000015ec.000016f8::2009/11/14-13:10:08.368 DBG   [NETFTAPI] received NsiInitialNotification
    000015ec.00000f1c::2009/11/14-13:10:08.384 DBG   [NETFTAPI] received NsiAddInstance  for 169.254.2.172
    0000192c.0000104c::2009/11/14-13:10:13.603 INFO  [RHS] Initializing.
    00001998.00001288::2009/11/14-13:10:13.634 INFO  [RHS] Initializing.
    0000192c.000019e0::2009/11/14-13:10:13.681 WARN  [RHS] ERROR_MOD_NOT_FOUND(126), unable to load resource DLL mqclus.dll
    00001998.00001630::2009/11/14-13:10:13.681 WARN  [RHS] ERROR_MOD_NOT_FOUND(126), unable to load resource DLL mqclus.dll
    000015ec.0000193c::2009/11/14-13:10:13.696 WARN  [RCM] Failed to load restype MSMQ: error 126.
    0000192c.000019e0::2009/11/14-13:10:13.696 WARN  [RHS] ERROR_MOD_NOT_FOUND(126), unable to load resource DLL mqtgclus.dll
    00001998.00001630::2009/11/14-13:10:13.696 WARN  [RHS] ERROR_MOD_NOT_FOUND(126), unable to load resource DLL mqtgclus.dll
    000015ec.0000193c::2009/11/14-13:10:13.696 WARN  [RCM] Failed to load restype MSMQTriggers: error 126.
    0000192c.0000104c::2009/11/14-13:10:13.884 WARN  [RES] Network Name <Cluster Name>: Trying to remove credentials for LocalSystem returned status C0000225, STATUS_NOT_FOUND is a non-critical failure for a remove operation
    00001568.00000f00::2009/11/14-13:10:13.899 INFO  [RHS] Initializing.
    000008bc.00000ef0::2009/11/14-13:10:13.915 INFO  [RHS] Initializing.
    00001568.00001748::2009/11/14-13:10:13.978 WARN  [RHS] ERROR_MOD_NOT_FOUND(126), unable to load resource DLL mqclus.dll
    000008bc.0000110c::2009/11/14-13:10:13.978 WARN  [RHS] ERROR_MOD_NOT_FOUND(126), unable to load resource DLL mqclus.dll
    000015ec.0000193c::2009/11/14-13:10:13.978 WARN  [RCM] Failed to load restype MSMQ: error 126.
    00001568.00001748::2009/11/14-13:10:13.978 WARN  [RHS] ERROR_MOD_NOT_FOUND(126), unable to load resource DLL mqtgclus.dll
    000008bc.0000110c::2009/11/14-13:10:13.978 WARN  [RHS] ERROR_MOD_NOT_FOUND(126), unable to load resource DLL mqtgclus.dll
    000015ec.0000193c::2009/11/14-13:10:13.978 WARN  [RCM] Failed to load restype MSMQTriggers: error 126.
    000015ec.0000193c::2009/11/14-13:10:13.993 WARN  [QUORUM] Node 2: One off quorum (2)
    000015ec.0000193c::2009/11/14-13:10:14.087 INFO  [RGP] Parameters set: opening 5 stage 7 pruning 5 tick 300 ms
    000015ec.0000193c::2009/11/14-13:10:14.087 INFO  [CORE] Node 2: Extracting state for Database Manager
    000015ec.0000193c::2009/11/14-13:10:14.181 INFO  [DM] Key \Registry\Machine\Cluster does not appear to be loaded (status STATUS_OBJECT_NAME_NOT_FOUND(c0000034)
    000015ec.0000193c::2009/11/14-13:10:14.181 INFO  [DM] Loading Hive, Key Cluster, FilePath C:\Windows\Cluster\CLUSDB
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  [DM] Paxos Tag Read from Hive: 262:242:113863498
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  [DM] Node 2: Exporting state <BoxedByteArray len='262144'/>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  [CORE] Node 2: Extracting state for Cluster Config Manager
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  [CORE] Node 2: Extracting state for Checkpoint Manager
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  [CORE] Node 2: Extracting state for Topology Manager
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  [TM] ExportState state: <class mscs::TopologyInfo>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO    <Networks><vector len='2'>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO      <item><obj sig='NETW' id='b79cdc3f-8589-4508-b9fe-01151e5cb0d8' name='Public Network 192.168.239.0'/></item>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO      <item><obj sig='NETW' id='b8af03d7-f6c5-4e4e-9b0c-c7ef436c39a6' name='Public Network 192.168.240.0'/></item>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  </vector>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  </Networks>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO    <Interfaces><vector len='2'>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO      <item><obj sig='ITFC' id='9acdd4a9-fc3c-4669-a7c2-d99e801ceb9c' name='LDNEXCL02 - Public'/>node Id:'2' adapterName:'HP NC7782 Gigabit Server Adapter' addrPrefixSet:'<set size='1'>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO      <item>192.168.239.7/24</item>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  </set>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  '
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  </item>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO      <item><obj sig='ITFC' id='5f16938c-6ca5-41d3-b3dd-e29ce9516ef2' name='LDNEXCL01 - Public'/>node Id:'1' adapterName:'HP NC373i Multifunction Gigabit Server Adapter' addrPrefixSet:'<set size='1'>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO      <item>192.168.240.7/24</item>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  </set>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  '
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  </item>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  </vector>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  </Interfaces>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO    <RouteAttributes><struct mscs::TmRouteAttributes>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO    <SameSubnetDelay>1000</SameSubnetDelay>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO    <CrossSubnetDelay>1000</CrossSubnetDelay>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO    <SameSubnetThreshold>5</SameSubnetThreshold>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO    <CrossSubnetThreshold>10</CrossSubnetThreshold>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  </struct mscs::TmRouteAttributes>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  </RouteAttributes>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  </class mscs::TopologyInfo>
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  [CORE] Node 2: Extracting state for Node State Manager
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  [CORE] Node 2: Extracting state for Resource Control Manager
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  [CORE] Node 2: Extracting state for Event Manager
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  [CORE] Node 2: Extracting state for Quorum Manager
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  [CORE] Node 2: Extracting state for Interface Manager
    000015ec.0000193c::2009/11/14-13:10:14.212 INFO  [CORE] Node 2: Extracting state for Network Manager
    000015ec.00001698::2009/11/14-13:10:14.212 INFO  [CORE] Node 2: Calling PostForm(true) for Global Update Manager
    000015ec.00001698::2009/11/14-13:10:14.212 INFO  [CORE] Node 2: Calling PostForm(true) for Causal Multicast
    000015ec.00001698::2009/11/14-13:10:14.212 INFO  [CORE] Node 2: Calling PostForm(true) for QuorumManagerFast
    000015ec.00001698::2009/11/14-13:10:14.212 INFO  [CORE] Node 2: Calling PostForm(true) for Database Manager
    000015ec.00001698::2009/11/14-13:10:14.212 INFO  [CORE] Node 2: Calling PostForm(true) for Cluster Config Manager
    000015ec.00001698::2009/11/14-13:10:14.212 INFO  [CORE] Node 2: Calling PostForm(true) for Checkpoint Manager
    000015ec.00001698::2009/11/14-13:10:14.212 INFO  [CM] mscs::CryptoCheckpoint::RestoreCheckpoint: Restoring crypto checkpoint 1\Microsoft Enhanced Cryptographic Provider v1.0\e8110555-9925-42b4-b90f-06675b351afd-Netname Resource Data.
    000015ec.00001698::2009/11/14-13:10:14.228 INFO  [CM] mscs::CryptoCheckpoint::RestoreCheckpoint: Restoring crypto checkpoint 1\Microsoft Enhanced Cryptographic Provider v1.0\421ecf41-d88b-42e8-b761-7a69e60dcc8e-Netname Resource Data.
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [CORE] Node 2: Calling PostForm(true) for Topology Manager
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [TM] PostForm, forming=true
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [TM] GUMing adapter list List of adapters for LDNEXCL02.*****eu.local, size=1:
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  adapter: HP NC7782 Gigabit Server Adapter (Public)
    000015ec.00001698::2009/11/14-13:10:14.243 INFO     guid: A9328832-D34B-4892-9CA2-1225D30AC347
    000015ec.00001698::2009/11/14-13:10:14.243 INFO     type 6 tunnelType 0 status 1
    000015ec.00001698::2009/11/14-13:10:14.243 INFO     DHCP-Enabled: false
    000015ec.00001698::2009/11/14-13:10:14.243 INFO     prefixes:
    000015ec.00001698::2009/11/14-13:10:14.243 INFO          192.168.239.0/24
    000015ec.00001698::2009/11/14-13:10:14.243 INFO     addresses:
    000015ec.00001698::2009/11/14-13:10:14.243 INFO          192.168.239.7
    000015ec.00001698::2009/11/14-13:10:14.243 INFO     gateways:
    000015ec.00001698::2009/11/14-13:10:14.243 INFO          192.168.239.254
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  Guids:
    000015ec.00001698::2009/11/14-13:10:14.243 INFO      1e7763f4-c017-456c-9f5d-18b317c4a367
    000015ec.00001698::2009/11/14-13:10:14.243 INFO      a89c75f8-66d6-488b-98d9-3fd82a787dd0
    000015ec.00001698::2009/11/14-13:10:14.243 INFO      bdc06763-a269-4077-a42b-709c6d3d91f6
    000015ec.00001698::2009/11/14-13:10:14.243 INFO      ea055aec-2c98-401d-9403-4557aa6e83b2
    000015ec.00001698::2009/11/14-13:10:14.243 INFO      d29641fd-d602-484d-802a-0721c6f5b969
    000015ec.00001698::2009/11/14-13:10:14.243 INFO      e222885a-31f0-41bf-9439-f7d971f89dbe
    000015ec.00001698::2009/11/14-13:10:14.243 INFO      3a2da0fc-313e-44e2-9a33-5c2ea6ce3a0c
    000015ec.00001698::2009/11/14-13:10:14.243 INFO      623b2a5a-703d-4e3f-b408-f8b199eb3e60
    000015ec.00001698::2009/11/14-13:10:14.243 INFO      4f7f43a6-0ddd-4d9a-b3b8-c6ad3536ad6a
    000015ec.00001698::2009/11/14-13:10:14.243 INFO      d784dfba-8f18-40c5-b09a-a95aa8d95fd9
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [ClNet] Preparing input network config for node 2
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [TM] PrepareInputConfigEntryList, examining network <obj sig='NETW' id='b79cdc3f-8589-4508-b9fe-01151e5cb0d8' name='Public Network 192.168.239.0'/>
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [ClNet] NetCfg input: node id 2 is connected to network Public Network 192.168.239.0 (b79cdc3f-8589-4508-b9fe-01151e5cb0d8) by interface LDNEXCL02 - Public (9acdd4a9-fc3c-4669-a7c2-d99e801ceb9c), adapter HP NC7782 Gigabit Server Adapter (A9328832-D34B-4892-9CA2-1225D30AC347)
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [TM] PrepareInputConfigEntryList, examining network <obj sig='NETW' id='b8af03d7-f6c5-4e4e-9b0c-c7ef436c39a6' name='Public Network 192.168.240.0'/>
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [ClNet] NetCfg input: node id 2 is not connected to network Public Network 192.168.240.0 (b8af03d7-f6c5-4e4e-9b0c-c7ef436c39a6)
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  ( Adapters ) = [ClNet] Configuring networks...
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [ClNet] Phase 1 - Examining previous network definitions
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [ClNet] Examining cluster network Public Network 192.168.240.0 (b8af03d7-f6c5-4e4e-9b0c-c7ef436c39a6)
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [ClNet] Ignoring configuration entry for cluster network Public Network 192.168.240.0 (b8af03d7-f6c5-4e4e-9b0c-c7ef436c39a6) because it has no previous matching adapter. Processing has not yet completed so an adapter may still be found for this network.
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [ClNet] Examining cluster network Public Network 192.168.239.0 (b79cdc3f-8589-4508-b9fe-01151e5cb0d8)
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [ClNet] This node was attached to the network by adapter HP NC7782 Gigabit Server Adapter, (A9328832-D34B-4892-9CA2-1225D30AC347)
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [ClNet] Checking if adapter HP NC7782 Gigabit Server Adapter is still installed
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [ClNet] Adapter A9328832-D34B-4892-9CA2-1225D30AC347 is still installed, checking if it is still attached to the same network
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [ClNet] Adapter HP NC7782 Gigabit Server Adapter is still attached to network Public Network 192.168.239.0.
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [ClNet] Phase 2 - Assigning or creating networks for all remaining adapters
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [TM] Adjusting heartbeat parameters
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [IM] Resetting interface state calculation state
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [IM] Sending reports for all interfaces
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [IM] Sending connectivity report to leader (node 2): <class mscs::InterfaceReport>
    000015ec.00001698::2009/11/14-13:10:14.243 INFO    <fromInterface>9acdd4a9-fc3c-4669-a7c2-d99e801ceb9c</fromInterface>
    000015ec.00001698::2009/11/14-13:10:14.243 INFO    <upInterfaces><vector len='1'>
    000015ec.00001698::2009/11/14-13:10:14.243 INFO      <item>9acdd4a9-fc3c-4669-a7c2-d99e801ceb9c</item>
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  </vector>
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  </upInterfaces>
    000015ec.00001698::2009/11/14-13:10:14.243 INFO    <downInterfaces><vector len='0'>
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  </vector>
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  </downInterfaces>
    000015ec.00001698::2009/11/14-13:10:14.243 INFO    <viewId>002</viewId>
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  </class mscs::InterfaceReport>
    000015ec.00000f1c::2009/11/14-13:10:14.243 INFO  [IM] Leader got report <class mscs::InterfaceReport>
    000015ec.00000f1c::2009/11/14-13:10:14.243 INFO    <fromInterface>9acdd4a9-fc3c-4669-a7c2-d99e801ceb9c</fromInterface>
    000015ec.00000f1c::2009/11/14-13:10:14.243 INFO    <upInterfaces><vector len='1'>
    000015ec.00000f1c::2009/11/14-13:10:14.243 INFO      <item>9acdd4a9-fc3c-4669-a7c2-d99e801ceb9c</item>
    000015ec.00000f1c::2009/11/14-13:10:14.243 INFO  </vector>
    000015ec.00000f1c::2009/11/14-13:10:14.243 INFO  </upInterfaces>
    000015ec.00000f1c::2009/11/14-13:10:14.243 INFO    <downInterfaces><vector len='0'>
    000015ec.00000f1c::2009/11/14-13:10:14.243 INFO  </vector>
    000015ec.00000f1c::2009/11/14-13:10:14.243 INFO  </downInterfaces>
    000015ec.00000f1c::2009/11/14-13:10:14.243 INFO    <viewId>002</viewId>
    000015ec.00000f1c::2009/11/14-13:10:14.243 INFO  </class mscs::InterfaceReport>
    000015ec.00000f1c::2009/11/14-13:10:14.243 INFO   from 2
    000015ec.00000f1c::2009/11/14-13:10:14.243 INFO  [IM] 1 reports in state calculator queue
    000015ec.000011c4::2009/11/14-13:10:14.243 INFO  [IM] State calculator got new report from 9acdd4a9-fc3c-4669-a7c2-d99e801ceb9c
    000015ec.000011c4::2009/11/14-13:10:14.243 INFO  [IM] 0 reports in state calculator queue
    000015ec.000011c4::2009/11/14-13:10:14.243 INFO  [IM] Ideal Matrix  = <SquareMatrix>
    000015ec.000011c4::2009/11/14-13:10:14.243 INFO    <Row id='0'1/>
    000015ec.000011c4::2009/11/14-13:10:14.243 INFO  </SquareMatrix>
    000015ec.000011c4::2009/11/14-13:10:14.243 INFO  [IM] Actual Matrix = <SquareMatrix>
    000015ec.000011c4::2009/11/14-13:10:14.243 INFO    <Row id='0'1/>
    000015ec.000011c4::2009/11/14-13:10:14.243 INFO  </SquareMatrix>
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [CORE] Node 2: Calling PostForm(true) for Node State Manager
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [CORE] Node 2: Calling PostForm(true) for Resource Control Manager
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [RCM] PostForm is called - forming = true.
    000015ec.00001698::2009/11/14-13:10:14.243 INFO  [VER] Updating node versions in the cluster database.
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  (allSplitGroups, splitGroups) = [IM] Calculated interface state result: <class mscs::InterfaceResult>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO    <up><vector len='1'>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO      <item>9acdd4a9-fc3c-4669-a7c2-d99e801ceb9c</item>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  </vector>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  </up>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO    <down><vector len='0'>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  </vector>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  </down>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO    <unreachable><vector len='0'>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  </vector>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  </unreachable>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  </class mscs::InterfaceResult>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  [IM] Issuing state change gum with result <class mscs::InterfaceResult>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO    <up><vector len='1'>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO      <item>9acdd4a9-fc3c-4669-a7c2-d99e801ceb9c</item>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  </vector>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  </up>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO    <down><vector len='0'>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  </vector>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  </down>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO    <unreachable><vector len='0'>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  </vector>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  </unreachable>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  </class mscs::InterfaceResult>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  [IM] Changing the state of adapters according to result: <class mscs::InterfaceResult>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO    <up><vector len='1'>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO      <item>9acdd4a9-fc3c-4669-a7c2-d99e801ceb9c</item>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  </vector>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  </up>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO    <down><vector len='0'>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  </vector>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  </down>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO    <unreachable><vector len='0'>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  </vector>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  </unreachable>
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  </class mscs::InterfaceResult>
    000015ec.0000193c::2009/11/14-13:10:14.259 INFO  [NM] Changing network state for network Public Network 192.168.239.0 to 3
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  [IM] State calculation complete without issuing pings
    000015ec.000011c4::2009/11/14-13:10:14.259 INFO  [IM] Resetting interface state calculation state
    000015ec.0000193c::2009/11/14-13:10:14.259 INFO  [NM] Received GUM to change state of network Public Network 192.168.239.0 (id b79cdc3f-8589-4508-b9fe-01151e5cb0d8) to 3
    00001568.00001858::2009/11/14-13:10:14.384 INFO  [RES] File Server: CLUSCTL_RESOURCE_TYPE_STARTING_PHASE1 Called
    00001568.00001858::2009/11/14-13:10:14.759 ERR   [RHS] RhsCall::Perform_NativeEH: ERROR_MOD_NOT_FOUND(126)' because of 'Unable to load resource DLL mqclus.dll.
    '
    00001568.00001858::2009/11/14-13:10:14.790 ERR   [RHS] RhsCall::Perform_NativeEH: ERROR_MOD_NOT_FOUND(126)' because of 'Unable to load resource DLL mqtgclus.dll.
    '
    000015ec.00001698::2009/11/14-13:10:14.977 INFO  [RCM] TransitionToState(Cluster IP Address) Initializing-->OpenCallIssued.
    000015ec.00001698::2009/11/14-13:10:14.977 INFO  [RCM] TransitionToState(Exchange Information Store Instance (LDNMBX01)) Initializing-->OpenCallIssued.
    000015ec.00001698::2009/11/14-13:10:14.977 INFO  [RCM] TransitionToState(Exchange System Attendant Instance (LDNMBX01)) Initializing-->OpenCallIssued.
    000015ec.00001698::2009/11/14-13:10:14.977 INFO  [RCM] TransitionToState(First Storage Group/Mailbox Database (LDNMBX01)) Initializing-->OpenCallIssued.
    000015ec.00001698::2009/11/14-13:10:14.977 INFO  [RCM] TransitionToState(Helpdesk/HelpDesk Mailbox (LDNMBX01)) Initializing-->OpenCallIssued.
    000015ec.00001698::2009/11/14-13:10:14.977 INFO  [RCM] TransitionToState(IPv4 Static Address 1 (LDNMBX01)) Initializing-->OpenCallIssued.
    00001568.00001984::2009/11/14-13:10:14.977 WARN  [RES] Microsoft Exchange Information Store <Exchange Information Store Instance (LDNMBX01)>: [EXRES] ExchangeOpen: Open requested.
    000015ec.00001698::2009/11/14-13:10:14.977 INFO  [RCM] TransitionToState(IPv4 Static Address 2 (LDNMBX01)) Initializing-->OpenCallIssued.
    000015ec.00001698::2009/11/14-13:10:14.977 INFO  [RCM] TransitionToState(JOURNALING/JOURNALING Mailbox (LDNMBX01)) Initializing-->OpenCallIssued.
    000015ec.00001698::2009/11/14-13:10:14.977 INFO  [RCM] TransitionToState(*****SEL Group/*****SEL Mailbox (LDNMBX01)) Initializing-->OpenCallIssued.
    000015ec.00001698::2009/11/14-13:10:14.977 INFO  [RCM] TransitionToState(*****FIN Group/*****FIN (LDNMBX01)) Initializing-->OpenCallIssued.
    000015ec.00001698::2009/11/14-13:10:14.977 INFO  [RCM] TransitionToState(*****NG Group/*****NG Mailbox (LDNMBX01)) Initializing-->OpenCallIssued.
    00001568.000015a0::2009/11/14-13:10:14.977 WARN  [RES] Microsoft Exchange System Attendant <Exchange System Attendant Instance (LDNMBX01)>: [EXRES] ExchangeOpen: Open requested.
    000015ec.00001698::2009/11/14-13:10:14.977 INFO  [RCM] TransitionToState(Network Name (LDNMBX01)) Initializing-->OpenCallIssued.
    000015ec.00001698::2009/11/14-13:10:14.977 INFO  [RCM] TransitionToState(Public Folders/Public Folders (LDNMBX01)) Initializing-->OpenCallIssued.
    000015ec.00001698::2009/11/14-13:10:14.977 INFO  [API] Online.
    000015ec.00001698::2009/11/14-13:10:14.977 INFO  [RCM] rcm::RcmGum::NodeIsUp(2)
    00001568.00001220::2009/11/14-13:10:14.977 WARN  [RES] Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (LDNMBX01)>: [EXRES] ExchangeOpen: Open requested.
    00001568.00001920::2009/11/14-13:10:14.977 WARN  [RES] Microsoft Exchange Database Instance <Helpdesk/HelpDesk Mailbox (LDNMBX01)>: [EXRES] ExchangeOpen: Open requested.
    00001568.00001614::2009/11/14-13:10:14.977 WARN  [RES] Microsoft Exchange Database Instance <JOURNALING/JOURNALING Mailbox (LDNMBX01)>: [EXRES] ExchangeOpen: Open requested.
    00001568.000018ec::2009/11/14-13:10:14.977 WARN  [RES] Microsoft Exchange Database Instance <*****SEL Group/*****SEL Mailbox (LDNMBX01)>: [EXRES] ExchangeOpen: Open requested.
    00001568.000019e8::2009/11/14-13:10:14.977 WARN  [RES] Microsoft Exchange Database Instance <*****NG Group/*****NG Mailbox (LDNMBX01)>: [EXRES] ExchangeOpen: Open requested.
    00001568.0000178c::2009/11/14-13:10:14.977 WARN  [RES] Microsoft Exchange Database Instance <*****FIN Group/*****FIN (LDNMBX01)>: [EXRES] ExchangeOpen: Open requested.
    00001568.000015a0::2009/11/14-13:10:14.977 WARN  [RES] Microsoft Exchange System Attendant <Exchange System Attendant Instance (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::AddRef: Count=1.
    00001568.00001984::2009/11/14-13:10:14.977 WARN  [RES] Microsoft Exchange Information Store <Exchange Information Store Instance (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::AddRef: Count=1.
    00001568.00001b70::2009/11/14-13:10:14.977 INFO  [RES] Network Name <Network Name (LDNMBX01)>: NetNameOpen Invoked
    00001568.00001560::2009/11/14-13:10:14.977 WARN  [RES] Microsoft Exchange Database Instance <Public Folders/Public Folders (LDNMBX01)>: [EXRES] ExchangeOpen: Open requested.
    00001568.00001920::2009/11/14-13:10:14.977 WARN  [RES] Microsoft Exchange Database Instance <Helpdesk/HelpDesk Mailbox (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::AddRef: Count=1.
    00001568.0000178c::2009/11/14-13:10:14.977 WARN  [RES] Microsoft Exchange Database Instance <*****FIN Group/*****FIN (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::AddRef: Count=1.
    00001568.00001614::2009/11/14-13:10:14.977 WARN  [RES] Microsoft Exchange Database Instance <JOURNALING/JOURNALING Mailbox (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::AddRef: Count=1.
    00001568.00001220::2009/11/14-13:10:14.977 WARN  [RES] Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::AddRef: Count=1.
    00001568.000019e8::2009/11/14-13:10:14.977 WARN  [RES] Microsoft Exchange Database Instance <*****NG Group/*****NG Mailbox (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::AddRef: Count=1.
    00001568.000018ec::2009/11/14-13:10:14.977 WARN  [RES] Microsoft Exchange Database Instance <*****SEL Group/*****SEL Mailbox (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::AddRef: Count=1.
    00001568.00001560::2009/11/14-13:10:14.977 WARN  [RES] Microsoft Exchange Database Instance <Public Folders/Public Folders (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::AddRef: Count=1.
    00001568.000018a8::2009/11/14-13:10:14.977 INFO  [RES] IP Address <IPv4 Static Address 2 (LDNMBX01)>: Deleting IP interface 6EFA8C0.
    00001568.000018a8::2009/11/14-13:10:14.977 WARN  [RES] IP Address <IPv4 Static Address 2 (LDNMBX01)>: Failed to delete IP interface 6EFA8C0, status 87.
    00001568.00001b10::2009/11/14-13:10:14.977 INFO  [RES] IP Address <IPv4 Static Address 1 (LDNMBX01)>: Resource open, resource ID = 2155520.
    00001568.00001858::2009/11/14-13:10:14.993 INFO  [RES] IP Address <Cluster IP Address>: Resource open, resource ID = 2391392.
    00001568.00001b70::2009/11/14-13:10:14.993 INFO  [RES] Network Name <Network Name (LDNMBX01)>: Successful open of resid 48663264
    00001568.00001b70::2009/11/14-13:10:15.102 INFO  [RES] File Server: CLUSCTL_RESOURCE_TYPE_STARTING_PHASE2 Called
    00001568.00001994::2009/11/14-13:10:15.446 WARN  [RHS] ERROR_MOD_NOT_FOUND(126), unable to load resource DLL mqclus.dll
    000015ec.00000888::2009/11/14-13:10:15.446 INFO  [RCM] rcm::RcmResType::LoadDll: Got error 126; will attempt to load mqclus.dll via Wow64.
    000008bc.0000110c::2009/11/14-13:10:15.446 WARN  [RHS] ERROR_MOD_NOT_FOUND(126), unable to load resource DLL mqclus.dll
    000015ec.00000888::2009/11/14-13:10:15.446 WARN  [RCM] Failed to load restype MSMQ: error 126.
    00001568.00001614::2009/11/14-13:10:15.477 ERR   [RHS] RhsCall::Perform_NativeEH: ERROR_MOD_NOT_FOUND(126)' because of 'Unable to load resource DLL mqclus.dll.
    '
    00001568.00001994::2009/11/14-13:10:15.477 WARN  [RHS] ERROR_MOD_NOT_FOUND(126), unable to load resource DLL mqtgclus.dll
    000015ec.00000888::2009/11/14-13:10:15.477 INFO  [RCM] rcm::RcmResType::LoadDll: Got error 126; will attempt to load mqtgclus.dll via Wow64.
    000008bc.0000110c::2009/11/14-13:10:15.477 WARN  [RHS] ERROR_MOD_NOT_FOUND(126), unable to load resource DLL mqtgclus.dll
    000015ec.00000888::2009/11/14-13:10:15.477 WARN  [RCM] Failed to load restype MSMQTriggers: error 126.
    00001568.00001614::2009/11/14-13:10:15.509 ERR   [RHS] RhsCall::Perform_NativeEH: ERROR_MOD_NOT_FOUND(126)' because of 'Unable to load resource DLL mqtgclus.dll.
    '
    00001568.00001614::2009/11/14-13:10:15.540 INFO  [RES] Network Name: RemoveVcoOnResCleanup set to 0
    000015ec.00001698::2009/11/14-13:10:15.696 INFO  [RCM] rcm::RcmGum::GroupMoveOperation(Cluster Group,2)
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] Cluster Name cannot be hosted on node 2
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] Possible owners: <vector len='1'>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN      <item>1</item>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  </vector>
    000015ec.00001698::2009/11/14-13:10:15.696 INFO  [RCM] rcm::RcmGroup::ProcessStateChange: (Cluster Group, Orphaned --> Offline)
    000015ec.00001698::2009/11/14-13:10:15.696 INFO  [RCM] Finding potential nodes.
    000015ec.00001698::2009/11/14-13:10:15.696 INFO  [RCM] Choosing node 2 randomly from list of 1 nodes for Available Storage.
    000015ec.00001698::2009/11/14-13:10:15.696 INFO  [RCM] Choosing node 2 for group 'Available Storage'.
    000015ec.00001698::2009/11/14-13:10:15.696 INFO  [RCM] rcm::RcmGum::GroupMoveOperation(Available Storage,2)
    000015ec.00001698::2009/11/14-13:10:15.696 INFO  [RCM] rcm::RcmGroup::ProcessStateChange: (Available Storage, Orphaned --> BringingToPersistentState)
    000015ec.0000193c::2009/11/14-13:10:15.696 INFO  [RCM] rcm::RcmGroup::ProcessStateChange: (Available Storage, BringingToPersistentState --> Offline)
    000015ec.0000193c::2009/11/14-13:10:15.696 INFO  [RCM] Group move for 'Available Storage' has completed.
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] IPv4 Static Address 1 (LDNMBX01) cannot be hosted on node 2
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] Possible owners: <vector len='1'>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN      <item>1</item>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  </vector>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] IPv4 Static Address 1 (LDNMBX01) cannot be hosted on node 2
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] Possible owners: <vector len='1'>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN      <item>1</item>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  </vector>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] IPv4 Static Address 1 (LDNMBX01) cannot be hosted on node 2
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] Possible owners: <vector len='1'>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN      <item>1</item>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  </vector>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] IPv4 Static Address 1 (LDNMBX01) cannot be hosted on node 2
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] Possible owners: <vector len='1'>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN      <item>1</item>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  </vector>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] IPv4 Static Address 1 (LDNMBX01) cannot be hosted on node 2
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] Possible owners: <vector len='1'>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN      <item>1</item>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  </vector>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] IPv4 Static Address 1 (LDNMBX01) cannot be hosted on node 2
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] Possible owners: <vector len='1'>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN      <item>1</item>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  </vector>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] IPv4 Static Address 1 (LDNMBX01) cannot be hosted on node 2
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] Possible owners: <vector len='1'>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN      <item>1</item>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  </vector>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] IPv4 Static Address 1 (LDNMBX01) cannot be hosted on node 2
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] Possible owners: <vector len='1'>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN      <item>1</item>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  </vector>
    000015ec.00001698::2009/11/14-13:10:15.696 INFO  [RCM] Choosing node 2 from preferred node order.
    000015ec.00001698::2009/11/14-13:10:15.696 INFO  [RCM] Choosing node 2 for group 'LDNMBX01'.
    000015ec.00001698::2009/11/14-13:10:15.696 INFO  [RCM] rcm::RcmGum::GroupMoveOperation(LDNMBX01,2)
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] IPv4 Static Address 1 (LDNMBX01) cannot be hosted on node 2
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] Possible owners: <vector len='1'>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN      <item>1</item>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  </vector>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] IPv4 Static Address 1 (LDNMBX01) cannot be hosted on node 2
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] Possible owners: <vector len='1'>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN      <item>1</item>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  </vector>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] IPv4 Static Address 1 (LDNMBX01) cannot be hosted on node 2
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] Possible owners: <vector len='1'>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN      <item>1</item>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  </vector>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] IPv4 Static Address 1 (LDNMBX01) cannot be hosted on node 2
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] Possible owners: <vector len='1'>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN      <item>1</item>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  </vector>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] IPv4 Static Address 1 (LDNMBX01) cannot be hosted on node 2
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] Possible owners: <vector len='1'>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN      <item>1</item>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  </vector>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] IPv4 Static Address 1 (LDNMBX01) cannot be hosted on node 2
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] Possible owners: <vector len='1'>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN      <item>1</item>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  </vector>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] IPv4 Static Address 1 (LDNMBX01) cannot be hosted on node 2
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] Possible owners: <vector len='1'>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN      <item>1</item>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  </vector>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] IPv4 Static Address 1 (LDNMBX01) cannot be hosted on node 2
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  [RCM] Possible owners: <vector len='1'>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN      <item>1</item>
    000015ec.00001698::2009/11/14-13:10:15.696 WARN  </vector>
    000015ec.00001698::2009/11/14-13:10:15.696 INFO  [RCM] rcm::RcmGroup::ProcessStateChange: (LDNMBX01, Orphaned --> BringingToPersistentState)
    000015ec.00001698::2009/11/14-13:10:15.696 INFO  [CORE] Node 2: Calling PostForm(true) for Event Manager
    000015ec.00001698::2009/11/14-13:10:15.696 INFO  [CORE] Node 2: Calling PostForm(true) for Quorum Manager
    000015ec.00000888::2009/11/14-13:10:15.696 WARN  [RCM] IPv4 Static Address 1 (LDNMBX01) cannot be hosted on node 2
    000015ec.00000888::2009/11/14-13:10:15.696 WARN  [RCM] Possible owners: <vector len='1'>
    000015ec.00000888::2009/11/14-13:10:15.696 WARN      <item>1</item>
    000015ec.00000888::2009/11/14-13:10:15.696 WARN  </vector>
    000015ec.00001698::2009/11/14-13:10:15.696 INFO  [CORE] Node 2: Calling PostForm(true) for Interface Manager
    000015ec.00001698::2009/11/14-13:10:15.696 INFO  [IM] Processing view change
    000015ec.00001698::2009/11/14-13:10:15.696 INFO  [IM] Resetting interface state calculation state
    000015ec.00001698::2009/11/14-13:10:15.696 INFO  [IM] Leader is sending request for all interfaces in the current view
    000015ec.00001698::2009/11/14-13:10:15.696 INFO  [CORE] Node 2: Calling PostForm(true) for Network Manager
    000015ec.00001698::2009/11/14-13:10:15.696 INFO  [CORE] Node 2: PostForm is completed for all components
    000015ec.0000193c::2009/11/14-13:10:15.696 INFO  [NM] Changing network state for network Public Network 192.168.240.0 to 1
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO  [IM] Got a request for interface state: <class mscs::InterfaceRequest>
    000015ec.0000193c::2009/11/14-13:10:15.696 INFO  [NM] Received GUM to change state of network Public Network 192.168.240.0 (id b8af03d7-f6c5-4e4e-9b0c-c7ef436c39a6) to 1
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO    <interfaces><vector len='1'>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO      <item>9acdd4a9-fc3c-4669-a7c2-d99e801ceb9c</item>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO  </vector>
    000015ec.00001698::2009/11/14-13:10:15.696 INFO  [IM] Ignoring down event for Network adapter Public Network 192.168.240.0 because it does not has any node in the current view
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO  </interfaces>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO  </class mscs::InterfaceRequest>
    000015ec.00000888::2009/11/14-13:10:15.696 WARN  [RCM] IPv4 Static Address 1 (LDNMBX01) cannot be hosted on node 2
    000015ec.00000888::2009/11/14-13:10:15.696 WARN  [RCM] Possible owners: <vector len='1'>
    000015ec.00000888::2009/11/14-13:10:15.696 WARN      <item>1</item>
    000015ec.00000888::2009/11/14-13:10:15.696 WARN  </vector>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO  [IM] Found the following interfaces on my machine <list len='1'>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO      <item><obj sig='ITFC' id='9acdd4a9-fc3c-4669-a7c2-d99e801ceb9c' name='LDNEXCL02 - Public'/>node Id:'2' adapterName:'HP NC7782 Gigabit Server Adapter' addrPrefixSet:'<set size='1'>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO      <item>192.168.239.7/24</item>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO  </set>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO  '
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO  </item>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO  </list>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO  [IM] Sending connectivity report to leader (node 2): <class mscs::InterfaceReport>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO    <fromInterface>9acdd4a9-fc3c-4669-a7c2-d99e801ceb9c</fromInterface>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO    <upInterfaces><vector len='1'>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO      <item>9acdd4a9-fc3c-4669-a7c2-d99e801ceb9c</item>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO  </vector>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO  </upInterfaces>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO    <downInterfaces><vector len='0'>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO  </vector>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO  </downInterfaces>
    000015ec.00000888::2009/11/14-13:10:15.696 INFO  [RCM] rcm::RcmResource::Online: bringing Public Folders/Public Folders (LDNMBX01)'s provider resource 'Exchange Information Store Instance (LDNMBX01)' online.
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO    <viewId>002</viewId>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO  </class mscs::InterfaceReport>
    000015ec.00000888::2009/11/14-13:10:15.696 INFO  [RCM] rcm::RcmResource::Online: bringing Public Folders/Public Folders (LDNMBX01)'s provider resource 'Network Name (LDNMBX01)' online.
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO  [IM] Leader got report <class mscs::InterfaceReport>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO    <fromInterface>9acdd4a9-fc3c-4669-a7c2-d99e801ceb9c</fromInterface>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO    <upInterfaces><vector len='1'>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO      <item>9acdd4a9-fc3c-4669-a7c2-d99e801ceb9c</item>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO  </vector>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO  </upInterfaces>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO    <downInterfaces><vector len='0'>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO  </vector>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO  </downInterfaces>
    000015ec.00000888::2009/11/14-13:10:15.696 WARN  [RCM] IPv4 Static Address 1 (LDNMBX01) cannot be hosted on node 2
    000015ec.00000888::2009/11/14-13:10:15.696 WARN  [RCM] Possible owners: <vector len='1'>
    000015ec.00000888::2009/11/14-13:10:15.696 WARN      <item>1</item>
    000015ec.00000888::2009/11/14-13:10:15.696 WARN  </vector>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO    <viewId>002</viewId>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO  </class mscs::InterfaceReport>
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO   from 2
    000015ec.00000f1c::2009/11/14-13:10:15.696 INFO  [IM] 1 reports in state calculator queue
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  [IM] State calculator got new report from 9acdd4a9-fc3c-4669-a7c2-d99e801ceb9c
    000015ec.00000888::2009/11/14-13:10:15.696 INFO  [RCM] rcm::RcmResource::Online: bringing Public Folders/Public Folders (LDNMBX01)'s provider resource 'IPv4 Static Address 2 (LDNMBX01)' online.
    000015ec.00000888::2009/11/14-13:10:15.696 INFO  [RCM] rcm::RcmResource::IssueOnlineCallIfReady: Waiting for open call to resource 'IPv4 Static Address 2 (LDNMBX01)' to complete (state is OpenCallIssued).
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  [IM] 0 reports in state calculator queue
    000015ec.00001698::2009/11/14-13:10:15.696 INFO  [IM] Ignoring down event for Network adapter Public Network 192.168.240.0 because it does not has any node in the current view
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  [IM] Ideal Matrix  = <SquareMatrix>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO    <Row id='0'1/>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  </SquareMatrix>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  [IM] Actual Matrix = <SquareMatrix>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO    <Row id='0'1/>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  </SquareMatrix>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  (allSplitGroups, splitGroups) = [IM] Calculated interface state result: <class mscs::InterfaceResult>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO    <up><vector len='1'>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO      <item>9acdd4a9-fc3c-4669-a7c2-d99e801ceb9c</item>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  </vector>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  </up>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO    <down><vector len='0'>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  </vector>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  </down>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO    <unreachable><vector len='0'>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  </vector>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  </unreachable>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  </class mscs::InterfaceResult>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  [IM] Issuing state change gum with result <class mscs::InterfaceResult>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO    <up><vector len='1'>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO      <item>9acdd4a9-fc3c-4669-a7c2-d99e801ceb9c</item>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  </vector>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  </up>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO    <down><vector len='0'>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  </vector>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  </down>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO    <unreachable><vector len='0'>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  </vector>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  </unreachable>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  </class mscs::InterfaceResult>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  [IM] Changing the state of adapters according to result: <class mscs::InterfaceResult>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO    <up><vector len='1'>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO      <item>9acdd4a9-fc3c-4669-a7c2-d99e801ceb9c</item>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  </vector>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  </up>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO    <down><vector len='0'>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  </vector>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  </down>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO    <unreachable><vector len='0'>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  </vector>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  </unreachable>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  </class mscs::InterfaceResult>
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  [IM] State calculation complete without issuing pings
    000015ec.000011c4::2009/11/14-13:10:15.696 INFO  [IM] Resetting interface state calculation state
    000015ec.00001344::2009/11/14-13:10:26.196 INFO  [NM] Received request from client address ::1.
    000015ec.00000f38::2009/11/14-13:10:26.212 INFO  [NM] Received request from client address ::1.
    000015ec.00000728::2009/11/14-13:10:29.305 INFO  [NM] Received request from client address ::1.
    000015ec.000010c4::2009/11/14-13:10:29.321 INFO  [NM] Received request from client address ::1.
    000015ec.000010c4::2009/11/14-13:10:29.883 INFO  [NM] Received request from client address LDNEXCL02.
    000015ec.000010c4::2009/11/14-13:10:30.274 INFO  [NM] Received request from client address LDNEXCL02.
    000015ec.00000f38::2009/11/14-13:10:31.180 INFO  [API] s_ApiGetQuorumResource final status 0.
    000015ec.00000f38::2009/11/14-13:10:34.180 INFO  [API] s_ApiGetQuorumResource final status 0.
    000015ec.00001344::2009/11/14-13:10:37.930 INFO  [RCM] rcm::RcmResource::Control: Waiting for open call to resource 'IPv4 Static Address 2 (LDNMBX01)' to complete (state is OpenCallIssued).
    000015ec.00000f74::2009/11/14-13:10:40.836 INFO  [NM] Received request from client address ::1.
    000015ec.00000f74::2009/11/14-13:11:08.601 INFO  [RCM] rcm::RcmResource::Control: Waiting for open call to resource 'IPv4 Static Address 2 (LDNMBX01)' to complete (state is OpenCallIssued).
    000015ec.0000196c::2009/11/14-13:11:08.757 INFO  [NM] Received request from client address ::1.
    000015ec.000017ec::2009/11/14-13:11:13.601 INFO  [JPM] Node 2: Found possible contact address 192.168.240.7:~0~ for node LDNEXCL01 in ClusDB.
    000015ec.000017ec::2009/11/14-13:11:13.601 INFO  [JPM] Node 2: Scheduling DNS query for node name LDNEXCL01
    000015ec.0000143c::2009/11/14-13:11:44.522 ERR   [QUORUM] Node 2: Death timer expired after 90 seconds (death timer started at 2009/11/14-13:10:13.993). Lost quorum.
    000015ec.0000143c::2009/11/14-13:11:44.522 ERR   lost quorum (status = 5925)
    000015ec.0000143c::2009/11/14-13:11:44.522 ERR   FatalError is Calling Exit Process.
    000015ec.000019e4::2009/11/14-13:11:44.522 INFO  [CS] About to exit process...
    00001568.000018a8::2009/11/14-13:11:44.522 WARN  [RES] IP Address <IPv4 Static Address 2 (LDNMBX01)>: Failed to delete IP interface information from database, status 1726.
    00001568.000018a8::2009/11/14-13:11:44.522 INFO  [RES] IP Address <IPv4 Static Address 2 (LDNMBX01)>: could not delete IP address [Unknown], probably it is no longer valid.
    00001568.000018a8::2009/11/14-13:11:44.522 INFO  [RES] IP Address <IPv4 Static Address 2 (LDNMBX01)>: Resource open, resource ID = 2396000.
    00001568.000018a8::2009/11/14-13:11:44.522 ERR   [RHS] Error 1722 calling RcmReceiveRhsReply().
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RHS] Cluster service has terminated.
    000008bc.00000ef0::2009/11/14-13:11:44.522 WARN  [RHS] Cluster service has terminated.
    000008bc.00000ef0::2009/11/14-13:11:44.522 INFO  [RHS] Exiting.
    00001568.00000f00::2009/11/14-13:11:44.522 INFO  [RES] IP Address <IPv4 Static Address 2 (LDNMBX01)>: IpaClose entry
    00001568.00000f00::2009/11/14-13:11:44.522 INFO  [RES] IP Address <IPv4 Static Address 2 (LDNMBX01)>: IpaReleaseResource entry
    00001568.00000f00::2009/11/14-13:11:44.522 INFO  [RES] IP Address <IPv4 Static Address 2 (LDNMBX01)>: Resource closed.
    00001568.00000f00::2009/11/14-13:11:44.522 INFO  [RES] IP Address <Cluster IP Address>: IpaClose entry
    00001568.00000f00::2009/11/14-13:11:44.522 INFO  [RES] IP Address <Cluster IP Address>: IpaReleaseResource entry
    00001568.00000f00::2009/11/14-13:11:44.522 INFO  [RES] IP Address <Cluster IP Address>: Resource closed.
    00001568.00000f00::2009/11/14-13:11:44.522 INFO  [RES] IP Address <IPv4 Static Address 1 (LDNMBX01)>: IpaClose entry
    00001568.00000f00::2009/11/14-13:11:44.522 INFO  [RES] IP Address <IPv4 Static Address 1 (LDNMBX01)>: IpaReleaseResource entry
    00001568.00000f00::2009/11/14-13:11:44.522 INFO  [RES] IP Address <IPv4 Static Address 1 (LDNMBX01)>: Resource closed.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange Database Instance <Public Folders/Public Folders (LDNMBX01)>: [EXRES] ExchangeClose: Close requested.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange Database Instance <Public Folders/Public Folders (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::Release: Count=0.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange Database Instance <Public Folders/Public Folders (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::Release: releasing resource private data.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange Database Instance <*****FIN Group/*****FIN (LDNMBX01)>: [EXRES] ExchangeClose: Close requested.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange Database Instance <*****FIN Group/*****FIN (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::Release: Count=0.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange Database Instance <*****FIN Group/*****FIN (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::Release: releasing resource private data.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange Database Instance <Helpdesk/HelpDesk Mailbox (LDNMBX01)>: [EXRES] ExchangeClose: Close requested.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange Database Instance <Helpdesk/HelpDesk Mailbox (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::Release: Count=0.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange Database Instance <Helpdesk/HelpDesk Mailbox (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::Release: releasing resource private data.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange Database Instance <*****NG Group/*****NG Mailbox (LDNMBX01)>: [EXRES] ExchangeClose: Close requested.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange Database Instance <*****NG Group/*****NG Mailbox (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::Release: Count=0.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange Database Instance <*****NG Group/*****NG Mailbox (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::Release: releasing resource private data.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange Database Instance <*****SEL Group/*****SEL Mailbox (LDNMBX01)>: [EXRES] ExchangeClose: Close requested.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange Database Instance <*****SEL Group/*****SEL Mailbox (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::Release: Count=0.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange Database Instance <*****SEL Group/*****SEL Mailbox (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::Release: releasing resource private data.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (LDNMBX01)>: [EXRES] ExchangeClose: Close requested.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::Release: Count=0.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange Database Instance <First Storage Group/Mailbox Database (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::Release: releasing resource private data.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange Database Instance <JOURNALING/JOURNALING Mailbox (LDNMBX01)>: [EXRES] ExchangeClose: Close requested.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange Database Instance <JOURNALING/JOURNALING Mailbox (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::Release: Count=0.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange Database Instance <JOURNALING/JOURNALING Mailbox (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::Release: releasing resource private data.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange Information Store <Exchange Information Store Instance (LDNMBX01)>: [EXRES] ExchangeClose: Close requested.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange Information Store <Exchange Information Store Instance (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::Release: Count=0.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange Information Store <Exchange Information Store Instance (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::Release: releasing resource private data.
    00001568.00000f00::2009/11/14-13:11:44.522 WARN  [RES] Microsoft Exchange System Attendant <Exchange System Attendant Instance (LDNMBX01)>: [EXRES] ExchangeClose: Close requested.
    00001568.00000f00::2009/11/14-13:11:44.538 WARN  [RES] Microsoft Exchange System Attendant <Exchange System Attendant Instance (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::Release: Count=0.
    00001568.00000f00::2009/11/14-13:11:44.538 WARN  [RES] Microsoft Exchange System Attendant <Exchange System Attendant Instance (LDNMBX01)>: [EXRES] EXCHANGE_RESOURCE::Release: releasing resource private data.
    00001568.00000f00::2009/11/14-13:11:44.538 INFO  [RES] Network Name <Network Name (LDNMBX01)>: Stopping credential update threads
    00001568.00000f00::2009/11/14-13:11:44.538 WARN  [RES] Network Name <Network Name (LDNMBX01)>: Trying to remove credentials for LocalSystem returned status C0000225, STATUS_NOT_FOUND is a non-critical failure for a remove operation
    00001568.00000f00::2009/11/14-13:11:44.538 INFO  [RES] Network Name <Network Name (LDNMBX01)>: ResID 48663264 closed.
    00001568.00000f00::2009/11/14-13:11:44.538 INFO  [RES] Network Name <Cluster Name>: Stopping credential update threads
    00001568.00000f00::2009/11/14-13:11:44.538 WARN  [RES] Network Name <Cluster Name>: Trying to remove credentials for LocalSystem returned status C0000225, STATUS_NOT_FOUND is a non-critical failure for a remove operation
    00001568.00000f00::2009/11/14-13:11:44.538 INFO  [RES] Network Name <Cluster Name>: ResID 2035456 closed.
    00001568.00000f00::2009/11/14-13:11:44.538 INFO  [RHS] Exiting.
     
    • Edited byAl Man Saturday, November 14, 2009 2:07 PM
    • Edited byAl Man Saturday, November 14, 2009 2:07 PM
    • Edited byAl Man Saturday, November 14, 2009 2:01 PM
    •  
     
  • Saturday, November 14, 2009 2:01 PMAl Man Users MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote
    After reading through the logs more, I suspect it could be something to do with the LocalSystem account.  Should the Cluster Service be using this account - or should it be a dedicated account?
     
  • Saturday, November 14, 2009 2:32 PMRajnish R Sharma Users MedalsUsers MedalsUsers MedalsUsers MedalsUsers Medals
     
    Sign In to Vote
    0
    Sign In to Vote
    Looking at the local system account errors , have a look into this post http://social.technet.microsoft.com/Forums/en/winserverClustering/thread/00d052cb-2c3b-4059-a26c-818748db14b7
    Raj