locked
DAG Cluster Events 1069 and 1558

    Question

  • We notice lot of cluster events 1069 and 1558 every 15 minutes for one of our DAG.  

     

    1069

    Cluster resource 'File Share Witness (\\CASHT1.internal.org\DAG1.internal.org)' in clustered service or application 'Cluster Group' failed.

     

    1558

    The cluster service detected a problem with the witness resource. The witness resource will be failed over to another node within the cluster in an attempt to reestablish access to cluster configuration data.

     

    I changed the FSW to a non-Exchange server and still see the same error pointing to problem on CASHT1.  The FSW witness file seems to be updated by the cluster as the last modified timestamp is current.  Anyone know what’s the problem?  Is it a false alarm?

     

    thanks

    Thursday, December 01, 2011 9:36 PM

All replies

  • Hi
       1. Please install this kb and test it again
      
    http://support.microsoft.com/?id=2194664
       2. Restart of clussvc on all  nodes
       3.It is known issue.
           We are seeing this issue on Windows Server 2008 and R2 Clusters. As a workaround to the issue, we changed the Cluster Quorum model from 'Node and Disk Majority' to 'Node and File Share Witness' and back.


    Terence Yu

    TechNet Community Support

    Friday, December 02, 2011 2:25 AM
    Moderator
  • We are running W2k8 R2 RTM, does SP1 include this hotfix?  The download link actually has SP1 as release number.

      Windows 7/WindowsServer 2008 R2 All (Global) x86 sp1 Fix326314

     

    Do I need the hotfix or SP on just the FSW or all the DAG members?   When I see 1069 and 1558 errors, the FSW folder actualy still being updated constantly.  Is that mean the FSW is still working and count as active cluster member in a Quorum? 

     

    Friday, December 02, 2011 4:16 AM
  • Hi
       1. please download it. It is not in Sp1
       2. On all member of DAG
       3. Yes FSW is still working.
       If you don't care for it , you can ignore it.

    Terence Yu

    TechNet Community Support

    Friday, December 02, 2011 4:21 AM
    Moderator
  • Is this a stretched cluster, spanning 2 or more AD sites?


    Jesper Bernle | Microsoft Community Contributor 2011 Awardee
    Saturday, December 03, 2011 1:49 PM
  • Yes, it is a stretched cluster on stretched VALN in two different physical locations but only one AD sites.  The two locations are connected by 1 or more gigibit WAN link.
    Saturday, December 03, 2011 10:48 PM
  • Perhaps you have some network issues (latency) that causes this behaviour. Try decreasing the sensitivity of the DAG by following: Configure Heartbeat and DNS Settings in a Multi-Site Failover Cluster

    Since your DAG members are on the same subnet, you'd be interested in the following:

    cluster /prop SameSubnetDelay=2000:DWORD

    cluster /prop SameSubnetThreshold=10:DWORD


    Jesper Bernle | Microsoft Community Contributor 2011 Awardee
    Sunday, December 04, 2011 1:15 PM
  • Yes, it is a stretched cluster on stretched VALN in two different physical locations but only one AD sites.  The two locations are connected by 1 or more gigibit WAN link.


    http://blogs.technet.com/b/exchange/archive/2011/11/20/recommended-windows-hotfix-for-database-availability-groups-running-windows-server-2008-r2.aspx

    Recommended Windows Hotfix for Database Availability Groups running Windows Server 2008 R2

     

     

     

    Sunday, December 04, 2011 1:35 PM
  • Fix372642 file says it is SP2 fix for Windows 2008 R2 SP1. We are running Windows 2008 R2, do we need the same fix? Fix372642 and Fix326314 are two different issues. Should I apply both? If yes, can we install them at the same time?
    Yes, it is a stretched cluster on stretched VALN in two different physical locations but only one AD sites.  The two locations are connected by 1 or more gigibit WAN link.


    http://blogs.technet.com/b/exchange/archive/2011/11/20/recommended-windows-hotfix-for-database-availability-groups-running-windows-server-2008-r2.aspx

    Recommended Windows Hotfix for Database Availability Groups running Windows Server 2008 R2

     

     

     

    Tuesday, December 06, 2011 5:52 PM
  • I'd update my servers so they are up to date. Then I'd install the http://support.microsoft.com/kb/2550886 and test it again. If it still won't work, go for changing the hearbeat delay settings I mentioned above.


    Jesper Bernle | Microsoft Community Contributor 2011 Awardee
    Tuesday, December 06, 2011 7:58 PM
  • Hi
       Do you have anything to update your issue ?

    Terence Yu

    TechNet Community Support

    Thursday, December 08, 2011 9:43 AM
    Moderator
  • Will Fix372642 and Fix326314 work on W2k8 R2 RTM?  

    Thursday, December 08, 2011 8:58 PM
  • Hi
      
    http://support.microsoft.com/kb/2550886
     
    Prerequisites

    <script type="text/javascript">// <![CDATA[ loadTOCNode(3, 'resolution'); ]]></script> To apply this hotfix, you must be running Windows Server 2008 R2 or Windows Server 2008 R2 Service Pack 1 (SP1).
      So it works on RTM.
        Can you post fix 326314 KB?

    Terence Yu

    TechNet Community Support

    Friday, December 09, 2011 1:46 AM
    Moderator
  • Thanks,

    http://support.microsoft.com/hotfix/KBHotfix.aspx?kbnum=2194664&kbln=en-us

     

    Will Microsoft release this updates to WSUS?  Especially for 372642?   Thanks

    Friday, December 09, 2011 5:51 AM
  • Hi
      
    http://support.microsoft.com/kb/2194664
      Prerequisites

      To apply this hotfix, you must be running Windows 7 or Windows Server 2008 R2.
      It should work on RTM.
         I don’t know whether WSUS will download this fix.  You had better receive the hot fix and install it on exchange server.


    Terence Yu

    TechNet Community Support


    Friday, December 09, 2011 6:07 AM
    Moderator
  • Hi
       Do you have anything to update your issue?

    Terence Yu

    TechNet Community Support

    Wednesday, December 14, 2011 5:46 AM
    Moderator
  • We notice lot of cluster events 1069 and 1558 every 15 minutes for our Biztalk servers .Cluster /logs :

    000008e4.00000a78::2014/08/04-15:55:54.478 INFO  [GUM] Node 2: Processing RequestLock 1:27680
    000008e4.00000a78::2014/08/04-15:55:54.478 INFO  [GUM] Node 2: Processing GrantLock to 1 (sent by 2 gumid: 185903)
    000008e4.00000794::2014/08/04-15:56:16.688 INFO  [GUM] Node 2: Processing RequestLock 2:43009
    000008e4.00000a78::2014/08/04-15:56:16.691 INFO  [GUM] Node 2: Processing GrantLock to 2 (sent by 1 gumid: 185904)
    000008e4.00000a78::2014/08/04-15:56:54.728 INFO  [GUM] Node 2: Processing RequestLock 1:27681
    000008e4.00000a78::2014/08/04-15:56:54.728 INFO  [GUM] Node 2: Processing GrantLock to 1 (sent by 2 gumid: 185906)
    000008e4.00000794::2014/08/04-15:57:16.480 INFO  [GUM] Node 2: Processing RequestLock 2:43011
    000008e4.00000a78::2014/08/04-15:57:16.483 INFO  [GUM] Node 2: Processing GrantLock to 2 (sent by 1 gumid: 185907)
    000008e4.00000a78::2014/08/04-15:57:54.738 INFO  [GUM] Node 2: Processing RequestLock 1:27682
    000008e4.00000a78::2014/08/04-15:57:54.738 INFO  [GUM] Node 2: Processing GrantLock to 1 (sent by 2 gumid: 185909)
    000008e4.00000794::2014/08/04-15:58:16.890 INFO  [GUM] Node 2: Processing RequestLock 2:43013
    000008e4.00000a78::2014/08/04-15:58:16.893 INFO  [GUM] Node 2: Processing GrantLock to 2 (sent by 1 gumid: 185910)
    000008e4.00001d04::2014/08/04-15:58:54.585 INFO  [NM] Received request from client address 10.151.97.135.
    000008e4.00000a78::2014/08/04-15:58:54.653 INFO  [GUM] Node 2: Processing RequestLock 1:27683
    000008e4.00000a78::2014/08/04-15:58:54.653 INFO  [GUM] Node 2: Processing GrantLock to 1 (sent by 2 gumid: 185912)
    000008e4.00001f74::2014/08/04-15:58:54.682 INFO  [NM] Received request from client address 10.151.97.135.
    000008e4.00001b60::2014/08/04-15:58:54.852 INFO  [NM] Received request from client address 10.151.97.135.
    00000d3c.000012d4::2014/08/04-15:58:55.335 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    000008e4.00001b60::2014/08/04-15:58:55.335 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    000008e4.00001b60::2014/08/04-15:58:55.335 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
    00000d3c.00000410::2014/08/04-15:58:55.341 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    000008e4.00000dd4::2014/08/04-15:58:55.342 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000d3c.000012d4::2014/08/04-15:58:55.347 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    000008e4.000012a0::2014/08/04-15:58:55.348 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000d3c.000012d4::2014/08/04-15:58:55.381 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    000008e4.00001b60::2014/08/04-15:58:55.381 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    000008e4.00001b60::2014/08/04-15:58:55.381 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
    00000d3c.000012d4::2014/08/04-15:58:55.387 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    000008e4.00000dd4::2014/08/04-15:58:55.387 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000d3c.00000410::2014/08/04-15:58:55.393 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    000008e4.00000dd4::2014/08/04-15:58:55.393 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    000008e4.00001368::2014/08/04-15:58:55.417 INFO  [API] s_ApiGetQuorumResource final status 0.
    00000d3c.00000410::2014/08/04-15:58:56.019 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    000008e4.00001368::2014/08/04-15:58:56.019 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    000008e4.00001368::2014/08/04-15:58:56.019 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
    00000d3c.00000410::2014/08/04-15:58:56.025 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    000008e4.00001b60::2014/08/04-15:58:56.025 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000d3c.00000410::2014/08/04-15:58:56.032 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    000008e4.00001368::2014/08/04-15:58:56.032 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    000008e4.00001368::2014/08/04-15:58:56.032 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
    00000d3c.00000410::2014/08/04-15:58:56.035 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    000008e4.00001b60::2014/08/04-15:58:56.035 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000d3c.00000410::2014/08/04-15:58:56.068 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    000008e4.00001368::2014/08/04-15:58:56.068 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000d3c.00000410::2014/08/04-15:58:56.079 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    000008e4.00001b60::2014/08/04-15:58:56.079 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000d3c.000012d4::2014/08/04-15:59:03.673 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    000008e4.00001368::2014/08/04-15:59:03.673 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    000008e4.00001368::2014/08/04-15:59:03.673 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
    00000d3c.00000410::2014/08/04-15:59:03.680 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    000008e4.00001f74::2014/08/04-15:59:03.680 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000d3c.00000410::2014/08/04-15:59:03.693 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    000008e4.00001368::2014/08/04-15:59:03.693 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    000008e4.00001368::2014/08/04-15:59:03.693 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
    00000d3c.000012d4::2014/08/04-15:59:03.699 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    000008e4.00001f74::2014/08/04-15:59:03.699 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    000008e4.00000794::2014/08/04-15:59:16.743 INFO  [GUM] Node 2: Processing RequestLock 2:43015
    000008e4.00000a78::2014/08/04-15:59:16.747 INFO  [GUM] Node 2: Processing GrantLock to 2 (sent by 1 gumid: 185913)
    00000d3c.00000410::2014/08/04-15:59:52.659 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    000008e4.00001b60::2014/08/04-15:59:52.659 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    000008e4.00001b60::2014/08/04-15:59:52.659 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
    00000d3c.00000410::2014/08/04-15:59:52.663 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    000008e4.00001368::2014/08/04-15:59:52.663 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000d3c.000012d4::2014/08/04-15:59:52.676 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    000008e4.00001368::2014/08/04-15:59:52.676 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    000008e4.00001368::2014/08/04-15:59:52.676 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
    00000d3c.00000410::2014/08/04-15:59:52.676 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    000008e4.00001b60::2014/08/04-15:59:52.676 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    000008e4.00001b60::2014/08/04-15:59:52.676 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
    00000d3c.000012d4::2014/08/04-15:59:52.683 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    000008e4.00000df8::2014/08/04-15:59:52.683 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000d3c.00000410::2014/08/04-15:59:52.683 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    000008e4.00001f74::2014/08/04-15:59:52.683 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000d3c.00000410::2014/08/04-15:59:52.696 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    000008e4.00001b60::2014/08/04-15:59:52.696 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    000008e4.00001b60::2014/08/04-15:59:52.696 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
    00000d3c.00000410::2014/08/04-15:59:52.702 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    000008e4.00001368::2014/08/04-15:59:52.702 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    00000d3c.00000410::2014/08/04-15:59:52.809 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    000008e4.00001f74::2014/08/04-15:59:52.809 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    000008e4.00001f74::2014/08/04-15:59:52.809 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQ's DLL is not present on this node.  Attempting to find a good node...
    00000d3c.00000410::2014/08/04-15:59:52.814 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQ returned 21.'
    000008e4.00001b60::2014/08/04-15:59:52.814 WARN  [RCM] Failed to load restype 'MSMQ': error 21.
    00000d3c.00000410::2014/08/04-15:59:52.826 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    000008e4.00000df8::2014/08/04-15:59:52.827 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    000008e4.00000df8::2014/08/04-15:59:52.827 WARN  [RCM] rcm::RcmApi::ResTypeControl: ResType MSMQTriggers's DLL is not present on this node.  Attempting to find a good node...
    00000d3c.00000410::2014/08/04-15:59:52.832 ERR   [RHS] s_RhsRpcCreateResType: ERROR_NOT_READY(21)' because of 'Startup routine for ResType MSMQTriggers returned 21.'
    000008e4.00001f74::2014/08/04-15:59:52.832 WARN  [RCM] Failed to load restype 'MSMQTriggers': error 21.
    000008e4.00000a78::2014/08/04-15:59:54.761 INFO  [GUM] Node 2: Processing RequestLock 1:27684
    000008e4.00000a78::2014/08/04-15:59:54.761 INFO  [GUM] Node 2: Processing GrantLock to 1 (sent by 2 gumid: 185915)
    000008e4.00000dd4::2014/08/04-16:00:01.591 INFO  [NM] Received request from client address 10.151.97.136.
    000008e4.00000dd4::2014/08/04-16:00:01.604 WARN  [API] s_ApiOpenResourceEx: Resource BIZTALK2UAT not found, status = 5007
    000008e4.00001368::2014/08/04-16:00:03.261 WARN  [API] s_ApiOpenResourceEx: Resource BIZTALK2UAT not found, status = 5007
    000008e4.00000794::2014/08/04-16:00:16.813 INFO  [GUM] Node 2: Processing RequestLock 2:43017
    000008e4.00000a78::2014/08/04-16:00:16.816 INFO  [GUM] Node 2: Processing GrantLock to 2 (sent by 1 gumid: 185916)
    000008e4.00000a78::2014/08/04-16:00:55.094 INFO  [GUM] Node 2: Processing RequestLock 1:27685
    000008e4.00000a78::2014/08/04-16:00:55.094 INFO  [GUM] Node 2: Processing GrantLock to 1 (sent by 2 gumid: 185918)
    000008e4.00000794::2014/08/04-16:01:17.024 INFO  [GUM] Node 2: Processing RequestLock 2:43019
    000008e4.00000a78::2014/08/04-16:01:17.026 INFO  [GUM] Node 2: Processing GrantLock to 2 (sent by 1 gumid: 185919)
    000008e4.00000a78::2014/08/04-16:01:54.950 INFO  [GUM] Node 2: Processing RequestLock 1:27686
    000008e4.00000a78::2014/08/04-16:01:54.950 INFO  [GUM] Node 2: Processing GrantLock to 1 (sent by 2 gumid: 185921)
    000008e4.00000794::2014/08/04-16:02:17.164 INFO  [GUM] Node 2: Processing RequestLock 2:43021
    000008e4.00000a78::2014/08/04-16:02:17.166 INFO  [GUM] Node 2: Processing GrantLock to 2 (sent by 1 gumid: 185922)
    000008e4.00000a78::2014/08/04-16:02:54.954 INFO  [GUM] Node 2: Processing RequestLock 1:27687
    000008e4.00000a78::2014/08/04-16:02:54.954 INFO  [GUM] Node 2: Processing GrantLock to 1 (sent by 2 gumid: 185924)
    000008e4.00000794::2014/08/04-16:03:17.220 INFO  [GUM] Node 2: Processing RequestLock 2:43023
    000008e4.00000a78::2014/08/04-16:03:17.223 INFO  [GUM] Node 2: Processing GrantLock to 2 (sent by 1 gumid: 185925)

    Monday, August 11, 2014 4:49 PM