locked
Windows Server 2008 Cluster Disk cannot bring online - System error 5018 RRS feed

 > 

  • Question

  • Question
    Sign in to vote
    0
    Sign in to vote
    I´ve created a Cluster Active/Passive over Windows Server 2008 Enterprise Edition x64 with two nodes . The specifications are:
    Servers: HP Blade ProLiant BL480c G1.

    HBAs:   QLogic QMH2462 4Gb FC HBA for HP c-Class BladeSystem

    Storage: NAS EMC win Powerpath 5.2 as my MPIO

    After i create the new cluster the disk attached are detected but not bring online with the message "The cluster resource could not be brought online by the resource monitor" and if a try to do it by the power shell with cluster res "Cluster Disk 2" /on it doesn´t work too, cause display the same message:

    Bringing resource 'Cluster Disk 2' online...

    System error 5018 has occurred (0x0000139a).
    The cluster resource could not be brought online by the resource monitor.


    Principaly the Cluster Log Says:
    00000e2c.00000c5c::2008/11/18-22:09:17.059 ERR   [RES] Physical Disk <Cluster Disk 1>: OnlineThread: Error 1 bringing resource online.
    00000e2c.00000c5c::2008/11/18-22:09:17.059 ERR   [RHS] Online for resource Cluster Disk 1 failed.
    00000d90.00000c10::2008/11/18-22:09:17.059 INFO  [RCM] HandleMonitorReply: ONLINERESOURCE for 'Cluster Disk 1', gen(7) result 5018.
    .... and the same for all the other disk attached.


    Disk <Cluster Disk 2>: Arbitrate request: FastPath 0
    00000834.00000a00::2008/11/14-21:20:47.968 ERR   [RES] Physical Disk <Cluster Disk 2>: Failed to register key, status 1
    00000834.00000a00::2008/11/14-21:20:52.086 ERR   [RES] Physical Disk <Cluster Disk 2>: OnlineThread: Unable to arbitrate for the disk. Error: 1.
    00000834.00000a00::2008/11/14-21:20:52.086 ERR   [RES] Physical Disk <Cluster Disk 2>: OnlineThread: Error 1 bringing resource online.
    00000834.00000a00::2008/11/14-21:20:52.086 ERR   [RHS] Online for resource Cluster Disk 2 failed.

    ---------------------------------------------------------------------------------------------------------------------------------
     I've tried a lot of conbinations with services offline on one node or powered off, etc.

    This nodes was on a Win2003 Cluster with the same storage, So i don´t know if this could be some driver, compatibiliti, MPIO upgrade needed, or some specific with the new version or process to build and handle resourses by Win2008.


    ¿Does anyone know what could be the cause or some configuration that could help?

    Until discover the specific problem or cause i´ll do the following i i´ll notice you if that work


    - Ensure Microsoft STORPort hotfix Q943545 (ref: http://support.microsoft.com/kb/Q943545/en-us) is installed to both nodes.

    - Upgrade to the latest EMC qualified HBA firmware (Boot code) 1.64 (BIOS v1.26)

    - Upgrade HBA driver to the latest EMC qualified ver 9.1.7.16

    - Flash HBA parameters NVRAM setting to correct HBA parameters to EMC approved settings.

    (Ref: page 37 on http://filedownloads.qlogic.com/files/Driver/67902/QLogic_Windows_04_23_08.pdf.  
    Also see page 124 to 126 for a procedure 'Updating the flash BIOS'... NVRAM settings for the HBAs can be flashed via SANsurfer by clicking Update NVRAM.)     




    Thanks and i hope any comment.
    H1R@M
    Wednesday, November 19, 2008 12:07 AM

Answers

  • Question
    Sign in to vote
    0
    Sign in to vote

     Thorsten Wujek & John Toner Thanks for all your guidance and helpful comments and im so sorry because i wrote some confused things. We´ve solved the issue in collaboration with EMC Support people. I'll describe the scenario, issue, cause and solution at following:

    Environment:
    OS:     Microsoft Windows Server 2008
    Application SW: Microsoft Cluster Server (MSCS)
    Product: EMC Symmetrix DMX-3

    Symptom 1
     Cluster Validation fails with error: Failed to Register PR key for cluster disk 0 from  node <nodename> status 1

    Symptom 2
     The persistent reserve test fails even though all of the cluster disks


    Symptom 1 Cause 
     The vcmdb is also visible to both hosts and the validation test chooses this disk to test  persistent reservations.  The vcmdb does not have SCSI-3 Persistent Reserve enabled by  default.

    Symptom 1 Fix       
     The vcmdb must be made invisible to disk management.  This can be done in one of the  following ways.

    • From a host perspective, this can be accomplished by right-clicking the disk in Device Manager and selecting disable. This can   also be done through Disk Manager by right-clicking on the VCM device, choose Properties > then select the Drivers tab > and click   Disable for this device.
    • From a Symmetrix perspective, this can be accomplished by unmapping the vcmdb from the FA port.  Note that this will affect all   hosts attached to this port.

    Symptom 2 Cause 

    •  For Windows 2008, you need the following director bits set: Common Serial Number (C), Enable Auto Negotiation (EAN), Enable Point-to-point (PP), Host SCSI Compliance 2007 (OS2007), SCSI-3 SPC-2  Compliance (SPC-2), Unique World Wide Name (UWN)
    •  For Windows 2008 failover clusters, you need the director bit SCSI-3 compliance (SC3),    and the "Persistent Reservation" (PER) bit must be  enabled on each Symmetrix DMX device used.

    Symptom 2 Fix

     How to set the SCSI3 Persistent reservation bit

     Host: Open Systems
      OS: Unix
      OS: Microsoft Windows

     Fact       SCSI3 persistent reservation it is also known as the PER bit
     Fact       Persistent Reservations - PER flag

     Fix  1. Create command (text) file with any text editor with syntax:

                    set device xxx   attribute=SCSI3_persist_reserv;     (xxx=Symmetrix device number)

            2. Run the "symconfigure" command with the "prepare" option. This will check the syntax of the command  file and verify whether the   Symmetrix is prepared to handle the requested configuration change. 

                   symconfigure -sid yyy -f commandfilename prepare   (yyy=symm serial number)

            3. Run the "symconfigure" command with the "commit" option to actually execute the change in the Symmetrix.

                   symconfigure -sid yyy -f commandfilename commit

    ------------------------------------------------------------------------------------------------------------------------------------------------------

    I really thank you very much for your support and disposal.

    Greetings!!
     

    H1R@M
    • Marked as answer by Hiram Celis Friday, November 21, 2008 6:18 PM
    Friday, November 21, 2008 6:17 PM

All replies

  • Question
    Sign in to vote
    1
    Sign in to vote
    Hi,

    what does the validation test tell you ??

    ThorstenWujek
    Wednesday, November 19, 2008 11:13 AM
  • Question
    Sign in to vote
    0
    Sign in to vote
    I agree with Thorston...make sure that you run thru the Validation tests and do not ignore the results.

    Just because the storage worked in Windows 2003 does not guarantee that it will work in a 2008 cluster. There are some vendor specific settings that might be required by the storage vendor in order to support persistent reservations on the array. I know on EMC Symmetrix storage, the devices need to have the PER flag set, and the FA ports need the SPC2 and OS2007 flags.

    Hope this helps.
    Visit my blog about multi-site clustering - http://msmvps.com/blogs/jtoner
    • Proposed as answer by JohnToner Wednesday, November 19, 2008 4:21 PM
    • Unproposed as answer by Hiram Celis Wednesday, November 19, 2008 11:54 PM
    Wednesday, November 19, 2008 4:21 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    On the Failover Cluster Validation Report, displays an error over "List Potential Cluster Disks" area. The specific message is:


    "Checking that multi-path I/O (MPIO) works for disk with identifier c70988a3 on node <NodeFQDN>.

    Multi-path I/O (MPIO) does not work for disk with identifier c70988a3 on node <NodeFQDN>
    Multi-path I/O (MPIO) software is not installed or not working propertly on node <NodeFQDN>"


    I understand that this indicates a MPIO software issue (in this case PowerPath) so Im trying to find some requierements to ask to EMC support people, because they says that EMC NS20 with PowerPath 5.2 works with Win2008 and they were checking the EMCReport to confirm.


    Thanks I´ll ask / check the following:

    • Storage Compatibility
    • Support of persistent reservations on the array
    • The config status about PRE falg and SPC2, OS2007 flags.



    I´ll share the awnsers or comments about it as soon as possible.

    Thanks again ThorstenWujek and JohnToner.

    H1R@M
    Wednesday, November 19, 2008 5:59 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    I´ve check this y then ru the wizard again:

    - Upgrade to the latest EMC qualified HBA firmware (Boot code) 1.64 (BIOS v1.26)

    - Upgrade HBA driver to the latest EMC qualified ver 9.1.7.16

    Both available at http://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/ResourceByOS.aspx?productid=1002&oemid=65&oemcatid=58604

    - flash HBA parameters NVRAM setting to correct HBA parameters to EMC approved settings.

    (Ref: page 37 on http://filedownloads.qlogic.com/files/Driver/67902/QLogic_Windows_04_23_08.pdf.  
    Also see page 124 to 126 for a procedure 'Updating the flash BIOS'... NVRAM settings for the HBAs can be flashed via SANsurfer by clicking Update NVRAM.)     



    I have 5 "Disk" attached; as i describe, the error display an issue with the disk with ID C70988A3, on the report too over the Disks description of all disks says "Disk partition style is MBR. Disk partition type is BASIC" except Disk 5 that display "Disk partition style is MBR. Disk partition type is BASIC. Disk is a redundant MPIO disk" I desactive it cause its the VCM that saves the Zone Masking Config and then the test continue but stop on "Validate SCSI-3 Persistent Reservation Test" with the following message

    "Failed to Register PR key for cluster disk 0
    Cluster Disk 0 does not support Persistent Reservation

    Failed to Register PR key for cluster disk 1
    Cluster Disk 1 does not support Persistent Reservation

    Failed to Register PR key for cluster disk 2
    Cluster Disk 2 does not support Persistent Reservation

    Failed to Register PR key for cluster disk 3
    Cluster Disk 3 does not support Persistent Reservation

    Failed to Register PR key for cluster disk 4
    Cluster Disk 4 does not support Persistent Reservation"

    So this is another specific issue
    H1R@M
    Wednesday, November 19, 2008 11:54 PM
  • Question
    Sign in to vote
    0
    Sign in to vote
    Hi,

    what kind of EMC are you using a DMC or a CLARION ?
     You have said it is a NAS. Normally with NAS devices you are using communication protocols like NFS or CIFS. These are no "block" protocols like SCSI transported by iSCSI or FC-SAN. Supported is only attached SAS (supporting SCSI 3 persistent reservation) or shared storage supporting the same. NAS devices are not supporting SCSI 3 persistent reservation.

    So from your validation test it seems that your storage is not supporting persistent reservation.
    The reasons could be:

    ++no block device
    ++actual firmware on storage is not supporting the OS

    I have used CLARION and DMC with Powerpath for Cluster 2008 and it is working fine.
    The same is with HP EVA but here you need the actual firmware and you must configure Windows LH OS support.

    Hope this helps

    Thorsten.
    ThorstenWujek
    Thursday, November 20, 2008 8:45 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

     Thorsten Wujek & John Toner Thanks for all your guidance and helpful comments and im so sorry because i wrote some confused things. We´ve solved the issue in collaboration with EMC Support people. I'll describe the scenario, issue, cause and solution at following:

    Environment:
    OS:     Microsoft Windows Server 2008
    Application SW: Microsoft Cluster Server (MSCS)
    Product: EMC Symmetrix DMX-3

    Symptom 1
     Cluster Validation fails with error: Failed to Register PR key for cluster disk 0 from  node <nodename> status 1

    Symptom 2
     The persistent reserve test fails even though all of the cluster disks


    Symptom 1 Cause 
     The vcmdb is also visible to both hosts and the validation test chooses this disk to test  persistent reservations.  The vcmdb does not have SCSI-3 Persistent Reserve enabled by  default.

    Symptom 1 Fix       
     The vcmdb must be made invisible to disk management.  This can be done in one of the  following ways.

    • From a host perspective, this can be accomplished by right-clicking the disk in Device Manager and selecting disable. This can   also be done through Disk Manager by right-clicking on the VCM device, choose Properties > then select the Drivers tab > and click   Disable for this device.
    • From a Symmetrix perspective, this can be accomplished by unmapping the vcmdb from the FA port.  Note that this will affect all   hosts attached to this port.

    Symptom 2 Cause 

    •  For Windows 2008, you need the following director bits set: Common Serial Number (C), Enable Auto Negotiation (EAN), Enable Point-to-point (PP), Host SCSI Compliance 2007 (OS2007), SCSI-3 SPC-2  Compliance (SPC-2), Unique World Wide Name (UWN)
    •  For Windows 2008 failover clusters, you need the director bit SCSI-3 compliance (SC3),    and the "Persistent Reservation" (PER) bit must be  enabled on each Symmetrix DMX device used.

    Symptom 2 Fix

     How to set the SCSI3 Persistent reservation bit

     Host: Open Systems
      OS: Unix
      OS: Microsoft Windows

     Fact       SCSI3 persistent reservation it is also known as the PER bit
     Fact       Persistent Reservations - PER flag

     Fix  1. Create command (text) file with any text editor with syntax:

                    set device xxx   attribute=SCSI3_persist_reserv;     (xxx=Symmetrix device number)

            2. Run the "symconfigure" command with the "prepare" option. This will check the syntax of the command  file and verify whether the   Symmetrix is prepared to handle the requested configuration change. 

                   symconfigure -sid yyy -f commandfilename prepare   (yyy=symm serial number)

            3. Run the "symconfigure" command with the "commit" option to actually execute the change in the Symmetrix.

                   symconfigure -sid yyy -f commandfilename commit

    ------------------------------------------------------------------------------------------------------------------------------------------------------

    I really thank you very much for your support and disposal.

    Greetings!!
     

    H1R@M
    • Marked as answer by Hiram Celis Friday, November 21, 2008 6:18 PM
    Friday, November 21, 2008 6:17 PM