none
5121 during DPM 2010 backup RRS feed

  • Question

  • I'm using DPM 2010 to backup our CSVs which reside on a HP P2000 G3 FC/iSCSI. CSVs on the Hyper-V cluster are accessed by FC only. iSCSI is used to connect the DPM server to the storage.

    Log Name:      System
    Source:        Microsoft-Windows-FailoverClustering
    Date:          17.4.2011 12:39:47
    Event ID:      5121
    Task Category: Cluster Shared Volume
    Level:         Error
    Keywords:     
    User:          SYSTEM
    Computer:      server2.domain.local
    Description:
    Cluster Shared Volume 'Volume1' ('CSV10k') is no longer directly accessible from this cluster node. I/O access will be redirected to the storage device over the network through the node that owns the volume. This may result in degraded performance. If redirected access is turned on for this volume, please turn it off. If redirected access is turned off, please troubleshoot this node's connectivity to the storage device and I/O will resume to a healthy state once connectivity to the storage device is reestablished.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Microsoft-Windows-FailoverClustering" Guid="{BAF908EA-3421-4CA9-9B84-6689B8C6F85F}" />
        <EventID>5121</EventID>
        <Version>0</Version>
        <Level>2</Level>
        <Task>38</Task>
        <Opcode>0</Opcode>
        <Keywords>0x8000000000000000</Keywords>
        <TimeCreated SystemTime="2011-04-17T10:39:47.521872700Z" />
        <EventRecordID>1197950</EventRecordID>
        <Correlation />
        <Execution ProcessID="2136" ThreadID="4360" />
        <Channel>System</Channel>
        <Computer>server2.domain.local</Computer>
        <Security UserID="S-1-5-18" />
      </System>
      <EventData>
        <Data Name="VolumeName">Volume1</Data>
        <Data Name="ResourceName">CSV10k</Data>
      </EventData>
    </Event>
     
     
    This happens during backup using Microsoft's Data Protection Manager 2010, which utilizes HP's VSS Hardware provider:
     
    C:\Users\administrator>vssadmin list providers
    vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
    (C) Copyright 2001-2005 Microsoft Corp.
     
    Provider name: 'Microsoft Software Shadow Copy provider 1.0'
       Provider type: System
       Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
       Version: 1.0.0.7
     
    Provider name: 'HP StorageWorks P2000/MSA2000 VSS Provider'
       Provider type: Hardware
       Provider Id: {bd04cbf9-212c-4553-9ea5-c5bfb05ccc8f}
       Version: 2.7.0.0
     

    I'm in contact with HP and was told the storage (all latest firmware) has full support for CSV. Why am I even getting this error and is it something I should worry about. What additional steps can I take to resolve this. Is additional information, logs needed. What can I test, ask HP...

     

     

    Wednesday, April 20, 2011 6:31 AM

Answers

All replies

  • Hi CypherMike,

    no worries, normal operation.

    you can find some great info here:

    http://technet.microsoft.com/en-us/library/ff634189.aspx

    kind regards,

    ALex Smits

    Wednesday, April 20, 2011 8:46 AM
  • Thank you for your reply and reasurance. It appears this is exactly it:

    To reduce the impact on VM1, we recommend that you use hardware snapshots, which enable the CSV to resume direct I/O mode as soon as the hardware snapshot has been created. The duration of this process is typically very short, about two minutes.

    Should there be an event log entry after resuming direct I/O, so I can double check everything?

    Wednesday, April 20, 2011 10:23 AM
  • Hi, I also get this event as well as event ID 5120 and it sometimes causes my VMs to stop during backups.

    I have a call open with MS regarding this issue. Apparantly 5121 is normal and can be ignored (then why it is logged as an error I will never know) but Ms have promised to provide a KB that explains further. I will post further details when I get the KB.

    How many protection groups do you have for VMs and how many VMs are in each group?


    Microsoft Partner
    Wednesday, April 20, 2011 4:24 PM
  • My backup started an hour ago and I received 5121 and quite a few of my VMs were Off (same as with you it appears).

    I first get - I'll only post the starting ones and the errors. BTW should I even see this one, if hardware VSS providers are used:

     

    Log Name: System
    Source: Service Control Manager
    Date:  20.4.2011 18:00:06
    Event ID: 7036
    Task Category: None
    Level:  Information
    Keywords: Classic
    User:  N/A
    Computer: server1.domain.local
    Description:
    The Volume Shadow Copy service entered the running state.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
     <System>
     <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" />
     <EventID Qualifiers="16384">7036</EventID>
     <Version>0</Version>
     <Level>4</Level>
     <Task>0</Task>
     <Opcode>0</Opcode>
     <Keywords>0x8080000000000000</Keywords>
     <TimeCreated SystemTime="2011-04-20T16:00:06.301703300Z" />
     <EventRecordID>957599</EventRecordID>
     <Correlation />
     <Execution ProcessID="608" ThreadID="6460" />
     <Channel>System</Channel>
     <Computer>server1.domain.local</Computer>
     <Security />
     </System>
     <EventData>
     <Data Name="param1">Volume Shadow Copy</Data>
     <Data Name="param2">running</Data>
     <Binary>5600530053002F0034000000</Binary>
     </EventData>
    </Event>
    

     

    Then 5121, as posted in my first post.

    Then:

    Log Name: System
    Source: Service Control Manager
    Date:  20.4.2011 18:03:55
    Event ID: 7036
    Task Category: None
    Level:  Information
    Keywords: Classic
    User:  N/A
    Computer: server1.domain.local
    Description:
    The HP StorageWorks VSS Hardware Provider service entered the running state.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
     <System>
     <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" />
     <EventID Qualifiers="16384">7036</EventID>
     <Version>0</Version>
     <Level>4</Level>
     <Task>0</Task>
     <Opcode>0</Opcode>
     <Keywords>0x8080000000000000</Keywords>
     <TimeCreated SystemTime="2011-04-20T16:03:55.708045300Z" />
     <EventRecordID>957602</EventRecordID>
     <Correlation />
     <Execution ProcessID="608" ThreadID="876" />
     <Channel>System</Channel>
     <Computer>server1.local</Computer>
     <Security />
     </System>
     <EventData>
     <Data Name="param1">HP StorageWorks VSS Hardware Provider</Data>
     <Data Name="param2">running</Data>
     <Binary>480050002000530074006F00720061006700650057006F0072006B00730020005600530053002000480061007200640077006100720065002000500072006F00760069006400650072002F0034000000</Binary>
     </EventData>
    </Event>
    
    Then:

    Log Name: System
    Source: volsnap
    Date:  20.4.2011 18:04:12
    Event ID: 8
    Task Category: None
    Level:  Error
    Keywords: Classic
    User:  N/A
    Computer: server1.domain.local
    Description:
    The flush and hold writes operation on volume \\?\Volume{2def8b40-caf4-4acd-8441-d276d5544ec4} timed out while waiting for a release writes command.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
     <System>
     <Provider Name="volsnap" />
     <EventID Qualifiers="49158">8</EventID>
     <Level>2</Level>
     <Task>0</Task>
     <Keywords>0x80000000000000</Keywords>
     <TimeCreated SystemTime="2011-04-20T16:04:12.513647300Z" />
     <EventRecordID>957603</EventRecordID>
     <Channel>System</Channel>
     <Computer>server1.domain.local</Computer>
     <Security />
     </System>
     <EventData>
     <Data>
     </Data>
     <Data>\\?\Volume{2def8b40-caf4-4acd-8441-d276d5544ec4}</Data>
     <Binary>000000000200300000000000080006C0000000000000000022000000000000000000000000000000</Binary>
     </EventData>
    </Event>
    
    Then:

    Log Name: System
    Source: volsnap
    Date:  20.4.2011 18:04:12
    Event ID: 8
    Task Category: None
    Level:  Error
    Keywords: Classic
    User:  N/A
    Computer: server1.domain.local
    Description:
    The flush and hold writes operation on volume C: timed out while waiting for a release writes command.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
     <System>
     <Provider Name="volsnap" />
     <EventID Qualifiers="49158">8</EventID>
     <Level>2</Level>
     <Task>0</Task>
     <Keywords>0x80000000000000</Keywords>
     <TimeCreated SystemTime="2011-04-20T16:04:12.513647300Z" />
     <EventRecordID>957604</EventRecordID>
     <Channel>System</Channel>
     <Computer>server1.domain.local</Computer>
     <Security />
     </System>
     <EventData>
     <Data>
     </Data>
     <Data>C:</Data>
     <Binary>000000000200300000000000080006C0000000000000000023000000000000000000000000000000</Binary>
     </EventData>
    </Event>
    

    I can provide the full event log from the time backup starts if needed.

    Looking at the VMs Event Logs there's not much to see, but there are obviously no entries from the time backup started to the time I manually turned them on.

     

    I have one PG and there are 12 VMs (all the servers on this two node cluster) in the group.

    I'd appreciate any and all assistance with what's becoming a major issue.



    Wednesday, April 20, 2011 5:19 PM
  • Hello Mike,

    Can you tell me how did you managed to get DPM 2010 backups working with MSA 2000 G3? I have the following scenario at one of my customers:

    Hyper-V cluster with 4 nodes with 20 VM`s, MSA 2000 G3 FC/iSCSI storage and DPM 2010 on a phisical server. All my nodes are connected via a separate iscsi vlan with jumbo frames, also my DPM 2010 is connected via iSCSI with the storage. I installed capi->VDS-VSS from hp.com website on all hypervisors, i created a vdisk->volume->snap pool for VM backups on the MSA 2000 and mapped it directly to DPM.

    After i create protection group for any VM in the cluster it enters OK state after 1 min ( considering that the smallest machine has 30 GB it`s impossible to create a replica under a minute, BTW i dont use 10 GB network) and if i do consistency check it fails with ID 30115 ( http://social.technet.microsoft.com/Forums/en-US/dpmhypervbackup/thread/a2ed4011-ce46-465c-bcb6-ecf2f34ff43b/ )

    I checked the storage and it`s licensed for 64 snapshots with default license.

    Please help me with this matter.

    Thank you.

    Thursday, June 30, 2011 9:43 AM
  • Hi traianuss,

    not sure I'll be able to assist you, but let's give it a go. If needed let me know where additional information can be exchanged (e-mail?), I'll gladly provide screenshots, configuration details if that might help.

    A snap pool that is created must be owned by the same controller as are the vdisks/volumes you're trying to backup. I didn't map it to DPM, since the purpose of the snap pool is so that the MSA has a space to create hardware snapshoot of the VMs on it. DPM will create the snapshots and delete them afterwards. Make sure that the volumes you're trying to snapshot are master not standard volumes or it won't work (it might be that DPM converts them but I'm not 100% sure).

    HP MSA2000 CLI guide if you need help with conversions http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c02520779/c02520779.pdf

    Do you see the snapshots being created in the web interface when DPM starts backing up?

    Thursday, June 30, 2011 3:11 PM
  • Thank you very much Mike, you enlightened me. I solved my problem, DPM is working flawlessly now.

    Traian

    Monday, July 4, 2011 9:39 PM
  • Hi, I have a same problem. What information from Mike do you help?

     

    jirka

    Friday, August 26, 2011 7:19 AM
  • Hi Mike,

     

    I have a few questions for you. This is also in reference to your screen shot from the question you posed yesterday.

     

    1) Do you need a SNAP POOL per vdisk or per controller?

    2) Can you confirm the volume type is master or standard? Currently mine are standard.

     

    Many thanks

    Thursday, September 1, 2011 8:35 AM
  • Hi Jarradbrown,

    they need to be master volumes in order for you to assign them to a snap-pool: "Converts a standard volume to a master volume; that is, it enables the volume for snapshots and associates it with an existing snap pool. The standard volume and the snap pool must be owned by the same controller, though they can be in different vdisks."

    Take a look at the HP MSA2000 CLI for additional information related to the show-volumesconvert std-to-master and snap-pool related commands. 

    Thursday, September 1, 2011 9:37 AM
  • Mike,

    Question. a CSV can have only one current owner, say node1 owns csv1. vm1 is on csv1, but vm1's current owner is node2.

    Is this the scenario when you get the 5121 event and is it being thrown by the node2?

    I can duplicate a 5121 in the above scenario. I don’t throw a 5121 if the vm’s and the csv’s have the same current node owner.

    I too am using HP VSS hardware providers with my EVA6100. The EVA's firmware is v.6.24, the latest.


    BChaffin
    Friday, December 23, 2011 3:29 PM
  • BChaffin, sorry for the late reply.

    What you describe doesn't seem to be the case here. All CSV's are owned by node1 and all VMs are on node1, I still receive quite a few 5121 events each day during DPM backup. The errors are being thrown by node2.

    Wednesday, February 22, 2012 2:06 PM