none
Windows Server Datacenter 2012 Hyper-V CSV VSS issues

    Question

  • My Setup:

    2 Node Cluster, both running Windows Server Datacenter 2012

    1 SAN running Windows Server Standard 2012

    2 CSV hosted on the SAN.  SAN offers the CSVs via ISCSI target software

    2 Nodes connect via ISCSI Intiator software

    Backup system DPM 2012 SP1 Beta running a seperate physical server from the Nodes and SAN. 

    Backups are working, somewhat, however not very effecently, the "Microsoft iSCSI Target VSS Hardware Provider" on the SAN continues to give an error everytime a backup is run for any of the VMs hosted on our CSVs.  It is returning the follow errors:

    EVENT ID 13

    Volume Shadow Copy Service information: The COM Server with CLSID {463948d2-035d-4d1d-9bfc-473fece07dab} and name HWPRV cannot be started. [0x80070005, Access is denied.]

    Operation:

       Creating instance of hardware provider

       Obtain a callable interface for this provider

       List interfaces for all providers supporting this context

       Query Shadow Copies

    Context:

       Provider ID: {3f900f90-00e9-440e-873a-96ca5eb079e5}

       Provider ID: {3f900f90-00e9-440e-873a-96ca5eb079e5}

       Class ID: {463948d2-035d-4d1d-9bfc-473fece07dab}

       Snapshot Context: -1

       Snapshot Context: -1

    and also,

    EVENT ID 12292

    Volume Shadow Copy Service error: Error creating the Shadow Copy Provider COM class with CLSID {463948d2-035d-4d1d-9bfc-473fece07dab} [0x80070005, Access is denied.].

    Operation:

       Creating instance of hardware provider

       Obtain a callable interface for this provider

       List interfaces for all providers supporting this context

       Query Shadow Copies

    Context:

       Provider ID: {3f900f90-00e9-440e-873a-96ca5eb079e5}

       Provider ID: {3f900f90-00e9-440e-873a-96ca5eb079e5}

       Class ID: {463948d2-035d-4d1d-9bfc-473fece07dab}

       Snapshot Context: -1

       Snapshot Context: -1

      Execution Context: Coordinator

    One of the main reasons for using Windows Server 2012 as our SAN OS is the use of the snapshots being offloaded to the SAN and not the Nodes.  The error that I am seeing and the time it takes for DPM to complete the backups, makes me think this is not happening.  How can I fixe this error?  Is the iSCSI hardware provider of Server 2012 able to be used for CSVs?  I am not seeing any of the redirected access indications I had in the past with other configuraitons.       

    Sunday, September 30, 2012 9:27 PM

All replies

  • Protecting Windows Server 2012 is not supported yet. You need to wait for DPM 2012 SP1.

    http://technet.microsoft.com/en-US/library/hh758036.aspx

    Sunday, September 30, 2012 9:43 PM
  • Yes, very much understood.  That is why I'm running the SP1 Beta.  Since there is no help here, is there a DPM 2012 SP1 Beta forum?
    Monday, October 01, 2012 2:07 PM
  • Hi, I am getting the exact same error (I have a similar setup as well). I am also usind DPM SP1 Beta.

    Microsoft Partner

    Tuesday, October 02, 2012 1:10 PM
  • It makes me feel better to know I'm not alone here.  As I get time today, I'm going to continue to search for a beta forum of some sort for DPM 2012 SP1.  It seems like it should be very easy to find, but I'm having a bit of difficulty.  If I find the proper location I will reply here for you as well. 

    Tuesday, October 02, 2012 1:13 PM
  • So I am having to revert to using the Microsoft Software Shadow Copy Provider until we find out why the hardware provider isn't working... maybe it is possible that as it is still Beta the hardware provider is not working yet? Maybe someone from Microsoft could find out for us?

    Good luck with finding a forum for the Beta version, you may have better luck trying on the connect site for the Beta and logging a bug.


    Microsoft Partner

    Tuesday, October 02, 2012 1:25 PM
  • Just as an update, the cause of the error above was two-fold:

    1. I had installed the VSS Hardware Provider on my SAN and not my nodes without noticing it. I fixed that issue quickly, and easily.

    2. Once I had the VSS Hardware Provider installed on my nodes correctly, I was receiving the same from above only now on both my
    Hyper-V nodes. To fix this error I followed the security configuration in the bottom of this article (http://technet.microsoft.com/en-us/library/hh848268.aspx). I created a domain account and made it an admin of my SAN server, and changed
    the Identity of the "WTSnapsthoProvider" and set it to my new iSCSI domain service account I had created. I also set the "WtServers" registry string
    value to the FQDN of my SAN. This cleared the errors.

    However, I began to see that the snapshots were being created, but there were NTFS errors on them. In DPM 2012 SP1 Beta it would
    stall at 0 MB transferred, and never back up the VMs on the cluster. I believe this is the same thing that rEMOTE_eVENT was seeing in his post. My hunch is that these errors are related to the VSS Hardware Snapshot provider not being CSV 2.0 "aware" the same way the new software provider seems to be. Maybe the Storage Team or the System Center team will give us some information on this on their blogs in the future.

    In the end, I had to revert back to my original configuration which consisted of removing the VSS Hardware Provider and just using the "Microsoft CSV
    Shadow Copy Provider". This allows multiple VMs to be backed up by DPM 2012 SP1 Beta without any redirected I/O. However, the speed of the backups is not near what I would expect from a VSS Hardware Provider. I have been unable to find any documentation on VSS Hardware Providers supported for
    backups of CSV 2.0 volumes in the notes of the DPM 2012 SP1 Beta release. I'm hoping with the release of DPM 2012 SP1 RTM possibly we will have more information on this.

    On a small side note, I did notice that the DPM 2012 SP1 Beta states that all protected systems should have .NET 3.5 installed. This is not installed on my Hyper-V nodes, and does not seem to be an issue. I don't think it is the "magic key" to getting VSS Hardware Snapshots to work. Just a curious entry to look up later down the road.


    Saturday, October 06, 2012 2:17 AM
  • I have just found a new blog post describing how to configure the iSCSI VSS Provider for Server 2012... looks like it needs credentials to fix the errors we have been getting:

    http://blogs.technet.com/b/filecab/archive/2012/10/08/iscsi-target-storage-vds-vss-provider.aspx

    Let me know if this works for you!


    Microsoft Partner

    • Proposed as answer by rEMOTE_eVENT Tuesday, October 09, 2012 8:11 PM
    Tuesday, October 09, 2012 8:10 PM
  • Thank you rEMOTE_eVENT.  I will give this a try.  It is pretty close to what I tired before, but there are a few differences.  I do not see where they specifically say this will work with CSVs as well, but I will definitely give it a shot.  Thanks for posting, and I will reply with my results!
    Tuesday, October 09, 2012 8:42 PM
  • Good news, I have just tested the credential settings and this works :) !!! DPM is now using the Microsoft iSCSi Hardware provider without any errors!!!

    Volume Shadow Copy Service information: A database of hardware snapshots was created.

    Operation:

    Processing BeginPrepareSnapshot

    Add a Volume to a Shadow Copy Set

    Context:

    Provider Name: Microsoft iSCSI Target VSS Hardware Provider

    Provider Version: 6.2.9200.16384

    Provider ID: {3f900f90-00e9-440e-873a-96ca5eb079e5}

    Volume Name: \\?\Volume{8b8a2f54-1127-47b5-8c34-50792d775132}\

    Snapshot ID: {3b4ff28b-abaa-4f93-adb5-59fb6292b622}

    Snapshot Context: 4194304

    Execution Context: Coordinator


    Microsoft Partner


    WOW... backups are now fast... REALLY FAST :)
    • Edited by rEMOTE_eVENT Wednesday, October 10, 2012 10:51 AM
    Wednesday, October 10, 2012 10:33 AM
  • I just wanted to post a reply here on what happened with our tests.  Using the hardware snapshot provider for our CSV VM backups did not work.  Not only did it not work, the process seemed to not be CSV aware, and cause corruption in one of our iSCSI VHDs on the iSCSI target.  We ended up losing all VMs on that LUN and had to restore from backups.  rEMOTE_eVENT are you running a cluster or single server?  At this point we cannot prove whether there was corruption on the LUN before we tried the hardware snapshot provider, or if this process cause this error.  It is only an assumption, although a logical one from the errors we have seen, that the hardware snapshot provider is not CSV aware, and caused the issue.  Hopefully we will see more information on this as these technology become more commonly used. 
    Thursday, November 01, 2012 1:40 PM
  • Hi, I have a similar setup to you:

    2 node Hyper-V cluster with one CSV and ISCSI hardware providers installed.
    1 SAN running ISCSI target.
    1 DPM 2012 SP1 Beta server.
    All servers are running Server 2012 RTM

    So I have been testing the Microsoft VSS hardware provider and have had nothing but errors and CSV disks going offline with errors and backups of VMs constantly failing.
    I am reverting back to the SystemSoftwareProvider until I get a definitive answer from Microsoft if the Microsoft ISCSI hardware provider is CSV aware / compatible / supported etc etc.


    Microsoft Partner

    Monday, November 19, 2012 11:45 AM