locked
Mirror Hyper-V Hosts config over WAN? RRS feed

  • Question

  • I know that Hyper-V can be setup in a cluster locally, but how can I backup my Hyper-V hosts over a WAN link?
    My plan is to setup a SAN in our main office (along with 3 hosts), and setup an async mirror of that SAN to our remote office 1000 miles away over MPLS (3MB) link. The SAN will hold all of my VMs, but I need a way to also mirror my Hosts in the event of a disaster. I want to manually switch over to the remote hosts so I don't need an expensive failover process. We are a non-profit so we get substantial discounts with Microsoft.
    Any suggestions for mirroring our hosts to the remote site?
    Sunday, October 11, 2009 6:39 PM

Answers

  • Hello,

    You have several different levels that this can be taken to.  These are the ways you can accomplish multi-site availability with Hyper-V:

    * SAN replication:  You can purchase a SAN which can replicate VMs to a different site.  These SANs are becoming increasingly common and good versions exist in many iSCSI products like Equallogic and LeftHand.  I've used those products to accomplish a site to site replication topology for VMs, such that they can be recovered at a second location from a second Hyper-V environment (basically that is waiting to be used).

    * Software Replication:  You can use a third party replication capability such as that from DoubleTake or SteelEye.  These products include some built-in auto-failover capabilities and are generally priced at a level that makes them more attainable than SAN based replication.  These can be an excellent choice for reliable recovery from a secondary site, or for another server in the same site.  Some examples are:

    DoubleTake:  http://www.doubletake.com/english/products/double-take-virtualization/Pages/Double-Take-for-Hyper-V.aspx

    SteelEye:  http://www.steeleye.com/hyper-v/

    *  Backup and Recovery:  You can always use Windows Server Backup, Data Protection Manager, Symantec, etc. to accomplish backup and recovery of Hyper-V virtual machines to a secondary location in conjunction with potentially data backups.  You'd certainly want these in any recovery scenario in the first place.  You need to continue to backup your system states, SQL data, Exchange data, etc. independantly so that you can recover it.

    I hope this answers your question.  Have a great day!

    Nathan Lasnoski
    http://nathanlasnoski.spaces.live.com
    Monday, October 12, 2009 4:28 PM

All replies

  • Hello,

    You have several different levels that this can be taken to.  These are the ways you can accomplish multi-site availability with Hyper-V:

    * SAN replication:  You can purchase a SAN which can replicate VMs to a different site.  These SANs are becoming increasingly common and good versions exist in many iSCSI products like Equallogic and LeftHand.  I've used those products to accomplish a site to site replication topology for VMs, such that they can be recovered at a second location from a second Hyper-V environment (basically that is waiting to be used).

    * Software Replication:  You can use a third party replication capability such as that from DoubleTake or SteelEye.  These products include some built-in auto-failover capabilities and are generally priced at a level that makes them more attainable than SAN based replication.  These can be an excellent choice for reliable recovery from a secondary site, or for another server in the same site.  Some examples are:

    DoubleTake:  http://www.doubletake.com/english/products/double-take-virtualization/Pages/Double-Take-for-Hyper-V.aspx

    SteelEye:  http://www.steeleye.com/hyper-v/

    *  Backup and Recovery:  You can always use Windows Server Backup, Data Protection Manager, Symantec, etc. to accomplish backup and recovery of Hyper-V virtual machines to a secondary location in conjunction with potentially data backups.  You'd certainly want these in any recovery scenario in the first place.  You need to continue to backup your system states, SQL data, Exchange data, etc. independantly so that you can recover it.

    I hope this answers your question.  Have a great day!

    Nathan Lasnoski
    http://nathanlasnoski.spaces.live.com
    Monday, October 12, 2009 4:28 PM
  • Carlton,

    If you put your VM config file on the replicated SAN volume, it will be replicated along with the VHD files.  However, you will not be able to simply use those replicated configuration files to re-provision new VMs.  Unless you plan on implementing a multi-site Hyper-v failover cluster with SteelEye DataKeeper or some other Microsoft multi-site cluster partner, your only option is to re-provision new virtual machines and use the existing replicated VHD files as the source for those new VMs.  One of the things you will lose during this re-provisioning process is all your network configuration, so you will need to go into each VM and set up the NIC cards with IP addresses for the new subnet of your DR site.

    Good luck and let us know how you make out.
    David A. Bermingham, Director of Product Management, SteelEye Technology
    Monday, October 12, 2009 6:51 PM
  • We are a non-profit so our failover doesn't need to be immediate. That said I don't think it would be cost effective for us to cluster our Hyper-V hosts over the WAN. We're going to have eight VMs so initally there will be a lot of setup involved, but I don't see making a lot of changes after the dust settles. If I make a change to a VM at our main site (more memory, CPUs, etc...), I can manually make that change in our Remote site. correct?
     

    Tuesday, October 13, 2009 12:38 AM
  • Like I said, I think you will be re-provisioning those VMs in the DR after a disaster, so you can assign the appropriate amount of RAM and CPUs at that time.  I would do some testing and see how things turn out.  If you already own Windows Enterprise you have paid for clustering already.  I'm not sure if your current array based replication supports WSFC, but I think the benefit of multi-site failover clustering will be worth your effort and investment in the long run.  Check out this blog post that describes the implementation of Hyper-V in a multi-site cluster in detail.
    David A. Bermingham, Director of Product Management, SteelEye Technology
    • Proposed as answer by laurajj Wednesday, December 30, 2009 6:11 AM
    Wednesday, October 14, 2009 1:29 AM
  • Hi David,

    Can SteelEye DataKeeper Cluster Edition v7.1 replicate between CSV and standard volumes?     For example if the production site has a 4 node Hyper-V R2 cluster using CSV and the DR site has a single server which is part of the same cluster

    Mark 
    Wednesday, November 18, 2009 9:00 AM
  • No, unfortunately all of the servers need to have access to the same storage resource type.  So it is either a DataKeeper Volume and one VM per volume or CSV and no replication.


    David A. Bermingham, Director of Product Management, SteelEye Technology
    Wednesday, November 18, 2009 2:08 PM
  • Ok makes sense, so does that mean that Clustered Shared Volumes and the advantages it provides (Live Migration, shared LUNs etc) cannot ever be used in conjuction with DataKeeper Cluster Edition.    Would this scenario work with DataKeeper Standard Edition?    If the DR site cannot be part of the production cluster can the virtual machines be replicated from a CSV to standard DataKeeper volume?

    Mark
    Wednesday, November 18, 2009 11:11 PM
  • CSV is not required for LiveMigration, so DataKeeper can be used with Live Migration.  Here is a video that shows DataKeeper being used to Live Migrate VMs without shared storage or CSV.

    http://clusteringformeremortals.com/2009/11/17/the-difference-between-hyper-v-live-migration-and-quick-migration/

    Without CSV, one VM per LUN is recommended in order to be able to control failover at the VM level.  If you have more than one VM per LUN, they must all fail over together.  I hear it is possible to configure more than one VM per volume via cluster.exe, but I have not tried it to verify it works.

    DataKeeper and CSV are mutually exclusive at this moment in any configuration.  The main problem is that we are tracking writes to a disk at the block level.  If multiple servers are writing to the same disk at the same time, we need a way to merge the changes into a single stream that can be replicated to the target.  We are not there yet, but may have a solution like that in the future.
    David A. Bermingham, Director of Product Management, SteelEye Technology
    Thursday, November 19, 2009 8:16 PM
  • Thanks David.    Using DataKeeper replicated volumes for Live Migration would be great in a situation where shared storage is not used however what if the production cluster is connected to a fibre channel SAN on CSV volumes, how would we achieve replicating those virtual machines to a one or more servers at the DR site which are attached to a fibre channel SAN at the DR site?

    Mark
    Friday, November 20, 2009 12:00 AM
  • Good luck.  Most of the vendors, including anti-virus, backup, encryption and replication vendors are working on solutions to support CSV, however it is a major paradigm change and support does not come easy, if at all in some cases.  If you find a replication solution that supports CSV, please post back.  Here is an interesting read regarding CSV...

    http://www.vcritical.com/2009/09/hands-off-that-csv/


    David A. Bermingham, Director of Product Management, SteelEye Technology
    Friday, November 20, 2009 2:03 AM
  • The Hyper-V R2 technology is very new so I can appreciate that support wouldn't be there yet but this limitation with replication will be a major setback for those looking to replicate Hyper-V R2 to a DR site in the near future as many if not all companies would be using Clustered Shared Volumes to leverage Live Migration and shared LUNs within the production cluster

    Backups of CSV volumes will be supported with DPM 2010 and Symantec Backup Exec 2010.    I would image antivirus wouldn't be much of a concern as it is not recommended on Hyper-V hosts and should be Core installs

    Mark
    Friday, November 20, 2009 8:27 AM
  • I think it is only a matter of time before realtime replication of the entire CSV volume from one array to another for disaster recovery will be available.  What this means is that in the event of a disaster, all of the VMs will be moved to the DR site at the same time.  What will be more difficult is being able to Live Migrate or Quick Migrate an individual VM from one CSV to another CSV, the same way you can today with SteelEye DataKeeper and other WSFC integrated replication technologies when VMs are on individual volumes.

    If you are interested in providing additional input and/or taking beta product that supports CSV replication, contact me at david dot bermingham at steeleye com
    David A. Bermingham, Director of Product Management, SteelEye Technology
    Friday, November 20, 2009 2:27 PM
  • Hi David,

    I would be very interested in testing the beta and have sent you an email

    Mark
    Tuesday, November 24, 2009 4:12 AM
  • I have not received an email.
    David A. Bermingham, Director of Product Management, SteelEye Technology
    Tuesday, November 24, 2009 2:19 PM
  • We have just set up something similar. Our configuration is as follows:

    Primary site: Win 2k8 DC +Hyper-V role (cluster - 3 servers; 1 VM per LUN)
    DR Site: Win2k8R2 DC + hyper-V role (cluster - 3 servers; CSV)

    First we took a backup of all the VHDs that we wanted to replicate, copied them to a USB hard drive and then copied them onto the CSV at the DR site.

    Next we wrote the scripts the do the following:

    1) enumerate all LUNs
    2) use diskshadow.exe to create a snapshot of each volume and expose as a drive letter
    3) use CWRsync to replicate files with mask *.vhd to the CWRsync server at the remote ste (directly into the CSV location).
    4) delete shadows

    We run the above process every few days keeping a relatively fresh copy of the data at the DR site. The total size of VHDs we are replicating is about 1TB, and the bi-weekly transfer is only about 15-20GB.

    All the VMs are already created at the remote site, and point to the correct VHDs on the CSV. At this stage, we are able to boot the VMs without a problem, but as mentioned by others in this thread, the network configuration is lost and must be recreated. I am currently working on a NETSH script which will reassign the correct IP address of the host on startup if it is missing, which will mean that I can simply start a host at the remote site and it should work without intervention.

    The above is certianly not realtime replication, but it does work, and is free.

    We will be upgrading our Hyper-V cluster at the primary site to Win2k8R2+Hyper-V soon. Diskshadow.exe does not work with CSV, but the VHD files appear not to be locked so I will RSYNC them out without taking a snapshot. This is the equivalent of hard resetting a server without flushing everything to disk - most of the time it's not a problem.

    Good luck with your implementation, and let us know how it goes!

    Ascendo
    Tuesday, December 1, 2009 6:03 AM
  • You can use the products of StarWind Software like StarWind or StarWind Enterprise according to your needs. The latest versions of this software support all of your requirements and are at affordable prices.

    ---
    iSCSI SAN software
    http://www.starwindsoftware.com
    • Proposed as answer by HKraft Thursday, March 11, 2010 3:58 PM
    Thursday, December 24, 2009 11:56 PM
  • Could you please share your scripts? I'm interested in doing something similar and would love it if I don't have to re-invent the wheel.

    thanks!

    p.s it looks like you're basically doing exactly what I want to do as I just stumbled on your postings here as well:

    http://social.technet.microsoft.com/Forums/en/windowsserver2008r2highavailability/thread/1f08c5a3-2366-4fe6-8cdb-e85a11f383c6

    Wednesday, March 17, 2010 11:07 PM
  • Ascendo,

    Great sharing.

    Do you have netsh script to assign the IP address?

    1) enumerate all LUNs
    2) use diskshadow.exe to create a snapshot of each volume and expose as a drive letter
    3) use CWRsync to replicate files with mask *.vhd to the CWRsync server at the remote ste (directly into the CSV location).
    4) delete shadows

    Can you provide the details on how to do the above steps?

     

     

    Tuesday, March 23, 2010 6:06 PM
  • Hi John

    We have since migrated from R1 to R2 on the primary cluster, so we now replicate from CSV to CSV. I can email you the scripts we currently use if you want?

    If you are on R1 and want to replicate to R2, you can enumerate all the LUNs by using 'mountvol|find /i "\\?\"', and then cycle through each one with a for loop in a batch file. This needs to be done on each host in the cluster.

    The rest of the sync is pretty much the same in terms of diskshadow (except for requiring my CSVB.exe utility to place the CSV on the correct node and prepare it for shadowing).

    Cheers

    Ascendo

     

    Wednesday, March 24, 2010 6:03 AM
  • Hi Ascendo,

     

    We currently have Hyper-V R2 running in our primary site on CSV.  We're looking at doing something like you describe above to a DR Site also using CSV.  Did you get Diskshadow working on your primary CSV?  Can you email those scrpts to me?

    hyperv at bintechpartners dot com

    Wednesday, March 24, 2010 9:17 PM
  • If you did, share those scripts with the rest of us ;)


    David A. Bermingham, Director of Product Management, SteelEye Technology
    Thursday, March 25, 2010 1:38 AM
  • Ascendo,

    Thanks for your help.  What's the utility CSVB.exe?

    Can you send me the script at jfriedmanx at yahoo dot com?

    Thank you very much!

    Thursday, March 25, 2010 2:24 PM
  • Hi Ascendo,

    Sorry to bother you again.

    I have checked my email but I haven't received your script.

    Here is my email address:  jfriedmanx at yahoo dot com

    Thank you.

    Monday, March 29, 2010 4:53 PM
  • Hi, we've just developed a small backup utility that should do what you want - it has a small footprint and manages WAN backups quite well (manually as per your interest above). More info at http://www.hyperoo.net

     

    Feel free to play around with the trial and let us know what you think.

    Monday, May 31, 2010 3:37 PM