none
Live Migration half as fast depending on source node

    Question

  • So this is pretty weird. Brand new pair of Dell R620 servers with Broadcom 57800 10GbE NICs set up in a cluster. All Windows Updates and NIC driver updates installed. If I Live Migrate from Node 1 to Node 2, I see >9.5Gb/s transfer speeds, which means that the 7 VMs I have running migrate over in about 17 seconds. Not too shabby. If I migrate the VMs back, I only see about 5Gb/s speeds, and it takes about twice as long: 34 seconds. Both servers' Hyper-V Manager are set for up to 8 simultaneous migrations. I can't find any difference in settings between the two. I'm stumped! Any ideas?
    Monday, July 29, 2013 11:06 PM

Answers

All replies

  • That's a very strange suggestion - this isn't a Storage Migration, it's a Live Migration. The storage is completely un-utilized, and it's a PS6110XS - 7 SSDs and 17 10k drives on a 10GbE network. I really doubt it's any sort of bottleneck.
    Tuesday, July 30, 2013 3:43 AM
  • Just a question: Is it maybe a Problem with the CPU that only one Core handles the traffic of the livemigration?

    To verify please have a look at the TaskManager use "Change graph to _> Logical processors" . If set look diring the livemigration and the sending and reciving host for the cpu. Is there a Saturation of one cpu?  


    Grüße/Regards Carsten Rachfahl | MVP Virtual Machine | MCT | MCITP | MCSA | CCA | Husband and Papa | www.hyper-v-server.de | First German Gold Virtualisation Kompetenz Partner ---- If my answer is helpful please mark it as answer or press the green arrow.

    Tuesday, July 30, 2013 1:09 PM
  • I did a few tests, and I'm seeing something interesting here. Not sure if it's a symptom or a cause, but when on the slow Live Migrations, none of the sending Node's CPUs get very heavily utilized. Here's a fast LM:

    And here's a slow LM:

    If these are hard to read, both of the images are available here, too:

    http://imgur.com/t8UjVek,YyZkoIJ#0

    • Edited by JakesterPDX Tuesday, July 30, 2013 4:43 PM Added URL for images
    Tuesday, July 30, 2013 4:40 PM
  • Still with the storage questions, eh? Yes, over iSCSI... still have no idea how this pertains to Live Migrations (not Storage Migrations).
    Tuesday, July 30, 2013 4:59 PM
  • Um, no, not better. A Live Migration barely touches the iSCSI storage network at all. It transfers the memory of a running VM over the LAN. I have no idea why you keep thinking that iSCSI would have any effect on this, but even if it did, as I explained in my first response to you, we have ample iSCSI bandwidth and performance.

    This feels like if I brought in my computer to say my graphics were slow, and you kept suggesting that it was the monitor's fault. Yes, they're technically connected, but one cannot impact the performance of the other.

    Tuesday, July 30, 2013 5:07 PM
  • I'm sorry, but are you not capable of basic math? You think that 10 = 5? I consistently get nearly 10Gbps going from Node 1 to Node 2, and I get about 5Gbps going from Node 2 to Node 1. I don't know how you run your shop, but where I come from I expect to get the full performance from my servers and switching.
    Tuesday, July 30, 2013 5:10 PM
  • I'm trying to be patient, but it feels like you're being difficult on purpose. I apologize if you're just not familiar with how this all works, but I'm looking for an actual answer here, not off-base ideas.
    Tuesday, July 30, 2013 5:11 PM
  • I didn't make any personal attacks. I'm sorry if you're offended, but it's one thing to throw out a guess, and it's another to keep insisting that you're on the right track when you're clearly not. You've highjacked this thread to make it all about your off-base idea and accusations of me attacking you, and I'm just trying to get to the bottom of my issue.
    Tuesday, July 30, 2013 6:51 PM
  • It seems like I guess: the livemigration is dependend on the cores. Try to compare the networkcard Settings and the bios Settings esspecialy the C-States and Performace. Here is an article that could help from my fellow MVP College Didier Van Hoye:

    http://workinghardinit.wordpress.com/2013/06/10/still-need-to-optimizing-power-settings-on-dell-12th-generation-servers-for-lightning-fast-hyper-v-live-migrations/

    Grüße/Regards Carsten Rachfahl | MVP Virtual Machine | MCT | MCITP | MCSA | CCA | Husband and Papa | www.hyper-v-server.de | First German Gold Virtualisation Kompetenz Partner ---- If my answer is helpful please mark it as answer or press the green arrow.

    • Marked as answer by JakesterPDX Tuesday, July 30, 2013 9:34 PM
    Tuesday, July 30, 2013 8:38 PM
  • Bingo! That was it. Thank you so much!
    Tuesday, July 30, 2013 9:34 PM