none
Some Windows Guests not demanding more memory

    Question

  • Scenario

    I have a 3 Windows 2012 R2 Hyper-V hosts which is participating in a cluster, all of the virtual machines are hosted on CSV's. We also use Dynamic Memory for most of the servers

    We do regular reboots every week for our virtual machines, but we've found that sometimes, a virtual guest will suddenly stop requesting additional memory.  At the time of writing, I have identified 10 machines - on different hosts, and even different clusters which are exhibiting this behaviour.  They're also running OS's ranging from Windows 7 to Windows 2008 R2 and Windows 2012.

    I have selected one to demonstrate below.

    Here, you can see the virtual machine is configured to use Dynamic Memory

    Dynamic Memory Configured

    and here, you can see that it's not actually demanding more than the start memory, despite the fact that I've overloaded the servers virtual memory by using MSPAINT.

    Workaround

    Rebooting the virtual machine does not resolve the issue, but live migrating the virtual machine from the host it is on to another host suddenly allows it to demand more memory.  One can then live-migrate it back to the original host and the problem is no longer there.

    Troubleshooting

    We have performed the following troubleshooting steps:

    • Verified that the Integration Services are up-to-date (currently running 6.3.9600.16384)
    • Verified that the virtual server has up-to-date Windows Updates
    • Checked the Hyper-V-Worker\Admin event log on the current host - no errors pertaining to this VM.
    • Verified that the Hyper-V host has sufficient RAM (currently only using 542 out of 768 GB)
    • Checked the System event log on the virtual machine - nothing out of the ordinary to report.
    • Verified that all services which start with the word "Hyper-V" on the guest machine is running.

    Using the following command, I can easily identify which machines are affected (this command applies to SCVMM only):

    Get-SCVirtualMachine | where {$_.DynamicMemoryEnabled -eq $true -and $_.DynamicMemoryDemandMB -eq $_.Memory} | ft Name,HostName,OperatingSystem,Memory,DynamicMemoryDemandMB,DynamicMemoryMinimumMB,DynamicMemoryMaximumMB


    I do not know any diagnostic program which can give me more information regarding the memory status of a server - specifically one which can tell me more information about Memory demand, and why it's not working. 

    Considering that the problem is resolved when I live migrate the affected machine, it would suggest that the issue is with the host, but as live migrating it back no longer exhibits the issue - that seems to negate that theory.  Additionally, the other 103 virtual machines which run on this host seem to not show this issue.

    It's almost as if the machine is being ran with static memory, and Hyper-V doesn't realise it's dynamic, but the moment it's live migrated, it suddenly realises this and enables dynamic memory.

    Any thoughts?



    • Edited by LesterClayton Tuesday, May 5, 2015 1:10 PM Code Block a Line
    Tuesday, May 5, 2015 12:32 PM

Answers

  • We have a public release of the private hotfix we have been testing...

    https://support.microsoft.com/en-us/kb/3095308

    Hopefully this will fix all your issues (it did for us).


    This posting is provided "AS IS" with no warranties, and confers no rights. Please remember, if you see a post that helped you please click "Vote as Helpful", and if it answered your question, please click "Mark as Answer". I do not work for Microsoft, I manage a large estate in the public sector, my views are generally first hand production experiences.

    • Proposed as answer by Microsoft Jim Friday, October 16, 2015 8:43 AM
    • Marked as answer by LesterClayton Friday, October 16, 2015 8:49 AM
    Friday, October 16, 2015 8:43 AM

All replies

  • Hiya,

    next time you see this behavior, could you try and run "Update-ClusterVirtualMachineConfiguration" on the affected server and see if that does change anything.

    https://technet.microsoft.com/en-us/library/ee461058.aspx

    Tuesday, May 12, 2015 6:17 AM
  • Why on earth is this thread marked as answered????

    Go speak to PSS or PG...

    Wednesday, June 17, 2015 2:20 PM
  • It's marked as Answered because I didn't respond to the suggestion made.  I have unfortunately been so busy that I've not had the opportunity to look into this since I posted my original question.

    That being said, I am now able to provide feedback.  Thanks for the nudge, it reminded me about it.

    At the time of writing, I have 3 virtual machines which is experiencing this issue, where it's not demanding any more than the startup RAM.  I have updated the configuration on one of them using the powershell command, and have attempted to demand more memory with no success.

    Any further input is welcome, but I suspect I am going to have to raise a support case for this.

    Wednesday, June 17, 2015 3:00 PM
  • There have been a few cases raised about this recently... See this thread which has a couple of CaseIDs in it. One mine, one a guy called Peter. Mine has gone to the PG and they are currently building a repro environment. by all means raise the case and use ours as reference.

    https://social.technet.microsoft.com/Forums/windowsserver/en-US/538b8898-5c29-4408-9fdd-531eff4241c5/dynamic-memory-is-not-working-all-the-time?forum=winserverhyperv

    Is it intermittent in your environment?
    Couple of "work arounds":
    1 reboot the machine or shut it down and start it back up!
    2 sometimes a live migration will kick the balloon driver into life!

    Wednesday, June 17, 2015 3:12 PM
  • WE HAVE A WINNER!!!

    Product Group have managed to reproduce the issue after we supplied the VMs, hardware specifications and scripts to auto reboot VMs until one isn't assigned Dynamic Memory...

    Response from our TAM:
    "PG has the repro and a fix has been developed, should be out soon once they are done with the regression testing. Will post more once it’s available to me."

    This means they have accepted it as a BUG and have created a hotfix for it. they are currently testing the hotfix then they will throw it over to me to test in our environment. If it looks good it will be released as a hotfix, i'll post as soon as I have a release date. I'll ask them for an RCA also, not sure if they will supply one though.


    Peter, you have my Case ID don't you? ask your engineer to contact Kartikay for an update.
    • Proposed as answer by Jesper Arnecke Thursday, August 27, 2015 9:00 AM
    Friday, July 17, 2015 12:45 PM
  • Thank you very much for updating my thread as well as the other one :)
    Friday, July 17, 2015 12:47 PM
  • No worries, I know there are about 4 threads for the same subject knocking around... Keep them all up to date :-)
    Friday, July 17, 2015 12:49 PM
  • Hotfix for this issue will be released on 13th October 2015. I'm pushing for a private hotfix and will update when I know more.
    Friday, July 31, 2015 10:08 AM
  • Thanks for keeping us up to date :)
    Wednesday, August 5, 2015 10:45 AM
  • Just been informed I will be getting a test hotfix on August 10th. The problem is host side not guest side so we should only have to patch hosts rather than guests.
    Wednesday, August 5, 2015 11:08 AM
  • Any news on this ?
    We also have the issue.
    Monday, August 24, 2015 2:33 PM
  • I am currently testing a private hotfix, which seems to have fixed the issue. It is currently aptly name KB9999999!

    As far as i'm aware 13th October is the official release date.

    I have asked them for the KB number this will be published under also.


    This posting is provided "AS IS" with no warranties, and confers no rights. Please remember, if you see a post that helped you please click "Vote as Helpful", and if it answered your question, please click "Mark as Answer". I do not work for Microsoft, I manage a large estate in the public sector, my views are generally first hand production experiences.


    Thursday, August 27, 2015 8:44 AM
  • Thank you for the reply Jim,

    Just got it with a third VM, 2008 R1. Live Migration does not help this time!

    Tuesday, September 15, 2015 7:29 AM
  • We have a public release of the private hotfix we have been testing...

    https://support.microsoft.com/en-us/kb/3095308

    Hopefully this will fix all your issues (it did for us).


    This posting is provided "AS IS" with no warranties, and confers no rights. Please remember, if you see a post that helped you please click "Vote as Helpful", and if it answered your question, please click "Mark as Answer". I do not work for Microsoft, I manage a large estate in the public sector, my views are generally first hand production experiences.

    • Proposed as answer by Microsoft Jim Friday, October 16, 2015 8:43 AM
    • Marked as answer by LesterClayton Friday, October 16, 2015 8:49 AM
    Friday, October 16, 2015 8:43 AM
  • The KB article seems to leave out a crucial detail - where is the hotfix supposed to be applied?  The Hyper-V hosts, the VMs using dynamic RAM, or both??
    Wednesday, January 27, 2016 7:44 PM
  • The KB article seems to leave out a crucial detail - where is the hotfix supposed to be applied?  The Hyper-V hosts, the VMs using dynamic RAM, or both??
    Hyper-V hosts :)
    Wednesday, January 27, 2016 7:45 PM
  • Thank you, the article didn't mention Hyper-V anywhere so wasn't sure if problem was also with guest handling of dynamic memory.  Another important detail not mentioned in the article, does the hotfix require a reboot of the HV host?  Good to know for planning/scheduling.
    Thursday, January 28, 2016 12:46 AM
  • Thank you, the article didn't mention Hyper-V anywhere so wasn't sure if problem was also with guest handling of dynamic memory.  Another important detail not mentioned in the article, does the hotfix require a reboot of the HV host?  Good to know for planning/scheduling.

    Yes, the hotfix will require a reboot - it changes a file which is in use (as virtual machines are undoubtedly running), and you cannot replace a file which is in use so it schedules for replacement on next reboot.  Hopefully you have a cluster that has enough capacity to allow for 1 node failure, so you could theoretically live migrate everything off - and update 1 node at a time.

    Remember to upvote comments if you find they add benefit to the conversation - I've updated both of your questions, because they do indeed add value :)

    Thursday, January 28, 2016 7:20 AM