locked
Dynamic memory allocation failure RRS feed

 > 

  • Question

  • Question
    Sign in to vote
    0
    Sign in to vote

    Dear reader,

    We sometimes have a problem where our windows 2012 r2 RDS virtual servers, that reside on windows 2012r2 hyper-v hosts, loose their dynamic memory and only have their startup memory left to work with. Users start complaining that things are very slow etc.

    If I check several screens (RDS Broker load balancing, hyper-v manager, cluster manager and the vm's task manager) it's clear that the vm only has its startup memory allocated. I'm not sure if this happens instantly or immidiatly after the nightly reboot. To resolve the problem we have to call all users on the vm where it happens and ask them to logoff (if they are even able to), and then we reboot the machine.

    I have checked the logs from the machine where the VM resides on and the logs from the vm itself. But I cannot find anything. We also have alot of windows 2008r2 vm's with dynamivc memory, but none of those have ever had this problem.

    Searched the internet, but so far it seems we are only.

    Can anyone give me a lead to troubleshoot this?

    Best regards,

    Ruud Boersma

    MCITP Enterprise administrator

    Monday, October 27, 2014 10:03 AM

Answers

  • Question
    Sign in to vote
    0
    Sign in to vote

    We worked around the issue by fixing all rds to 25Gb, and disable dynamic mem. In fact we do not actually need dynamic memory, becaue we do not have an environment that justifies it.

    But offcourse the issue exists, and should be resolved by MS. Maybe someone in the future stumbles over this post and has a solution. If one does be so kind to let us know.Thanks in advance.

    Best regards.

    Ruud Boersma

    MCITP Enterprise administrator

    Hiya,

    I think the issues resolves to something specific for your environment and not to a Microsoft related component. As previously stated RDS does support dynamic memory, however not all applications handle this well.

    • Proposed as answer by Alex Lv Monday, November 3, 2014 2:10 AM
    • Marked as answer by Alex Lv Sunday, November 9, 2014 7:42 AM
    Tuesday, October 28, 2014 9:13 AM

All replies

  • Question
    Sign in to vote
    1
    Sign in to vote
    Some workloads just does not work well with dynamic memory. SQL is one of them. RDS should work OK. On the host, check the current amount of memory available to virtual machines in Performance Monitor counter: Hyper-V Dynamic Memory Balancer – Available Memory.

    Sam Boutros, Senior Consultant, Software Logic, KOP, PA http://superwidgets.wordpress.com (Please take a moment to Vote as Helpful and/or Mark as Answer, where applicable) _________________________________________________________________________________ Powershell: Learn it before it's an emergency http://technet.microsoft.com/en-us/scriptcenter/powershell.aspx http://technet.microsoft.com/en-us/scriptcenter/dd793612.aspx

    Monday, October 27, 2014 10:36 AM
  • Question
    Sign in to vote
    0
    Sign in to vote
    So, what your saying is that a third party app running on the RDS could be responsible for this error?

    MCITP Enterprise administrator

    Monday, October 27, 2014 12:59 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    No, I'm talking about RDS itself.


    Sam Boutros, Senior Consultant, Software Logic, KOP, PA http://superwidgets.wordpress.com (Please take a moment to Vote as Helpful and/or Mark as Answer, where applicable) _________________________________________________________________________________ Powershell: Learn it before it's an emergency http://technet.microsoft.com/en-us/scriptcenter/powershell.aspx http://technet.microsoft.com/en-us/scriptcenter/dd793612.aspx

    Monday, October 27, 2014 1:30 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Hiya,

    A few bullets from the top of my head:

    1: RDS itself does not have any problems with dynamic memory.

    2: The 2008 R2 servers needs an update to support dynamic memory.

    3: SQL server will, by nature, consume all memory available, unless min/max memory setting is  configured. Even in newer versions of SQL, this is still a consideration.

    4: Not all applications support dynamic memory. Most applications utilizing a memory cache will suffer from this. So yes, this could indeed be a third party application causing this.

    Monday, October 27, 2014 2:25 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    We worked around the issue by fixing all rds to 25Gb, and disable dynamic mem. In fact we do not actually need dynamic memory, becaue we do not have an environment that justifies it.

    But offcourse the issue exists, and should be resolved by MS. Maybe someone in the future stumbles over this post and has a solution. If one does be so kind to let us know.Thanks in advance.

    Best regards.

    Ruud Boersma

    MCITP Enterprise administrator

    Tuesday, October 28, 2014 8:51 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    We worked around the issue by fixing all rds to 25Gb, and disable dynamic mem. In fact we do not actually need dynamic memory, becaue we do not have an environment that justifies it.

    But offcourse the issue exists, and should be resolved by MS. Maybe someone in the future stumbles over this post and has a solution. If one does be so kind to let us know.Thanks in advance.

    Best regards.

    Ruud Boersma

    MCITP Enterprise administrator

    Hiya,

    I think the issues resolves to something specific for your environment and not to a Microsoft related component. As previously stated RDS does support dynamic memory, however not all applications handle this well.

    • Proposed as answer by Alex Lv Monday, November 3, 2014 2:10 AM
    • Marked as answer by Alex Lv Sunday, November 9, 2014 7:42 AM
    Tuesday, October 28, 2014 9:13 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Hi all,

    I'm going to be "one of those people" who revives dead posts for something that is relevant but obviously not fixed... sorry in advance!

    We have the exact same situation, a bunch of RDSH guests with Dynamic memory turned on (60+ of them). every day between 1-8 of them will fail to allocate Dynamic memory and will be stuck on their startup RAM amount. This really hurts our users at peak times.

    I have engaged our TAM and have raised a case with PSS, Usual story "your the only one with this problem". Which obviously isn't true.

    So, we have tons of free RAM on the hosts, 600GB+ on most of them, the issue affects RDSH hosts at random, across multiple hosts and clusters.

    The screen shots attached are of one of our hosts from this morning. it has 8GB startup, 8GB minimum, 32GB Maximum RAM configured, with a 23% buffer. the host has 752GB RAM FREE. Notice how the perf counter "Hyper-V Dynamic Memory Integration Service" is reporting "0", it should be reporting "32768". also under task manager on the VM we are missing "Maximum memory" which should be just below "Hardware reserved" in the bottom right hand corner.

    Looks like the balloon driver is being delayed at boot time, we are going to XPerf all the servers in the hope that we can catch the critter. It's an unusual problem.

    We only have Acrobat PDF viewer, word viewer, excel viewer and two custom dot.NET applications installed on the guests. Some of the servers are also just dumb RDSH hosts, with not connection broker configured, using an F5 loadbalancer for load distribution and session management. All guests are 2012R2 patched up to March 2015, integration Services are installed and up to date (its an enlightened OS remember).

    Friday, May 1, 2015 8:23 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    We have been having this same issue, it happens after a reboot and it's as if a VM doesn't know it is set to use dynamic memory - it gets "stuck" at whatever it's startup memory is.  It happens a lot with Chitrix boxes but I've had it happen with other VMs too.  Usually a reboot fixes it but sometimes I've had to shut down the VM and restart it.

    It started happening a couple of months ago and it hasn't been a huge deal so far but something is definitely going wrong and it's not just "something specific" to the OP's environment.

    -edit-

    This post seems to refer to the same issue.

    https://social.technet.microsoft.com/Forums/en-US/cc4b4672-473b-4ebf-994f-93034dfdff42/some-windows-guests-not-demanding-more-memory?forum=winserverhyperv


    • Edited by Matt Br Monday, May 11, 2015 5:36 PM
    Monday, May 11, 2015 1:47 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Our case is progressing nicely MS have admitted that it looks like something in dmvsc.sys which is the dynamic memory driver in 2012 R2. Product team is currently investigating...

    A live migration usually kicks dynamic memory back into life (forces a reset in the driver stack for dynamic memory) so currently we are just live migrating machines around (a lot!) to get around the issue. Hopefully an update will follow very soon, i'm due a call with our TAM for a catchup on the issue as it is raised as SEV B.

    Wednesday, May 13, 2015 3:00 PM
  • Question
    Sign in to vote
    0
    Sign in to vote
    Thanks for the tip about the live migration, I'll try that next time it happens instead of a reboot!  In the meantime I hope they get it sorted so we can get a hotfix.
    Wednesday, May 13, 2015 6:29 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Incidentally this is what we are seeing on the "memory" tab in the Hyper-V manager when one of the VMs is having the dynamic memory issue.


    Thursday, May 14, 2015 8:16 PM
  • Question
    Sign in to vote
    0
    Sign in to vote
    I have the exact same issue. I hope MS will get a fix soon. 
    Friday, May 15, 2015 6:31 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    @ Matt Br,

    We see the exact same behaviour in Hyper-V Manager...

    Friday, May 15, 2015 11:53 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Had the issue on a VM this morning and tried the Live Migration suggestion but unfortunately it didn't solve the problem.  

    From what I can remember I've only seen this on 2008 R2 and older VMs - has anyone had the issue with 2012+?

    Friday, May 15, 2015 12:46 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    We are seeing this on several 2012R2 Hyper-v clusters with 2012 R2 RDSH guests.

    Just chased our TAM, next stage is live debugging and diag of one of our production hosts and guests! :-S

    Friday, May 15, 2015 2:58 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Matt,

    What workload are you running in your VMs? Ours are mainly RDSH (only seen these VMs affected).
    What underlying storage and NICs are you using on the hyper-V hosts? Are they clustered?

    FYI we are running QLogic 8262 CNA's for storage and networking and we have EMC Powerpath installed. That's all we have on the hosts (apart from SCEP \ SCOM \ SCCM clients).

    The guests have:
    Acrobat Reader,
    SCEP Client
    SCOM Client
    Excel Viewer
    Word Viewer
    Classic Start Menu
    Couple of non installed custom dot.net applications written in house

    Friday, May 15, 2015 3:02 PM
  • Question
    Sign in to vote
    0
    Sign in to vote
    I'm afraid it doesn't seem to be be confined to any one particular type of workload and it has cropped up on both clustered and standalone hosts.
    Friday, May 15, 2015 7:42 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Hi everybody,

    Great to have found this topic, we have exactly the same problem. See https://social.technet.microsoft.com/Forums/en-US/538b8898-5c29-4408-9fdd-531eff4241c5/dynamic-memory-is-not-working-all-the-time?forum=winserverhyperv&prof=required .

    We have currently opened a case at MS but are not as far as i have seen in this topic. We are still in the process of gathering info en logfiles etc.

    At this time we are also live migrating the VM around, it is still the "best/fastest" fix to resolve the problem at the time it happens. Hope to see a fix soon.

    Grtx

    Peter Camps

    Thursday, May 21, 2015 1:22 PM
  • Question
    Sign in to vote
    0
    Sign in to vote
    This tread really should be unmarked as answered!
    Wednesday, June 17, 2015 2:16 PM
  • Question
    Sign in to vote
    0
    Sign in to vote
    Out of interest are you guys using VMM alongside HyperV??
    Friday, June 26, 2015 9:47 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Yes, we are using VMM 2012R2 (rollup 4).

    Today got the answer to open a new case????? Microsoft is absolutely treating this case ridiculously.

    At this moment we are trying to escalate this within Microsoft through a few contacts we have at MS.

    Monday, July 13, 2015 8:52 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    WE HAVE A WINNER!!!

    Product Group have managed to reproduce the issue after we supplied the VMs, hardware specifications and scripts to auto reboot VMs until one isn't assigned Dynamic Memory...

    Response from our TAM:
    "PG has the repro and a fix has been developed, should be out soon once they are done with the regression testing. Will post more once it’s available to me."

    This means they have accepted it as a BUG and have created a hotfix for it. they are currently testing the hotfix then they will throw it over to me to test in our environment. If it looks good it will be released as a hotfix, i'll post as soon as I have a release date. I'll ask them for an RCA also, not sure if they will supply one though.


    Peter, you have my Case ID don't you? ask your engineer to contact Kartikay for an update.
    • Edited by Microsoft Jim Friday, July 17, 2015 11:11 AM Case ID
    • Proposed as answer by Jesper Arnecke Thursday, August 27, 2015 8:58 AM
    Friday, July 17, 2015 11:10 AM
  • Question
    Sign in to vote
    0
    Sign in to vote
    Hotfix for this issue will be released on 13th October 2015. I'm pushing for a private hotfix and will update when I know more.
    Friday, July 31, 2015 10:09 AM
  • Question
    Sign in to vote
    0
    Sign in to vote
    I have the private hotfix which we are testing in our environment, so far, the issue has not come back. I'm pushing to get the KB number for this issue to share with the community.

    This posting is provided "AS IS" with no warranties, and confers no rights. Please remember, if you see a post that helped you please click "Vote as Helpful", and if it answered your question, please click "Mark as Answer". I do not work for Microsoft, I manage a large estate in the public sector, my views are generally first hand production experiences.

    Thursday, August 27, 2015 8:51 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    This sounds great. We switched to fixed memory for all machines, but definitely want to use dynamic for some of them.

    Thumbs up.

    MCITP Enterprise administrator

    Thursday, August 27, 2015 10:02 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Hello Jim,

    Have you heard any news about the release of this hotfix, is it still planned for October 13th?

    It would be about time for MS to release a fix for this issue.

    Wednesday, October 7, 2015 11:24 AM
  • Question
    Sign in to vote
    4
    Sign in to vote

    The public release is now available

    https://support.microsoft.com/en-us/kb/3095308

    We have ran it and it fixes the issue. thanks to all who have followed this saga!

    This posting is provided "AS IS" with no warranties, and confers no rights. Please remember, if you see a post that helped you please click "Vote as Helpful", and if it answered your question, please click "Mark as Answer". I do not work for Microsoft, I manage a large estate in the public sector, my views are generally first hand production experiences.

    • Proposed as answer by Microsoft Jim Friday, October 16, 2015 8:41 AM
    Friday, October 16, 2015 8:41 AM