none
scheduled pruneshadowcopies2010.ps1 and autoprotectinstances.ps1 never completes RRS feed

  • Question

  • dpm 2012 ur3, secondary dpmserver.

    The only mentioning i've seen about his has been that long hostnames need to be adressed, but our hostname as returned by 'hostname' is 9 chars.

    Every night at 00:00 DPM kicks theese jobs, but they never ends, they'll build up a 2,5 GB working memory set and then just sit there. 

    Besides of having to do this manually, this will kill our dpmserver after a couple of days uptime, having two powershell.exe-processes added each night, occupying 2,5 gb each.

    I believe this phenomenon has been around since we upgraded from 2010 to 2012.

    we did a diskmigration two months ago and had to run pruneshadowcopies2010 manually as domainadmin a couple of times, then the rp's slowly got removed..


    Ivarson

    Wednesday, June 5, 2013 10:30 AM

Answers

  • Hi Ivarsson (Joel)

    Start of by looking in the local logs of your DPM server, the problem could also be a combination of related matters. Youi need to filter the sources down.

    Please post any information regarding your progress.


    Best Regards

    Robert Hedblom

    MVP DPM


    Check out my DPM blog @ http://robertanddpm.blogspot.com

    Saturday, June 8, 2013 9:24 PM
    Moderator

All replies

  • Hi Ivarsson (Joel)

    Start of by looking in the local logs of your DPM server, the problem could also be a combination of related matters. Youi need to filter the sources down.

    Please post any information regarding your progress.


    Best Regards

    Robert Hedblom

    MVP DPM


    Check out my DPM blog @ http://robertanddpm.blogspot.com

    Saturday, June 8, 2013 9:24 PM
    Moderator
  • I looked around in Temp\ at that time but I didn't figure out what to filter on.. I had no idea when the process bailed out and stopped doing its work nor what source to search for.

    This matter is still an Active case with MS and according to them this is pointing to a bug in DPM that has yet to be fixed in either version.

    In addition to having the processes hanging, the 'bug' also causes our tempdb to grow out of hand. We implemented a workaround from MS that throttled the concurrent sql-connections byt adding a maxpoolsize-attribe to the connectionstring for dpmdb. That helped for a month or two, til the issues came back.

    FYI. This time i ticked 'Alert me when someone responds' so reminders through phonecalls should be unnecessary


    Ivarson

    Thursday, September 5, 2013 8:18 PM
  • Any resolution for this Ivarson, or effective workaround?
    Monday, November 25, 2013 4:28 AM
  • Hard to say. We ended up reinstalling dpm 2012sp1 on 2k12 servers, In my case it's a bug with a fix being planned to be included in next UR with a releasedate I'm not sure is public. The issue was said to exist in all dpm-versions so we didn't upgrade prmarly to get rid of this particular problem. But yeah, appending a MaxPoolSize-value to the connectionstrngs was a known workaround and did a major improvement.

    Ivarson

    Monday, November 25, 2013 5:09 AM
  • Hard to say. We ended up reinstalling dpm 2012sp1 on 2k12 servers, In my case it's a bug with a fix being planned to be included in next UR with a releasedate I'm not sure is public. The issue was said to exist in all dpm-versions so we didn't upgrade prmarly to get rid of this particular problem. But yeah, appending a MaxPoolSize-value to the connectionstrngs was a known workaround and did a major improvement.

    Ivarson

    The default pool size is 100.

    The Internet has many comments from people who feel the need to increase this (eg to 200) as a workaround for general miss-designed applications which fail to close connections when done using them.  This buys the application usually just a few additional hours before failing again for the same reason (pool exhausted).

    I can imagine that reducing the pool size to 50 would cause the pool to exhaust available connections sooner, resulting in the application involved having MORE trouble getting an available connection to it's database.  I'm not sure this is what you meant, and I can't imagine DPM working better afterwards, although it would throttle DPM significantly, but more in the meaning "to choke until it stops breathing", rather than the throttle on a car.  :)

    What did MS suggest you change this to, did they suggest increasing or decreasing it, and to what value?

    The actual powershell scripts themselves don't appear to have changed between versions. (2010 through 2012SP1cu3), although I haven't checked DPM2012R2 yet.
    Monday, November 25, 2013 6:23 AM
  • You're spot on. We saw through SQL mgmtstd that there was lots of unfinished connections hanging around and therefor I believe we set the limit to 200 (not sure, could have been more). No the prunescript hasn't changed but be advised not to modify them in their original location since I think they are to be left in future updates (if the name of the scripts isn't updated too of course. Also, there's recommendations not to schedule jobs close to midnight cause if the maintanancescripts, but moving the jobs around didn't help for us. You might be right in your assumptions that an increase only buys time. We did get the issues again after a while, it seemed to disappear completely when we finally reinstalled OS/dpm (though we restored the same db)

    Ivarson

    Monday, November 25, 2013 3:42 PM
  • Here's the proof that this was the problem.  This graph below is of user connections to the dedicated SQL instance on the DPM server, used exclusively by DPM itself.  Keep in mind that the default pool size is 100 connections.  In this case we've increased the max pool size to 500, allowing it to consume the full number of connections it needs to function.  One can clearly see a huge spike to 400 connections occurring at midnight.  This is caused by the product designers choosing to schedule all the housekeeping scripts to execute concurrently at exactly midnight.

    the list of scripts executed at that time include:

    pruneshadowcopiesDpm2010.ps1
    AutoProtectInstances.ps1
    StopProtectionOfStaleClients.ps1

    Once we raised the limit of connections up from the default 100 to 500, then suddenly auto-grow started working again as well, which hadn't worked for months.  Also, the above thee scripts actually finish executing each night, instead of lingering on until manually killed.  Previously, we'd have 10 copies of the pruneshadowcopiesDpm2010.ps1 and AutoProtectInstances.ps1 still running if we didn't manually kill them each day.  Curiously, manually running each script during the day worked fine, and was the workaround we had been using.

    graph of SQL user connections to the DPM SQL instance

    Tuesday, December 3, 2013 9:10 PM
  • "

    Every night at 00:00 DPM kicks theese jobs, but they never ends, they'll build up a 2,5 GB working memory set and then just sit there. 

    Besides of having to do this manually, this will kill our dpmserver after a couple of days uptime, having two powershell.exe-processes added each night, occupying 2,5 gb each.

    "
    We have same issue.

    DPM 2012 R2 (clear installation)

    Windows Server 2012 R2 Standard  (clear installation)


    Have a nice day !!!

    Thursday, January 16, 2014 7:26 AM
    Moderator
  • The registry keys you need to edit to fix these are as follows.  Note, only add the last value ";Max Pool Size=500" at the end:

    [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Data Protection Manager\DB]
    "ConnectionString"="Integrated Security=SSPI;Initial Catalog=DPMDB;Application Name=MSDPM;server=yourserver\\MSDPM2012;Connect Timeout=90;Max Pool Size=500"
    "GlobalDbConnectionString"="Integrated Security=SSPI;Initial Catalog=DPMDB;Application Name=MSDPM;server=yourserver\\MSDPM2012;Connect Timeout=90;Max Pool Size=500"

    Friday, January 17, 2014 12:53 PM