none
Client Settngs Wake Up Proxy DANGER< DANGER

    Question

  • Microsoft's SCCM sp1 has a new feature called "wake up proxy" whereby host a pretends to be host b (host a spoof's the mac address of host b) when host b goes to sleep. Needless to say, we were seeing tons of mac address flapping and port-security issues.

    On the client settings that you deploy to your devices,  If you have power management selected and have wake up proxy as yes or enabled, it wll cause network card " flapping" or rebroadcast of your mac address accross the wan.  This will trigger port security and bring down your network. 

    Refer to the Cisco Tac Below.  Only in sp1 sccm 2012.  Dont enable this feature, your network and security team will hate you once they found out SCCM 2012 sp1 caused this!  What a nightmare. This is apparently a pretty new issue.  over 2k users experiecned rolling blackouts of there ports becuase we have port security turned on our switches. 

    Can anyone confirm this on there network?

    https://supportforums.cisco.com/thread/2220560

    Friday, June 21, 2013 6:49 PM

Answers

All replies

  • There is nothing surprising here, this is expected behavior for the proxy wake-up feature.

    It is explicitly called out in the TechNet documentation and they explicitly recommend working with your network folks to ensure nothing like this happens. WoL is a network centric technology; not working with network folks to enable it is well ...


    Jason | http://blog.configmgrftw.com

    Friday, June 21, 2013 9:29 PM
  • Thank you for your reply, however I believe  there is a bug in Sccm 2012 sp1.  

    If you search for flapping and sccm 2012 sp1 you will see only recent search results.  Sp1 only contains this changed feature, So it would be correct to assume that only early adopters are seeing this issue.  

    To say that I did not read all the documentation is incredible, as the sp1 has only been our for 4 months now.

    This is a dangerous feature and needs to be more clearly marked, and explained in terms mere mortal can understand.  I realize you should not be installing Sccm 2012 if you are not a seasoned It person, but even for us a well marked and explained choice is much better and so easy to fix in the GUI.  Ive listened to many technet broadcast from MVP's and microosft lectures and not heard this feature explained once.

    Please check this mans article, just written out , it explains the whole process and how sccm network activity on layer 2, or syn packets can act as ddos actacks.  For now I have disabled this feature.

    http://www.1e.com/blogs/2013/05/03/configuration-manager-2012-sp1-wol-proxy-feature-overview/

    My interpretation of the above article!

    When and if a client disappears unexpectedly or goes to sleep, the election process becomes aggressive. The clients or pcs, start sending other mac address of a sleeping client a magic packet to wake them up. If the first manager client does not find the sleeping client it goes to the next so called manager ( which is just another pc! ) This is interpreted as  port spoffing another mac address! This behavior is duplicating virus behavior.  Incredible bad design if you use smart switches and port security as the port goes down.

    The next issue is the client does not update for unknown reasons, even though Ive sent out a new client by updating the client policy either manually or through Sccm console. 

    When a client does not process an client policy change and goes to sleep , whatever changes you made wont propagate until the client wakes up again.  Meanwhile the manager computer on a subnet that has the mac address of the sleeping clients is causing excessive traffic! Its like a circular effect, clients sleeping, other clients broadcasting, more managers broadcasting looking for mac address that are sleeping, or in active! Does it ever stop--NO.

    Respectfully to Microsoft,  now you have whole network and security departments wary of the very brand there supposed to trust, with the click of one button on the client settings you can cause so much damage.

    Monday, June 24, 2013 3:26 PM
  • This is the part of the documentation that Jason was referring to: http://technet.microsoft.com/en-us/library/dd8eb74e-3490-446e-b328-e67f3e85c779#BKMK_PlanToWakeClients. There's even a *warning* on that page.
    What's missing from the official docs?

    Torsten Meringer | http://www.mssccmfaq.de

    Monday, June 24, 2013 3:47 PM
    Moderator
  • I guess you are technically right, since the documention your talking about is dated May-1st of this year. My bookmark shows march 1st 2013  http://technet.microsoft.com/en-us/library/gg682067.aspx.  Link provided and shows no such warning!

    As of today we have no clients "flapping" which is a good thing!

    Thank you for you help, hopefully this will save another person alot of grief!

    Monday, June 24, 2013 6:44 PM
  • hi, i just wondering if the problem was fixed by someone.

    is that the correction is made on the cisco switch or SCCM.

    Wednesday, August 14, 2013 6:22 PM
  • In sccm under client settings, we just turned off all power management of devices, specifically wake up proxy, we set it to NO. 

    On the Cisco side, I don't think its there issue to address, I guess you could turn off port security but that would be a mistake.

    Wednesday, August 14, 2013 7:44 PM
  • You can also simply increase the number of allowed MACs on each port instead of disabling port security completely.

    Jason | http://blog.configmgrftw.com

    Wednesday, August 14, 2013 7:50 PM
  • We have one location with ports shutting down due to flapping\arp inspection errors. We have no Wake options turned on. This location has different security settings on the switches so assuming it is SCCM, our other locations are not suffering.

    Is there anything else in SCCM that could cause issues similar to the Wake features? Like branch cache?

    Thanks!

    Tuesday, August 12, 2014 6:23 PM
  • Is there anything else in SCCM that could cause issues similar to the Wake features? Like branch cache?


    No as far as I know.

    Torsten Meringer | http://www.mssccmfaq.de

    Tuesday, August 12, 2014 7:58 PM
    Moderator
  • Concur with Torsten. These are not the droids you are looking for.

    Jason | http://blog.configmgrftw.com

    Tuesday, August 12, 2014 8:02 PM
  • Agreed, ConfigMgr has the Wake Up Proxy feature that can cause map flap issue. If you've disabled that feature, then nothing else in ConfigMgr would be causing it.

    Wally Mead

    Tuesday, August 12, 2014 8:04 PM
  • Thanks guys. I am disabling the clients for a few days to appease management but I am relieved to see such a consensus!
    Wednesday, August 13, 2014 2:11 PM
  • I know you said you are appeasing management, but just so everyone knows, changing the startup of the SMS Agent Host service (or any of the site server/system services) is completely unsupported by Microsoft.

    Wally Mead

    Wednesday, August 13, 2014 2:53 PM
  • From what I can see GPO is turning them back on anyhow... I may need to uninstall. Would blocking silence the client?
    Wednesday, August 13, 2014 3:02 PM
  • Could also be the Client Health feature that ensures that the agent is enabled and started :-)

    The proper way to control this would be to turn off the Wake-Up Proxy feature (which you've done) make sure all clients have retrieved policies (likely already happened as happens hourly by default) and then ConfigMgr is not doing anything like this at all now.

    Outside that, you'd really need to run ccmsetup /uninstall to remove the Configuration Manager client agent (which is certainly not necessary).

    The bottom line is that this stuff is all well documented, and a warning message appears on the console screen when you attempt to enable the feature - telling you to read the docs as to what the feature is going to do to the environment.


    Wally Mead

    Wednesday, August 13, 2014 3:37 PM
  • Wake up was never on which makes me think this is all a waste of time as far as SCCM causing the flapping.

    Health feature... super.

    This location's network we do not manage. Same company but inconsistent security configurations and antiquated equipment. It really sucks.

    Thanks! I'll let you know what happens.

    Wednesday, August 13, 2014 4:01 PM
  • If Wake-Up Proxy was never enabled, then it absolutely is not Configuration Manager. It has no other feature that does anything with MAC flapping, etc.

    Good luck and please do report back to ease everyone's mind (though should be at ease with all the statements about it can't be ConfigMgr).

    And be sure to mark the thread as answered when you do respond.


    Wally Mead

    Wednesday, August 13, 2014 4:03 PM
  • Update:

    Error reported by network group: Packet Rate Exceeded and Arp-Inspection Violation

    Most that we have caught have been having a lot of communication with peer machines.

    The bulk of port issues started over a day or two after I enabled BanchCache via GP. This office has had port drop issues for years but nothing grouped like this (1-3 per day). I'm not saying that's it, just pointing it out the timing. We're going to leave the clients removed for a couple of more workdays then maybe add 10 back at a time next week and see what happens.

    Thanks again!

    Friday, August 15, 2014 4:36 PM
  • Update: No dropped ports since removing SCCM. I'm going to reinstall a few a day and watch.
    Tuesday, August 19, 2014 2:34 PM
  • Have you had time to do any additional testing? Would be interesting to hear the results.

    Wally Mead

    Monday, August 25, 2014 9:30 PM
  • No reported drops. We've installed only 3 machines at this point. Will add more this week.
    Tuesday, August 26, 2014 1:54 PM
  • Any update to this? Can't believe it would a ConfigMgr issue :-)

    Wally Mead

    Wednesday, September 3, 2014 2:58 AM
  • It's not looking good for SCCM although I'm still skeptical... I'm waiting to get the approval to install on a few more again. They requested to wait for end-of-month work to be completed. We have 7 out of around 60 installed. If there have been drops I have not been told. This is in a remote office so I'm not there to monitor unfortunately.
    Wednesday, September 3, 2014 2:06 PM
  • Had a port pop this morning during an agent install. It popped while downloading files ultimately leading the install to fail.

    Tuesday, September 9, 2014 6:14 PM
  • Define "pop"?

    Jason | http://blog.configmgrftw.com | @jasonsandys

    Tuesday, September 9, 2014 6:45 PM
  • Update from my original post. We have no other issues with network activity or ports locking up. We have changed all routers and switches as a part of a normal network upgrade. All sccm client settings are the same as the original post.

    My issues are gone and have not come back with over 3k users and 17 remote locations.

    Tuesday, September 9, 2014 6:56 PM
  • Security shut down due to ARP-Inspection Violation, or Packet Rate Exceeded which is new to me... They don't care to share much info. It could be something else for all I know... This is so frustrating. It could be a day or two before I get the logs if I get them at all.

    So with the packet rate violation concern I'm going to drop the bit rates and continue the client install with some additional machines.

    Tuesday, September 9, 2014 8:08 PM
  • So this really has nothing to do with the network but is instead due to, in this case, an over-aggressive IPS.

    With ConfigMgr managing 100,000+ organizations worldwide, I can safely say that it's not the culprit here.


    Jason | http://blog.configmgrftw.com | @jasonsandys

    Tuesday, September 9, 2014 8:22 PM
  • Yeah. I'm marking this as answered. I really do appreciate everyone's input!
    Wednesday, September 10, 2014 12:32 PM