none
Help! - BIG Issue with Primary DP After Updating to SP1 RRS feed

  • Question

  • Good Morning - 

    Last night, I updated our production SCCM 2012 to SP1.  I was very careful to check all I could before doing so.  The install seemed to go fine and a few tests passed when I was on the primary server.

    This morning, I tried to open the Admin COnsole isntalled on my workstation and found it wouldn't connect to the site.  After running the console installation again, it worked.  Easy, right?  Just deploy the console to a collection to all who have it installed?  I tried to do so, but it never deployed.

    Here's why - My primary DP (only local one) isn't updating content.  My other 3 remote ones are.  I checked the logs and here's what I found:

    Console / Monitoring

    • Distribution Manager failed to find or create the defined share or volume on distribution 
    • Distribution Manager failed to process package "SCCM Console SP1"

    distmgr.log

    • Failed to set share security on share \\ABCSCCM02.domain.com\SMSSIG$. Error = 5
    • Failed to set access security on share SMSSIG$ on server ABCSCCM02.domain.com
    • DPConnection::Disconnect: Revert to self
    • Cannot find or create the signature share.
    • STATMSG: ID=2324 SEV=E LEV=M SOURCE="SMS Server" COMP="SMS_DISTRIBUTION_MANAGER" SYS=ABCSCCM02.domain.com SITE=ABC PID=2340 TID=6220 GMTDATE=Tue Jan 08 15:16:35.474 2013 ISTR0="["Display=\\BNASCCM02.bassberry.com\"]MSWNET:["SMS_SITE=BBS"]\\ABCSCCM02.domainy.com\" ISTR1="ABC001A1" ISTR2="SMSSIG$" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=2 AID0=400 AVAL0="ABC001A1" AID1=404 AVAL1="["Display=\\ABCSCCM02.domain.com\"]MSWNET:["SMS_SITE=ABC"]\\ABCSCCM02.domain.com\"
    • Error occurred. Performing error cleanup prior to returning.

    So far, I opened the SMSSIG$ & SMSPKGE$ shares on the server and compared NTFS & Share permissions to other DPs which looked good.  I also added the hostmane of the primary site server as well as the sccm service account with full rights to them, but it didn't help any.

    All post I've found so far about this say to uninstall/reinstall the DP.  I'm sure that would work, but I have tons of content which would take forever to redistribute.

    Any suggestions?  I'm continuing to search, but wanted to go ahead and post.  If uninstall / reinstall of DP is only option, suggested procedures for doing so would be helpful.

    Thanks!


    Ben K.

    EDIT:

    Here is a link to my distmgr.log.  It's ~2mb and names were changed to protect the innocent :)

    Tuesday, January 8, 2013 3:23 PM

Answers

  • OK - 

    Just got off the phone with Microsoft and think we found the issue...  I'm now working!

    I had my Site System Installation Account set to my service account (DA) which I use for most everything.  We changed that back to "Use the site server's computer account to install the site system."

    Screenshot

    After doing that, I restarted the SMS_Executive services, updated my packages, and they worked!

    Microsoft has now marked it as a "Known Bug"

    Please try it out and let me know if this resolves your issues, too - 

    Thanks - 


    Ben K.

    • Proposed as answer by KnightNZ Thursday, January 10, 2013 7:47 PM
    • Marked as answer by Ben Kellermann Friday, January 11, 2013 10:49 PM
    Thursday, January 10, 2013 5:41 PM

All replies

  • Is ABCSCCM02.domain.com a remote DP of the primary site? Can you upload the entire distmgr.log to skydrive?

    Torsten Meringer | http://www.mssccmfaq.de

    Tuesday, January 8, 2013 3:29 PM
    Moderator
  • Thanks for reply - 

    Just edited original post with link to log file.  Here's the breakdown:

    ABCSCCM02 = Primary Site Server / Dist Point Local to largest office / Win 08 R2

    ABCSCSQL01 = Dedicated SQL server for SCCM @ primary site / Win 08 R2 / SQL 2008

    ABCFILE01 = Remote Dist Point in Knoxville / Win 08 R2

    ABDFILE01 = Remote Dist Point in Memphis / Win 03 R2

    ABEFILE01 = Remote Dist Point in Washington DC / Win 08 R2

    ABCSCCM02 is the only DP giving this issue.  All others are working as they should...

    Hope that helps...  Thanks again!


    Ben K.


    Tuesday, January 8, 2013 3:33 PM
  • UPDATE 1

    After more research, I took screenshots of SMSSIG$'s share and NTFS permissions, then Stopped sharing it.  As the post I read said, SCCM did reshare it automatically - but - it still didn't fix the issue.

    UPDATE 2

    Correction - the stopping sharing fixed it - somwhat...  If I stop sharing SMSSIG$ then Update my DPs, I here's what happens:

    1. Stop Sharing SMSSIG$

    2. Update DPs on package failing

    3. SMSSIG$ is automatically re-shared with correct (supposedly) permissions

    4. The package is added to SMSSIG$ (opened folder and verified new sig files are created) + console reports it successfully deployed to DP

    5. No other packages can be added - they get the same error.

    Basically, every time I need to add a package to the DP, I must Stop the SMSSIG$ share once per package. Wierd.

    Any Ideas?  Thanks


    Ben K.


    Tuesday, January 8, 2013 4:03 PM
  • Nice troubleshooting there! Really weird problem :) But!

    Is your content not located at the site server or at least in the same fast network? Is it really that much data?

    If you really have read that reinstalling the DP will solve it why don't you just do that friday afternoon and let it distribute all weekend? I can't take that long time can it?

    Tuesday, January 8, 2013 5:44 PM
  • Thanks for your reply!

    When working with SCCM before, I've always had the source content physically on the primary site server (if that's what you are talking about.)  However, I brought SCCM to this environment which already existed with LANDesk and a few others (which SCCM's has trumped and they are now gone thank goodness)

    Here are the stats:

    • E: part of SCCM server is mostly for DP content and contains no source data except for downloaded software updates.  It's currently  57gb free of 349gb
    • Packages / Applications / OS Images / - 95% source data on server share (local gpbs network but diff server)
    • Drivers - all source data on same share as above
    • Soft Updates - Everything on E: part of SCCM server as mentioned above

    True about Friday afternoon.  

    One more thing I forgot to mention and just thought of...

     I just remembered that about 3-4 months ago, I started getting errors where everything I tried to distribute gave me hash mismatch errors.  Due to this, I finally deleted EVERY item which syncs with a DP, deleted the DP, reinstalled it, and rebuilt the packages all from scratch.  I had tried to just uninstall / reinstall the DP, but even then still got mismatches. New packages were the only thing that worked.  Wasn't a fun weekend!

    Since then, I've used my MDT boot images since I couldn't rebuild the native ones as they couldn't be deleted for rebuild.

    Wonder if issue stems from this?  I watched the SP1 upgrade log last night and it did throw error when updating one of the native boot images.

    May have to redo DP Friday as you mentioned.  If I did, could I just uninstall the DP role and wait for everything to be processed or would I have to remove the DP from each individual package first?  

    Any other suggestions to prevent having to rebuild are also welcome :)  Thanks!

    Here's a link to the latest distmgr.log.  This was running when I stopped the share, it automatically re-created the share, allowed one package to be added to DP, then threw errors again if anyone's interested...


    Ben K.



    Tuesday, January 8, 2013 6:01 PM
  • Any errors in eventlog?

    Torsten Meringer | http://www.mssccmfaq.de

    Tuesday, January 8, 2013 6:53 PM
    Moderator
  • No Errors at all in Application or System logs...

    Had to create a temp VM Site Server with small DP until fixed.  Still, it's causing some big issues.

    Any more suggestions?  

    Also...

    If I do have to reinstall DP (local on Primary SCCM Server), what's the suggested process?  Which option below would be the scenario that I need to use?

    1. Remove DP role from primary site server.  Add role back a few minutes later
    2. Go to properties of each package, local content, remove each individually from local DP, do step 1 above, then re-add packages
    3. Recreate all packages which sync to DP (had to do that before due to hash mismatch error - hopefully not this time!

    I have my 4 DPs in a DP Group.  This is what most packages are distributed to - not the individual DPs (even though I think it adds all 4 individually when choosing a group)

    Which would be the option I'd need to use?

    Thanks!


    Ben K.



    Tuesday, January 8, 2013 9:50 PM
  • If I were you, I would do the following:

    1. Remove the DP from all package assignments - If you have a lot of packages that this DP is assigned to, you can use a VBscript like this one (It still works for SCCM 2012... I've used it):  http://blogs.msdn.com/b/rslaten/archive/2006/03/01/removing-a-retired-dp-from-all-your-packages.aspx
    2. Remove the DP role from the Primary Site server - Monitor the DistMgr.log for the removal - This shouldn't take too long.
    3. Re-deploy the DP role to the Primary Site server - Monitor the DistMgr.Log/Distribution Point Configuration State for the installation - This shouldn't take too long.
    4. Add the Distribution Point back to the Distribution Point groups it was a member of - If you assigned packages to DPs via Distribution Point groups, it will add the DP back to all of those packages.

    You should not need to re-create any packages, as that doesn't sound like it is a problem because you aren't having distribution issues with any other Distribution Points.

    Hope this helps.

    Wednesday, January 9, 2013 1:28 AM
  • Hello

    Same issue over here.

    Had a 2008r2 server wich I upgraded to sccm sp1 and got the same messages as mentioned in the first post ...

    Then I installed a new server with server 2012, sql 2012 and sccm 2012 sp1 and i have the same messages as in the first post.

    I Think this is a big bug into sp1 ...

    Anyone already a working work arround?

    regards

    Sven

    Update:

    Even when i create a new package I get the same error ...

    • Edited by Sven Bosmans Wednesday, January 9, 2013 8:57 AM update
    Wednesday, January 9, 2013 7:32 AM
  • Thanks - 

    I just may have to use that solution.  So... just to make sure, is this how I use it?

    1. Delete any advertisements which may run during window in which I am performing reinstall
    2. Run script from link referencing my DP on Primary site server
    3. Monitor DP log until uninstall & other activity is caught up / complete
    4. Reinstall DP SS role

    The part I'm unsure on is #1.  If nothing is set to advertise, am I golden and ready to go? 

    Thanks


    Ben K.

    Wednesday, January 9, 2013 11:04 PM
  • FWIW I've got the exact same problem on a pre-production server, SQL Server 2012 SP1/CU2, Windows Server 2008 R2 SP1, SCCM 2012 SP1 RTM.  I'd rather use server 2012 but we'd need to upgrade ESX to 5.0 which isn't very practical right now.

    I've tried the workaround of stopping SMSSIG$ and then I can redistribute a package correctly, from then on everything fails again though.  Any ideas on a more robust solution would be fantastic.

    I tried removing & re-adding the DP role on the local system using the script above, but the issue persists. As it's a fresh system there's only half a dozen Applications/advertisements on there so not too big a job.

    Not ideal given that it's my first week on the job and the massive time spent debugging wierd issues isn't making me look good ;)

    Edit:

    Relevant portion of the distmgr.log from a failed update:

    CreateSignatureShare, connecting to DP SMS_DISTRIBUTION_MANAGER
    Signature share exists on distribution point path \\sccmserver\SMSSIG$ SMS_DISTRIBUTION_MANAGER
    Set share security on share \\sccmserver\SMSSIG$ SMS_DISTRIBUTION_MANAGER
    Failed to set share security on share \\sccmserver\SMSSIG$. Error = 5 SMS_DISTRIBUTION_MANAGER
    Failed to set access security on share SMSSIG$ on server sccmserver SMS_DISTRIBUTION_MANAGER

    However - if I unshare SMSSIG$ and do another refresh, it works, and the relevant text is:

    Successfully created the directory for the signature export - \\sccmserver\D$\SMSSIG$. SMS_DISTRIBUTION_MANAGER
    Setting default permissions on NAL path MSWNET:["SMS_SITE=PRI"]\\sccmserver\D$\ with relative path SMSSIG$.
    Setting permissions on path MSWNET:["SMS_SITE=PRI"]\\sccmserver\D$\SMSSIG$\. SMS_DISTRIBUTION_MANAGER
    Successfully created share SMSSIG$ on server sccmserver SMS_DISTRIBUTION_MANAGER
    Set share security on share \\sccmserver\SMSSIG$ SMS_DISTRIBUTION_MANAGER

    Looks like an access issue which is okay when it's accessing the folder via D$ as opposed to via directly?

    • Edited by KnightNZ Thursday, January 10, 2013 4:16 AM
    Thursday, January 10, 2013 2:55 AM
  • Just asking... Is the Remote Differential Compression feature installed on the site server and the DP?


    http://www.blogmynog.com

    Thursday, January 10, 2013 5:40 AM
  • Hello Stephen

    Yes is is installed and enabled ...


    Thursday, January 10, 2013 7:33 AM
  • Have you already checked the default permissions?
    - Share: System: full control + Administrators: full control + Users: Read
    - NTFS: Administrators: full control + Users: read/execute, list folder, read
    Have you already tried a site reset?

    Torsten Meringer | http://www.mssccmfaq.de

    Thursday, January 10, 2013 7:37 AM
    Moderator
  • Site reset did'nt solve the problem

    and default permissions are also OK.

    Thursday, January 10, 2013 7:53 AM
  • Hi Everybody,

    change your start as Account on Service SMS_EXECUTIVE to an Administrator Account. After change your dp works fine.

    Thursday, January 10, 2013 10:46 AM
  • change your start as Account on Service SMS_EXECUTIVE to an Administrator Account. After change your dp works fine.


    That is absolutely NOT supported and might cause other issues as well. I don't recommend doing that!

    Torsten Meringer | http://www.mssccmfaq.de

    Thursday, January 10, 2013 10:55 AM
    Moderator
  • But it works and can be a perfect workarround until someone find and solves the real problem ...

    Thursday, January 10, 2013 11:20 AM
  • Morning Guys -

    I am going to try to "change your start as Account on Service SMS_EXECUTIVE to an Administrator Account" here in a few.  Just to make sure, though, you are talking about changing the "Log on as" in the SMS_Executive to something like the SCCM Servive account which in turn is/can be a domain admin - right?

    Screenshot

    Torsten:  What are the disadvantages of doing so?

    Thanks

    UPDATE 1

    I changed the "Log On Account" as noted above, then updated DP for a package which a added drivers to this morning.  I also watched the distmgr.log at the same time.

    When it started on the DP with the issue, I got the below errors in my log:

    Set share security on share \\ABCSCCM02.domain.com\SCCMContentLib$ SMS_DISTRIBUTION_MANAGER 1/10/2013 9:39:34 AM 9268 (0x2434)

    Failed to set access security for content 8037BF3A-45EA-43B6-BE81-5B37FC2429A0

    SetObjectSecurity failed; 0x8007051b

    ... and so on - the 2 above lines for each driver I added I believe since the content ID was different each time...

    I then added Full Access to the same DA user which I used for the "Log on as" for both Share & NTFS on the SCCMContentLib$ share.  I applied it and it took a couple of minutes to commit the changes.

    Once committed, I updated the DP again and watched the distmgr.log.

    I'm waiting on it to run and will update this post once I have results...


    Ben K.


    Thursday, January 10, 2013 3:31 PM
  • Torsten:  What are the disadvantages of doing so?

    It's not supported, not tested and the product was not designed to run in that context. I would call CSS before to fix the root cause before doing those ind of things.

    Torsten Meringer | http://www.mssccmfaq.de


    Thursday, January 10, 2013 3:44 PM
    Moderator
  • UPDATE 2

    The Update DP just ran again - again for a driver package where I added drivers this morning and after I changes the service log on as as well as added full permissions for the same DA user to SCCMContentLib$.

    I got the same errors as above.

    Here's a link to the newest distmgr.log file after making the changes discussed recently.  I changed the names of some things for security, but when opening the edited version to verify in CMTrace, the errors weren't highlighted for some reason - so - just in case, the errors mentioned above are at timestamp 1/10/13 9:47:22

    Also - Even though I received the errors above, the Content Status pie chart for the same driver package whose "slice" was red easlier is now green showing it was successful - even though the log file may diagree...

    Thoughts?  Thanks!


    Ben K.



    Thursday, January 10, 2013 3:53 PM
  • UPDATE 3

    Know I'm sending a lot of info, but believe the more, the better...

    I just created a new package using an OEM Adobe Flash MSI, didn't make any changes, and added it to my DP group where one of the 4 DPs is the one with issues (what I normally do)

    Again, the Admin Console's pie chart showed all 4 DPs as green this time after refresh, but received the errors mentioned above again in the distmgr.log file - only for the DP which is having issues.

    • Set share security on share \\BNASCCM02.bassberry.com\SCCMContentLib$
    • SetObjectSecurity failed; 0x8007051b
    • CContentDefinition::SetSecurity failed; 0x8007051b
    • Failed to set access security for content ABC003B0.1

    I read that the error code means INVALID OWNER.  Could it be that the SCCMContentLib$'s share need to have it's owner set to the same account?  Not going to try that yet as don't want to get in too deep with workarounds - unless someone has any other ideas...



    Ben K.


    Thursday, January 10, 2013 4:00 PM
  • Try to change the log on Service Account on each DP`s.
    Thursday, January 10, 2013 4:14 PM
  • Ok - I will - but - don't see how that would help.  The distmgr.log showed all of the 3 other DPs by name and that they updated correctly.  Only the 4th (one with issues which I changed the service log on on) gave the error.  

    Still, I'm desperate so will try.

    I just opened a support request with Microsoft, too, so we'll see where that leads.  I'll post anything I find - if there's anything...

    Thanks


    Ben K.

    Thursday, January 10, 2013 4:54 PM
  • OK - 

    Just got off the phone with Microsoft and think we found the issue...  I'm now working!

    I had my Site System Installation Account set to my service account (DA) which I use for most everything.  We changed that back to "Use the site server's computer account to install the site system."

    Screenshot

    After doing that, I restarted the SMS_Executive services, updated my packages, and they worked!

    Microsoft has now marked it as a "Known Bug"

    Please try it out and let me know if this resolves your issues, too - 

    Thanks - 


    Ben K.

    • Proposed as answer by KnightNZ Thursday, January 10, 2013 7:47 PM
    • Marked as answer by Ben Kellermann Friday, January 11, 2013 10:49 PM
    Thursday, January 10, 2013 5:41 PM
  • Thanks for the feedback. Much appreciated!


    Torsten Meringer | http://www.mssccmfaq.de

    Thursday, January 10, 2013 5:49 PM
    Moderator
  • For me it worked! All packages are now distributed succesfully again. Thanks!

    Peter

    Thursday, January 10, 2013 6:23 PM
  • The MS solution also worked for me.

    besides my sccm machine account is member of the domain admins ...

    Cool: we found a known bug in sccm 2012 sp1 ...


    Thursday, January 10, 2013 7:22 PM
  • Marvellous, works perfectly, thanks very much!

    Thursday, January 10, 2013 7:48 PM
  • Well, you don't have to add the machine account to the domain admins group beacause of this!

    Why don't you just put the site servers machine account into the local administrators group?

    Thursday, January 10, 2013 7:53 PM
  • Hi , 

    Guys , could you try to recreate the GUID of ur packages by adding a space anywhere ? 

    Could also be fixing the issue

    Friday, January 11, 2013 10:55 AM
  • Guys , could you try to recreate the GUID of ur packages by adding a space anywhere ? 

    Could also be fixing the issue


    Which GUID are you talking about? And why trying something different if Microsoft already confirmed that this behavior is a bug?

    Torsten Meringer | http://www.mssccmfaq.de

    Friday, January 11, 2013 11:14 AM
    Moderator
  • Where is that screen?  During the install of the DP?

    Tuesday, January 15, 2013 4:11 PM
  • Yes - or configuration of it.

    1. Click on Administration in the lower left
    2. Expand Site Configuration on left
    3. Select Servers and Site System Roles
    4. Click once to select primary server on right
    5. Below server list, right click Site System, then choose properties

    It's in the middle of the window that appears.  Good Luck! 


    Ben K.

    Tuesday, January 15, 2013 10:14 PM
  • This worked for us... unfortunately we went all the way to point of completely deleting and re-creating all of our distribution packages, which was a huge pain, and didn't work.  Changing the site system account ultimately solved our issue.  Thanks for posting!

    Monday, February 11, 2013 2:46 PM
  • Anyone still having these issues? i have it on a new installation of SCCM 2012 SP1.

    I cant dist anything i get the error you are describing, AND i have changed the Site System setting, (they where set right from the get-go)

    Any ideas?


    • Edited by Johan Atea Friday, March 8, 2013 3:40 PM
    Friday, March 8, 2013 3:31 PM
  • Hi there, We still have the same problem, installed SP1 at the beginning of this month. After DP didn't work, we removed the DP and it worked again. Now 2 weeks later we have the same problem again. Also de Site System account is set as Use the site server's computer account to install the site system."

    Is there anyone who has got other ideas?

    Wednesday, April 17, 2013 10:47 AM
  • Worked like a treat. I was struggling since long time to figure out this issue. Thanks very much.

    Vijay Ramshetty MCITP

    Thursday, April 18, 2013 9:15 AM
  • if you still have issues...do you have a file called "program" on c:\ ?? We had an issue with this file on c: and that a single package could not be processed because of it....renaming the file solved it.....but still i do not know how this file is created and what it is for...
    Tuesday, June 18, 2013 9:59 AM
  • Can you please help me find the location in SCCM to make this change? I've looked everywhere and can't seem to find it. I'm pretty sure that I changed the setting from the computer account to my service account as well. Thanks 

    Jason

    Friday, January 10, 2014 10:16 PM