none
OSD failing after 1910 upgrade RRS feed

  • Question

  • After upgrading my SCCM site to 1910, my Windows 10 OS Upgrade task sequences are failing. It completes the Upgrade Operating System step, then hangs at "Working on Updates 100%". It is supposed to restart at this point, and continue the task sequence. I have not been able to find any relevant errors in the log files. This was working fine until the site upgrade. I am upgrading OS from 1803 or 1809 to 1903 or 1909.  They all hang at the same spot.  I was able to roll back an SCCM client to 1906, and the upgrade ran fine.  Is this a bug in the 1910 client?
    Thursday, January 9, 2020 8:24 PM

All replies

  • following this thread as we are experiencing the same issues since upgrading our Config Mgr to 1910. 

    6.5.0
    Friday, January 10, 2020 3:11 AM
  • Hi,

    Thanks for posting in TechNet.

    If possible, please help install the hotfix KB4535819 for Configuration Manager version 1910 to have a try.



    Thanks for your time.

    Best regards,
    Simon 

    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Friday, January 10, 2020 8:55 AM
  • We are facing the exact same issue. TS for Windows 10 upgrade to 1903 was created on SCCM 1906 and it worked well. But recently we upgraded SCCM to 1910 and since then the TS is getting in hung state for the devices which has 1910 client updated.

    However the machines with 1906 client it is working fine. Any bug with 1910 client?


    Rohit

    Friday, January 10, 2020 9:23 AM
  • We are also facing excat the same issue after upgrading our SCCM to Version 1910.
    The  hotfix KB4535819 doesn't apply to us, as we are not in the fast ring and our client version already is on version 5.00.8913.1012

    The very same upgrade Task Sequence runs fine with the prvious client version 5.00.8853.1006.

    So we also suspect a bug in the latest client version.

    Please examine this with priority as is has some kind of impact....

    Friday, January 10, 2020 9:40 AM
  • We already have this version installed, We did the SCCM upgrade after the 17th December.

    Still an issue


    Rohit

    Friday, January 10, 2020 10:54 AM
  • Thank you, but we are not using the early update ring, so this hotfix will not apply.
    Friday, January 10, 2020 2:49 PM
  • Please examine this with priority as is has some kind of impact....

    No one monitoring these forums can or will examine this. If you need priority help with this, you need to open a support case with Microsoft customer support services.


    Jason | https://home.configmgrftw.com | @jasonsandys

    Friday, January 10, 2020 3:45 PM
  • I am unable to install this hotfix, as we are not in the early update ring.
    Friday, January 10, 2020 6:19 PM
  • We are in the same boat (see https://social.technet.microsoft.com/Forums/en-US/4c66e602-bc6c-4ce9-89da-d74e2ec277f7/1909-feature-update-woes?forum=configmanagerosd)

    We need to update more than  400 machines with 1709 soon (EOL in 04/20) and cannot do so due to broken Inplace Upgrade in 1910. Please Microsoft. Fix this now!!!

    Saturday, January 11, 2020 10:37 AM
  • Hi,

    Thanks for your reply.

    I will do more research about this issue and if there is any update, I will let you know. Thanks for your understanding.

    Best regards,
    Simon

    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Monday, January 13, 2020 8:02 AM
  • We have installed the Hotfix KB4535818 today, then updated the SCCM client on several reference machines.and subsequently started the 1909 inplace upgrade again on serveral machines.

    Unfortunately the Hotfix seems NOT to solve our problem, not being able to install an Inplace Upgrade any more. 

    We are still seeing the dreadful error 0x8000FFFF in smsts.log at the time where tsmbootstrapp should resume the Task Sequence, right after the OS upgrade has completed. Still this does not happen. The entire Task Sequence is stuck at "Waiting for updates ... 100%..." and after a timeout of ca. 30-45 minutes bombs out and shows the login screen. All upgrade steps post the OS upgrade (i.e. language packs) have been skipped and Software Center show that the upgrade has failed with error 0xFFFF(65535).

    Having done lots of inplace upgrades in the past this is really annoying, since it worked great until 1910. 

    Can we have a solution soon for this?

    Monday, January 13, 2020 9:43 PM
  • > Can we have a solution soon for this?

    Have you opened a support case for this? No one here can fix this for you. You need to let Microsoft know directly that this is an issue for you by opening a support case.


    Jason | https://home.configmgrftw.com | @jasonsandys

    Tuesday, January 14, 2020 2:11 AM
  • We have opened a support case with Microsoft today. I will post any findings here as soon as we have some ...
    Tuesday, January 14, 2020 8:30 PM
  • Yes please keep us posted. I'm still in the process of trying to find out who can open support cases in our organization or i would have one open for the exact same issues. 


    Tuesday, January 14, 2020 8:52 PM
  • Hi,

    Thanks for your reply. Looking forward to hearing from you. Thanks for your time.

    Best regards,
    Simon


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Wednesday, January 15, 2020 1:53 AM
  • I am also experiencing this issue since upgrading to 1910 slow ring, upgrade completes bug hungs at setupcomplete.cmd at the following line: "Running C:\windows\CCM\\TSMBootstrap.exe to resume task sequence" and it never resumes the task sequence. Bootstrap process has to be killed to get to the logon prompt.

    Gabriel Lopez

    Thursday, January 16, 2020 9:02 PM
  • Hello,

    Same issue :

    1. SCCM version & Client version : 1910
    2. Try to upgrade Windows 10 to 1909 (with Task Sequence) --> The Task Sequence is stuck at "Waiting for updates ... 100%..."

    We opened a Microsoft case, we are waiting for a response... (SCCM support overload...)

    I let you know as soon as I have a solution


    Florent

    Friday, January 17, 2020 6:32 AM
  • After talking to Microsoft support they advised us to first disable 3rd party Antivirus and then to test if the Inplace upgrade from 1709 to 1809 or 1809 to 1909 is any better. They suspected that the "big jump" from 1709 to 1909 is too big (however this has been working in October). 

    Having tested all these different scenarios I can unfortunately only confirm that NONE of this helped at all. It does not matter from or to which OS version I'm trying to upgrade or if 3rd party AV is installed or not. The inplace upgrade does not work at all, no matter what we try. Our support case is still open and I'll keep you up to date on the progress. 

    Seems like a severe SCCM bug to me ...

    Friday, January 17, 2020 2:56 PM
  • I can now prove this is a bug, after many failures I was able to test with a 1906 client on the same machine same environment, same task sequence. I created some exclusions to avoid the client from being upgraded to 1910 and the task sequence finished with no issues. Hopefully Microsoft releases a hotfix soon. :(

    Gabriel Lopez

    Friday, January 17, 2020 9:22 PM
  • Have you opened a support case with Microsoft to show them your issue and proof? If not, honestly, your proof and the work you putting into getting it is pretty much worthless. The only way Microsoft knows about issues is if customers report them through the official support channels.

    Jason | https://home.configmgrftw.com | @jasonsandys

    Friday, January 17, 2020 10:09 PM
  • This is not currently any type of known issue or bug at Microsoft regarding this issue. Applying KB4535819 will not fix the problem since there is no fix in KB4535819 for the issue (since the issue is currently not even known at Microsoft). To get the issue properly investigated and if needed, fixed, please make sure to open a support case with Microsoft. Posting here will not get the issue investigated or fixed.

    Are by any chance any of you swapping out or making any modifications to the SetupComplete.cmd script? If so can you disable swapping out the file/disable modifications/revert to the default SetupComplete.cmd file of 1910 and see if the issue goes away? The SetupComplete.cmd script was changed in 1910 to support the new SetupCompletePause variable in 1910:

    https://docs.microsoft.com/en-us/configmgr/osd/understand/task-sequence-variables?redirectedfrom=MSDN#SetupCompletePause

    If you are making any modifications to the SetupComplete.cmd file or swapping it out with an older version it could lead to issues like what you are seeing. Also take a look at the SetupComplete.log at the root of C:\Windows to see if it gives any indication as to why the issue is occurring.


    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation


    Saturday, January 18, 2020 1:09 AM
  • we have never modified setupcomplete.cmd in the past, so this should not be the cause for the error (at least for us) 
    Saturday, January 18, 2020 9:03 AM
  • While working with MS support and performing different tests, we now found out that

    uninstalling our Virusscan solution (McAfee Endpoint Security) beforehand makes the upgrade Task Sequence to smoothly run through again......

    Hope we're not stuck now in a ping-pong game between McAfee and MS.

    However, the strange thing is, that it perfectly worked with SCCM version 1906.

    No changes from the McAfee side have been implemented since then.

    Will keep you posted.

    Tuesday, January 21, 2020 1:51 PM
  • I strongly recommend that you reach out to McAfee and report the problem to them.

    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation

    Tuesday, January 21, 2020 1:55 PM
  • This is not in any way surprising. McAfee has been responsible for killing ConfigMgr in many, many environments, many, many times over the past 10 years or so. IMO, they take an overly aggressive approach that results in often stopping legitimate processes (like ConfigMgr).I know that doesn't help you explicitly, however, I would certainly expect this to be a root cause for more issues in the future (and even some current issues you didn't realize were related).

    Jason | https://home.configmgrftw.com | @jasonsandys

    Tuesday, January 21, 2020 1:56 PM
  • This is good info! I've been too busy with the OOB update this month to open a support call. I was trying to upgrade from Win 10 Ent 1809 to 1909. I did upgrade McAfee ENS to the "supported version" for Win 10 1909 but now I have to validate by removing ENS. I will give this a try and report back. Thank you!

    Gabriel Lopez

    Tuesday, January 21, 2020 4:18 PM
  • Removing McAfee ENS solved the issue. The weird thing is that it works fine with SCCM 1906 and Win 10 1909 but not with MECM 1910 and Win 10 1909.

    Gabriel Lopez

    Tuesday, January 21, 2020 11:06 PM
  • Uninstalled McAfee from the SCCM server solved the task sequence from stalling for me. Was looking at this since i upgraded to 1910. The moment i uninstalled McAfee the OSD deployment stated working again. 
    Wednesday, January 22, 2020 2:31 AM
  • Hi,

    Thanks for all your reply and sharing sharing. This may help the users who have similar issue. Thank you!

    Best regards,
    Simon

    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Wednesday, January 22, 2020 3:43 AM
  • Just to be sure.

    are we talking about the McAfee ENS installed on the clients, or on the SCCM Server, or a both components (Client+Server) bugged?

    Thanks!

    Wednesday, January 22, 2020 6:35 AM
  • Its the ENS on the client.

    Wednesday, January 22, 2020 10:05 AM
  • ENS on the client. In my case Endpoint Security 10.6.1 July Update Repost (v10.6.1.1666), that is listed on their website as compatible with Win 10 1909.

    Gabriel Lopez

    Wednesday, January 22, 2020 2:34 PM
  • I noticed a new update in our ConfigMgr console this morning. Has anybody else see/tried installing it yet. Wonder if the issues are still present after installing. 

    https://support.microsoft.com/en-us/help/4538166/client-update-for-configuration-manager-current-branch-version-1910

    Wednesday, January 22, 2020 2:44 PM
  • This currently is not any type of known issue or bug at Microsoft. As most evidence suggests it is McAfee inadvertently blocking something during the in-place upgrade. In other words there currently is no fix from the Microsoft side since this does not appear to be a Microsoft issue. Gor this reason any new updates that are showing up in the ConfigMgr console would have nothing to do with fixing this issue.

    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation

    Wednesday, January 22, 2020 3:32 PM
  • we have also been encouraged by MS support to remove our McAfee before the upgrade - which we did. Unfortunately this did not change anything. Removing ENS prior to the update did NOT resolve the problem.

    We even installed 1709 machines from scratch without McAfee and then tried the upgrade - and it failed with the 1910 client.

    1906 (though not perfect) worked somehow. 1910 broke inplace upgrade entirely - at least for us.

    It doesn't make a difference if McAfee is installed or not - at least for us.

    Wednesday, January 22, 2020 8:05 PM
  • Most likely you are running into a different issue than what is being discussed in this thread since we have had multiple confirmations that disabling/removing McAfee resolves the problem. I would encourage you to continue working with MS support on finding a resolution.

    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation

    Wednesday, January 22, 2020 8:12 PM
  • might be a different issue ..., however it's still not working for us - Therfore we're still in close discussion with MS support to get this fixed - whatever the reason might be, we need a solution or workaround to upgrade our machines before they'll be running out of support ...
    Wednesday, January 22, 2020 8:21 PM
  • I am not disagreeing with you that there is not an issue for you and agree that you need to be fixed. The proper way to get the issue investigated is through your support case. As mentioned this most likely is an environmental issue. In your case it does not appear to be McAfee. However currently there does not appear to be any type of bug or issue with the product. If there was a bug or issue in the product we would be getting a much higher reports of the problem which we are not. Again I am not saying that this is not a problem for you - just that this does not appear to be a general issue with the product. This should help you focus troubleshooting the issue to something environmental and which is different in your environment vs. the majority of other customers (for example McAfee was the cause of the issue for the above other customers).
    Wednesday, January 22, 2020 8:32 PM
  • totally agree with you

    will report here how we are proceeding ...

    Wednesday, January 22, 2020 8:38 PM
  • Morning all,

    I've been following this thread as we were experiencing Task Sequences stuck after the Setup Configuration Manager step and a reboot. I got a little dismayed when McAfee started appearing as we're not running AV. However, after running the latest hotfix: https://support.microsoft.com/help/4538488 KB4538488 and a reboot all of our Task Sequences are running as normal again.

    Hope this info help someone out.

    Thursday, January 23, 2020 9:12 AM
  • There were no fixes in the 4538488 hotifx regarding anything having to do with Task Sequences or in-place upgrades so probably just a coincidence. Possibly the problem could have been that the client was trying to be upgraded during the IPU Task Sequence which led to the problem, and why this hotfix could have potentially fixed it.

    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation

    Thursday, January 23, 2020 12:45 PM
  • McAfee refers me back to Microsoft.
    Is it recommended to exclude the following files from the SelfProtection?

    The McAfee self protection blocks the C:\Windows\CCM\CcmExec.exe and the C:\$WINDOWS.~BT\Sources\SetupHost.exe

    Log:

    2020-01-23 08:57:45.026Z|Activity|ApBl                |mfeesp                                  |      4492|     10088|SP                  |XModuleEvents.cpp(821)                  | NT-AUTORITÄT\SYSTEM hat C:\Windows\CCM\CcmExec.exe ausgeführt. Dieser Prozess hat versucht, auf den Prozess mfetp.exe zuzugreifen, hat damit die Regel "Kernschutz – Schutz von McAfee-Prozessen vor unautorisiertem Zugriff und Abbruch" verletzt und wurde daher blockiert. In KB85494 wird erläutert, wie Sie auf dieses Ereignis reagieren können.


    2020-01-23 09:26:50.696Z|Activity|ApBl                |mfeesp                                  |      4492|     10096|SP                  |XModuleEvents.cpp(821)                  | NT-AUTORITÄT\SYSTEM hat C:\$WINDOWS.~BT\Sources\SetupHost.exe ausgeführt. Dieser Prozess hat versucht, auf die Datei C:\Windows\System32\drivers\mfeepmpk.sys zuzugreifen, hat damit die Regel "Bedrohungsschutz – Schutz von McAfee-Treibern" verletzt und wurde daher blockiert. In KB85494 wird erläutert, wie Sie auf dieses Ereignis reagieren können.


    • Edited by unblack Friday, January 24, 2020 11:04 AM
    Friday, January 24, 2020 10:51 AM
  • What was the technical reason they referred you back to Microsoft? Wouldn't excluding files from protection be a setting in their product? Did you explain to them that the IPU works just fine if McAfee is not enabled?

    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation

    Friday, January 24, 2020 2:32 PM
  • If they are being blocked, then yes, as these are certainly critical for the success of the upgrade -- ccmexec is critical for the operation of the client in general as well.

    As a note here, this isn't the first time, it's actually just the most recent in a string of many times going all the way back to ConfigMgr 2007 and 2012 that AV products (the one called out in this thread in particular) have caused issues by blocking known good processes in ConfigMgr.

    I know that doesn't solve the issue in any way, but it's just reality.


    Jason | https://home.configmgrftw.com | @jasonsandys

    Friday, January 24, 2020 2:34 PM
  • Yes as you can imagine I am overly familiar with 3rd party AV products causing issues with ConfigMgr. I am trying to understand why McAfee is referring the customer back to Microsoft when the issue is clearly with their product, and the information given to the customer seems to be some configuration within the McAfee product. I am not sure what McAfee wants us Microsoft to do regarding the information they gave to the customer. If McAfee had some technical analysis and data that shows it may be something on the Microsoft side, then we will gladly take a look at that data and further troubleshoot.

    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation

    Friday, January 24, 2020 2:58 PM
  • We are also still having the same problem.  We were already uninstalling McAfee ENS with previous upgrades, and it worked fine.  However, after upgrading to 1910, uninstalling McAfee does not fix the problem.  After reading this thread, I modified to have the task sequence uninstall ALL McAfee products before the upgrade starts, and verified this was done.  The OS Upgrade still hangs up.  Removing McAfee made no difference.  The ONLY thing I have found that works is to downgrade the SCCM client to 1906 first, then run the OS Upgrade task sequence.  

    Friday, January 24, 2020 4:19 PM
  • As mentioned with the other customer above, it may be a different issue.  Currently there is no known issue with the product so I would recommend opening a case with Microsoft support so that we can fully investigate.

    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation

    Friday, January 24, 2020 4:28 PM
  • Update:

    i tried the tasksequence with the exclusion.

    In did not work :-(

    There must be something else in the McAfee ENS.

    Even if I deactivate all modules, but do not uninstall it does not work.

    So i have to contact McAfee again.

    Saturday, January 25, 2020 7:26 AM
  • in our case we also entire removed McAfee from machines before starting the upgrade and the upgrade still fails. Seems to be a tricky combination that breaks things:

    • SCCM 1906 Agent + McAfee installed = Upgrade OK
    • SCCM 1910 Agent + McAfee installed = Upgrade BROKEN
    • SCCM 1910 Agent + McAfee removed = Upgrade BROKEN

    Something must have changed in the SCCM agent from 1906 to 1910 that the upgrade process is no longer working for us ...

    Saturday, January 25, 2020 5:18 PM
  • Of course something changed, it's an upgraded version and on average all new versions of ConfigMgr contain 3,000+ changes. That's the entire point of upgrading. As Frank has called out, if it works without McAfee installed (which it clearly does), then the issue is something on McAfee's side.

    For your third line item above, as Frank has also called out, you need to work with Microsoft support to identify the issue in your environment . It's certainly possible that this is a bug, defect, design flaw, etc in ConfigMgr, but without hand-on investigation and troubleshoot to identify the source, it's just as likely that's its an environmental issue. No one in this thread, at this point, can tell you which.


    Jason | https://home.configmgrftw.com | @jasonsandys

    Saturday, January 25, 2020 10:10 PM
  • As Jason pointed, yes every version of ConfigMgr has changes in it. That is the point of an upgrade - to add functionality and fix known bugs. I can say that one of the changes/additions done around the point of failure is the addition of the SetupCompletePause variable and the SetupComplete.cmd script being different in 1910 vs. previous versions. That is why early on in the thread I recommended taking a look at that.

    With that said, with the complexity of a product like ConfigMgr and a process like an In-Place Upgrade, an issue like this is not going to be solved via forum posts. This is why I am making the recommendation to open a case with Microsoft support. A customer earlier in this thread did exactly that and they found the cause of the issue for them - McAfee. We probably would have never been able to uncover that through forum posts. In your instance the cause of failure may be something else, but again without opening a case and allowing Microsoft support properly investigate it, we will never know.

    Remember that when Microsoft tests the product we do so with basic out of box documented functionality. We do not test with 3rd party products. Maybe some change in the product did break it working with McAfee, but it is up to McAfee to investigate that and if necessary, reach out to Microsoft so that we can work with them in fixing whatever the problem is.

    One suggestion regarding McAfee. In my experience it is sometimes not enough to just disable McAfee or add exclusions to fix problems like this. You have to completely uninstall and remove all McAfee components before it starts working. I know some of you have already taking this action (completely uninstalling McAfee), but for those who have not I would recommend taking this action to narrow down the problem. Again we are not suggesting this as the permanent fix but instead as a testing/troubleshooting step to narrow down what is causing the issue.

    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation

    Sunday, January 26, 2020 1:00 AM
  • We already have an official case open with Microsoft for some weeks now. I will post our progress here and once we have a solution, I'll let everyone participate. 
    Sunday, January 26, 2020 2:40 PM
  • Send your case # to frankroj@microsoft.com and I will follow up with the case owner.

    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation

    Sunday, January 26, 2020 10:00 PM
  • I have narrowed our environment down to the ENS Threat Prevention module being installed. When we removed Threat Prevention from the add remove programs before the upgrade started the upgrade task seems to work like normal. 

    • SCCM 1906 Agent + McAfee installed = Upgrade OK
    • SCCM 1910 Agent + McAfee installed = Upgrade BROKEN
    • SCCM 1910 Agent + McAfee installed with no ENS TP module = OK

    I tried adding the exclusions recommended by McAfee https://community.mcafee.com/t5/Endpoint-Security-ENS/SCCM-1910-Win10-In-Place-Upgrade-Tasksequence-failing-if-ENS-is/m-p/647533#M7225  but the only work around that I have found so far is to remove Threat Prevention. 

    Monday, January 27, 2020 1:04 PM
  • Thanks for the update and the info Jason.

    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation

    Monday, January 27, 2020 1:50 PM
  • I can confirm that this is happening in my environment as well. I've tried both v10.6 and v10.7 of Threat Prevention and our Task Sequence (fresh install, not upgrade) is failing. As soon as I remove Threat Prevention, it works fine. If I use Latest versions of McAfee VSE instead of McAfee ENS, then it works fine.

    We added an exclusion for ccmexec.exe, but this didn't help.

    In our situation, after the final reboot in the Task Sequence, just before it goes to 'Report Done' and shows Deployment Complete screen, it sits at the 'spinning dots' part of the boot forever. On some machines, it sits at the spinning dots and then reboots and goes back to spinning dots, and reboots etc....

    Ourbuild works perfectly with SCCM 1906, but then fails with SCCM 1910. 

    Versions we are using when build fails -

    Windows 10 1903 / 1909

    SCCM 1910

    McAfee ENS 10.6 or 10.7

    • Edited by Roger_H Monday, January 27, 2020 2:36 PM
    Monday, January 27, 2020 2:32 PM
  • If you add exclusions for all of the exe's in the C:\Windows\CCM folder does it work? In addition to CCMExec.exe at the very least also exclude TSMBootstrap.exe, TsProgressUI.exe, TsManager.exe, OSDUpgradeOS.exe, smsappinstall.exe, smsswd.exe,  ccmrepair.exe, tsenv.exe, and TSInstallSWUpdate.exe. CCMExec actually does not come into place until after the Task Sequence resumes after the IPU is done and the client is repaired.

    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation

    Monday, January 27, 2020 3:19 PM
  • Thanks for reply - will try it (assuming I can get security team to approve the exclusions :-) ). Only reason I excluded this is because on my machine (not a new build or upgrade), I noticed that ENS was blocking ccmexec.exe the day after I upgraded SCCM, and looking back through logs, it seems to do the same thing, the day after each SCCM upgrade for the last 12 months.
    • Edited by Roger_H Monday, January 27, 2020 3:25 PM
    Monday, January 27, 2020 3:22 PM
  • Roger I noticed you mentioned your issue is with a Fresh OSD, do you happen to have any computer restarts in your task after the Setup Configuration Mgr Client step? 

    We had to move our McAfee steps to the very last step and disable all computer reboots in those steps to make our task sequence finish. 

    Monday, January 27, 2020 4:31 PM
  • FYI thread going on over at McAfee:

    https://community.mcafee.com/t5/Endpoint-Security-ENS/SCCM-1910-Win10-In-Place-Upgrade-Tasksequence-failing-if-ENS-is/td-p/647052

    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation

    Monday, January 27, 2020 4:37 PM
  • As mentioned several times above, that hotfix does not contain a fix for this issue. That fix fixes a completely different issue as described in the KB article. At the current time, there is no issue to fix from the Microsoft side. This currently completely appears to be a McAfee issue. Microsoft cannot provide a fix for something that there is no issue with from the Microsoft side.

    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation

    Tuesday, January 28, 2020 3:08 PM
  • Hello, 

    We tried with the hotfix --> https://support.microsoft.com/en-us/help/4538166/client-update-for-configuration-manager-current-branch-version-1910

    OSD Task Sequence and Upgrade in place doesn't work with McAfee.

    It works without Mcafee

    Our Microsot case has been escalated and a Technical Advisor from the support team is currently reviewing it 


    Florent

    Tuesday, January 28, 2020 7:55 PM
  • Well installing a hotfix that isn't meant to correct the problem you are having will never fix the problem.

    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation

    Tuesday, January 28, 2020 8:05 PM
  • Also I don't understand why you are escalating the case with Microsoft? A TA or anyone above them at Microsoft is not going to be able to help you. This is an issue with McAfee. You need to open a case with McAfee and work with them.

    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation

    Tuesday, January 28, 2020 8:06 PM
  • @Frank Rojas :

    • Microsoft support asked us to try with the hotfix
    • We also have a case with McAfee support

    Florent


    • Edited by Florentflo Tuesday, January 28, 2020 8:14 PM
    Tuesday, January 28, 2020 8:13 PM
  • Please send me your case # to frankroj@microsoft.com. I will make sure that this is addressed with the engineer who gave you that action plan. I apologize for the erroneous action plan.

    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation

    Tuesday, January 28, 2020 8:16 PM
  • Roger I noticed you mentioned your issue is with a Fresh OSD, do you happen to have any computer restarts in your task after the Setup Configuration Mgr Client step? 

    We had to move our McAfee steps to the very last step and disable all computer reboots in those steps to make our task sequence finish. 

    Hi. In our TS, the last reboot is the last step in the 'OSD Results and Branding' folder. The build freezes after this reboot with the spinning dots. If I put the McAfee install steps AFTER this final reboot, then the build finishes. However, as this area is outside of the 'main task sequence', I am concerned that if (for some reason) McAfee applications fail to install, the build will still report as 'Deployment Complete', and our policy is that if McAfee fails for whatever reason, the build must error and report as failed.

    To explain.... This TS hangs up at spinning dots

    And this TS works fine

    Wednesday, January 29, 2020 11:52 AM
  • Hi - I tried exclusions of all EXE files C:\Windows\CCM under CCM and the same happens. From what I can see in the McAfee logs, even though McAfee seems to be blocking something, or stopping something from working, it's not being logged in the McAfee Activity logs.

    Another thing I noticed - if I use my build on a VM, using Offline Media in the form of an ISO image, hosted on a fast SSD drive, then the build works without any issues. So it seems that McAfee ENS on it's own, with default settings does not cause issues, but once the agent dowenloads and applies the policies from ePO, then it starts causing issues. I suspect that in the case of the successful VM, the final reboot happens too quickly for the ePO Agent to get its policy and apply it, because if I use the exact same VM, n the same host, but use an 'online build' using a local DP, then the build takes longer (as expected) and fails as well.

    Wednesday, January 29, 2020 11:56 AM
  • Roger what you could also do is duplicate your McAfee ENS group and copy it down under the Gather Logs and StateStore on Failure. The way an MDT integrated Task Sequence is designed is that it will drop down into that group if anything above it fails. Although this will not 100% guarantee that the McAfee agent will be installed, it will at least install it in most cases when something in the Task Sequence fails.

    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation

    Wednesday, January 29, 2020 1:58 PM
  • We now have also a support case open with McAfee. Their first response was unfortunately not very promising ("it's due to Microsoft has changed something, please ask Microsoft, .... bla bla").

    We also tried the latest 10.7 McAfee version which has the same error. Would be interesting to hear, how others are proceeding in their support case with McAfee. 

    If we don't get a solution within the next month we will probably roll back the SCCM client to good old 1906 version on all our  Windows 10 1709 machines so that we can continue with the inplace upgrade regardless of the issue, before this Windows edition goes out of support ...

    Friday, January 31, 2020 12:09 PM
  • You can tell McAfee that yes we definitely did change something. In fact we changed a lot of things. That is what an upgrade does. We could go through and tell you what all of this changes were, for example the following article:

    https://docs.microsoft.com/en-us/configmgr/core/plan-design/changes/whats-new-in-version-1910

    but that does nothing to advance fixing the issue nor pinpoints which one of those changes the McAfee product is blocking or having issues with. It is up to them to make their product compatible with our updated products so they should be investigating what their product is doing to block ConfigMgr.

    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation

    Friday, January 31, 2020 1:42 PM
  • you're right Frank, it won't fix anything and as I'm not expecting a fix from McAfee to be available any time soon, downgrading to the old SCCM client seems the only viable solution for now ... which is sad, but the only way for now to move forward. 
    Friday, January 31, 2020 3:41 PM
  • I'm not expecting a resolution anytime soon either, but I have no idea how to downgrade my SCCM client. What I'm going to try is to use McAfee VSE Patch 13 in the build, and then use ePO to upgrade to VSE to ENS.
    Friday, January 31, 2020 3:44 PM
  • All I am working with another engineer on a case where McAfee wants to do a conference call with Microsoft. I am working on being part of that call. I will update you with the status after that call.

    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation

    Friday, January 31, 2020 3:51 PM
  • can't wait to get an update after that call
    Friday, January 31, 2020 7:24 PM

  • I think I have found what was causing the issue for us.  The Task Sequence was uninstalling ENS successfully before the OS upgrade.  Then the OS upgrade runs, and hangs at 100%.  I was able to RDP into the client, and noticed that ENS was reinstalled.  This was because in EPO, it was set to evaluate at each agent-server communication.  So, EPO was apparently reinstalling ENS during the upgrade, as the McAfee Agent was still installed.  I unchecked the 'evaluate at each agent-server communication' for ENS and made sure that the 'Install ENS' task would only run at a time when we will not be running upgrades.  I then moved my 'Install ENS' task to the very end of the TS.  That seems to have resolved the issue.  I am still testing, but other than downgrading the client to 1906, this is the only workaround I have found.  The confusing part is that this setting is not an issue with the 1906 client.
    Friday, January 31, 2020 10:11 PM
  • Thank you for the information and for sharing Jamey. This is good info to know and hopefully will help other customers. My guess is that McAfee does not like something about one of the changes/additions we did to the 1910 client. What specifically that is I am not sure.


    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation


    Friday, January 31, 2020 10:25 PM
  • Good point. We also tried to uninstall McAfee before the upgrade, but it didn't help, without an additional reboot before starting the upgrade. However this is kind of ruining the entire upgrade procedure for the end user. The ideal solution would be that it simply works with McAfee as it did using the 1906 client and nothing to care about. We're currently preparing to downgrade the old 1906 SCCM client to our machines as a pre-project to the inplace upgrade - technically not ideal, but currently I see no other chance to move forward
    Saturday, February 1, 2020 12:47 PM
  • > The ideal solution would be that it simply works with McAfee

    This statement is backwards. ConfigMgr does not and should not work *with* McAfee. That's simply not how it works. McAfee should with and not interfere with the proper operations of applications on the systems.

    Why would a legitimate application change its behavior to accommodate the functionality of McAfee? That simply makes no sense and is reinforced by the fact that no other AV products are causing this issue. The entire point of an AV product is to monitor a system and block malicious behavior and activity. If it (the AV product) is blocking non-malicious behavior or activity, then it is what is and failing and what is the source of the issue. Kind of like a lock or security guard that doesn't let you into your own home -- what's the point in even having the home or security guard/lock at that point?


    Jason | https://home.configmgrftw.com | @jasonsandys

    Saturday, February 1, 2020 11:47 PM
  • @Jason: Please excuse, my statement might have been misunderstood or not well explained. It was not meant to be backwards and I'm not expecting Configmgr to follow the McAfee standards (as you might have suspected). My expectation is (exactly what you mentioned): The upgrade process should work with McAfee (or any other AV product) installed, meaning that McAfee should be aware of distinguishing good and bad activity going on - nothing else. I should have pointed that out more clearly to avoid confusion.

    Our focus is now clearly on McAfee to solve this finally and as long as there is no final solution, we are simply looking for workarounds.

    From a pure Microsoft standpoint we might end the discussion here at this point as it's not up to Microsoft to solve this and come back here, once McAfee has released a solution.


    Monday, February 3, 2020 8:20 AM
  • Update from my side... McAfee has asked for Failed VM Dump. They are analyzing and will share the results tomorrow. We tried with complete McAfee Disable but still the result is the same. No Luck.

    Only uninstalling AV is giving the positive result. 


    Rohit

    Tuesday, February 4, 2020 2:43 PM
  • I have my call with McAfee early tomorrow morning.

    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation

    Tuesday, February 4, 2020 2:44 PM
  • Unfortunately the call this morning did not produce much results. However we have agreed to look into possibly doing some data gathering for McAfee to assist them with determining why their product is causing the Task Sequence to fail. I will update as I have more info.

    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation

    Wednesday, February 5, 2020 3:50 PM
  • Thank You Frank Rojas for your Support!!
    Thursday, February 6, 2020 7:51 AM
  • Just got here from the McAfee Community, where they were only finger pointing at Microsoft.. even though it is very obvious who should be fixing this.

    Will definitely follow this thread though. Good luck!

    Thursday, February 6, 2020 4:59 PM
  • I can confirm that removing McAfee solved the same issue in our environment. We did open a support call with Microsoft, and they put the onus on Mcafee, even thought the upgrade to 1910 is what started the problem.  OSD now working again.

    DeployGuy

    Friday, February 7, 2020 2:16 PM
  • For our situation we have "solved" this for now by removing ENS before the Inplace Upgrade (via McAfee removal tool) and finally reinstalling ENS as the last step of the Inplace Upgrade Task Sequence, when the entire upgrade stuff is done. 

    Once, McAfee comes out with a solution we might consider that. However for now we're fine doing it this way, though it might not seem the ideal way and putting additional delays into the entire upgrade process for our users.

    The tricky thing about this issue is, that it seems not just simply to be a "Windows 10 vs. McAfee" issue, but instead turns out to be somewhat of an unlucky triangle of "Windows 10 + SCCM 1910 Task Sequence vs. McAfee" not working nicely together, maybe it's just simply "McAfee no longer working with SCCM Inplace Task Sequences after 1910"....

    Btw.: Upgrading Windows 10 with McAfee ENS installed works flawlessly, when there is no SCCM Task Sequence involved. 

    Sunday, February 9, 2020 9:55 PM
  • This workaround seems to work for us too.

    We were planning to look at alternatives for McAfee in the second half of this year anyway, because we had alot of strange problems with it lately and ePO is still running on an old 2008R2 server and needs a migration.
    This made us change the priority and we will start ASAP.

    Tuesday, February 11, 2020 9:14 AM
  • Did anyone have a workaround for a Fresh OSD deployment?, I did place McAfee last in our task sequence but I get the never ending spinning dots at the end of the build (never connects back to ConfigMGR).

    Is it worth opening another case with Microsoft? or would they put the onus back on McAfee support.

    Thursday, February 13, 2020 10:46 PM
  • (Tagging the discussion because we have the same issue)

    @josti, it seems that the current workaround is not to install McAfee ENS in the OSD if you have Endpoint Manager 1910.

    Seems that McAfee is still good at blaming others instead of checking what's wrong.
    That's why, when you create a case with Microsoft, they often ask you if you can reproduce the issue without the AV installed ^^
    Friday, February 14, 2020 9:12 AM
  • We have opened case with McAfee since 3 weeks now, They asked for VM dumps and there engineering team is analyzing the dumps, It's been 2 weeks and now they have extended the follow up to 24th Feb. 

    Hope by 24th we get some news with an update from McAfee.


    Rohit

    Monday, February 17, 2020 5:45 AM
  • Just tested the latest 10.7 released for February 2020, still the same issue...
    The Security Center bug seems fixed (Where it report that ENS is not installed even if it is) so that's at least something good...

    Gérald

    Monday, February 17, 2020 2:40 PM
  • We installed the latest rollup for 1910, KB4537079 and so far so good. Our Win10 TS now continue after the ENS install with the 5.0.0.8913.1032 SCCM client. 

    https://support.microsoft.com/help/4537079

    • Proposed as answer by josti Monday, February 24, 2020 10:00 AM
    • Unproposed as answer by josti Monday, February 24, 2020 10:00 AM
    • Proposed as answer by josti Monday, February 24, 2020 10:00 AM
    Monday, February 24, 2020 5:46 AM
  • What version of ENS you are using?

    Rohit

    Monday, February 24, 2020 7:32 AM
  • We also installed the newest rollup for ConfigMgr 1910 KB4537079 and has seemed to fix all our McAfee Upgrade task failures. We have tested both versions Endpoint Security 10.6.1 and 10.7.0 since the KB installed and haven't seen any failures. 


    6.7.0
    • Proposed as answer by nolajp Thursday, February 27, 2020 5:27 PM
    Tuesday, February 25, 2020 12:14 AM
  • Hi all,

    I can also confirm that the latest hotfix 4537079 made our upgrade TS work again.

    So much for the MS statement "its not us, its McAfee".

    I'd figure sometimes it was better they'd clean up there own backyard....

    best regards

    Wednesday, February 26, 2020 12:47 PM
  • KB4537079 fixed the issue, Don't understand why MS was not accepting the bug and kept on blaming McAfee. It was complete waste of time logging the ticket and following up with multiple test.

    Rohit

    Thursday, February 27, 2020 6:33 AM
  • I completely agree. This has caused us weeks of wasted work and we have been heavily told dozens of times not to blame Microsoft.

    I'd love to hear back now from all those defending Microsoft's position so heavily in the past, what exactly the reason was and if we can expect that this does not happen again when the next SCCM version comes out.

    Anyway, from now on we will rollout any new SCCM updates VERY reluctantly in the future.

    Thursday, February 27, 2020 9:59 AM
  • Still can't be ignored, that this only happened with McAfee and not any other Antivirus solution...
    Monday, March 2, 2020 7:38 AM
  • We have a similar issue.

    Upgraded to SCCM 1910, and now our Inplace Upgrade Task Sequence rollback each time (1809>1909) (smsts.log shows the Upgrade part return a successful 0 code, but its on the first reboot, that suddenly it goes into rollback.

    TS is a standard one from SCCM wizard.

    We run Symantec, have tried uninstalling that, upgrade fails

    Installed the hot fix KB4537079, and upgraded all clients , upgrade fails.

    The only way we can get the upgrades to work is to disable Secure Boot. But obviously this isnt workable with hundreds of laptops.

    Any other ideas i can try?

    thanks

    Monday, March 2, 2020 11:24 PM
  • Sound like another issue than the one we are talking here...

    Is your upgrade failing with an error 0xC1900101 - 0x20017 ?
    Tuesday, March 3, 2020 8:56 PM
  • Microsoft has had reports that the HFRU for 1910 (KB4537079) has "fixed" the issue for customers. However from the Microsoft side we did nothing in KB4537079 that specifically addresses this issue mainly because there was never any evidence that this was an issue that was caused by anything in Microsoft code. This was never a bug from the Microsoft side.

    Our theory is that something in the original 1910 code matched a series of bits that triggered the McAfee software to block the ConfigMgr Task Sequence and not let it continue. With the HFRU update in 1910 and binaries being recompiled, whatever series of bits that McAfee did not like previously were changed enough so that McAfee stopped blocking the Task Sequence.

    To clarify this was never a problem with the in-place upgrade itself. That worked just fine. This was a problem with McAfee blocking the Task Sequence from continuing after the in-place upgrade was done. This is why running the in-place upgrade manually outside of a Task Sequence "worked". McAfee was not blocking the in-place upgrade - it was blocking the Task Sequence.

    Microsoft in general does not test their software against 3rd party solutions including 3rd party AV. If the process did not work or the issue still occurred without McAfee installed, then Microsoft would have definitely taken the lead in investigating the issue. However the fact that the issue only occurred when McAfee was installed is the reason Microsoft was asking customer to involve McAfee to lead the investigation into the issue.

    I am happy to hear that the issue does seem to be finally resolved by the HFRU, but to reiterate this was not anything done on Microsoft's side on purpose. This was some type of race condition caused by McAfee and something in the original 1910 code that McAfee did not like and ended up blocking. Fortunately those conditions no longer exist in 1910 HFRU.

    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation

    Wednesday, March 4, 2020 5:26 PM
  • We have a similar issue.

    Upgraded to SCCM 1910, and now our Inplace Upgrade Task Sequence rollback each time (1809>1909) (smsts.log shows the Upgrade part return a successful 0 code, but its on the first reboot, that suddenly it goes into rollback.

    TS is a standard one from SCCM wizard.

    We run Symantec, have tried uninstalling that, upgrade fails

    Installed the hot fix KB4537079, and upgraded all clients , upgrade fails.

    The only way we can get the upgrades to work is to disable Secure Boot. But obviously this isnt workable with hundreds of laptops.

    Any other ideas i can try?

    thanks

    I would recommend you open a new thread on this issue or a case with Microsoft since this issue is different than what is being discussed in this thread. The issue in this thread is in regards to McAfee blocking the Task Sequence from continuing after the in-place upgrade is done.

    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation

    Wednesday, March 4, 2020 5:28 PM