none
1909 Feature Update woes RRS feed

  • Question

  • Hi all,

    now that Windwos 10 v 1909 is out, we have started preparing our environment (SCCM Current Branch 1906) to deploy this new version and updating existing clients. 

    OSD Task Sequences for new machines with 1909 seem to run pretty smooth (after figuring out some Language Pack annoyances). However, we are facing issues with SCCM Feature Update Task Sequences that bomb out after the base os installation. We are trying to update Windows 1709 and also 1809 versions. This is what we have encountered so far:

    The Task Sequence for the OS upgrade starts fine, performs a few expected reboots, then displaying "Windows is working in updates 0%, 15%, ..." When it reaches 100%, this Windows Upgrade step remains at 100% for a long long time and then suddenly shows the Windows login screen. All TS Post Processing steps (i.e. Language Packs, etc.) are skipped.

    SMSTS.LOG indicates, that the base OS upgrade seems to have finished without errors:

    • Successfully completed the action (Upgrade Operating System) with the exit win32 code 0

    After that we can discover several "Start top compile TS policy" steps, but suddenly the following errors happen:

    <![LOG[Failed to initate policy evaluation for namespace 'root\ccm\policy\machine', hr=0x80041010]LOG]!><time="10:07:10.271-60" date="11-25-2019" component="TSManager" context="" type="3" thread="6284" file="utils.cpp:5007">
    <![LOG[Error compiling client config policies. code 80041010]LOG]!><time="10:07:10.271-60" date="11-25-2019" component="TSManager" context="" type="3" thread="6284" file="tsmanager.cpp:1577">
    <![LOG[Task Sequence Manager could not initialize Task Sequence Environment. code 80041010]LOG]!><time="10:07:10.271-60" date="11-25-2019" component="TSManager" context="" type="3" thread="6284" file="tsmanager.cpp:1654">
    <![LOG[Task sequence execution failed with error code 80041010]LOG]!><time="10:07:10.271-60" date="11-25-2019" component="TSManager" context="" type="3" thread="6284" file="tsmanager.cpp:1305">
    

    After that the Task Sequence stops any further processing without showing any errors. Digging through all other SCCM logs I have found out that at the time time when the error above happens there is the following in CCMEXEC.LOG:

    <![LOG[Detected faulty configuration. Repair will be started in 5 minutes.]LOG]!><time="10:07:11.521-60" date="11-25-2019" component="CcmExec" context="" type="3" thread="6560" file="entrypoint.cpp:3079">
    <![LOG[Entering main message loop.]LOG]!><time="10:07:11.521-60" date="11-25-2019" component="CcmExec" context="" type="1" thread="6560" file="entrypoint.cpp:3104">
    <![LOG[WM_QUIT received in the main message loop.]LOG]!><time="10:12:11.521-60" date="11-25-2019" component="CcmExec" context="" type="1" thread="6560" file="entrypoint.cpp:3121">
    <![LOG[Initiated repair of configuration. Service will now exit.]LOG]!><time="10:12:11.599-60" date="11-25-2019" component="CcmExec" context="" type="1" thread="6560" file="entrypoint.cpp:3130">
    <![LOG[Shutting down CCMEXEC...]LOG]!><time="10:12:11.599-60" date="11-25-2019" component="CcmExec" context="" type="1" thread="6560" file="shutdown.cpp:53">
    

    This looks to me, that immediately after the OS upgrade, the system seems to start a repair of the SCCM client (for unknown reason). This repair seems to break the entire Task Sequence. 

    have triple-checked countlessly that the SCCM client is healthy before the upgrade but something seems to happen during the 1909 upgrade that makes Windows trying to repair the SCCM client. 

    Any thoughts what might go wrong here? 

    Thanks in advance

    Regards

    Udo



    Monday, November 25, 2019 7:41 PM

All replies

  • Hi Udo,

    This may be a timing issues with the Window 10 in-place upgrade task sequence on high performance devices when Windows setup is complete. Please help try to add an additional task sequence step with a 5 minute wait to allow the client fully functioning after the Setup Windows and ConfigMgr step. 

    This issue may be fixed with a new task sequence variable SetupCompletePause in next SCCM release. Here is the new feature in SCCM technical preview 1910:


    For more information, please refer to: New variable for Windows 10 in-place upgrade

    Thanks for your time.

    Best regards,
    Simon

    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Tuesday, November 26, 2019 3:16 AM
  • This new variable "SetupCompletePause" is unfortunately not yet available for us (we are on Configmgar 1906 (1910 is not yet available). Is there any release date already known, when this will become available?

    How would I otherwise add a 5 minute delay then? You mentioned to add a 5 minute delay after the "Setup Windows and Configmgr" step. However, as we have an Upgrade TS, there is no such step. The last thing that gets executed in the Task Sequence is the "Upgrade Operating System" step. The TS gets no further, which means that putting a 5 min delay command AFTER the Upgrade Operating System step, will never be executed. 

    The strange thing that is happening, is in the CCMEXEC.LOG, that is detecting a "faulty configuration", which seems to trigger the client repair. Any ideas why the faulty configuration might be detected? We have checked CCM client health throroughly before starting the Upgrade?

    Any other ideas?

    Wednesday, November 27, 2019 9:52 AM
  • Hi,

    Thanks for your reply.

    May we know if the problematic device are running the latest CM agent of SCCM 1906(5.00.8853.1006)? If not, please upgrade the CM agent to have a try.

    Thanks and regards,
    Simon

    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Thursday, November 28, 2019 8:58 AM
  • All clients are running the latest 1906 SCCM client (exactly the version you mentioned), no further upgrade necessary. 
    Friday, November 29, 2019 7:32 PM
  • Hi,

    Thanks for your reply.

    I will consult senior resource for the confirmation and do more research about this question, if there is any update, I will let you know. Thanks for your understanding.

    Best regards,
    Simon

    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Monday, December 2, 2019 2:57 AM
  • In the meantime we have updated to SCCM 1910 and tested the new variable - unfortunately without success. However the error 80041010 seems to be gone, albeit the Task Sequence is still bombing out and never completes the OS Upgrade step.

    What I could find out so far:

    • 1909 setup is finished successfully with errorlevel 0
    • setupcomplete.cmd is started. setupcomplete.log logs the following entry:
    Wed 01/08/2020- 8:28:42.78 Running C:\WINDOWS\CCM\\TSMBootstrap.exe to resume task sequence  
    • 2 seconds later smsts.log says the following:
    Waiting for CcmExec service to be fully operational    TSManager    08.01.2020 08:28:45    5796 (0x16A4) 
    • Hereafter ccmexec detects a "faulty configuration and tries to initiate a ccmrepair
    • After this point ccmsetup.log indicates, that ccmsetup seems to be running. After 3 minutes the last entry in ccmsetup.log is:
    CcmInitializePolicy. Sets and initializes default policy.
    • At this point the system does basically nothing for ~25 minutes (ccmsetup seems to be still running the whole time)
    • After this long 25 minutes delay ccmsetup seems to continue logging
    CcmBootstrapSqlCEDatabases. Create Sql CE databases that were queued by CcmBootstrapSqlCEDatabasesInit.
    • After lots of more logging entries ccmsetup completes successfully with errorlevel 0
    • A few seconds later smsts.log is telling us the following:

    Timed out waiting for CcmExex service to be fully operational    TSManager    08.01.2020 08:59:02    5796 (0x16A4)
    Failed to wait for CcmExec service to be fully operational (0x80004005)    TSManager    08.01.2020 08:59:02    5796 (0x16A4)
    Cleaning up any active Task Sequence requests in WMI.    TSManager    08.01.2020 08:59:02    5796 (0x16A4)
    Deleting CCM_TSExecutionRequest instance: CCM_TSExecutionRequest.RequestID="86f1058d-c87c-43c6-a1ef-60a06030d272".    TSManager    08.01.2020 08:59:02    5796 (0x16A4)
    Deleting CCM_TSExecutionRequest instance: CCM_TSExecutionRequest.RequestID="69257602-dcad-43f0-b90f-ffb4e0de5e73".    TSManager    08.01.2020 08:59:02    5796 (0x16A4)
    Removed 2 instances of CCM_TSExecutionRequest for task sequence.    TSManager    08.01.2020 08:59:02    5796 (0x16A4)
    Cleaning up any maintenance task requests in WMI.    TSManager    08.01.2020 08:59:02    5796 (0x16A4)
    Deleting SMS_MaintenanceTaskRequests instance: SMS_MaintenanceTaskRequests.TaskID="{69257602-DCAD-43F0-B90F-FFB4E0DE5E73}".    TSManager    08.01.2020 08:59:02    5796 (0x16A4)
    Removed 1 instance of SMS_MaintenanceTaskRequests for tasksequence.    TSManager    08.01.2020 08:59:02    5796 (0x16A4)
    Start to cleanup Task Sequence policy    TSManager    08.01.2020 09:00:53    5796 (0x16A4)
    Wmi query 'select * from CCM_Policy where PolicySource = 'CcmTaskSequence'' failed, hr=0x80041032    TSManager    08.01.2020 09:05:58    5796 (0x16A4)
    End Task Sequence policy cleanup    TSManager    08.01.2020 09:05:58    5796 (0x16A4)
    RegQueryValueExW is unsuccessful for Software\Microsoft\SMS\Task Sequence, SMSTSEndProgram    TSManager    08.01.2020 09:05:58    5796 (0x16A4)
    GetTsRegValue() is unsuccessful. 0x80070002.    TSManager    08.01.2020 09:05:58    5796 (0x16A4)
    End program:     TSManager    08.01.2020 09:05:58    5796 (0x16A4)
    Error Task Sequence Manager failed to execute task sequence. Code 0x8000ffff    TSManager    08.01.2020 09:05:58    5796 (0x16A4)

    • At this point the upgrade suddenly shows the Windows logon screen.
    • In Software Center I can see that the OS Upgrade has failed with error 0xFFFF(65535)

    We have spent countless hours of troubleshooting, unfortunately we don't get a single OS upgrade to complete successfully. We have a few hundreds of 1709 machines that need to be updated very soon and are completely stuck a this point.

    Any ideas or help is highly appreciated.

    Thanks Udo

    Wednesday, January 8, 2020 9:13 AM
  • here's another snippet from smsts.log. We cannot get a single Update to complete as the SCCM client is always trying to repair immediately after the OS upgrade completes. I have not found anything regarding the error 8000FFFF in combination with smsts.log.

    Successfully registered Task Sequencing COM Interface.          TSManager       10.01.2020 18:53:10      6232 (0x1858)
    Executing a standalone module (tsmanager.exe) version 5.0.8913.1000 from location 'C:\WINDOWS\CCM\TsManager.exe'     TSManager       10.01.2020 18:53:10      6232 (0x1858)
    Client push lock file found.        TSManager       10.01.2020 18:53:10      6232 (0x1858)
    UEFI: false       TSManager       10.01.2020 18:53:10     6232 (0x1858)
    Reading logging settings from Task Sequence environment to set Task Sequence logging.        TSManager            10.01.2020 18:53:10      6232 (0x1858)
    Deleting volume ID file C:\_SMSTSVolumeID.7159644d-f741-45d5-ab29-0ad8aa4771ca ...        TSManager            10.01.2020 18:53:10      6232 (0x1858)
    Compiling Config policy TSManager       10.01.2020 18:53:11      6232 (0x1858)
    Start to compile TS policy         TSManager       10.01.2020 18:53:11      6232 (0x1858)
    Failed to load CCM_Scheduler_ScheduledMessage object from XML. Instance is ignored.         TSManager            10.01.2020 18:53:11      6232 (0x1858)
    Policy complied successfully in WMI 'root\ccm\policy\defaultmachine\requestedconfig' namespace         TSManager            10.01.2020 18:53:11      6232 (0x1858)
    End Task Sequence policy compilation TSManager       10.01.2020 18:53:11      6232 (0x1858)
    Start to compile TS policy         TSManager       10.01.2020 18:53:11      6232 (0x1858)
    Failed to load CCM_UUPPSFXConfigEx object from XML. Instance is ignored.  TSManager       10.01.2020 18:53:11            6232 (0x1858)
    Failed to load CCM_SUMContentFlags object from XML. Instance is ignored.    TSManager       10.01.2020 18:53:11            6232 (0x1858)
    Policy complied successfully in WMI 'root\ccm\policy\defaultmachine\requestedconfig' namespace         TSManager            10.01.2020 18:53:11      6232 (0x1858)
    End Task Sequence policy compilation TSManager       10.01.2020 18:53:11      6232 (0x1858)
    Start to compile TS policy         TSManager       10.01.2020 18:53:11      6232 (0x1858)
    Failed to load CCM_Scheduler_ScheduledMessage object from XML. Instance is ignored.         TSManager            10.01.2020 18:53:11      6232 (0x1858)
    Policy complied successfully in WMI 'root\ccm\policy\defaultmachine\requestedconfig' namespace         TSManager            10.01.2020 18:53:11      6232 (0x1858)
    End Task Sequence policy compilation TSManager       10.01.2020 18:53:11      6232 (0x1858)
    Start to compile TS policy         TSManager       10.01.2020 18:53:11      6232 (0x1858)
    Failed to load CCM_Scheduler_ScheduledMessage object from XML. Instance is ignored.         TSManager            10.01.2020 18:53:11      6232 (0x1858)
    No policy to compile     TSManager       10.01.2020 18:53:11      6232 (0x1858)
    End Task Sequence policy compilation TSManager       10.01.2020 18:53:11      6232 (0x1858)
    Error compiling client config policies. code 8000FFFF    TSManager       10.01.2020 18:53:11     6232 (0x1858)
    Task Sequence Manager could not initialize Task Sequence Environment. code 8000FFFF        TSManager            10.01.2020 18:53:11      6232 (0x1858)
    Task sequence execution failed with error code 8000FFFF         TSManager       10.01.2020 18:53:11      6232 (0x1858)
    Cleaning Up.    TSManager       10.01.2020 18:53:11     6232 (0x1858)
    Removing Authenticator            TSManager       10.01.2020 18:53:11      6232 (0x1858)
    Cleaning up task sequence folder         TSManager       10.01.2020 18:53:11      6232 (0x1858)
    Deleting volume ID file C:\_SMSTSVolumeID.7159644d-f741-45d5-ab29-0ad8aa4771ca ...        TSManager            10.01.2020 18:53:12      6232 (0x1858)
    Successfully unregistered Task Sequencing Environment COM Interface.          TSManager       10.01.2020 18:53:12            6232 (0x1858)
    Executing command line: "C:\WINDOWS\CCM\TsProgressUI.exe" /Unregister with options (0, 0)            TSManager            10.01.2020 18:53:12      6232 (0x1858)
    ==========[ TsProgressUI started in process 6468 ]==========       TsProgressUI    10.01.2020 18:53:12      6472 (0x1948)
    Unregistering COM classes       TsProgressUI    10.01.2020 18:53:12      6472 (0x1948)
    Shutdown complete.     TsProgressUI    10.01.2020 18:53:12      6472 (0x1948)
    Process completed with exit code 0       TSManager       10.01.2020 18:53:12      6232 (0x1858)
    Successfully unregistered Task Sequence Progress UI. TSManager       10.01.2020 18:53:12     6232 (0x1858)
    Trying to restore client to the state it can be managed    TSManager       10.01.2020 18:53:12     6232 (0x1858)
    CCMExec service startup type is set to disabled TSManager       10.01.2020 18:53:12     6232 (0x1858)
    Modifying CCMExec Service to auto start.         TSManager       10.01.2020 18:53:12     6232 (0x1858)
    CCMExec Service started         TSManager       10.01.2020 18:53:13      6232 (0x1858)
    Waiting for CcmExec service to be fully operational       TSManager       10.01.2020 18:53:13     6232 (0x1858)
    
    Nobody out there with ideas? We have spent countless hours of troubleshooting.

    Friday, January 10, 2020 6:20 PM
  • Hello Simon,

    do you have an update for us?

    we're facing the same issue like Udo.

    Thanks
    Walter

    Tuesday, January 14, 2020 11:42 AM
  • We have the same issue.  OSD worked correctly until we upgraded to 1910.  Now all in-place upgrades are failing just as you mentioned. 
    Tuesday, January 14, 2020 4:04 PM
  • Same here. All Upgrades from 1607,1703,1709 failing.
    The only working workaround is to edit the setupcomplete.cmd and add exit at the beginning of the file.

    https://sysmanrec.com/what-to-do-if-computer-is-stuck-on-running-setupcomplete-cmd-after-windows-upgrade

    We have SCCM 1910 (No fast ring). There are no Hotfixes available for this release.
    The Fast Ring Edition has received several updates to resolve this issue.


    • Edited by unblack Friday, January 17, 2020 10:22 AM
    Friday, January 17, 2020 9:21 AM
  • Please see this related thread:

    https://social.technet.microsoft.com/Forums/en-US/d5f8f899-9c8d-47f5-a267-1f8062c986df/osd-failing-after-1910-upgrade

    So far the common thread with all customers having this issue is that they have McAfee AV installed. Disabling McAfee has fixed the issue for all known customers having the issue. We have also had a few internal cases where the customers also had McAfee installed and disabling it resolved the issue. If you are using McAfee and disabling it resolves the issue, I would recommend opening a case with McAfee to report the issue. It could potentially also be other 3rd party AVs or security software so our suggestion is to disable any such AV or security software, test to see if it fixes the problem, and if it does report the issue to the 3rd party.

    FYI some of the information from the above logs does not make sense. For example when setupcomplete.cmd launches the Task Sequence for the FIRST time after the IPU is done, it should not be waiting for CCMExec to start up because CCMExec (SMS Agent Host) should be disabled. Instead the Task Sequence runs a client repair followed by re-enabling the client. If you are seeing different behavior make sure to disable any type of client health or remediation tools that may be running during the Task Sequence that may be re-enabling the ConfigMgr client during the IPU.

    Also I would not follow the advise from the above link. Some of the information at that site is incomplete. For example taking the client out of provisioning mode does not involve just changing one registry value from true to false. I also strongly recommend against any modifications to the Setupcomplete.cmd, especially to force it to exit out. This most likely will break the Task Sequence and not allow it to continue properly after reboots.

    As of right now based on cases opened by a few customers we have found no evidence that this is an issue caused by ConfigMgr 1910 or a bug in ConfigMgr 1910. If you need any additional assistance with this issue, please open a case with Microsoft support. We will be more than happy to to investigate further if the issue continues after disabling McAfee and/or any 3rd party security software.

    Note: We are not suggesting to permanently disable McAfee or any 3rd party AV. We are only making this suggestion as a troubleshooting step and to narrow down root cause. If you find the issue is with your 3rd party AV or security software, we recommend working with that 3rd party on being able to re-enable the software without it interfering with the IPU upgrade including the ConfigMgr Task Sequence process.


    Frank Rojas Sr. Support Escalation Engineer | Cloud & Infrastructure Solutions | Microsoft Endpoint Configuration Manager Customer Services & Support | Microsoft Corporation

    Tuesday, January 21, 2020 9:36 PM
  • You are right!

    THX!

    McAfee ENS causes this issue!

    Uninstalling McAfee is a workaround:

    McAfeeEndpointProductRemoval.exe --accepteula --ENS

    i've raised a ticket with mcafee and Microsoft
    • Edited by unblack Wednesday, January 22, 2020 8:44 AM
    Wednesday, January 22, 2020 8:42 AM
  • Please keep this thread updated if either McAfee or Microsoft come up with a fix for this issue. 
    Wednesday, January 22, 2020 12:50 PM
  • https://social.technet.microsoft.com/Forums/en-US/d5f8f899-9c8d-47f5-a267-1f8062c986df/osd-failing-after-1910-upgrade?forum=configmanagerosd#985341e8-2c64-4410-bc2e-a42424f86756

    McAfee refers me back to Microsoft.
    Is it recommended to exclude the following files from the SelfProtection?

    The McAfee self protection blocks the C:\Windows\CCM\CcmExec.exe and the C:\$WINDOWS.~BT\Sources\SetupHost.exe

    Log:

    2020-01-23 08:57:45.026Z|Activity|ApBl                |mfeesp                                  |      4492|     10088|SP                  |XModuleEvents.cpp(821)                  | NT-AUTORITÄT\SYSTEM hat C:\Windows\CCM\CcmExec.exe ausgeführt. Dieser Prozess hat versucht, auf den Prozess mfetp.exe zuzugreifen, hat damit die Regel "Kernschutz – Schutz von McAfee-Prozessen vor unautorisiertem Zugriff und Abbruch" verletzt und wurde daher blockiert. In KB85494 wird erläutert, wie Sie auf dieses Ereignis reagieren können.

    2020-01-23 09:26:50.696Z|Activity|ApBl                |mfeesp                                  |      4492|     10096|SP                  |XModuleEvents.cpp(821)                  | NT-AUTORITÄT\SYSTEM hat C:\$WINDOWS.~BT\Sources\SetupHost.exe ausgeführt. Dieser Prozess hat versucht, auf die Datei C:\Windows\System32\drivers\mfeepmpk.sys zuzugreifen, hat damit die Regel "Bedrohungsschutz – Schutz von McAfee-Treibern" verletzt und wurde daher blockiert. In KB85494 wird erläutert, wie Sie auf dieses Ereignis reagieren können.

    Friday, January 24, 2020 11:04 AM
  • Update: the exclusion did not fix it :-(
    Saturday, January 25, 2020 7:06 AM