none
Exchange Server 2016 & Windows Server 2016 Datacenter IIS AppPool's constantly crashing. ECP/Powershell inaccessible

    Question

  • Hello everyone.

    I am having a serious issue with my Exchange Servers.

    The W3WP.exe application constantly crashes and spams the Event Log with crashes every few seconds.

    This happens on not one, both two exchange servers running similar OS's but on different physical servers.

    Here is the Event Log:

    Faulting application name: w3wp.exe, version: 10.0.14393.0, time stamp: 0x57899b8a
    Faulting module name: ntdll.dll, version: 10.0.14393.0, time stamp: 0x578997b2
    Exception code: 0xc0000409
    Fault offset: 0x00000000000951ff
    Faulting process id: 0x1838
    Faulting application start time: 0x01d22a2bbdd32496
    Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
    Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
    Report Id: 7e0a3e16-396d-42d4-8aab-8e5a3ce2eb14
    Faulting package full name: 
    Faulting package-relative application ID: 

    I cannot access Powershell/ECP (I can access PowerShell by changing the Front End location to point to the non-profixed Front-End, but the default Powershll does not work)

    I have completely reinstalled two other VM's for Exchange and the result ends up being the same eventually. This is not an immediate problem, it only seems to occur after awhile. 

    I cannot find anything that would cause this. Mostly things are default.

    Additionally, I have these showing up, probably due to the AppPool crashing:

    A process serving application pool 'MSExchangeOWAAppPool' suffered a fatal communication error with the Windows Process Activation Service. 

    (contains a few other ones, like the PowerShell app pool)

    Please Help!!


    xmarsbarsx


    • Edited by xmarsbarsx Wednesday, October 19, 2016 5:18 PM
    Wednesday, October 19, 2016 5:15 PM

Answers

All replies

  • Hello everyone.

    I am having a serious issue with my Exchange Servers.

    The W3WP.exe application constantly crashes and spams the Event Log with crashes every few seconds.

    This happens on not one, both two exchange servers running similar OS's but on different physical servers.

    Here is the Event Log:

    Faulting application name: w3wp.exe, version: 10.0.14393.0, time stamp: 0x57899b8a
    Faulting module name: ntdll.dll, version: 10.0.14393.0, time stamp: 0x578997b2
    Exception code: 0xc0000409
    Fault offset: 0x00000000000951ff
    Faulting process id: 0x1838
    Faulting application start time: 0x01d22a2bbdd32496
    Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
    Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
    Report Id: 7e0a3e16-396d-42d4-8aab-8e5a3ce2eb14
    Faulting package full name: 
    Faulting package-relative application ID: 

    I cannot access Powershell/ECP (I can access PowerShell by changing the Front End location to point to the non-profixed Front-End, but the default Powershll does not work)

    I have completely reinstalled two other VM's for Exchange and the result ends up being the same eventually. This is not an immediate problem, it only seems to occur after awhile. 

    I cannot find anything that would cause this. Mostly things are default.

    Additionally, I have these showing up, probably due to the AppPool crashing:

    A process serving application pool 'MSExchangeOWAAppPool' suffered a fatal communication error with the Windows Process Activation Service. 

    (contains a few other ones, like the PowerShell app pool)

    Please Help!!


    xmarsbarsx


    I have heard of others having this issue and don't know if there is an easy resolution. I would open a ticket with Microsoft support. ( I assume of course this is Exch 2016 CU3)


    Blog:    Twitter:   



    Wednesday, October 19, 2016 5:30 PM
    Moderator
  • Could you direct me to the web page for that? It is Exchange Server 2016 CU3


    xmarsbarsx

    Wednesday, October 19, 2016 6:11 PM
  • Could you direct me to the web page for that? It is Exchange Server 2016 CU3


    xmarsbarsx

    https://support.microsoft.com/en-us/getsupport?oaspworkflow=start_1.0.0.0&wf=0&wfName=productselection&gprid=18596&ccsid=636124987940632882

    Blog:    Twitter:   

    Wednesday, October 19, 2016 6:33 PM
    Moderator
  • Gotchya, I suppose this requires payment doesn't it?

    Was this a common problem? Is this perhaps a bug with Windows Server 2016 + Exchange Server 2016 CU3?

    I'm surprised I haven't heard more about this issue, I cannot seem to find anything online. 


    xmarsbarsx

    Wednesday, October 19, 2016 6:48 PM
  • Gotchya, I suppose this requires payment doesn't it?

    Was this a common problem? Is this perhaps a bug with Windows Server 2016 + Exchange Server 2016 CU3?

    I'm surprised I haven't heard more about this issue, I cannot seem to find anything online. 


    xmarsbarsx

    I dont know if its a common issue - prob not many have moved to W2016. 

    If its a bug, you will be reimbursed. 


    Blog:    Twitter:   

    Wednesday, October 19, 2016 7:08 PM
    Moderator
  • I too have setup Exchange 2016 CU3 on Server 2016 and am seeing the same issue
    Friday, October 21, 2016 8:28 AM
  • Yes, this has to be a compatibility problem.

    I am not having this issue on Windows Server 2012 R2 Datacenter.

    I have installed 3 Windows Server 2016 Vm's with Exchange 2016 CU3 and all 3 have now failed, thankfully I kept my Windows Server 2012 R2 VM to quickly get it back up and running.

    As previously stated, this is NOT an immediate problem. It works for awhile (this time a day) and then stops, without any additional intervention.


    xmarsbarsx


    • Edited by xmarsbarsx Friday, October 21, 2016 1:10 PM
    Friday, October 21, 2016 1:08 PM
  • Yeah, I was planning a migration from Exchange 2013 but I spotted the issue before I started actually moving mailboxes to the 2016 environment so it's not causing any issues at the moment.  So for now the Exchange 2016 VMs are sitting there waiting until I find a solution.  

    Friday, October 21, 2016 2:06 PM
  • Someone may have found a solution, but no documentation provided:

    https://www.reddit.com/r/exchangeserver/comments/57c89y/blank_page_when_accessing_owaecp_after_rebooting/

    Seems to be the same issue that I am having at least, same error with PowerShell as well...

    Hoping for an answer... 


    xmarsbarsx

    Friday, October 21, 2016 4:33 PM
  • Someone may have found a solution, but no documentation provided:

    https://www.reddit.com/r/exchangeserver/comments/57c89y/blank_page_when_accessing_owaecp_after_rebooting/

    Seems to be the same issue that I am having at least, same error with PowerShell as well...

    Hoping for an answer... 


    xmarsbarsx

    I thought that was my solution the other day as well as I had the same issue (backend websites were missing an SSL binding) but unfortunately the issue didn't go away after fixing that up. 

    If you want to try it, open up IIS Manager on your Exchange 2016 server, expand Sites in the list and then select "Exchange Back End".  On the far right, choose "Bindings" and then in the list select the HTTPS listing (should be on port 444) and choose Edit.  For the SSL Certificate, this should be the same as what is being used on the Default Website, which you probably assigned within the ECP/Powershell when you were first setting up Exchange.

    Saturday, October 22, 2016 12:58 AM
  • Thought it was this too but I tried this, and many different combinations, I never really messed with the Default Web Sites as I create my own virtual directories. Even a server that I never touched became a problem.

    Now Im having an issue with IIS WOrker Processes using 100% CPU (split between 3-4 processes using 25%+ per process)


    xmarsbarsx

    Saturday, October 22, 2016 5:50 PM
  • Ive just run into this issue, Installed 2x windows server 2016 datacenter with Exchange 2016 cu3 and was in the middle of setting them up and one has this exact issue "w3wp.dll" crashes.

    Ive also tried the SSL binding on the Exchange backend site but it didnt work for me, Luckily i hadnt migrated to these new servers yet so Email still works but I hope MS can come up with a fix ASAP.

    JK


    JK MCP

    Tuesday, October 25, 2016 10:24 AM
  • I have the same problem. Problem occurs after adding the server to the DAG cluster. When I remove server from the DAG, the server works correctly again.

    MC

    Wednesday, October 26, 2016 9:00 AM
  • Mine happened before i setup a DAG, had just installed Exchange 2016 cu3 and completed setup ready for migration and new DAG then this issue arose. It happened to one server then the 2nd but not at the same time.

    Now the W3wp.dll keeps crashing everytime IIS tries to start.

    We need a solution ASAP, i cant commision our new mail servers till its fixed.

    Ive updated both servers as far as they will go.

    Thanks

    JK 


    JK MCP

    Wednesday, October 26, 2016 10:29 PM
  • I am experiencing the same issues. I configured a DAG on Exch2016-CU3 running on Server 2016. Everything looked great until I started migrating user mailboxes. I rebooted and now cannot access exchange management shell on these 2 hosts. I also see the w3wp.exe /ntdll.dll errors in application log along with WAS/5011 App Pool Warnings in the system log. This directory is rolling with AppCrash logs:

    C:\ProgramData\Microsoft\Windows\WER\ReportQueue

    Version=1
    EventType=BEX64
    EventTime=131221494704684843
    ReportType=2
    Consent=1
    ReportIdentifier=5044f1da-9d34-11e6-836e-00155d005f00
    IntegratorReportIdentifier=520c0245-fc46-4247-9ef1-4114655f2c88
    NsAppName=w3wp.exe
    AppSessionGuid=00005688-0000-000d-33b0-660e4131d201
    TargetAppId=W:0000f519feec486de87ed73cb92d3cac802400000000!000054443c275eddd9bbe75c6d47cf9c24076b5d785c!w3wp.exe
    TargetAppVer=2016//07//16:02:27:22!15ee0!w3wp.exe
    BootId=4294967295
    Response.type=4
    Sig[0].Name=Application Name
    Sig[0].Value=w3wp.exe
    Sig[1].Name=Application Version
    Sig[1].Value=10.0.14393.0
    Sig[2].Name=Application Timestamp
    Sig[2].Value=57899b8a
    Sig[3].Name=Fault Module Name
    Sig[3].Value=ntdll.dll
    Sig[4].Name=Fault Module Version
    Sig[4].Value=10.0.14393.351
    Sig[5].Name=Fault Module Timestamp
    Sig[5].Value=5801a332
    Sig[6].Name=Exception Offset
    Sig[6].Value=000000000009520f
    Sig[7].Name=Exception Code
    Sig[7].Value=c0000409
    Sig[8].Name=Exception Data
    Sig[8].Value=000000000000000a
    DynamicSig[1].Name=OS Version
    DynamicSig[1].Value=10.0.14393.2.0.0.400.8
    DynamicSig[2].Name=Locale ID
    DynamicSig[2].Value=1033
    DynamicSig[22].Name=Additional Information 1
    DynamicSig[22].Value=318f
    DynamicSig[23].Name=Additional Information 2
    DynamicSig[23].Value=318f8eb61e176e217a9ae2192c7b640f
    DynamicSig[24].Name=Additional Information 3
    DynamicSig[24].Value=23de
    DynamicSig[25].Name=Additional Information 4
    DynamicSig[25].Value=23ded2732a75ed3d0d1da6bc85112415
    UI[2]=c:\windows\system32\inetsrv\w3wp.exe
    UI[5]=Check online for a solution (recommended)
    UI[6]=Check for a solution later (recommended)
    UI[7]=Close
    UI[8]=IIS Worker Process stopped working and was closed
    UI[9]=A problem caused the application to stop working correctly. Windows will notify you if a solution is available.
    UI[10]=&Close

    Patrick

    Friday, October 28, 2016 5:33 PM
  • If you are experiencing problems, you should open a ticket with Microsoft Support.


    Blog:    Twitter:   

    Friday, October 28, 2016 5:55 PM
    Moderator
  • Already working on that. Thanks!
    Friday, October 28, 2016 6:47 PM
  • Just a +1 for this issue. I've reproduced it precisely on 3x different test machines (fully patched), IIS seems to start to fail after approx 30-60 mins of operation. I am running a DAG configuration - not tried it without a DAG.
    Friday, October 28, 2016 10:13 PM
  • Removing the server from the DAG and rebooting seems to 'solve' the issue... I can launch the Exchange shell successfully (no longer receive Object Reference Not Set to an Instance of an Object exception), and w3wp.exe is no longer faulting.

    Interestingly, the same exception was present on the right hand side pane of the ECP when browsing to the DAG in question - but only when the DAG had a member server. 

    Work in progress :)

    Friday, October 28, 2016 10:32 PM
  • I didnt even get my servers into a DAG, I had left them to finish the day after but when I came back to them I had this issue on one then the other.

    So its not exclusively affecting DAG members, but time factor does seem to play a part as I didnt see this issue till a good while after setting them up.

    Any news on a fix yet MS?

    JK


    JK MCP

    Saturday, October 29, 2016 8:31 AM
  • +1 2 W2016 servers (std), Exchange 2016 CU3. As soon as I've added them to DAG, they do not behave. Rob Hardman, how did you manage to take the servers out of DAG?
    Tuesday, November 01, 2016 6:12 AM
  • Hi Chatzki,

    This is from memory so apologies if it's not comprehensive;

    1. Ensure all mailboxes (including arbitration etc) are removed from mailbox databases residing in the DAG.
    2. Get-MoveRequest -movestatus completed | Remove-Moverequest to clear down any migration artifacts (assuming migration was successful).
    3. Delete mailbox database copies from passive nodes of the DAG
    4. Remove AuditLog mailboxes from mailbox databases in the DAG
    5. Delete the mailbox databases from the DAG
    6. Remove each server from membership of the DAG, once the last server is removed from the DAG the cluster service will be destroyed automatically and you'll be left with standalone exchange server(s)
    7. Delete the empty DAG
    8. Reboot each server that was in the DAG

    Points 2,3 were handled with Exchange Shell. I'm in a coexistence scenario with Ex2013 thus I was able to use a PSSession to an Ex2013 server to complete the commands. Other points were completed using EAC, aside from the big hammer at point 8 ;)

    If you cannot complete the removal this way, so long as all your mailboxes are safely moved off, you can force removal of servers after mailbox database deletion using ADSIEdit (caution!) so that you can start again. Some people have had to remove their Ex2016 servers in this way, as the product seemingly cannot be uninstalled from Server 2016. I certainly had to. If you do end up destroying the servers in this manner, it is also good practice to remove the DAG cluster network object from your AD along with the orphaned computer accounts.

    HTH,

    Rob


    • Edited by Rob Hardman Tuesday, November 01, 2016 2:18 PM Insertion of 7)
    Tuesday, November 01, 2016 11:55 AM
  • Thanks, Rob! I would add:

    0. Start Powershell with Add-PSSnapin *exchange* ; since the management console throws the error upon the launch.

    What a shame for Microsoft. I've contacted them and they said (quote):

    As discussed this problem has been already found when using Windows Server 2016 + Exchange Server 2016 + DAG. This is currently under investigation so no ETA can be provided right now. Since this is a bug we will proceed to close this ticket for free (no charge).

    Why did MS state that WS2016 is compatible with Exchange 2016 CU3?


    Wednesday, November 02, 2016 1:38 PM
  • +1 2 servers 2016 Datacenter, Exchange 2016 CU3. After adding both servers to the DAG they worked until the next reboot. Since then, OWA, ECP and EMS are inaccessible

    Wednesday, November 02, 2016 5:01 PM
  • Same issue here too.  I have had to just shut down the boxes since I cant unwind this mess.  Good thing its a lab environment.
    Wednesday, November 02, 2016 5:13 PM
  • +1 We are also seeing this as soon as we establish a DAG on 2016 server with exchange 2016CU3
    Thursday, November 03, 2016 3:36 PM
  • +1

    We are seeing this exact same behavior as well. A fully patched Server 2016 with Exchange 2016 CU 3 will start to spew "MSExchangeOWAAppPool" errors constantly, and Exchange management shell will not launch. If I remove the server from the DAG, then everything returns to normal. I have built three Exchange 2016 nodes and they all exhibit the exact same behavior.

    Unreal.

    Friday, November 04, 2016 2:33 PM
  • Official from Microsoft:

    If you attempt to run Microsoft Exchange 2016 CU3 on Windows Server 2016, you will experience errors in the IIS host process W3WP.exe. There is no workaround at this time. You should postpone deployment of Exchange 2016 CU3 on Windows Server 2016 until a supported fix is available.

    https://technet.microsoft.com/en-us/windows-server-docs/get-started/windows-server-2016-ga-release-notes


    Blog:    Twitter:   

    • Proposed as answer by Rob Hardman Friday, November 04, 2016 4:48 PM
    • Marked as answer by xmarsbarsx Friday, November 04, 2016 7:39 PM
    Friday, November 04, 2016 2:41 PM
    Moderator
  • Thanks for the official response! While its not really what many of us were looking for, it is nice to know its a known issue.

    xmarsbarsx

    Friday, November 04, 2016 3:16 PM
  • Anyone with an open case with Microsoft or know something of an upcoming patch to fix this before we go ahead and deploy Windows Server 2016 + Exchange 2016?
    Monday, December 05, 2016 12:00 PM
  • This is my home lab.  I have installed Windows Server 2016 and Exchange 2016 CU3.  So far have rebooted the server few times.  No issue with ECP or EMS.  

    I am not running DAG.

    Ram

    Tuesday, December 06, 2016 4:09 PM
  • So CU4 is out but it states you need a patch in order to install it on Server 2016 however the patch is 404 on MS site.  Anyone had any luck fixing their borked CU3 install on Server 2016 (with DAG) yet?

    Tuesday, December 13, 2016 6:15 PM
  • So CU4 is out but it states you need a patch in order to install it on Server 2016 however the patch is 404 on MS site.  Anyone had any luck fixing their borked CU3 install on Server 2016 (with DAG) yet?


    Give it some time before the links are valid.

    Blog:    Twitter:   

    Tuesday, December 13, 2016 7:24 PM
    Moderator
  • you need do download KB 3206632  for Windows 2016 and install it on Exchange 2016 CU3 in order to resolve the problem of IIS crash after implementing the DAG

    RANA

    Thursday, December 15, 2016 3:38 PM
  • Hi

    You can download Exchange 2016 CU4 from this link https://www.microsoft.com/en-us/download/details.aspx?id=54450

    Monday, December 19, 2016 12:38 PM