none
Exchange 2003 and SMTP dying RRS feed

  • Question

  • I've Exchange 2003 which now all of the sudden have SMTP service dying every few seconds after restart of service. It started some days ago but after couple of hours i was able to stop this madness by deleting russian email from the queue (spam). Basically every single day some Russian email is coming which breaks SMTP. I delete the offending email, restart all IIS Admin services and it's up and running for a day or two.  Exchange is at latest service pack (version 6.5 (Build 7638.2: Service Pack 2)) so the issues related to my earlier readings about this problem seem to be not related? I tried to reinstall service pack but it was complaining about earlier IMF being installed yet in directory of Exchange there's no IMFv1 but only IMFv2 so I would presume it's the newest version.

    Does anyone have a clue how to solve this? Not really sure where to look for an answer?

    The Simple Mail Transfer Protocol (SMTP) service terminated unexpectedly. It has done this 97 time(s). Event ID 7034

    The IIS Admin Service service terminated unexpectedly. It has done this 57 time(s). The following corrective action will be taken in 1 milliseconds: Run the configured recovery program. Event ID 7031


    I also see Virtual Server 2:

    SMTP server cannot read metabase key MailQueueDir. from SMTPSVC with EVENT ID 418

    And
    Virtual Server 2: SMTP server cannot read metabase key MailPickupDir with EVENT ID 418

    Application pool 'DefaultAppPool' is being automatically disabled due to a series of failures in the process(es) serving that application pool. event id 1002

    Inetinfo terminated unexpectedly and the system was not configured to restart IIS Admin. The World Wide Web Publishing Service has shut down. event id 1030

    The application-specific permission settings do not grant Local Launch permission for the COM Server application with CLSID
    {A9E69610-B80D-11D0-B9B9-00A0C922E750}
    to the user NT AUTHORITY\NETWORK SERVICE SID (S-1-5-20). This security permission can be modified using the Component Services administrative tool. EVENT ID 10016

    For the sake of example email.

     

    Server doesn't have any antivirus software on it. Only Policy Patrol 6which was there for 2 years+ with same version. After things started to happen I've upgraded it to version 7.05.

     

     


    My little website www.pro-solutions.pl with some simple/small projects.


    Tuesday, May 10, 2011 6:12 AM

Answers

  • I've worked with both Microsoft and Policy Patrol. Microsoft offered 2 patches and also suggested after some diagnosis and similar cases in their database that Policy Patrol might be the case. Indeed Policy Patrol was interested in this and provide few versions on new smtp sinks. We're still working things thru as there are some errors still rarely but it's going good direction.

    I was told to disable the following functionality and to replace it with new sink that they sent me:

     - Bayesian filtering (including Automatic Bayesian filter learning)

    - Address verification (Verify existence of MX record & Verify sender's SMTP connection)

    So far I've had one crash today and there was one day when i had 100 per day. It's weird that something like this worked for so long without any problems and now out of the sudden started crashing (without me touching Policy Patrol or Exchange).


    My little website www.pro-solutions.pl with some simple/small projects.
    • Marked as answer by Gen Lin Wednesday, May 25, 2011 2:06 AM
    Thursday, May 19, 2011 8:27 AM

All replies

  • How is the mailflow in general? Do you see frequently mails are backing up? Also when you telnet do you get all the SMTP verbs?

     

    You also mentioned something about Russian email and deleting the same & restarting the IIS would solve this problem temporarily so assuming this is a spam, I'll suggest enable the sender/Recipient filtering.

    Since your Inetinfo is crashing let’s make sure that you've antivirus on the system that is configured correctly and has the exclusion set accordingly as well.

     Now to be able to pinpoint where the issue is, you need to configure IIS crash dumps & when the issue will occur, it’ll create the information dump & later that can be analysed and we’ll be able to decisively troubleshoot the issue.


    Regards, Pushkal MishrA
    Tuesday, May 10, 2011 6:48 AM
  • @Pushkal

    I've Policy Patrol version 6 which after smtp started dying was upgraded to Policy Patrol 7. This worked for nearly 2 years without much of a problem and we're not receiving any spam to our inboxes as PP cleans it up nicely. We have no antivirus on the server so there should be nothing preventing server to work.

    We have about 50 people using Exchange so there's not so much emails going out or in (except spam which PP deals with). But the queue is empty most of the time and it was working fine for most time. We do have sometimes problems with queue trying to send emails but this is when someone sends 5mb email to 100 people and we've only 1.5mbit/s out connection so it takes a while to send but it never made SMTP to die. Especially that even if it waited in the queue rest of services were working correctly.. But now if the offending email gets to the queue (I can only verify this from the directory on drive) SMTP service dies and keeps on dying all the time bringing other services down as well (OWA and other Exchange Routing services die as well). As soon as I remove the email and restart everything smtp and other services are up and running. 

    Can you provide me some information how can i configure IIS crash dumps?


    My little website www.pro-solutions.pl with some simple/small projects.
    Tuesday, May 10, 2011 6:56 AM
  • Ok. It's installed. Please let me know if there's anything I need to do now? I've did this steps:

     

    a. Run Gflags.exe.
    b. For Image File Name, type the name of the process that you want to debug. For an IIS 5.0 Web site, the name of this process is Inetinfo.exe, Dllhost.exe, or Aspnet_wp.exe. For an IIS 6.0 Web site, the name of this process is Inetinfo.exe or W3wp.exe.
    c. Under Destination, click the Image File Options option.
    d. In the lower pane of the Global Flags dialog box, click Enable page heap.
    e. Click Apply, and then click OK.

    C:\IISDebugTools>iisreset

    Attempting stop...
    Internet services successfully stopped
    Attempting start...
    Internet services successfully restarted

    C:\IISDebugTools>iisdump.exe -I -p 5088

    IIS Debug Dump Utility v01.01.00.3800 (File Version 2003.07.22.08)
    Microsoft Corporation (c)2003. All Rights Reserved.

    Command line:
            iisdump.exe -I -p 5088


    Log files will be placed at:
    C:\IISDebugTools\logs\20110510-094540\

    Running specified commands. This may take a few minutes...
    Processing Application Event Log.
    Processing IIS Metabase...
    Processing debug script for target process: 3248...
    Processing debug script for target process: 1836...
    Processing debug script for target process: 5272...
    Processing debug script for target process: 5088...
    IIS Metabase log completed.
    Processing modules in: C:\WINDOWS\SYSTEM32\INETSRV\...
    Finished processing modules in: C:\WINDOWS\SYSTEM32\INETSRV\.
    Processing modules in: C:\WINDOWS\SYSTEM32\...
    Debug script for target process 1836 completed.
    Application Event Log completed.
    Processing System Event Log.
    Debug script for target process 5272 completed.
    Debug script for target process 3248 completed.
    Finished processing modules in: C:\WINDOWS\SYSTEM32\.
    Processing modules in: CLSID...
    Debug script for target process 5088 completed.
    Finished processing modules in: CLSID.
    Processing modules for target process: 3248...
    Finished processing modules for target process: 3248.
    Processing modules for target process: 1836...
    Finished processing modules for target process: 1836.
    Processing modules for target process: 5272...
    Finished processing modules for target process: 5272.
    Processing modules for target process: 5088...
    Finished processing modules for target process: 5088.
    SysInfo log completed.
    System Event Log completed.

    IIS Debug Dump completed. Log files may be found at:
    C:\IISDebugTools\logs\20110510-094540\

     

    Although this seems like the problem appeared, some files are created but SMTP is working...

     

     


    My little website www.pro-solutions.pl with some simple/small projects.


    Tuesday, May 10, 2011 7:07 AM
  • After installing and configuring the dump when issue will occur it will create the dump files and now these files need to be analyzed for possible cause.

    Unfortunately I am not an IIS expert so request you to get the files analyzed either by IIS specialist in MS support center or post the data to IIS forum

    http://social.technet.microsoft.com/Forums/en-US/windowsserver2008r2webtechnologies/threads although this is for windows 2008 web services but they might be knowing stuff for 2003 !


    Regards, Pushkal MishrA
    Tuesday, May 10, 2011 9:55 AM
  • Events id 7034, 7031, 1002 and 1030 are just side effects of inetinfo service crashing so the million dollar question is, why does it crash. The A9E69610-B80D-11D0-B9B9-00A0C922E750 CLSID in event 10016 corresponds to the same inetinfo (IIS Admin Service). The comments for event id 418 at http://www.eventid.net/display.asp?eventid=418&eventno=1085&source=smtpsvc&phase=1 suggest that the metabase might be corrupted. The comment about accounts no longer having enough rights into the IIS metabase might be worth investigating, especially considering the 10016 event id.

    It is unfortunate that Policy Patrol has been upgraded as you don't know now if the upgrade itself is not creating some problems.


    Thursday, May 12, 2011 3:38 PM
  • I'm working with Microsoft to resolve the issue. Got tired of the service breaks.I'm not sure about the metabase problem because I believe it was caused later on when i followed some advice to create 2nd smtp service and then deleted it and after that some errors regarding smtp2 started to show up.

     

    Also I've noticed last days:

     

    Faulting application inetinfo.exe, version 6.0.3790.3959, stamp 45d69692, faulting module pp4_smtpsink.dll, version 4.3.0.553, stamp 4cecb44b, debug? 0, fault address 0x000b61c1.
    For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

    which seems to relate to PP.

     

    However before it was just:

     

    Faulting application inetinfo.exe, version 6.0.3790.3959, stamp 45d69692, faulting module c4dll.dll, version 0.0.0.0, stamp 43e9f457, debug? 0, fault address 0x00036145.

    and only recently PP4_SMPTSINK started to show up. So might be related to upgrading PP.

     

    Anyways I've reported this to Microsoft as I had one free coupon that was expiring end of June so we will see. I'll post here what was it. Unless someone has idea to fix it :-)


    My little website www.pro-solutions.pl with some simple/small projects.
    Thursday, May 12, 2011 3:47 PM
  • From the error it does appear that the faulting module is "pp4_smtpsink.dll"....so I would suggest talking to PP tech support as well and finding out if there are any known issues with PP's this latest version v/s exchange.

     

    Also Can you telnet onto problem server on port 25 and see if you are getting all the SMTP verbs, I hope this product hadn't corrupted SMTP stack. Make sure that you've all of them as mentioned in the KB article

    http://technet.microsoft.com/en-us/library/bb124688(EXCHG.65).aspx

     

    If you've missed any single verb then you will need to go all the way from IIS reinstall, exchange binary install and SP update etc...

     

    If you haven't miss any then perhaps you might want to re-apply the service pack for exchange, reboot the server and see if that makes any difference.

     

    Update us before reapplying please.


    Thursday, May 12, 2011 5:08 PM
  • Any Update?
    Regards, Pushkal MishrA
    Thursday, May 19, 2011 6:21 AM
  • I've worked with both Microsoft and Policy Patrol. Microsoft offered 2 patches and also suggested after some diagnosis and similar cases in their database that Policy Patrol might be the case. Indeed Policy Patrol was interested in this and provide few versions on new smtp sinks. We're still working things thru as there are some errors still rarely but it's going good direction.

    I was told to disable the following functionality and to replace it with new sink that they sent me:

     - Bayesian filtering (including Automatic Bayesian filter learning)

    - Address verification (Verify existence of MX record & Verify sender's SMTP connection)

    So far I've had one crash today and there was one day when i had 100 per day. It's weird that something like this worked for so long without any problems and now out of the sudden started crashing (without me touching Policy Patrol or Exchange).


    My little website www.pro-solutions.pl with some simple/small projects.
    • Marked as answer by Gen Lin Wednesday, May 25, 2011 2:06 AM
    Thursday, May 19, 2011 8:27 AM
  • I wanted to relay that we are having the exact same issues. It appears to happen to us once every day or two. If I grab the piece of russian spam from C:\Program Files\Exchsrvr\Mailroot\vsi 1 then do a iisreset then things are fine again until another offending message. We are running Exchange 2003 standard on Windows Server 2003 Enterprise -- and the common link . . . Policy Patrol.

     

    We administer dozens of Exchange servers for dozens of companies and have never seen this issue before (and appears to be fairly rare). Interestingly enough, this is the only server that we run Policy Patrol on. I think you hit the nail on the head. Thanks for at least helping point us in the right direction.

     

    Please post back anything you find out and we'll do the same!

    Thursday, June 9, 2011 2:15 AM
  • Hello Andrew,


    It's still not solved for me. I've had Policy Patrol sending me couple of new versions of smtp sink and couple of other files to replace. They even rebuild the whole databases for me. I think you need to contact them yourselfs since they were diagnosing logs / crash dumps and policy patrol settings and based on that they were providing me some solutions.

    Right now I have still

     - Bayesian filtering (including Automatic Bayesian filter learning)

    - Address verification (Verify existence of MX record & Verify sender's SMTP connection)

    turned off... and latly they asked if I have enough ram on the machine (I have 1gb ram only there). I've bought more ram but didn't upgraded yet.

    Crashes however aren't happening as often as before. Haven't had one since couple of days. Hopefully this will end soon.. but they will have to still provide me solution since I want to turn back on features they told me to disable :-)

    I can give you files/updates/tips that they sent me but it may not be apriopriate for your situation before talking to them.. (unless you don't have support on the product).


    Let me know.


    My little website www.pro-solutions.pl with some simple/small projects.
    Thursday, June 9, 2011 6:17 AM
  • Hi!

    I have the same problem and tried to solve this problem with policy patrol/red earth software but no luck, in the beginning is was all the time but now maybe once or twice every day. still unacceptable for us.

    i Noticed if i tried uninstalling PP then everything works so the problem has to be PP.

    Did you get any further with this problem? is it solved?

    Today i sent another support ticket to PP and i hope maybe they have the solution to fix this now, this has been going on for almost a year now :(

    Thanks

    Jan

    Monday, February 13, 2012 8:31 AM
  • Hello Jan,

    Well they worked with me since May till now. They send me a dozen of possible fixes and it sometimes did fix things for couple of days. All in all it's not yet fixed but I get crashes very ocassionally now where simple restart of SMTP fixes it (where befoer I had to cleanup queue and do a lot of magic). I sent them a lot of crash dumps and I will send them 4 more crash dumps that I gathered from a month to analyze. They are about to releasing PP 8 which I hope won't have a problem with this.

    However as a workaround I installed www.mobilepcmonitor.com on my server (actually using it on all servers since then since it's a great app) that gives me instant (1 minute delay max or so) notifications on things that happen like SMTP being down or PORT being closed etc. So I get this notification anytime I just make the in app restart of SMTP service (from a car even) and it starts working again. It's not a fix but it's far better then have to login and pay attention all the time :) I can probably send you some fixes they send me and you can apply them to your machine but not sure if it's a best idea.

    With regards,

    Przemek


    My company website www.evotec.pl My other website www.pro-solutions.pl with some simple/small projects.


    Monday, February 13, 2012 3:36 PM